Document release notes procedure

.pre-commit-config.yaml

@@ -2,10 +2,10 @@
 exclude: "^.venv$|^.cache$|^.pytest_cache$"
 fail_fast: false
 default_language_version:
-  python: python3.11
+  python: python3.10
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.2.0
+    rev: v4.6.0
     hooks:
       - id: check-added-large-files
         args: ["--maxkb=1024"]
@@ -21,20 +21,26 @@ repos:
       - id: debug-statements
   # NOTE: Keep order as pyupgrade (will update code) then pycln (remove unused imports), then isort (sort them) and black (final formatting)
   - repo: https://github.com/asottile/pyupgrade
-    rev: v2.34.0
+    rev: v3.17.0
     hooks:
       - id: pyupgrade
         args:
           - "--py39-plus"
         name: upgrade code
+  - repo: https://github.com/hadialqattan/pycln
+    rev: v2.4.0
+    hooks:
+      - id: pycln
+        args: [--all, --expand-stars]
+        name: prune imports
   - repo: https://github.com/PyCQA/isort
-    rev: 5.12.0
+    rev: 5.13.2
     hooks:
       - id: isort
         args: ["--profile", "black"]
         name: sort imports
   - repo: https://github.com/psf/black
-    rev: 22.3.0
+    rev: 24.8.0
     hooks:
       - id: black
         name: black format code

release-notes/README.md

@@ -0,0 +1,118 @@
+# README
+
+### TL;DR
+- This folder contains **release notes** divided by **product**.
+- They must be **comprehended by any user** of the platform/product.
+- Scrum master orchestrates their process of creation.
+
+## How is this content created?
+
+All **release notes** are created using an iterative approach involving members of multiple teams: developers, App Team, product owners.
+
+The process is coordinated by the current scrum master. They are responsible for coordinating the team effort and making sure everything is in place before the release.
+
+
+## Scrum master procedures
+
+The scrum master takes care of enforcing deadlines. Their goal is to produce an email sent to the App Team containing information about:
+- where, when and what to test
+- ask App Team to rewrite the `draft release notes` into user readable `release notes`
+
+The following procedure is suggested (more details below):
+
+1. ask PO(s) (Nik) for the next review date. Be insistent. I mean it: **BE INSISTENT!**
+1. coordinate with team when to deploy to `staging` and `production`
+1. once dates are available create the **deadlines calendar** and post it to the team on Mattermost (reminders about important deadlines)
+1. ask developers to fill out the release drafts per product
+1. ensure NIH Staging is properly deployed
+1. inform App Team after the review when the deployment is ready for testing
+1. ensure all drafts have been converted to a non draft form (including the rewording) before release
+
+### 1. `review date`
+
+Ask Nik (be insistent) for the review date as soon as possible.
+It is impossible to organise without a release date.
+
+### 2. `pick staging and production release dates`
+
+Ask team members if it is possible to release to:
+- `staging` 1 day before the review
+- `production` after the App Team finishes testing staging (usually one or two days after)
+
+### 3. `deadlines calendar`
+
+The following deadlines must be established (pass them in this order to the below cli):
+
+- `prs_merge` (pull requests merge date) `usually one day before release to staging`: users have to get PRs (which must be testes by the App Team) merged in master by the end of this day
+- `staging_release` (staging release date) `decided with team`: PRs are moved to staging environment to which App Team will have access
+- `review_meeting` (review date) `decide by PO(s)`: the day of the review provided by Nik
+- `start_app_team` (first day of testing) `first day after review`: this day (included) App Team will have exclusive access to the NIH Staging deployment
+- `end_app_team` (last day of testing) `second day after review`: this day (included) by the end of the day App Team will be done testing
+- `prod_release` (production release) `decided with team`: staging changes are moved to production on this day
+
+```shell
+python scripts/3_deadlines_calendar.py <DATE> <DATE> <DATE> <DATE> <DATE> <DATE>
+```
+- `<DATE>` format: `dd.mm` where `dd` is a valid calendar day and `mm` is valid calendar month
+
+**NOTE:** make sure to enforce important deadlines (1/2 days before they expire) like:
+- date and time when `all PRs` need to be `merged` before release to staging
+- date and time when `release drafts` need to be `completed`
+
+
+### 4. `release drafts compilation`
+
+Ideally these are done before the review. The App Team will most likely see them only during testing.
+The email to the App Team should be sent briefly after the review is over.
+
+Use the following script to generate the new release drafts for all the products.
+A Mattermost message is also provided.
+
+:warning: Do not forget to commit and push the changes in the repository! :rotating_light:
+
+```shell
+python scripts/4_make_release_drafts.py <VERSION> <DATE> <TIME>
+```
+- `<VERSION>` format: `X.X.X` where `X` is a number
+- `<DATE>` format: `dd.mm` where `dd` is a valid calendar day and `mm` is valid calendar month
+- `<TIME>` format: `HH.MM` where `HH` is a valid hour and `MM` is valid minute
+
+
+:warning: Do not forget to commit and push the changes in the repository! :rotating_light:
+
+
+### 5. `staging deployed`
+
+Ensure with the team that staging is properly deployed. Check the following:
+- staging release to NIH Staging deployment occurred
+- sim4life services released
+
+### 6. `email App Team`
+
+Compose and send out email to inform the App Team when they can start testing the deployment.
+
+```shell
+python scripts/6_app_team_email.py <DATE> <DATE> <VERSION>
+```
+- `<DATE>` format: `dd.mm` where `dd` is a valid calendar day and `mm` is valid calendar month
+- `<VERSION>` format: `X.X.X` where `X` is a number
+
+### 7. `no drafts left before release`
+
+Check all products and make sure all the release drafts have been converted. Talk with the responsible people:
+- `s4l`: App Team
+- `tip`: (not decided yet), so it's the current scrum master
+- `osparc`: (not decided yet), so it's the current scrum master
+
+# After Hot-fixing
+
+Please note, that if you hot-fix a change to a production deployment, you have to run the below command and follow it's instructions.
+
+:warning: Do not forget to commit and push the changes in the repository! :rotating_light:
+
+```shell
+python scripts/run_after_hotfix_to_prod.py <VERSION>
+```
+- `<VERSION>` format: `X.X.X` where `X` is a number
+
+:warning: Do not forget to commit and push the changes in the repository! :rotating_light:

scripts/3_deadlines_calendar.py

@@ -0,0 +1,51 @@
+import argparse
+from typing import Final
+
+from utils import validate_date
+
+TEMPLATE: Final[
+    str
+] = """
+
+## Deadlines & upcoming schedule
+`{prs_merge}`: @all have all your PRs intended to be tested merged by the end of the day
+`{staging_release}`: STAGING release
+`{review_meeting}`: REVIEW MEETING
+`{review_meeting}`: by the end of the day created draft of the release notes
+`{start_app_team}`: -> {end_app_team}: App Team has exclusive access to the AWS Staging environment
+`{prod_release}`: PROD release
+"""
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Assemble deadelines calendar")
+    for date in [
+        "prs_merge",
+        "staging_release",
+        "review_meeting",
+        "start_app_team",
+        "end_app_team",
+        "prod_release",
+    ]:
+        parser.add_argument(
+            date,
+            type=validate_date,
+            help="The date is in the format dd.mm",
+        )
+    args = parser.parse_args()
+    print(f"\nDeadlines: {args}")
+
+    mattermost_message = TEMPLATE.format(
+        prs_merge=args.prs_merge,
+        staging_release=args.staging_release,
+        review_meeting=args.review_meeting,
+        start_app_team=args.start_app_team,
+        end_app_team=args.end_app_team,
+        prod_release=args.prod_release,
+    )
+
+    print(f"\n{mattermost_message}\n")
+
+
+if __name__ == "__main__":
+    main()

scripts/4_make_release_drafts.py

@@ -0,0 +1,126 @@
+import argparse
+import json
+from pathlib import Path
+
+import requests
+from utils import (
+    INITIALS_TO_USERNAMES,
+    list_folders_in_path,
+    validate_date,
+    validate_time,
+    validate_version,
+)
+
+CURRENT_DIR: Path = Path(__file__).resolve().parent
+
+INITIALS: list[str] = list(INITIALS_TO_USERNAMES.keys())
+
+_TEMPLATE: str = """Click on the link for a list of all the PRs released since `{version_tag}`
+https://github.com/ITISFoundation/osparc-simcore/compare/{version_tag}...master
+
+Please check your name if finished:
+{names_to_check}
+
+**Draft release notes go below the line**
+---
+# Release Notes
+### Highlights
+
+
+<details>
+<summary>Show detailed release notes</summary>
+
+{collapsable_content}
+</details>
+"""
+
+
+_MATTERMOST_TEMPLATE: str = """
+@all please contribute in compiling the `Release Drafts`.
+There are 3 release drafts to fill out: `s4l` (which also includes s4li-lite), `osparc`, `tip`.
+Instructions:
+- Please go through all your last changes in master since the last release for each product: [osparc](https://github.com/ITISFoundation/osparc-issues/blob/master/release-notes/osparc/{version_tag}.md), [s4l](https://github.com/ITISFoundation/osparc-issues/blob/master/release-notes/s4l/{version_tag}.md) and [tip](https://github.com/ITISFoundation/osparc-issues/blob/master/release-notes/tip/{version_tag}.md) (each product contains a link to these changes)
+- For each product please chose those features which it makes sense to highlight.
+- When highlighting a feature please provide a meaningful `USER` readable description and the link to the PR
+- deadline `{deadline_day} by {deadline_time}`
+- please :white_check_mark: your name when done in all products so I can keep track of who is missing.
+
+:pray: Thanks everybody for your kind cooperation and help.
+"""
+
+
+def _get_previous_version(vtag: str) -> int:
+    version_parts = vtag.strip("v").split(".")
+    version_parts[1] = f"{int(version_parts[1])-1}"
+
+    return "v" + ".".join(version_parts)
+
+
+def _get_names_to_check() -> str:
+    return "\n".join([f"- [ ] {name}" for name in INITIALS])
+
+
+def _get_collpasable_notes(vtag: str) -> str:
+    response = requests.get(
+        f"https://api.github.com/repos/ITISFoundation/osparc-simcore/releases/tags/{vtag}",
+        timeout=10,
+    )
+    json_response = json.loads(response.text)
+    return json_response["body"]
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Process a version number in the format X.X.X"
+    )
+    parser.add_argument(
+        "version", type=validate_version, help="The version number in the format X.X.X"
+    )
+    parser.add_argument(
+        "deadline_day",
+        type=validate_date,
+        help="The date is in the format dd.mm",
+    )
+    parser.add_argument(
+        "deadline_time",
+        type=validate_time,
+        help="The time is in the format HH:MM",
+    )
+
+    args = parser.parse_args()
+    tag = f"{args.version}"
+
+    vtag = f"v{tag}"
+
+    print("\nTotal [STEPS] 2\n")
+
+    print(f"Will generate form tag: {vtag}")
+
+    new_draft_content = _TEMPLATE.format(
+        version_tag=_get_previous_version(vtag),
+        names_to_check=_get_names_to_check(),
+        collapsable_content=_get_collpasable_notes(vtag),
+    )
+    for product_folder in list_folders_in_path(CURRENT_DIR / ".." / "release-notes"):
+        draft_file = product_folder / f"{vtag}.md"
+        if draft_file.exists():
+            msg = f"{draft_file} already exists, make sure you are creating the draft for the correct release"
+            # raise ValueError(msg)
+        draft_file.write_text(new_draft_content)
+
+    print(
+        "\n[STEP] 1/2 post this message on mattermost, please edit the DEADLINE accordingly\n"
+    )
+    print(
+        _MATTERMOST_TEMPLATE.format(
+            version_tag=vtag,
+            deadline_day=args.deadline_day,
+            deadline_time=args.deadline_time,
+        )
+    )
+
+    print("\n[STEP] 2/2 commit and push changes generated in this repo!!!!\n")
+
+
+if __name__ == "__main__":
+    main()