{Release} Auto generate history notes

doc/authoring_command_modules/README.md

@@ -201,12 +201,12 @@ Submitting Pull Requests
 
 ### Format PR Title
 
-History notes are auto-generated based on commit messages and descriptions starting from [S165](https://github.com/Azure/azure-cli/milestone/82). The commmit message for the squashed merge commit defaults to the PR title. Starting from 01/30/2020, we require all the PR titles to follow the below format:
-1. [**Mandatory**] Each commit message must start with `[Component Name]` or `{Component Name}`. 
+History notes are auto-generated based on PR titles and descriptions starting from [S165](https://github.com/Azure/azure-cli/milestone/82). Starting from 01/30/2020, we require all the PR titles to follow the below format:
+1. [**Mandatory**] Each PR title must start with `[Component Name]` or `{Component Name}`. 
     * `Component Name` shall be replaced by the real ones such as `Storage`, `Compute`. It could be the name of a command module, but in title case with necessary spaces for better readability, such as `API Management`, `Managed Service`. Other possible component names include but are not limited to: `Packaging`, `Misc.`, `Aladdin`.
     * `[]` means this change is customer-facing and the message will be put into `HISTORY.rst`. `{}` means this change is not customer-facing and the message will **NOT** be included in `HISTORY.rst`.
     * If the component name is `Core`, the message will be written in `src/azure-cli-core/HISTORY.rst`. Otherwise, the message will be written in `src/azure-cli/HISTORY.rst`.
-2. [**Mandatory**] If it's a breaking change, the second part should be `BREAKING CHANGE` followed by a colon. In the case of hotfix, put `Hotfix` in this part. For other cases, this part should be empty.
+2. [**Mandatory**] If it's a breaking change, the second part should be `BREAKING CHANGE` followed by a colon. In the case of hotfix, put `Hotfix` in this part. If it's related to fixing an issue, put `Fix #number` in this part. For other cases, this part could be empty.
 3. [**Recommendation**] If the change can be mapped into a command, then the next part could be the command name, followed by a colon.
 4. [**Recommendation**] Use the right verb with present-tense in original form to descibe what is done:
     * **Add** for new features.
@@ -215,22 +215,23 @@ History notes are auto-generated based on commit messages and descriptions start
     * **Remove** for deprecated features removed in this release.
     * **Fix** for any bug fixes.
 
-An example title of customer-facing change PR:
+Examples of customer-facing change PR title:
 
->[Storage] BREAKING CHANGE: az storage remove: remove --auth-mode argument
+>[Storage] BREAKING CHANGE: az storage remove: remove --auth-mode argument  
+>[ARM] Fix #10246: az resource tag crashes when the parameter --ids passed in is resource group ID
 
-An example title of non-customer-facing change PR:
+An example of non-customer-facing change PR title:
 
 >{Aladdin} Add help example for dns
 
 ### Format PR Description
 
-If you would like to write multiple history notes for one PR, please write the notes under `History Notes` section in the PR description,  following the same format described above. In this case, the PR title should be a summary of all the changes in this PR and will not be put into `HISTORY.rst`.
+If you would like to write multiple history notes for one PR, please write the notes under `History Notes` section in the PR description, following the same format described above. The PR template already contains the history note template, just change it if needed. In this case, the PR title should be a summary of all the changes in this PR and will not be put into `HISTORY.rst`. You can delete the `History Notes` section if not needed.
 
 Other than that, you can put any reasonable information in the description above the `History Notes` part.
 
-### Only Modify HISTORY.rst for Hotfix
-As described above, history notes will be auto-generated and inserted into `HISTORY.rst` during release. When submitting a normal PR, you don't need to modify `HISTORY.rst` manually. In the case of hotfix, you should create a hotfix branch based on release branch and submit a PR to merge hotfix into release. In this PR, if you have customer-facing changes, you need to modify `HISTORY.rst` to add history notes. Later on you also need to merge the hotfix branch into dev branch and the second part of the PR title should be `Hotfix`. The auto-generation of history notes for the next release will ignore the commit that contains `Hotfix`.
+### Submit Hotfix
+In the case of hotfix, you should create a hotfix branch based on release branch and submit a PR to merge hotfix into release. In this PR, the second part of the PR title should be `Hotfix`. If you have customer-facing changes, you need to modify `HISTORY.rst` to add history notes. The auto generated history notes for the next release will ignore the PR that contains `Hotfix`. You also need to submit a PR to merge the release branch back into dev branch before next release. Do **NOT** squash and merge this PR. After the PR gets approved by code owners, you should fast forward dev to release on your local machine and then push dev to upstream repository. 
 
 An example title of hotfix change PR:
 

scripts/release/generate_history_notes.py

@@ -0,0 +1,191 @@
+#!/usr/bin/env python
+
+# --------------------------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# --------------------------------------------------------------------------------------------
+#
+# This script is used to generate history notes for the commits on dev branch since last release.
+# The history notes are generated based on the title/description of the Pull Requests for the commits.
+# Make sure you have added the remote repository as upstream and fetched the latest code, i.e. you
+# have done:
+# git remote add upstream git@github.com:Azure/azure-cli.git
+# git fetch upstream
+
+import fileinput
+import json
+import re
+import subprocess
+import requests
+
+base_url = 'https://api.github.com/repos/azure/azure-cli'
+commit_pr_url = '{}/commits/commit_id/pulls'.format(base_url)
+
+history_line_breaker = '==============='
+history_notes = {}
+
+
+def generate_history_notes():
+    dev_commits = get_commits()
+    for commit in dev_commits:
+        prs = get_prs_for_commit(commit['sha'])
+        # parse PR if one commit is mapped to one PR
+        if len(prs) == 1:
+            process_pr(prs[0])
+        else:
+            process_commit(commit)
+
+    cli_history = ''
+    core_history = ''
+    for component in sorted(history_notes, key=str.casefold):
+        if component.casefold() == 'core':
+            core_history += construct_core_history(component)
+        else:
+            cli_history += construct_cli_history(component)
+    if core_history == '':
+        core_history = '* Minor fixes\n'
+    else:
+        core_history = core_history[:-1]  # remove last \n
+
+    print("azure-cli history notes:")
+    print(cli_history)
+    print("azure-cli-core history notes:")
+    print(core_history)
+
+    cli_history = cli_history[:-1]  # remove last \n
+    with fileinput.FileInput('src/azure-cli/HISTORY.rst',
+                             inplace=True) as file:
+        modify_history_file(file, cli_history)
+
+    with fileinput.FileInput('src/azure-cli-core/HISTORY.rst',
+                             inplace=True) as file:
+        modify_history_file(file, core_history)
+
+
+def modify_history_file(file: fileinput.FileInput, new_history: str):
+    write = True
+    for line in file:
+        if line == '{}\n'.format(history_line_breaker):
+            print(line.replace(
+                history_line_breaker,
+                '{}\n\n{}'.format(history_line_breaker, new_history)),
+                  end='')
+            write = False
+        else:
+            # remove any history notes written above previous release version
+            # make the generation of history notes idempotent
+            if re.match(r'^[0-9]+\.[0-9]+\.[0-9]+$', line):
+                write = True
+            if write:
+                print(line, end='')
+
+
+def construct_cli_history(component: str):
+    history = '**{}**\n\n'.format(component)
+    for note in history_notes[component]:
+        history += '* {}\n'.format(note)
+    history += '\n'
+    return history
+
+
+def construct_core_history(component: str):
+    history = ''
+    for note in history_notes[component]:
+        history += '* {}\n'.format(note)
+    history += '\n'
+    return history
+
+
+def get_commits():
+    out = subprocess.Popen([
+        'git', 'log', 'upstream/release...upstream/dev',
+        '--pretty=format:"%H %s"'
+    ],
+                           stdout=subprocess.PIPE,
+                           stderr=subprocess.STDOUT)
+    stdout, _ = out.communicate()
+    dev_commits = []
+    for line in stdout.decode('utf-8').splitlines():
+        words = line.strip('"').split(None, 1)
+        sha = words[0]
+        subject = words[1]
+        if not subject.startswith('{'):
+            dev_commits.append({'sha': sha, 'subject': subject})
+    dev_commits.reverse()
+    return dev_commits
+
+
+def get_prs_for_commit(commit: str):
+    headers = {'Accept': 'application/vnd.github.groot-preview+json'}
+    url = commit_pr_url.replace('commit_id', commit)
+    response = requests.get(url, headers=headers)
+    if response.status_code != 200:
+        raise Exception("Request to {} failed with {}".format(
+            url, response.status_code))
+    prs = json.loads(response.content.decode('utf-8'))
+    return prs
+
+
+def process_pr(pr):
+    lines = [pr['title']]
+    body = pr['body']
+    search_result = re.search(r'\*\*History Notes:\*\*(.*)---',
+                              body,
+                              flags=re.DOTALL)
+    if search_result is None:
+        search_result = re.search(r'\*\*History Notes:\*\*(.*)',
+                                  body,
+                                  flags=re.DOTALL)
+        if search_result is not None:
+            body = search_result.group(1)
+    else:
+        body = search_result.group(1)
+    lines.extend(body.splitlines())
+    process_lines(lines)
+
+
+def process_commit(commit):
+    lines = commit['subject'].splitlines()
+    process_lines(lines)
+
+
+def process_lines(lines: [str]):
+    # do not put note of hotfix here since it's for last release
+    if re.search('hotfix', lines[0], re.IGNORECASE):
+        return
+    note_in_desc = False
+    for desc in lines[1:]:
+        component, note = parse_message(desc)
+        if component is not None:
+            note_in_desc = True
+            history_notes.setdefault(component, []).append(note)
+    # if description has no history notes, parse PR title/commit message
+    # otherwise should skip PR title/commit message
+    if not note_in_desc:
+        component, note = parse_message(lines[0])
+        if component is not None:
+            history_notes.setdefault(component, []).append(note)
+
+
+def parse_message(message: str) -> (str, str):
+    # do not include template
+    if message.startswith('[Component Name'):
+        return None, None
+    m = re.search(r'^\[(.+)\](.+)$', message)
+    if m is not None:
+        component = m.group(1)
+        note = m.group(2).strip()
+        #remove appended PR number in commit message
+        note = re.sub(r' \(#[0-9]+\)$', '', note)
+        note = re.sub('BREAKING CHANGE:',
+                      '[BREAKING CHANGE]',
+                      note,
+                      flags=re.IGNORECASE)
+        if note.endswith('.'):
+            note = note[:-1]
+        return component, note
+    return None, None
+
+
+if __name__ == "__main__":
+    generate_history_notes()