Ensure that README.md documentation states the latest release

[asraa]

See #369

to ensure that the documentation is up-to-date with the latest release in SHA256SUM.md and the latest GitHub releases

[ianlewis]

We have a pull request outstanding on slsa-github-generator for this. The same thing could be applied here.
slsa-framework/slsa-github-generator#1266

/cc @pnacht

[pnacht]

I've taken a gander at the code and this shouldn't be too complex. However, I believe I'll only be able to look at it early 23Q1.

[pnacht]

I've written a bash script similar in spirit to slsa-github-generator's references.sh, but fit for purpose in this repo.

It checks that SHA256SUM.md and README.md always refer to the same slsa-verifier version, and that this version matches the major version declared in go.mod.

The README.md check is a bit fragile. It only detects version numbers that come after the string "slsa-verifier". Something of the sort is necessary since the file refers to versions of multiple different projects, but would love to hear better solutions. It works for the file's current state, but significant rewrites may require tinkering with the check.

I now want to add this script to a workflow. Similar to slsa-github-generator (after slsa-framework/slsa-github-generator#1266), my idea would be to add a pre-release check to the release.yml workflow. However, looking at RELEASE.md, I have a few questions:

1. Should the documentation be updated prior to the release or after it? Just asking because the "Publish release" step seems independent of the "Update documentation" step.
   Copying the flow from slsa-github-generator would mean updating the docs prior to the "Final Release" stage. This could be done by parsing the tag name within release.yml and only running the doc checks if it's not a release candidate.
2. Should updating the e2e test in example-package's e2e.installer-action.yml be included? Since that action is a nightly run, we can use this check to ensure that it's updated, but it'd have to be immediately prior to a new release to avoid false positives.

[ianlewis]

• Should the documentation be updated prior to the release or after it? Just asking because the "Publish release" step seems independent of the "Update documentation" step.
  Copying the flow from slsa-github-generator would mean updating the docs prior to the "Final Release" stage. This could be done by parsing the tag name within release.yml and only running the doc checks if it's not a release candidate.

I think updating the docs prior to the release should be ok. Though I don't see that we create a PR before creating the release candidate so you couldn't do it in a pre-submit unless you did checked the PR description for a #label:release tag like we do in slsa-github-generator. Doing it in release.yml would only fail to create the release artifacts so you wouldn't catch the issue until after the release was created (I suppose that might be ok for RC releases).

• Should updating the e2e test in example-package's e2e.installer-action.yml be included? Since that action is a nightly run, we can use this check to ensure that it's updated, but it'd have to be immediately prior to a new release to avoid false positives.

I think it can be run manually and the RELEASE.md has instructions for that. Since the references need to exist before you can update and run the workflow, I think those instructions are ok where there are. We should catch most issues with the RC releases.

[pnacht]

Ah yes, you're right. pre-submit.actions is a much better place to run these tests. Ok, I'll send a PR adding this to that workflow (with the #label:release filter) next week.