chore: script that discovers minimum tested versions

[emmettbutler]

This script discovers the minimum version of every package referenced in riotfile.py's Venv tree (excluding those envs that do not execute a pytest command) and writes that information to a CSV file. It also takes into account the installation dependencies in pyproject.toml.

For use in #9323

Checklist

• Change(s) are motivated and described in the PR description
• Testing strategy is described if automated tests are not included in the PR
• Risks are described (performance impact, potential for breakage, maintainability)
• Change is maintainable (easy to change, telemetry, documentation)
• Library release note guidelines are followed or label changelog/no-changelog is set
• Documentation is included (in-code, generated user docs, public corp docs)
• Backport labels are set (if applicable)
• If this PR changes the public interface, I've notified @DataDog/apm-tees.

Reviewer Checklist

• Title is accurate
• All changes are related to the pull request's stated goal
• Description motivates each change
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Change is maintainable (easy to change, telemetry, documentation)
• Release note makes sense to a user of the library
• Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[brettlangdon]

do we need to do this without riot?

are we able to use riot to generate an artifact during build time? e.g. the Build deploy job could generate sdist, wheels, and some other artifacts like supported library versions

then that can be packaged with the other artifacts into the OCI artifact.

(we could even generate the result as Python and have it just embedded into the sitecustomize.py file if we don't want to read something from disk)

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: emmett.butler/min_versions
Commit report: 5ade18a
Test service: dd-trace-py

✅ 0 Failed, 140838 Passed, 36443 Skipped, 7h 50m 47.26s Total duration (2h 36m 3.2s time saved)
❄️ 1 New Flaky

New Flaky Tests (1)

• test_otel_trace_across_fork - test_context.py - Last Failure

  Expand for error

   At request <Request GET /test/session/snapshot >:
      At snapshot (token='tests.opentelemetry.test_context.test_otel_trace_across_fork'):
       - Directory: /snapshots
       - CI mode: 1
       - Trace File: /snapshots/tests.opentelemetry.test_context.test_otel_trace_across_fork.json
       - Stats File: /snapshots/tests.opentelemetry.test_context.test_otel_trace_across_fork_tracestats.json
       At compare of 1 expected trace(s) to 1 received trace(s):
        At trace 'internal' (2 spans):
   Received fewer spans (1) than expected (2). Expected unmatched spans: 'internal'
  

[emmettbutler]

@brettlangdon I realized a reason using import riotfile isn't ideal: it would require the script to live in the same directory as the riotfile, because the riotfile does not exist inside of a python module with an __init__ function.

I think this change is stable now.

[brettlangdon]

@brettlangdon I realized a reason using import riotfile isn't ideal: it would require the script to live in the same directory as the riotfile, because the riotfile does not exist inside of a python module with an __init__ function.

I think this change is stable now.

it just means the script needs to be executed from the root directory no?

we could also just add the root directory to the python path either via shell script or directly in python.

🤷🏻 not trying to argue for one specific way vs the other

[emmettbutler]

@brettlangdon yup, you're right. Updated.

[brettlangdon]

overall riot parsing looks good, other than I have a question about how to convert specifiers into actual minimum versions

[emmettbutler]

@brettlangdon I've adjusted the script to avoid stripping out the specifier information (<, >=, etc) from the version string that it writes to the csv file. Because the determination of "minimum" is done based on naive gt/lt comparison there are probably some edge cases where the wrong thing gets written to the file, but I think that in the majority of cases this approach will be solid enough for use in the single-step guardrail logic.

[brettlangdon]

I think that in the majority of cases this approach will be solid enough for use in the single-step guardrail logic.

@emmettbutler what are you thinking, just parsing the specifiers in SSI and then comparing actual versions against them?

if yes, we need to keep in mind that packaging might not be present on the system.

[emmettbutler]

@brettlangdon pretty much, yes. It seems to me that including the range markers in the minimum versions file is the best we can do without going to PyPI during that file's creation. I'm trying to avoid having the script go to PyPI because I suspect the large overhead it would add is unnecessary to achieve the guardrails goal.

I think we can compare version specifiers to the degree necessary without the packaging module. That comparison really just boils down to whether or not the minimum version specifier includes a less-than sign. See my other change for an illustration of what I mean.

[emmettbutler]

@erikayasuda @ZStriker19 @brettlangdon let me know if you'd like any changes to this approach

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-06-07 14:16:41

Comparing candidate commit ade7bec in PR branch emmett.butler/min_versions with baseline commit c035b91 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 221 metrics, 9 unstable metrics.

[brettlangdon]

lgtm, but I am worried about including out test dependencies, and it isn't clear to me how to identify those from our riotfile.py 🤔