Use Codspeed for continous benchmarking

[MichaReiser]

Summary

This PR replaces our custom benchmarking CI step with CodSpeed.

CodSpeed requires us to use their criterion API wrapper. However, only using their wrapper has the downside that running the benchmarks locally becomes impossible (because it is unsupported). I worked around this by adding a feature flag to our benchmark crate that controls whether we use criterion's API directly or codespeed's wrapper.

Test Plan

CI run Running the benchmarks is significantly faster. I hope that build time will improve with a hot cache.

The data shows up in the dashboard

[MichaReiser]

Current dependencies on/for this PR:

• main
  • PR Use Codspeed for continous benchmarking #6896 👈

This comment was auto-generated by Graphite.

[codspeed-hq]

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

🆕 16 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

• formatter[numpy/globals.py] (1.4 ms)
• linter/default-rules[numpy/globals.py] (2.2 ms)
• formatter[large/dataset.py] (72.6 ms)
• formatter[pydantic/types.py] (26.3 ms)
• linter/default-rules[pydantic/types.py] (38.8 ms)
• linter/default-rules[numpy/ctypeslib.py] (18.7 ms)
• linter/default-rules[large/dataset.py] (94.1 ms)
• linter/all-rules[numpy/ctypeslib.py] (34.3 ms)
• parser[numpy/ctypeslib.py] (12.5 ms)
• parser[pydantic/types.py] (26.8 ms)
• parser[large/dataset.py] (68.8 ms)
• parser[numpy/globals.py] (1.4 ms)
• linter/all-rules[pydantic/types.py] (71.7 ms)
• formatter[numpy/ctypeslib.py] (13.9 ms)
• linter/all-rules[large/dataset.py] (156.2 ms)
• linter/all-rules[numpy/globals.py] (4 ms)

[github-actions]

PR Check Results

Ecosystem

✅ ecosystem check detected no changes.

[MichaReiser]

@Boshen you might enjoy this. Or you can wait and check in with me to ask about our experience with it.

[charliermarsh]

Looks great. Excited to try it :)