ref(seer grouping): Use new `CircuitBreaker` class for circuit breaking #74563

lobsterkatie · 2024-07-19T15:46:52Z

The current implementation of circuit breaking for Seer similarity requests, using with_circuit_breaker, doesn't work because any errors the breaker might track are caught much farther down the call stack and never make it to the breaker. This fixes that by switching from using with_circuit_breaker to using an instance of the new CircuitBreaker class, which allows us to track the errors at a low level, before they're caught, while still using the breaker to decide whether to make the request at a much higher level.

codecov · 2024-07-19T16:19:08Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.16%. Comparing base (0d530a6) to head (8186b35).
Report is 6 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #74563      +/-   ##
==========================================
+ Coverage   78.14%   78.16%   +0.01%     
==========================================
  Files        6731     6731              
  Lines      300214   300257      +43     
  Branches    51642    51648       +6     
==========================================
+ Hits       234608   234684      +76     
+ Misses      59285    59250      -35     
- Partials     6321     6323       +2

Files	Coverage Δ
src/sentry/conf/server.py	`87.78% <100.00%> (+0.02%)`	⬆️
src/sentry/grouping/ingest/seer.py	`100.00% <100.00%> (ø)`
src/sentry/options/defaults.py	`100.00% <ø> (ø)`
src/sentry/seer/similarity/similar_issues.py	`97.40% <100.00%> (+0.21%)`	⬆️
src/sentry/utils/circuit_breaker2.py	`97.61% <100.00%> (+30.60%)`	⬆️

... and 5 files with indirect coverage changes

src/sentry/event_manager.py

src/sentry/grouping/ingest/seer.py

armenzg · 2024-07-22T17:24:10Z

src/sentry/grouping/ingest/seer.py

+        metrics.incr(
+            "grouping.similarity.did_call_seer",
+            sample_rate=1.0,
+            tags={"call_made": False, "blocker": "circuit-breaker"},


Suggestion: You could use a centralized constant to keep track of all the reasons that could block

e.g.

class SEER_BLOCKING_REASONS(Enum): CIRCUIT_BREAKER = "circuit-breaker" ...

I could, it's true. It'd have to be imported from utils, since different versions of the metric are recorded there, too.

I'm not sure there's a huge advantage there if I'm only using each one once, though, is there?

src/sentry/grouping/ingest/seer.py

asottile-sentry · 2024-07-24T13:30:46Z

src/sentry/seer/similarity/similar_issues.py

@@ -30,6 +36,11 @@
    timeout=settings.SEER_GROUPING_TIMEOUT,
 )

+seer_similarity_circuit_breaker = CircuitBreaker(
+    SEER_SIMILARITY_CIRCUIT_BREAKER_KEY,
+    options.get("seer.similarity.circuit-breaker-config"),


an options call at the module is only reflected at import time (and is always the default because the database connection isn't set up)

this is breaking mypy locally because the database isn't set up

and is always the default because the database connection isn't set up

Oh, very good to know. Thanks for catching that! I'll switch it to initializing on the fly.

ah yeah -- thought it would've stuck from the last time

Ah, missed that - thanks for pointing it out!

…t breaking (#74563)" This reverts commit e117cd0. Co-authored-by: asottile-sentry <103459774+asottile-sentry@users.noreply.github.com>

getsentry-bot · 2024-07-24T13:31:18Z

PR reverted: 1bb6a50

This is a redo of #74563 (which got reverted), with the difference that the circuit breaker is now reinstantiated before each use, rather than being instantiated once at the module level. (The problem with that setup was that the module turns out to initialize before our database connections are established, meaning we only get the hardcoded default config rather than the one provided by the options automator.)

…t breaking (#74563)" This reverts commit e117cd0. Co-authored-by: asottile-sentry <103459774+asottile-sentry@users.noreply.github.com>

This is a redo of #74563 (which got reverted), with the difference that the circuit breaker is now reinstantiated before each use, rather than being instantiated once at the module level. (The problem with that setup was that the module turns out to initialize before our database connections are established, meaning we only get the hardcoded default config rather than the one provided by the options automator.)

vercel bot deployed to Preview July 19, 2024 15:47 View deployment

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 19, 2024

lobsterkatie force-pushed the kmclb-switch-seer-similarity-to-use-new-circuit-breaker branch from aa80763 to e88dfb9 Compare July 19, 2024 17:16

vercel bot deployed to Preview July 19, 2024 17:18 View deployment

lobsterkatie marked this pull request as ready for review July 19, 2024 18:42

lobsterkatie requested a review from a team as a code owner July 19, 2024 18:42

armenzg approved these changes Jul 22, 2024

View reviewed changes

lobsterkatie force-pushed the kmclb-add-core-CircuitBreaker-methods branch from 6bd2c86 to 526edc4 Compare July 23, 2024 00:48

lobsterkatie force-pushed the kmclb-switch-seer-similarity-to-use-new-circuit-breaker branch from e88dfb9 to 3762351 Compare July 23, 2024 00:50

vercel bot deployed to Preview July 23, 2024 00:54 View deployment

lobsterkatie force-pushed the kmclb-switch-seer-similarity-to-use-new-circuit-breaker branch from 3762351 to c286d0c Compare July 23, 2024 06:00

vercel bot deployed to Preview July 23, 2024 06:02 View deployment

lobsterkatie force-pushed the kmclb-add-core-CircuitBreaker-methods branch from e0d7108 to 347a4b1 Compare July 23, 2024 06:17

lobsterkatie force-pushed the kmclb-switch-seer-similarity-to-use-new-circuit-breaker branch from c286d0c to 46f8edb Compare July 23, 2024 06:20

vercel bot deployed to Preview July 23, 2024 06:22 View deployment

lobsterkatie force-pushed the kmclb-add-core-CircuitBreaker-methods branch from 347a4b1 to a6d8786 Compare July 23, 2024 17:36

lobsterkatie force-pushed the kmclb-switch-seer-similarity-to-use-new-circuit-breaker branch from 46f8edb to 2328eb0 Compare July 23, 2024 17:40

vercel bot deployed to Preview July 23, 2024 17:42 View deployment

lobsterkatie force-pushed the kmclb-add-core-CircuitBreaker-methods branch from a6d8786 to 1713e4b Compare July 23, 2024 18:13

lobsterkatie force-pushed the kmclb-switch-seer-similarity-to-use-new-circuit-breaker branch from 2328eb0 to 8186b35 Compare July 23, 2024 18:15

vercel bot deployed to Preview July 23, 2024 18:17 View deployment

Base automatically changed from kmclb-add-core-CircuitBreaker-methods to master July 23, 2024 18:48

lobsterkatie added 6 commits July 23, 2024 11:51

stop using with_circuit_breaker

85e252f

consolidate metrics calls

66dd5e6

add circuit breaker for seer similarity

192966c

check circuit breaker in should_call_seer_for_grouping

64067c7

track errors with circuit breaker in get_similarity_data_from_seer

36b6321

update tests

f7ac804

lobsterkatie force-pushed the kmclb-switch-seer-similarity-to-use-new-circuit-breaker branch from 8186b35 to f7ac804 Compare July 23, 2024 18:54

vercel bot deployed to Preview July 23, 2024 18:56 View deployment

lobsterkatie merged commit e117cd0 into master Jul 23, 2024
49 checks passed

lobsterkatie deleted the kmclb-switch-seer-similarity-to-use-new-circuit-breaker branch July 23, 2024 19:36

asottile-sentry reviewed Jul 24, 2024

View reviewed changes

asottile-sentry added the Trigger: Revert add to a merged PR to revert it (skips CI) label Jul 24, 2024

lobsterkatie mentioned this pull request Jul 24, 2024

ref(seer grouping): Use new CircuitBreaker class, take 2 #74898

Merged

github-actions bot locked and limited conversation to collaborators Aug 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ref(seer grouping): Use new `CircuitBreaker` class for circuit breaking #74563

ref(seer grouping): Use new `CircuitBreaker` class for circuit breaking #74563

lobsterkatie commented Jul 19, 2024

codecov bot commented Jul 19, 2024 •

edited

Loading

armenzg Jul 22, 2024

lobsterkatie Jul 23, 2024

asottile-sentry Jul 24, 2024

lobsterkatie Jul 24, 2024

asottile-sentry Jul 24, 2024

lobsterkatie Jul 24, 2024

getsentry-bot commented Jul 24, 2024

ref(seer grouping): Use new CircuitBreaker class for circuit breaking #74563

ref(seer grouping): Use new CircuitBreaker class for circuit breaking #74563

Conversation

lobsterkatie commented Jul 19, 2024

codecov bot commented Jul 19, 2024 • edited Loading

Codecov Report

armenzg Jul 22, 2024

Choose a reason for hiding this comment

lobsterkatie Jul 23, 2024

Choose a reason for hiding this comment

asottile-sentry Jul 24, 2024

Choose a reason for hiding this comment

lobsterkatie Jul 24, 2024

Choose a reason for hiding this comment

asottile-sentry Jul 24, 2024

Choose a reason for hiding this comment

lobsterkatie Jul 24, 2024

Choose a reason for hiding this comment

getsentry-bot commented Jul 24, 2024

ref(seer grouping): Use new `CircuitBreaker` class for circuit breaking #74563

ref(seer grouping): Use new `CircuitBreaker` class for circuit breaking #74563

codecov bot commented Jul 19, 2024 •

edited

Loading