fix(iast): improve overhead control logic (#8452)

[gnufede]

IAST: Improve overhead control logic so the decision to analyze a request is done at span start and is saved at the span level using the core API. This should fix issues where requests were analyzed when they shouldn't be and viceversa.

• Change(s) are motivated and described in the PR description

• Testing strategy is described if automated tests are not included in the PR

• Risks are described (performance impact, potential for breakage, maintainability)

• Change is maintainable (easy to change, telemetry, documentation)

• Library release note guidelines are followed or label changelog/no-changelog is set

• Documentation is included (in-code, generated user docs, public corp docs)

• Backport labels are set (if applicable)

• If this PR changes the public interface, I've notified @DataDog/apm-tees.

• If change touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.

• Title is accurate

• All changes are related to the pull request's stated goal

• Description motivates each change

• Avoids breaking API changes

• Testing strategy adequately addresses listed risks

• Change is maintainable (easy to change, telemetry, documentation)

• Release note makes sense to a user of the library

• Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment

• Backport labels are set in a manner that is consistent with the release branch maintenance
  policy

(cherry picked from commit 7e8aaac)

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: backport-8452-to-2.5
Commit report: f90965c
Test service: dd-trace-py

✅ 0 Failed, 36436 Passed, 1397 Skipped, 1h 19m 22.07s Total duration (38m 27.09s time saved)

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-02-22 15:46:52

Comparing candidate commit f90965c in PR branch backport-8452-to-2.5 with baseline commit 22fcb7f in branch 2.5.

Found 7 performance improvements and 15 performance regressions! Performance is the same for 173 metrics, 9 unstable metrics.

scenario:coreapiscenario-core_dispatch_with_results_listeners_and_all_listeners

• 🟩 max_rss_usage [-826.263KB; -702.774KB] or [-2.805%; -2.386%]

scenario:coreapiscenario-core_dispatch_with_results_no_listeners

• 🟩 max_rss_usage [-805.781KB; -652.805KB] or [-2.746%; -2.225%]

scenario:coreapiscenario-set_item

• 🟥 max_rss_usage [+606.839KB; +763.683KB] or [+2.113%; +2.659%]

scenario:httppropagationextract-datadog_tracecontext_tracestate_not_propagated_on_trace_id_no_match

• 🟥 max_rss_usage [+745.244KB; +916.094KB] or [+2.609%; +3.207%]

scenario:httppropagationextract-datadog_tracecontext_tracestate_propagated_on_trace_id_match

• 🟥 max_rss_usage [+652.627KB; +837.498KB] or [+2.283%; +2.929%]

scenario:httppropagationextract-empty_headers

• 🟩 max_rss_usage [-787.136KB; -594.854KB] or [-2.677%; -2.023%]

scenario:httppropagationextract-invalid_priority_header

• 🟥 max_rss_usage [+610.627KB; +779.145KB] or [+2.130%; +2.718%]

scenario:httppropagationextract-invalid_trace_id_header

• 🟩 max_rss_usage [-932.274KB; -732.340KB] or [-3.169%; -2.489%]

scenario:httppropagationextract-large_header_no_matches

• 🟥 max_rss_usage [+830.197KB; +1027.339KB] or [+2.919%; +3.612%]

scenario:httppropagationextract-medium_header_no_matches

• 🟥 max_rss_usage [+841.607KB; +987.257KB] or [+2.961%; +3.473%]

scenario:httppropagationextract-none_propagation_style

• 🟥 max_rss_usage [+838.746KB; +1022.886KB] or [+2.952%; +3.600%]

scenario:httppropagationinject-ids_only

• 🟩 max_rss_usage [-828.374KB; -684.688KB] or [-2.828%; -2.338%]

scenario:httppropagationinject-with_all

• 🟥 max_rss_usage [+909.750KB; +1034.621KB] or [+3.203%; +3.642%]

scenario:httppropagationinject-with_tags

• 🟩 max_rss_usage [-719.598KB; -589.074KB] or [-2.459%; -2.013%]

scenario:sethttpmeta-obfuscation-regular-case-explicit-query

• 🟥 max_rss_usage [+718.360KB; +812.725KB] or [+2.462%; +2.786%]

scenario:sethttpmeta-obfuscation-regular-case-implicit-query

• 🟥 max_rss_usage [+678.804KB; +779.372KB] or [+2.325%; +2.669%]

scenario:sethttpmeta-obfuscation-worst-case-explicit-query

• 🟥 max_rss_usage [+723.989KB; +828.804KB] or [+2.477%; +2.836%]

scenario:sethttpmeta-useragentvariant_exists_1

• 🟥 max_rss_usage [+645.301KB; +743.243KB] or [+2.224%; +2.561%]

scenario:sethttpmeta-useragentvariant_not_exists_2

• 🟩 max_rss_usage [-745.240KB; -640.027KB] or [-2.506%; -2.152%]

scenario:tracer-large

• 🟥 max_rss_usage [+1.095MB; +1.209MB] or [+3.755%; +4.144%]

scenario:tracer-medium

• 🟥 max_rss_usage [+657.846KB; +750.769KB] or [+2.307%; +2.633%]

scenario:tracer-small

• 🟥 max_rss_usage [+876.653KB; +988.665KB] or [+3.088%; +3.483%]