Request: Allow configuration of Spring Scheduling to not instrument some @Scheduled methods

[GrahamLea]

(Yeah, I'm raising this a 2nd time. I've explained why below.)

Problem

Spring Scheduling instrumentation can only be either on or off. A function @Scheduled to run every second gets traced every second, even if it does nothing interesting. There is no way to ask the instrumentation not to instrument individual function.

Critically, just because I don't want a @Scheduled function instrumented, doesn't mean that I want to see no tracing from the work it might do. Sometimes, I want to make a decision deeper in the stack about whether to start a trace.

Possible Solutions

It'd be great to be able to annotate a @Scheduled method with something like @WithoutSpan and have it not start a trace. (I believe the Spring scheduling instrumentation always starts a new trace? Not because the code is written to do that, but because of the nature of how Spring Scheduling works. I could be wrong about that, though.)

Alternatively, a configuration option for the Agent could work, though is less desirable than an annotation. The Agent appears to already have over 70 config variables programmed into it, including in Spring components like Integration and Batch.

Alternatives Considered

I originally raised this in #5008, and received a lot of help from @trask to implement a Sampler extension. However, I found two problems with that solution:

1. The implementation of the Sampler is non-trivial and confusing. Because I sometimes want to trace some of the work within the @Scheduled function, but not the function itself, I need to be selective about when I do and don't fall back to the ParentBasedSampler. That decision is made in part by communicating between multiple invocations of the Sampler using custom attributes that are pulled out by casting the parent Span to ReadableSpan. 😝
2. Even when it works, the result is a bit dodgy anyway. When I RECORD_ONLY the @Scheduled function then RECORD_AND_SAMPLE its children, what I end up with in my tracer is of course a rootless trace and a warning about missing spans.

So, the recommendations in #5008, while given in good faith, have lead down the path of creating a convoluted Sampler that, when functioning as designed, creates rootless traces.

When I sat back and looked at how it worked, I realised what I've created is a kludge to work around (poorly) the fact that all @Scheduled functions are being instrumented when I just don't want them to be.

I suppose another alternative is to disable Spring Scheduling tracing altogether and annotate almost all my @Scheduled functions with @WithSpan. It would work (probably?), but it's not ideal because it's only a minority of tasks that I want to avoid instrumenting, and it opens the codebase up to human error when the default is to not trace rather than to trace.

For the moment, I'm going to return to my Spring aspect hack which undoes the OTel Spring Scheduling instrumentation wrapping based on the presence of my own custom annotation.

Contributing

I'd be happy to do the work on this (with some guidance) if you decide to go ahead with implementing it.

[trask]

Critically, just because I don't want a @Scheduled function instrumented, doesn't mean that I want to see no tracing from the work it might do. Sometimes, I want to make a decision deeper in the stack about whether to start a trace.

ya, this does make the sampler approach problematic

some kind of generic @WithoutSpan is going to be tough because the instrumentation point for spring scheduling isn't really the @Scheduled method

I wonder if in this case of needing something more custom, it would help if we had a "spring scheduling library instrumentation" which could more naturally facilitate more (programmatic) hooks, and you could use that instead and disable the spring scheduling java agent instrumentation (https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/#suppressing-specific-agent-instrumentation)