Semaphore timeout in FranzGoVerifiableWithSiTest.test_si_with_timeboxed

[VladLazar]

A semaphore timed out in the NTP archiver upload loop and the exception bubbled up stopping the loop:

upload loop error: seastar::semaphore_timed_out (Semaphore timedout)

The latest occurence is from Friday (5 Aug), but there's more in the past: https://buildkite.com/redpanda/vtools/builds/3123#01826bf6-06ed-43d7-b0cf-5124782ac879.

[VladLazar]

The important clue in the error message is that the semaphore is not named. All the semaphores explicitly created by redpanda are named. This means that the time-out comes from some other concurrency primitive that uses seastar::semaphore under the hood. Some digging later I found the the segment_read_lock in the archiver which matches the description.

The reason for the failure is that the test causes contention on the lock. In the current configuration every segment fetch from SI causes a cache eviction (see #5915). That fix might help here too.

The underlying problem of not handling semaphore time outs still remains though. Perhaps we should catch the exception here and silence it as we do for other exceptions in the upload loop.