Slurm timeouts that happen during MiniWDL container runs are not detected as timeouts

[adamnovak]

Cecilia reported a workflow run where the jobs were printing a message from MiniWDL saying that they saw signal 2, and failing.

I think they are getting the timeout signal from Slurm, but MiniWDL's signal handler is replacing the one we install/the default Python one and not letting us see the timeout signal that we expect to make the worker actually fail the job in a way we recognize as a timeout. So we don't get any of the useful user-facing timeout logging and the user thinks the job is actually failing and not just timing out.

We either need to hack MiniWDL's signal handlers, or detect when MinIWDL is raising its WDL.runtime.error.Terminated exception and treat it as a timeout (at least under Slurm).

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1637