Actively tail worker stdio from supervisor agent

[ranweiler]

In the supervisor agent, incrementally read from the running worker agent's redirected stderr and stdout, instead of waiting until it exits.

The worker agent's stderr and stdout are piped to the supervisor when tasks are run. The supervisor's WorkerRunner does not use wait_with_output(), which handles this (at the cost of blocking). Instead, it makes repeated calls to to try_wait() on timer-based state transitions, and does not try to read the pipes until the worker exits. But when one of the child's pipes is full, the child can block forever waiting on a write(2), such as in a log facade implementation.

This bug has not been caught because we control the child worker agent, and until recently, it mostly only wrote to these streams using env_logger at its default log level. But recent work: (1) set more-verbose INFO level default logging, (2) logged stderr/stdout lines of child processes of the worker, and (3) some user targets logged very verbosely for debugging. This surfaced the underlying issue.

[ranweiler]

This is probably the root cause for #542.

[ghost]

Hello @bmc-msft!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.