--message-format= should also imply a corresponding flag to test harnesses

[nagisa]

Describe the problem you are trying to solve

I currently invoke cargo with --message-format=json in order to parse its output and present it to development/review tooling as in-line messages. Cargo currently does not support presenting the test output in a different way. In order to obtain machine-readable output from tests, I need to build the tests with --no-run, collect the test artifacts, and run them manually, while passing --format=json to the harnesses and parse that output, instead.

Describe the solution you'd like

Running with --message-format= should output test progress, status and similar information as the requested message format.
Notes

This would also enable some progress towards #5609. See also #5609 (comment)

[ehuss]

It would be nice to get JSON output from libtest. There are a few other issues where Cargo could benefit from using the JSON output (#1983, #2832, #4324). There are a number of issues that make this tricky:

• libtest JSON is still unstable, and there is a lot of uncertainty about what the eventual format will look like (Tracking issue for libtest JSON output rust#49359).
• There isn't a well-defined CLI protocol between Cargo and the test harness. This is primarily an issue if harness=false or if using a custom test framework. As noted in Disabling colored output requires convoluted cargo test --color=never -- --color=never incantation #1983 (comment), there are quite a few other flags that would be nice to forward to the harness. I think it would be good to have some strategy on how that should work with custom test frameworks (should Cargo just avoid passing the flags? should custom test frameworks define a common set of command-line flags?)
• I'm uncertain how to make a transition for something like this, since existing tools will be expecting normal text output, changing the behavior would likely break them. I suspect Cargo may need to add something like rustc's json flag to add options for changing the JSON behavior.

[nagisa]

I've been thinking about this further recently, and given then custom harnesses, it seems to me that not trying to interact with or handle test suites is probably a good intermediate step. That is, we could initially implement json messages that report status of test suite itself, in a way that is not dependent on how test suite outputs it data. For example:

When test suite run begins:

{"reason": "test-suite", "kind": "begin", "executable": "...", "etc": "etc"}

While test suite is printing stuff to their stdout:

{"reason": "test-suite", "kind": "output-line", "executable": "...", "line": "running N tests"}

And once its done:

{"reason": "test-suite", "kind": "finished", "executable": "....", "success": true}

That way support and format of output from test harnesses is entirely decoupled from cargo and tools that are interested in parsing said output can still do so line-by-line by looking at test-suite/output-line events (and possibly adding -- --output=json -Zunstable-options to the CLI themselves).