Should examples be unit-testable?

[ehuss]

As part of #5384, I changed it so that examples aren't unit-tested (to be consistent), but I'm having second thoughts since it is a breaking change (and probably doesn't make complete sense).

If they should be unit-testable, under which circumstances should they be tested? I think the old behavior was a little confusing (sometimes they were tested, sometimes they weren't).

We could treat them like bins (compile twice, once as a test, once regularly), but that would cause cargo test to build examples twice for the vast majority of users who don't use this.

Another idea is to only compile them as tests if test = true is set for the example in Cargo.toml.

I'd also be fine with restoring the old behavior.

For reference, here is a list of crates on crates.io that have #[test] in an example:

vst-0.0.1
alga-0.5.3
algebra-0.2.0
algs4-0.7.0
combine-3.1.0
dessert-0.1.2
error_def-0.3.16
gauthz-0.1.0
i2cdev-0.4.0
iron-test-0.6.0
jsonnet-rs-0.5.0
nickel-0.10.1
parrot-0.1.0
partial-io-0.3.0
pathfinding-0.7.1
plague-0.6.3
proptest-0.5.1
rustcov-0.0.0
simple-munin-plugin-0.1.0
topdown-rs-0.3.3

[alexcrichton]

I personally like the idea of test = true being an opt-in for examples, I think that makes a lot of sense!

[Mark-Simulacrum]

I agree with making this opt-in by default, but perhaps it would be nice to have a heuristic that warns if #[test] is present in examples but not enabled in Cargo's config? I'm not sure if that's viable from a performance standpoint but I'd otherwise be somewhat worried about discoverability of this change.