feat: add retries & update ping/go interop test

[laurentsenta]

• Add a retry the build step
  • Note we use a few lines of shell instead of adding a new, external, dependency.
• Add a status check on Testground setup
• Reduce the run timeout
  • Test succeeds in ~ 2 min at the moment, this timeout means we're losing at worst 4 min when we detect a breaking change that hangs the test.
• Update the ping/go/compat and add the corresponding go.v0.22.mod file to fix the test.
• Reorder the go.toml and rust.toml resources
• Add a trigger on push & PR
  • This checks the interop tests on pull requests.

[laurentsenta]

Results

Rust

• no custom (master): https://github.com/laurentsenta/test-plans/actions/runs/2920569374 (15m)
• + demo/pass: https://github.com/laurentsenta/test-plans/actions/runs/2920569458 (17m)
• + demo/break: https://github.com/laurentsenta/test-plans/actions/runs/2920569546 (23m)

Go

• no custom (master): https://github.com/laurentsenta/test-plans/actions/runs/2920570006 (16m)
• + demo/pass: https://github.com/laurentsenta/test-plans/actions/runs/2920569664 (18m)
• + demo/break: https://github.com/laurentsenta/test-plans/actions/runs/2920569734 (19m)

Rust + Go

2 tests here:

• rust master + go master + a custom
• all rust, including master + all go, including master + a custom

Results:

• no custom (masters): https://github.com/laurentsenta/test-plans/actions/runs/2920569822 (15m, 30m)
• + rust - demo/pass: https://github.com/laurentsenta/test-plans/actions/runs/2920569900 (15m, 28m)
• + rust - demo/break: https://github.com/laurentsenta/test-plans/actions/runs/2920569946 (26m, 33m)
• + go - demo/pass: https://github.com/laurentsenta/test-plans/actions/runs/2920570079 (19m, 27m)
• + go - demo/break: https://github.com/laurentsenta/test-plans/actions/runs/2920570169 (22m, 40m)

[laurentsenta]

@marten-seemann @mxinden Ready for review,

In the result I shared above,
the largest test takes more than 30m on failures. That's the one where we build and test 13 versions together.

I would like to:

• enable the cross-versions test on PRs for go-libp2p (again) and for rust-libp2p.
• enable the "small" cross-implementation tests on PR.
  • this one tests go mater + rust master + the current branch

We could call the "large" cross-implementation tests nightly.

Next I'll work on stability, Testground releases, and caching to get all these stats below 10 min.

[laurentsenta]

I ran 200 tests tonight and got 2 errors:

• still a 404 during a package download
• an EOF during testground setup

I added a few more temporary tweaks.

I'll re-run 400 tests (100 for every configuration) which should take ~ 20 hours, I hope to get below 1% error rate for all the tests except "all interop".

[marten-seemann]

I ran 200 tests tonight and got 2 errors:

That seems very high. That's a 1% error rate on 3 retries, which would suggest that every single run has a 22% probability of failing. Do we have any idea why that is?

[laurentsenta]

I ran 200 tests tonight and got 2 errors:

That seems very high. That's a 1% error rate on 3 retries, which would suggest that every single run has a 22% probability of failing. Do we have any idea why that is?

My first intuition is that this simplification is incorrect:

• we didn't retry the testground setup,
• 404 errors during a package download are not independent events (probabilistically speaking).

We observed 0.5% failed workflows with the testground setup, so I added a (temporary) retry on the testground install.

We observed 0.5% failed workflows during the build because goproxy.io started throwing 404s errors for a package. The build step failed 3 times in a row, but the probability of having a 404 on attempts 2 and 3 is much MUCH higher when you got a 404 on attempt 1.

[marten-seemann]

Why are we getting 404s in the first place? That's different from a connection timeout, and seems to indicate a more fundamental problem.

[laurentsenta]

@marten-seemann To me that's a case of our proxy (proxy.golang.org btw, not goproxy.io) not having a 100% uptime SLA, but you might have more insight on this:

That's the run:
https://github.com/laurentsenta/test-plans/runs/8000694045?check_suite_focus=true

go: github.com/yuin/goldmark@v1.4.1: reading https://proxy.golang.org/github.com/yuin/goldmark/@v/v1.4.1.info: 404 Not Found
	server response: not found: temporarily unavailable

[marten-seemann]

That's annoying, in this case this shouldn't even be a 4xx error, but rather a 5xx error.
I'm still wondering why this occurs that frequently. We have other CI builds that download a lot of dependencies (every run of go-libp2p, kubo, lotus etc. will), and they don't seem to be plagued with those issues.

[laurentsenta]

@marten-seemann Agreed, we might have caught a very unlikely error with the goproxy 404.

Results for the tests I run yesterday:

• go versions interop: 100 / 100 successes
• rust versions interop: 100 / 100 successes
• rust - go interop (only masters): 100 / 100 successes
• rust - go interop (all tests): 94 / 100 successes
  • Reason: I added a 25 min build timeout, as a safeguard, and it was reached 6 times (see actions).

[mxinden]

This looks good to me. Not sure my review should have much weight for the Go related changes.

[mxinden]

Why do we need to sleep in between retries?

[laurentsenta]

The retry might happen almost instantaneously because of the docker caching. The goal here is to wait some time to give remote services a chance to improve the situation (like the 404 above).

I'll merge as is, but there will be at least one follow-up PR for improvements, where we can remove this sleep if we don't want it.

[laurentsenta]

@mxinden I updated with v0.47