flaky TestRSASignZero (using OpenSSL)

[marten-seemann]

=== RUN   TestRSASignZero
    rsa_test.go:74: failed to generate RSA key
--- FAIL: TestRSASignZero (1.38s)

[marten-seemann]

This is where the failure comes from: https://github.com/libp2p/go-openssl/blob/00b60808a1ace0fb1c21c840e3e30f98a88cb2ae/key.go#L428

[marten-seemann]

I have no idea why this occurs, and I don't really know how to debug this further, as the problem happens inside OpenSSL, not in our code.

We should add this to our list of reasons to deprecate OpenSSL in favor of BoringCrypto.

[marten-seemann]

@MarcoPolo pointed out that there's ERR_get_error, which returns an error code. Not sure how to do that in a race-free way without locking a global mutex, but at least that allows further debugging.

[marten-seemann]

Partial progress. Here's the error code:

=== RUN   TestKeys
    key_test.go:283: failed to generate RSA key: 33554536

Just no idea what it means. Google is not very helpful on this one.

[marten-seemann]

rsa routines is the human readable error code, obtained with ERR_lib_error_string. Not very helpful.

See libp2p/go-openssl@master...libp2p:go-openssl:debug-generate-rsa for the code.

[marten-seemann]

@Jorropo Do you by any chance have an idea what's going on here?

[marten-seemann]

I tried to reproduce this both in a VM and on a Ubuntu server, but it works. Seems to be related to GitHub Actions CI setup.

I've disabled the go-crypto workflow for now: https://github.com/libp2p/go-libp2p/actions/workflows/go-crypto.yml