GPG signing tests fail on my machine (when gpg is not installed)

[ilyagr]

I'm getting these errors at commit 84685a4. I haven't investigated it, but perhaps @julienvincent has some idea? They seem to pass in the CI.

🐠 RUST_BACKTRACE=1 cargo insta test --accept --workspace --test-runner nextest -N -- test_gpg
    Finished test [unoptimized + debuginfo] target(s) in 0.11s
    Starting 4 tests across 9 binaries (1040 skipped)
        FAIL [   0.053s] jj-lib::runner test_gpg::unknown_key

--- STDOUT:              jj-lib::runner test_gpg::unknown_key ---

running 1 test
test test_gpg::unknown_key ... FAILED

failures:

failures:
    test_gpg::unknown_key

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 309 filtered out; finished in 0.05s


--- STDERR:              jj-lib::runner test_gpg::unknown_key ---
thread 'test_gpg::unknown_key' panicked at lib/tests/test_gpg.rs:56:14:
called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
stack backtrace:
   0: rust_begin_unwind
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/result.rs:1653:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/result.rs:1077:23
   4: runner::test_gpg::GpgEnvironment::new
             at ./tests/test_gpg.rs:48:23
   5: runner::test_gpg::unknown_key
             at ./tests/test_gpg.rs:145:15
   6: runner::test_gpg::unknown_key::{{closure}}
             at ./tests/test_gpg.rs:144:17
   7: core::ops::function::FnOnce::call_once
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/ops/function.rs:250:5
   8: core::ops::function::FnOnce::call_once
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

        FAIL [   0.054s] jj-lib::runner test_gpg::gpg_singing_roundtrip

--- STDOUT:              jj-lib::runner test_gpg::gpg_singing_roundtrip ---

running 1 test
test test_gpg::gpg_singing_roundtrip ... FAILED

failures:

failures:
    test_gpg::gpg_singing_roundtrip

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 309 filtered out; finished in 0.05s


--- STDERR:              jj-lib::runner test_gpg::gpg_singing_roundtrip ---
thread 'test_gpg::gpg_singing_roundtrip' panicked at lib/tests/test_gpg.rs:56:14:
called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
stack backtrace:
   0: rust_begin_unwind
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/result.rs:1653:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/result.rs:1077:23
   4: runner::test_gpg::GpgEnvironment::new
             at ./tests/test_gpg.rs:48:23
   5: runner::test_gpg::gpg_singing_roundtrip
             at ./tests/test_gpg.rs:90:15
   6: runner::test_gpg::gpg_singing_roundtrip::{{closure}}
             at ./tests/test_gpg.rs:89:27
   7: core::ops::function::FnOnce::call_once
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/ops/function.rs:250:5
   8: core::ops::function::FnOnce::call_once
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

        FAIL [   0.054s] jj-lib::runner test_gpg::invalid_signature

--- STDOUT:              jj-lib::runner test_gpg::invalid_signature ---

running 1 test
test test_gpg::invalid_signature ... FAILED

failures:

failures:
    test_gpg::invalid_signature

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 309 filtered out; finished in 0.05s


--- STDERR:              jj-lib::runner test_gpg::invalid_signature ---
thread 'test_gpg::invalid_signature' panicked at lib/tests/test_gpg.rs:56:14:
called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
stack backtrace:
   0: rust_begin_unwind
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/result.rs:1653:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/result.rs:1077:23
   4: runner::test_gpg::GpgEnvironment::new
             at ./tests/test_gpg.rs:48:23
   5: runner::test_gpg::invalid_signature
             at ./tests/test_gpg.rs:176:15
   6: runner::test_gpg::invalid_signature::{{closure}}
             at ./tests/test_gpg.rs:175:23
   7: core::ops::function::FnOnce::call_once
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/ops/function.rs:250:5
   8: core::ops::function::FnOnce::call_once
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

        FAIL [   0.054s] jj-lib::runner test_gpg::gpg_signing_roundtrip_explicit_key

--- STDOUT:              jj-lib::runner test_gpg::gpg_signing_roundtrip_explicit_key ---

running 1 test
test test_gpg::gpg_signing_roundtrip_explicit_key ... FAILED

failures:

failures:
    test_gpg::gpg_signing_roundtrip_explicit_key

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 309 filtered out; finished in 0.05s


--- STDERR:              jj-lib::runner test_gpg::gpg_signing_roundtrip_explicit_key ---
thread 'test_gpg::gpg_signing_roundtrip_explicit_key' panicked at lib/tests/test_gpg.rs:56:14:
called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
stack backtrace:
   0: rust_begin_unwind
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/result.rs:1653:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/result.rs:1077:23
   4: runner::test_gpg::GpgEnvironment::new
             at ./tests/test_gpg.rs:48:23
   5: runner::test_gpg::gpg_signing_roundtrip_explicit_key
             at ./tests/test_gpg.rs:114:15
   6: runner::test_gpg::gpg_signing_roundtrip_explicit_key::{{closure}}
             at ./tests/test_gpg.rs:113:40
   7: core::ops::function::FnOnce::call_once
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/ops/function.rs:250:5
   8: core::ops::function::FnOnce::call_once
             at /rustc/d44e3b95cb9d410d89cb8ab3233906a33f43756a/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

------------
     Summary [   0.055s] 4 tests run: 0 passed, 4 failed, 1040 skipped
        FAIL [   0.054s] jj-lib::runner test_gpg::gpg_signing_roundtrip_explicit_key
        FAIL [   0.054s] jj-lib::runner test_gpg::gpg_singing_roundtrip
        FAIL [   0.054s] jj-lib::runner test_gpg::invalid_signature
        FAIL [   0.053s] jj-lib::runner test_gpg::unknown_key
error: test run failed
done: no snapshots to review

• Platform: MacOS

🐠 rustc -Vv
rustc 1.78.0-nightly (d44e3b95c 2024-02-09)
binary: rustc
commit-hash: d44e3b95cb9d410d89cb8ab3233906a33f43756a
commit-date: 2024-02-09
host: aarch64-apple-darwin
release: 1.78.0-nightly
LLVM version: 17.0.6

[ilyagr]

I get the same errors (and sometimes, randomly, only two of the errors?) on stable rust:

🐠 rustc +1.76 -Vv
rustc 1.76.0 (07dca489a 2024-02-04)
binary: rustc
commit-hash: 07dca489ac2d933c78d3c5158e3f43beefeb02ce
commit-date: 2024-02-04
host: aarch64-apple-darwin
release: 1.76.0
LLVM version: 17.0.6

[ilyagr]

I think the issue is simply that I don't have gpg installed. Installing it fixed the problem.

We should either mention this in CONTRIBUTING.md or, better, find a way to not require it.

[yuja]

We'll need something like this for gpg and ssh-keygen:

jj/lib/tests/test_git_backend.rs

Lines 47 to 50 in 106483a

	// TODO: Better way to disable the test if git command couldn't be executed
	if Command::new("git").arg("--version").status().is_err() {
	eprintln!("Skipping because git command might fail to run");
	return;

This is ugly, but I don't think cargo natively supports skipping tests conditionally.

[ilyagr]

We could also check the CI environment variable that GitHub CI (and I think other CIs) sets, and fail the test in that case if gpg/git do not run.

I'm also slightly confused about why I sometimes had 4 tests fail and sometimes 2, but I'll assume this has to do with some testing infrastrucutre "fail fast" logic.

[julienvincent]

@yuja What about using features for this? Then we could run cargo test --features signing or something like that. Not sure how good/bad of an idea that is.

[yuja]

What about using features for this? Then we could run cargo test --features signing or something like that.

Doing that would make the test failures reported late in CI because I wouldn't bother to set the feature flags. These features could be detected by build.rs, but it'll be another mess.

FWIW, cargo appears to implement custom proc macro for this purpose.

• Add requirements to cargo_test. rust-lang/cargo#9892
• Runtime skipping of tests la10736/rstest#162