Improve key decryption times for validator client keystores

[philknows]

Problem description

Reference: https://discord.com/channels/593655374469660673/1197575814494035968/1261285696157782116

As of July 12, 2024, the EF DevOps team using kurtosis for their tests require loading 64 validator keys to the client for each test seeking a faster solution. Although improvements such as #4179 was implemented to improve this UX, data shows as of today we are still lagging behind comparatively to other client implementations.

loading in 64 validator keys:
teku: ~3sec
prysm: sub 1 sec
nimbus: sub 1 sec
lh: sub 1 sec
lodestar: 11sec (this varies quite a bit)
grandine: sub 1 sec

Currently, we use an already optimized cryptography library (https://github.com/ethereum/js-ethereum-cryptography), but we should investigate how to further get to a baseline closer to Teku at ~3s to a standard acceptable by larger node operators which run hundreds/thousands of keys per client instance.

Solution description

Investigate a solution.

Additional context

Run kurtosis locally with other clients to observe key decryption times:

kurtosis run github.com/ethpandaops/ethereum-package --args-file https://raw.githubusercontent.com/ethpandaops/ethereum-package/main/.github/tests/geth-all.yaml

[nflaig]

For the record, these are the latest benchmarks from unstable branch

Importing 2000 pbkdf2 keystores takes ~20 seconds

Jul-16 16:53:36.351[]                 info: Connecting to LevelDB database path=/home/nico/.local/share/lodestar/mainnet/validator-db
Jul-16 16:53:46.393[]                 info: 50% of local keystores imported. current=1007 total=2000 rate=6041.40keys/m
Jul-16 16:53:55.148[]                 info: 100% of local keystores imported. current=2000 total=2000 rate=6805.25keys/m
Jul-16 16:53:56.039[]                 info: 2000 local keystores

Importing 2000 scrypt keystores takes ~4 minutes

Jul-16 15:58:22.276[]                 info: Connecting to LevelDB database path=/home/devops/.local/share/lodestar/holesky/validator-db
Jul-16 15:58:32.316[]                 info: 4% of keystores imported. current=71 total=2000 rate=426.00keys/m
Jul-16 15:58:42.316[]                 info: 8% of keystores imported. current=150 total=2000 rate=474.00keys/m
Jul-16 15:58:52.316[]                 info: 12% of keystores imported. current=234 total=2000 rate=504.00keys/m
Jul-16 15:59:02.317[]                 info: 16% of keystores imported. current=312 total=2000 rate=467.95keys/m
// ....
Jul-16 16:02:02.335[]                 info: 88% of keystores imported. current=1754 total=2000 rate=431.96keys/m
Jul-16 16:02:12.335[]                 info: 92% of keystores imported. current=1836 total=2000 rate=492.00keys/m
Jul-16 16:02:22.336[]                 info: 96% of keystores imported. current=1927 total=2000 rate=545.95keys/m
Jul-16 16:02:31.771[]                 info: 100% of keystores imported. current=2000 total=2000 rate=464.23keys/m
Jul-16 16:02:32.140[]                 info: 2000 local keystores

And in Kurtosis with --disableKeystoresThreadPool flag, it's pretty much instant (sub 1 second)

Jul-16 16:07:41.657[]                 info: Connecting to LevelDB database path=/root/.local/share/lodestar/testnet/validator-db
Jul-16 16:07:41.960[]                 info: 100% of local keystores imported. current=64 total=64 rate=12842.81keys/m
Jul-16 16:07:42.569[]                 info: 64 local keystores