-
-
Notifications
You must be signed in to change notification settings - Fork 281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Beacon Chain Harness For Testing #6518
Comments
Really like the idea. I know I've personally wanted this for testing and would find it useful. |
we have what is it do you think from the above mentioned points that is lacking there? would love to see how we can make it more powerful / complete |
yes it'll produce something similar to |
Looks like this would facilitate integration testing. The 'feature' is testing infrastructure that lets us use realistic data rather than stubs. |
Agree with what @wemeetagain said. Should be integration testing. But realistically, this new test utils can be used in various places such as
Oh that's a good point. Off the top of my head:
We can discuss this elsewhere |
Tagging as help-wanted to signal as a potential project for an EPF candidate. |
Problem description
Currently when it comes to testing the correctness of new changes that involve interacting with
BeaconChain
we rely on 1) unit tests 2) simulation tests 3) spec tests.Unit tests require mocking up a chain, blocks and/or states. However since these objects contain many fields, one can only mock small parts of them that are related to the new changes. The nature of Ethereum protocol mandates validation in many places and a chain/block/state is only considered valid if the combination of the value in all fields pass the validation. Therefore passing unit tests with mocked chain/block/state does not guarantee the changes will work with a valid chain/block/state. Lodestar provides
MockedBeaconChain
but it only provides a chain that is TS type check correct and not protocol correct.Simulation test creates an ephemeral devnet to test the overall functionality of Lodestar. It might be an overkill to test a small change. Also its error outputs might not be useful to tell which part of the new change is wrong. It is also resource intensive that is not designed to run on local machines for sanity checks but on the CIs.
Spec tests cover the correctness of the consensus protocol only. Things like caches, regen, Beacon api also interact with chain/block/state but not part of the protocol.
It is not hard to see the benefit of the ability to easily create a valid and functioning chain/block/state that is interact-able during unit tests.
Solution description
Create a suite of test utilities that can
IBeaconChain
interfaceThe test suite in general should be:
Additional context
Lighthouse's harness for reference: https://github.com/sigp/lighthouse/blob/b5bae6e7a2f52310617a3b93ab7579f3dc9d7880/beacon_node/beacon_chain/src/test_utils.rs#L604
The text was updated successfully, but these errors were encountered: