-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: adding codebase/deployed tests scenarios [PoC] #378
base: master
Are you sure you want to change the base?
Conversation
Hey, thanks for the contribution. A couple of observations here:
The spell crafter is expected to follow the Crafter's Checklist. In that document, there is an explicit set of steps that would prevent this issue. For an outsider, it might be not that obvious that such a checklist exists, but the spell teams make sure to go through them when onboarding a new team member or a new spell team. In the recent past, we briefly discussed the possibility of migrating the spell-related checklist to this very repo, so it becomes self-contained, but we didn't reach a conclusion back then. Maybe we should revisit that.
That would be yet another thing crafters and reviewers would have to keep in mind: "am I running the tests for a new spell or for an existing one?". This adds to the already huge pile of cognitive overhead involved in the process. Personally, I would try to avoid that as much as possible unless there is a significant gain in terms of security. The tests that don't make much sense during the development cycle, such as |
ey @amusingaxl thanks again for the review and detailed answer! ❤️
this could be programatically skipped using #379, with a the cognitive overhead reduction would be visible on the test logs:
I agree that explicitness in this test would be a step forward in security, given that it adds a layer of extra complexity to PASS when the spell is not deployed, and if for some reason, after deployment the test suite is being run on the Codebase (and not the Deployed scope), the reviewer will see a ✅ , on a very critical test, that could potentially jeopardize the whole protocol (if for example, the DssExecLib differs in the deployed spell than in the codebase one). under the same criteria, the rest of the tests could benefit from being redundantly called, despite the fact that if bytecode matches, the contracts should behave exactly the same, I give you the point there that, this is a pretty safe assumption to make |
That is not a bad idea! We could implement that if we decide go with the explicit skip.
I understand where you stand on this, but please notice that the deployment of the spell is handled by the There are also some items in the reviewer check list for the Deployed Stage that would catch such problem. |
The objective of this PoC strategy, is to enable both testing scenarios while developing and debugging the spells. Benefitting from contract inheritance, the testing codebase would be exactly the same for one or another, without requiring to multiply codebase (which is a pain to maintain).
When working on a spell, a developer may need to make some changes to debug what's happening on chain, and the current workflow for him is to change the deployed address to
address(0)
to declare that the test suite should read (and deploy) the codebase spell, instead of reading the forked one.Forgetting to delete this address, could result in a frustrating DX experience, while developer is making changes to the codebase, but no changes in tests appear. Instead, with the current PoC, the deployer could separate which test to perform, doing either
make test match=Deployed
ormake test match=Codebase
, or enable the CI to run different scopes at different steps.Also, this strategy, could benefit from running certain tests only when it makes sense to run, instead of hacking that test to pass even when it's run on the Codebase, for example,
testBytecode
in this PoC is set to run only on the Deployed scope (notice: it has 23 tests, while Codebase has 22).Of course, this needs to be addressed after optimizing the tests in such a way that they don't last minutes, but afterwards, the benefits for the developer or the CI process could overweight the waiting period of running these "duplicated" tests.