[TESTING] Script for benchmarking builders locally #137
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Runner script Allows image building, kurtosis ethereum-package starting and stopping using the generated image * feat: update geth builder overview dashboard Created a Go script that follows the following API and is expected to exist in ./scripts inside builder repo: Main features: - Runs on Windows and Linux using Go run - Does progressive command log printing - Allows to set ImageTag and provide ImageArgs when building starting the network Example commands: - build ../ repo into docker image go run emulate_network.go build - run network and see deployment progress go run emulate_network.go run -n=enclaveName -k="C:/Users/olegj/kurtosis.exe” - stop network go run emulate_network.go stop -n=enclaveName -k="C:/Users/olegj/kurtosis.exe” Co-authored-by: Carlos Bermudez Porto <cbermudez.dev@gmail.com>
new docker debug file
…nfig (#2) * Increase the number of generated transactions. Update custom flood config
* Update builder metrics flags * fix: update metrics flags * feat: update dashboard tooltips * feat: use ethereum package main branch * feat: use version 0.6.1 with builder fix
…a better peering of transactions. Minor improvements: * Use flashbots/builder for participant * Hardcode tag instead of using `latest` * Use capella 0
This reverts commit 40ac12b.
Fix builders not submitting blocks. Use geth as it appear to deliver…
This reverts commit 67bef29.
no port forwarding
* Using default Dockerfile for build Also updated readme
kamilchodola
requested review from
dvush,
TymKh,
Ruteri and
avalonche
as code owners
Increase fuzzing rate
|
Some updated results as we've increased fuzzing rate since the description was added. |
avalonche
reviewed
Nov 21, 2023
scripts/readme.md
Outdated
There was a problem hiding this comment.
nit: file name should be README.md
avalonche
reviewed
Nov 22, 2023
scripts/README.md
Outdated
| ``` | ||
| ## Architecture | ||
| Test network consists of 3 builders and a regular client. Dev and baseline are builders being compared. Dev builder is a candidate to be merged to main while baseline is the current version of builder. The 2 remaining clients are validator which is a builder connected to the relay as validator and a regular el client which submits blocks in epochs 0-3. | ||
|  |
There was a problem hiding this comment.
this image links to the nethermind fb-builder, could add the image as part of the PR?
avalonche
reviewed
Nov 22, 2023
scripts/README.md
Outdated
|
|
||
|
|
||
| ## Running in Docker in Docker | ||
| See this note for refrence here |
There was a problem hiding this comment.
is there a missing link?
avalonche
reviewed
Nov 22, 2023
| ``` | ||
| go run emulate_network.go help | ||
| ``` | ||
| ## Architecture |
There was a problem hiding this comment.
I think it'll be useful to expand a bit more on the architecture. For example, how the broadcast architecture works (a bit of an explainer of what each of the fuzzers do) and also why a third validation builder is needed (due to performance reasons).
avalonche
reviewed
Nov 22, 2023
| ``` | ||
| go run emulate_network.go help | ||
| ``` | ||
| ## Architecture |
There was a problem hiding this comment.
Additionally, I think it'll be super useful if there's a "dashboard" section here explaining all the metrics that are being collected and what it is measuring
minimal edited files
Dind updates WIP.
|
@avalonche fixed |
kamilchodola
added a commit
to kamilchodola/documentation
that referenced
this pull request
Jun 18, 2024
Name: Carlos Bermudez Project: Nethermind Client DevOps Team: Nethermind Joined: 05/04/2022(without Internship 05/07/2022) Discord handle: cbermudez97 Summary of their work/eligibility: Carlos started working in Nethermind before the Merge. He is contributing as DevOps to the Infrastructure requirements for the Nethermind Client, especially in creating workflows for improving testing in the Post-Merge scenario. Carlos also helps a lot in QA activities and builds new tools which are improving the way on how well Nethermind Client can be tested and monitored. Public: https://github.com/NethermindEth/nethermind/commits/master/?author=cbermudez97 (not much here sadly) https://github.com/NethermindEth/sedge/pulls?q=is%3Apr+author%3Acbermudez97 https://github.com/NethermindEth/metrics-infrastructure/commits/master/?author=cbermudez97 flashbots/builder#137 https://github.com/kurtosis-tech/ethereum-package/pulls?q=is%3Apr+author%3Acbermudez97 Other than above ones Carlos contributes a lot to currently private repositories which are about to be changed to public ones from which other EL/CL clients can benefit a lot like: 1. Easy scripting way to configure all EL and CL clients in Virtualized environment and sync on all testnets and mainnet. 2. Creating a big library of playbooks in Ansible to easily maintain already created nodes. 3. Tool which allows to add new nonValidator nodes to existing devnets (very useful to test SnapSync etc on Devnets without touching existing validators). 4. Infrastructure monitoring dashboard - tool where all spawned VMs with nodes installed can be browsed, edited (without need to jump directly to VM) or removed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📝 Summary
Script created gives a opportunity to test performance of builder implementation and compare it to latest builder version.
This gives an opportunity to ensure that builder works correctly and did not introduce any performance degradation.
Usage of a script is well described in Readme.md
High level usage:
Here is a image of comparison of latest docker image vs latest docker image (Baseline to see if results are the same)
Next one is comparison of latest vs deneb branch built with command
go run emulate_network.go build -t deneb- deneb seems to be a source for latest image so result should also be the same.And the last one shows a difference between latest image and main branch (main seems to be outdated and tool clearly exposes a memory usage difference which on main is more than 2x higher, slower bid production on main branch and a bit different disk usage)
📚 References
Ethereum-package from Kurtosis which is used as a engine to run a devnet:
https://github.com/kurtosis-tech/ethereum-package
CONTRIBUTING.md