Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bench: Add separate benchmarks for execution only #343

Merged
merged 2 commits into from
Jun 15, 2021
Merged

Conversation

chfast
Copy link
Member

@chfast chfast commented Jun 14, 2021

For Advanced and Baseline builtin VMs this adds "execute" benchmarks which uses cached code analysis and pre-allocated execution state to much usage pattern in Ethereum clients (like Silkworm).

The existing EVMC-like execution has been renamed to "total" as it performs analysis and execution in each iteration.

@chfast
Copy link
Member Author

chfast commented Jun 14, 2021

Current Advanced vs Baseline comparison:

Comparing advanced (from o/master) to baseline (from o/master)
Benchmark                                                                               Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
[advanced vs. baseline]/execute/main/blake2b_huff/empty_mean                         +0.3034         +0.3034            12            15            12            15
[advanced vs. baseline]/execute/main/blake2b_huff/2805nulls_mean                     +0.3054         +0.3054           253           330           253           330
[advanced vs. baseline]/execute/main/blake2b_huff/5610nulls_mean                     +0.3143         +0.3143           493           648           493           648
[advanced vs. baseline]/execute/main/blake2b_huff/8415nulls_mean                     +0.3126         +0.3126           722           948           722           948
[advanced vs. baseline]/execute/main/blake2b_huff/65536nulls_mean                    +0.3088         +0.3088          5614          7347          5614          7347
[advanced vs. baseline]/execute/main/blake2b_shifts/2805nulls_mean                   -0.0788         -0.0789          3409          3140          3409          3140
[advanced vs. baseline]/execute/main/blake2b_shifts/5610nulls_mean                   -0.0803         -0.0803          6810          6263          6810          6263
[advanced vs. baseline]/execute/main/blake2b_shifts/8415nulls_mean                   -0.0796         -0.0796         10202          9390         10202          9390
[advanced vs. baseline]/execute/main/blake2b_shifts/65536nulls_mean                  -0.0807         -0.0807         79038         72659         79037         72659
[advanced vs. baseline]/execute/main/sha1_divs/empty_mean                            +0.0768         +0.0768            50            54            50            54
[advanced vs. baseline]/execute/main/sha1_divs/1351_mean                             +0.1031         +0.1031          1047          1155          1047          1155
[advanced vs. baseline]/execute/main/sha1_divs/2737_mean                             +0.1050         +0.1050          2042          2256          2042          2256
[advanced vs. baseline]/execute/main/sha1_divs/5311_mean                             +0.1031         +0.1031          3989          4400          3989          4400
[advanced vs. baseline]/execute/main/sha1_divs/65536_mean                            +0.1060         +0.1060         48584         53734         48585         53734
[advanced vs. baseline]/execute/main/sha1_shifts/empty_mean                          +0.1377         +0.1377            29            33            29            33
[advanced vs. baseline]/execute/main/sha1_shifts/1351_mean                           +0.1464         +0.1465           612           702           612           702
[advanced vs. baseline]/execute/main/sha1_shifts/2737_mean                           +0.1512         +0.1512          1192          1372          1192          1372
[advanced vs. baseline]/execute/main/sha1_shifts/5311_mean                           +0.1484         +0.1484          2328          2673          2328          2673
[advanced vs. baseline]/execute/main/sha1_shifts/65536_mean                          +0.1419         +0.1419         28467         32507         28467         32508
[advanced vs. baseline]/execute/main/weierstrudel/0_mean                             +0.0423         +0.0423           161           168           161           168
[advanced vs. baseline]/execute/main/weierstrudel/1_mean                             +0.0390         +0.0390           363           377           363           377
[advanced vs. baseline]/execute/main/weierstrudel/3_mean                             +0.0408         +0.0408           567           590           567           590
[advanced vs. baseline]/execute/main/weierstrudel/9_mean                             +0.0433         +0.0433          1176          1227          1176          1227
[advanced vs. baseline]/execute/main/weierstrudel/14_mean                            +0.0401         +0.0401          1690          1758          1690          1758

@codecov
Copy link

codecov bot commented Jun 14, 2021

Codecov Report

Merging #343 (3b5d214) into master (2ad2213) will not change coverage.
The diff coverage is n/a.

❗ Current head 3b5d214 differs from pull request most recent head f48bb09. Consider uploading reports for the commit f48bb09 to get more accurate results

@@           Coverage Diff           @@
##           master     #343   +/-   ##
=======================================
  Coverage   99.78%   99.78%           
=======================================
  Files          29       29           
  Lines        4133     4133           
=======================================
  Hits         4124     4124           
  Misses          9        9           
Flag Coverage Δ
consensus 91.09% <ø> (ø)
unittests 99.78% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

@chfast chfast merged commit d81a8fd into master Jun 15, 2021
@chfast chfast deleted the bench_exec_only branch June 15, 2021 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant