Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix storage #3064

Merged
merged 16 commits into from
Jan 12, 2023
Merged

fix storage #3064

merged 16 commits into from
Jan 12, 2023

Conversation

synzhu
Copy link
Contributor

@synzhu synzhu commented Aug 23, 2022

Updates the execution data provider to track fulfilled height.

@codecov-commenter
Copy link

codecov-commenter commented Aug 23, 2022

Codecov Report

Merging #3064 (e2187d3) into master (d7ab76d) will increase coverage by 0.01%.
The diff coverage is 25.00%.

@@            Coverage Diff             @@
##           master    #3064      +/-   ##
==========================================
+ Coverage   55.26%   55.27%   +0.01%     
==========================================
  Files         760      760              
  Lines       69716    69718       +2     
==========================================
+ Hits        38526    38536      +10     
+ Misses      28024    28017       -7     
+ Partials     3166     3165       -1     
Flag Coverage Δ
unittests 55.27% <25.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
module/executiondatasync/tracker/storage.go 58.60% <ø> (ø)
module/executiondatasync/provider/provider.go 66.87% <25.00%> (-1.51%) ⬇️
consensus/hotstuff/eventloop/event_loop.go 73.46% <0.00%> (-1.37%) ⬇️
state/protocol/badger/state.go 52.65% <0.00%> (-0.67%) ⬇️
engine/consensus/compliance/core.go 74.60% <0.00%> (ø)
state/protocol/badger/validity.go 48.96% <0.00%> (+1.24%) ⬆️
...s/hotstuff/votecollector/staking_vote_processor.go 87.09% <0.00%> (+3.22%) ⬆️
...sus/approvals/assignment_collector_statemachine.go 48.55% <0.00%> (+7.24%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@peterargue
Copy link
Contributor

@smnzhu can you add some context to the description on what this change is fixing?

@synzhu
Copy link
Contributor Author

synzhu commented Oct 19, 2022

bors merge

bors bot added a commit that referenced this pull request Oct 19, 2022
3064: fix storage r=smnzhu a=smnzhu

Updates the execution data provider to track fulfilled height.

Co-authored-by: Simon Zhu <simon.zsiyan@gmail.com>
Co-authored-by: Kan Zhang <kan@axiomzen.co>
Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
@bors
Copy link
Contributor

bors bot commented Oct 19, 2022

Build failed:

@peterargue
Copy link
Contributor

@smnzhu Looks like there are some legit unit test failures

Copy link
Contributor

@peterargue peterargue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @Kay-Zee!

trackerStorage := mocktracker.NewMockStorage()

// Assume SetFulfilledHeight succeeds unless otherwise indicadted in a test requriement
trackerStorage.On("SetFulfilledHeight", mock.Anything).Return(nil)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this line is also run in mocktracker.NewMockStorage() so you can probably skip it here

@github-actions
Copy link
Contributor

github-actions bot commented Nov 25, 2022

FVM Benchstat comparison

This branch with compared with the base branch onflow:master commit 02e9bc3

The command (for i in {1..7}; do go test ./fvm ./engine/execution/computation --bench . --tags relic -shuffle=on --benchmem --run ^$; done) was used.

Collapsed results for better readability

old.txtnew.txt
time/opdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeNFTBatchTransfer-2116ms ± 3%117ms ± 1%~(p=0.537 n=6+5)
RuntimeTransaction/reference_tx-231.6ms ±21%31.5ms ±14%~(p=0.535 n=7+7)
RuntimeTransaction/convert_int_to_string-230.5ms ± 4%30.5ms ± 3%~(p=0.628 n=6+7)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-232.1ms ± 2%32.2ms ± 3%~(p=0.836 n=6+7)
RuntimeTransaction/get_signer_address-230.1ms ± 2%29.9ms ± 5%~(p=0.937 n=6+6)
RuntimeTransaction/get_public_account-232.1ms ± 5%33.3ms ±10%~(p=0.165 n=7+7)
RuntimeTransaction/get_account_and_get_balance-2260ms ± 2%261ms ± 3%~(p=0.805 n=7+7)
RuntimeTransaction/get_account_and_get_available_balance-2238ms ± 1%238ms ± 2%~(p=0.902 n=7+7)
RuntimeTransaction/get_account_and_get_storage_used-234.6ms ± 3%34.0ms ± 4%~(p=0.383 n=7+7)
RuntimeTransaction/get_account_and_get_storage_capacity-2214ms ± 3%215ms ± 2%~(p=0.902 n=7+7)
RuntimeTransaction/get_signer_vault-236.5ms ± 3%36.2ms ± 2%~(p=0.445 n=7+6)
RuntimeTransaction/get_signer_receiver-246.8ms ± 3%47.1ms ± 4%~(p=0.805 n=7+7)
RuntimeTransaction/transfer_tokens-2190ms ± 2%190ms ± 2%~(p=0.902 n=7+7)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-235.7ms ± 4%35.0ms ± 0%~(p=0.106 n=7+5)
RuntimeTransaction/load_and_save_long_string_on_signers_address-276.9ms ± 6%78.1ms ± 3%~(p=0.318 n=7+7)
RuntimeTransaction/create_new_account-2802ms ± 1%803ms ± 1%~(p=0.620 n=7+7)
RuntimeTransaction/call_empty_contract_function-231.9ms ± 4%32.1ms ± 3%~(p=0.535 n=7+7)
RuntimeTransaction/emit_event-246.1ms ± 3%45.8ms ± 6%~(p=0.805 n=7+7)
RuntimeTransaction/borrow_array_from_storage-2127ms ± 2%126ms ± 2%~(p=0.259 n=7+7)
RuntimeTransaction/copy_array_from_storage-2126ms ± 2%128ms ± 1%~(p=0.097 n=7+7)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-24.31s ± 1%4.30s ± 1%~(p=0.295 n=7+6)
 
alloc/opdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/get_account_and_get_balance-2119MB ± 2%122MB ± 0%+1.76%(p=0.018 n=7+5)
RuntimeTransaction/get_signer_address-235.0MB ± 2%35.4MB ± 1%+1.33%(p=0.035 n=7+6)
RuntimeNFTBatchTransfer-254.9MB ± 3%54.5MB ± 3%~(p=0.295 n=6+7)
RuntimeTransaction/reference_tx-235.4MB ± 4%35.2MB ± 3%~(p=0.805 n=7+7)
RuntimeTransaction/convert_int_to_string-234.5MB ± 2%34.8MB ± 2%~(p=0.209 n=7+7)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-235.7MB ± 5%35.3MB ± 3%~(p=0.259 n=7+7)
RuntimeTransaction/get_public_account-235.9MB ± 2%36.4MB ± 3%~(p=0.209 n=7+7)
RuntimeTransaction/get_account_and_get_available_balance-2106MB ± 2%107MB ± 3%~(p=0.097 n=7+7)
RuntimeTransaction/get_account_and_get_storage_used-236.8MB ± 2%36.4MB ± 4%~(p=0.383 n=7+7)
RuntimeTransaction/get_account_and_get_storage_capacity-2102MB ± 5%102MB ± 4%~(p=0.710 n=7+7)
RuntimeTransaction/get_signer_vault-237.3MB ± 4%37.0MB ± 2%~(p=0.318 n=7+7)
RuntimeTransaction/get_signer_receiver-240.9MB ± 4%41.7MB ± 6%~(p=0.383 n=7+7)
RuntimeTransaction/transfer_tokens-283.9MB ± 3%84.3MB ± 3%~(p=1.000 n=7+7)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-235.8MB ± 1%35.9MB ± 1%~(p=0.710 n=7+7)
RuntimeTransaction/load_and_save_long_string_on_signers_address-252.5MB ± 5%52.7MB ± 3%~(p=0.620 n=7+7)
RuntimeTransaction/create_new_account-2184MB ± 3%181MB ± 1%~(p=0.259 n=7+7)
RuntimeTransaction/call_empty_contract_function-235.6MB ± 2%35.5MB ± 1%~(p=0.755 n=7+5)
RuntimeTransaction/emit_event-240.4MB ± 4%40.4MB ± 3%~(p=0.805 n=7+7)
RuntimeTransaction/borrow_array_from_storage-270.1MB ± 2%69.7MB ± 5%~(p=0.805 n=7+7)
RuntimeTransaction/copy_array_from_storage-281.2MB ± 1%81.7MB ± 3%~(p=0.902 n=7+7)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-21.21GB ± 2%1.20GB ± 1%~(p=0.902 n=7+7)
 
allocs/opdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-282.7k ± 0%82.8k ± 0%+0.18%(p=0.001 n=7+7)
RuntimeTransaction/get_signer_address-286.9k ± 0%87.0k ± 0%+0.17%(p=0.001 n=7+7)
RuntimeTransaction/convert_int_to_string-294.9k ± 0%95.0k ± 0%+0.16%(p=0.001 n=7+7)
RuntimeTransaction/call_empty_contract_function-298.2k ± 0%98.3k ± 0%+0.15%(p=0.001 n=7+7)
RuntimeTransaction/get_public_account-2109k ± 0%109k ± 0%+0.14%(p=0.001 n=7+7)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2106k ± 0%106k ± 0%+0.14%(p=0.001 n=7+7)
RuntimeTransaction/get_account_and_get_storage_used-2119k ± 0%119k ± 0%+0.13%(p=0.001 n=7+6)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2122k ± 0%122k ± 0%+0.12%(p=0.001 n=7+7)
RuntimeTransaction/get_signer_vault-2123k ± 0%123k ± 0%+0.12%(p=0.001 n=7+7)
RuntimeTransaction/emit_event-2138k ± 0%138k ± 0%+0.11%(p=0.001 n=7+7)
RuntimeTransaction/get_signer_receiver-2198k ± 0%198k ± 0%+0.08%(p=0.001 n=7+7)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2194k ± 0%194k ± 0%+0.08%(p=0.001 n=7+7)
RuntimeTransaction/copy_array_from_storage-2293k ± 0%293k ± 0%+0.05%(p=0.001 n=7+7)
RuntimeTransaction/borrow_array_from_storage-2336k ± 0%336k ± 0%+0.04%(p=0.001 n=7+7)
RuntimeTransaction/get_account_and_get_available_balance-21.23M ± 0%1.23M ± 0%+0.01%(p=0.001 n=7+7)
RuntimeTransaction/get_account_and_get_storage_capacity-21.11M ± 0%1.11M ± 0%+0.01%(p=0.001 n=7+7)
RuntimeTransaction/get_account_and_get_balance-21.30M ± 0%1.30M ± 0%+0.01%(p=0.001 n=7+6)
RuntimeTransaction/transfer_tokens-2842k ± 0%842k ± 0%+0.01%(p=0.001 n=7+6)
RuntimeNFTBatchTransfer-2269k ± 0%268k ± 1%~(p=0.094 n=6+7)
RuntimeTransaction/create_new_account-22.27M ± 0%2.27M ± 0%~(p=0.165 n=7+7)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-217.2M ± 0%17.2M ± 0%~(p=0.128 n=7+7)
 
computationdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-2202 ± 0%202 ± 0%~(all equal)
RuntimeTransaction/convert_int_to_string-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2502 ± 0%502 ± 0%~(all equal)
RuntimeTransaction/get_signer_address-2302 ± 0%302 ± 0%~(all equal)
RuntimeTransaction/get_public_account-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_balance-21.00k ± 0%1.00k ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_available_balance-23.10k ± 0%3.10k ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_storage_used-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_storage_capacity-21.70k ± 0%1.70k ± 0%~(all equal)
RuntimeTransaction/get_signer_vault-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/get_signer_receiver-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/transfer_tokens-23.50k ± 0%3.50k ± 0%~(all equal)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/create_new_account-2202 ± 0%202 ± 0%~(all equal)
RuntimeTransaction/call_empty_contract_function-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/emit_event-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/borrow_array_from_storage-22.60k ± 0%2.60k ± 0%~(all equal)
RuntimeTransaction/copy_array_from_storage-22.60k ± 0%2.60k ± 0%~(all equal)
 
interactionsdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/convert_int_to_string-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/get_signer_address-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/get_public_account-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_balance-224.3M ± 0%24.3M ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_available_balance-25.64M ± 0%5.64M ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_storage_used-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_storage_capacity-25.64M ± 0%5.64M ± 0%~(all equal)
RuntimeTransaction/get_signer_vault-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/get_signer_receiver-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/transfer_tokens-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2334k ± 0%334k ± 0%~(all equal)
RuntimeTransaction/create_new_account-212.3M ± 0%12.3M ± 0%~(all equal)
RuntimeTransaction/call_empty_contract_function-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/emit_event-2331k ± 0%331k ± 0%~(all equal)
RuntimeTransaction/borrow_array_from_storage-2336k ± 0%336k ± 0%~(all equal)
RuntimeTransaction/copy_array_from_storage-2336k ± 0%336k ± 0%~(all equal)
 
us/txdelta
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-22.11k ± 1%2.10k ± 1%~(p=0.314 n=7+6)
 

@Kay-Zee
Copy link
Member

Kay-Zee commented Nov 28, 2022

bors merge

bors bot added a commit that referenced this pull request Nov 28, 2022
3064: fix storage r=Kay-Zee a=smnzhu

Updates the execution data provider to track fulfilled height.

3596: [Metrics] Report finalized height and executed height on startup r=zhangchiqing a=zhangchiqing

This PR fixes an issue that the finalized height and executed height metrics is not reported on startup until there is block being finalized and executed.

For EN which takes a while to load the execution state, the finalized height and executed height is not shown on metrics.

This PR fixes it by reporting finalized height and executed height on startup.

Co-authored-by: Simon Zhu <simon.zsiyan@gmail.com>
Co-authored-by: Kan Zhang <kan@axiomzen.co>
Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
Co-authored-by: Kay-Zee <kan@axiomzen.co>
Co-authored-by: Leo Zhang (zhangchiqing) <zhangchiqing@gmail.com>
Co-authored-by: Leo Zhang <zhangchiqing@gmail.com>
@bors
Copy link
Contributor

bors bot commented Nov 28, 2022

Build failed (retrying...):

bors bot added a commit that referenced this pull request Nov 28, 2022
3064: fix storage r=Kay-Zee a=smnzhu

Updates the execution data provider to track fulfilled height.

Co-authored-by: Simon Zhu <simon.zsiyan@gmail.com>
Co-authored-by: Kan Zhang <kan@axiomzen.co>
Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
Co-authored-by: Kay-Zee <kan@axiomzen.co>
@bors
Copy link
Contributor

bors bot commented Nov 28, 2022

@yhassanzadeh13 yhassanzadeh13 merged commit 63cec23 into master Jan 12, 2023
@yhassanzadeh13 yhassanzadeh13 deleted the smnzhu/storage-fix branch January 12, 2023 00:44
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants