[pallet_contracts] Add support for transient storage in contracts host functions #4566

smiasojed · 2024-05-24T10:26:25Z

Introduce transient storage, which behaves identically to regular storage but is kept only in memory and discarded after every transaction. This functionality is similar to the TSTORE and TLOAD operations used in Ethereum.

The following new host functions have been introduced:
get_transient_storage
set_transient_storage
take_transient_storage
clear_transient_storage
contains_transient_storage
Note: These functions are declared as unstable and thus are not activated.

substrate/frame/contracts/src/lib.rs

substrate/frame/contracts/src/wasm/runtime.rs

substrate/frame/contracts/src/transient_storage.rs

substrate/frame/contracts/src/lib.rs

substrate/frame/contracts/src/transient_storage.rs

smiasojed · 2024-06-18T13:18:27Z

bot bench substrate-pallet --pallet=pallet_contracts

command-bot · 2024-06-18T13:18:32Z

@smiasojed https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6494983 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 1-8c3cb4fc-c31f-4ea7-93cb-5a956db63737 to cancel this command or bot cancel to cancel all commands in this pull request.

…nto HEAD

…=dev --target_dir=substrate --pallet=pallet_contracts

command-bot · 2024-06-18T14:11:12Z

@smiasojed Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6494983 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6494983/artifacts/download.

smiasojed · 2024-06-18T18:38:41Z

bot bench substrate-pallet --pallet=pallet_contracts

command-bot · 2024-06-18T18:38:46Z

@smiasojed https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6497164 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 2-2506ce03-9cd3-46a1-b7a6-3a10d6498720 to cancel this command or bot cancel to cancel all commands in this pull request.

…nto HEAD

…=dev --target_dir=substrate --pallet=pallet_contracts

pgherveou · 2024-07-08T12:57:52Z

substrate/frame/contracts/src/benchmarking/mod.rs

+		let value = Some(vec![42u8; max_value_len as _]);
+		let mut setup = CallSetup::<T>::default();
+		let (mut ext, _) = setup.ext();
+		CallSetup::<T>::with_transient_storage(&mut ext, T::MaxTransientStorageSize::get())?;


would be a bit nicer to have with_transient_storage be a method of the builder

let mut setup = CallSetup::<T>::default(); setup.transient_storage_size(T::MaxTransientStorageSize::get()) let (mut ext, _) = setup.ext();

pgherveou · 2024-07-08T13:01:54Z

substrate/frame/contracts/src/benchmarking/call_builder.rs

+		let amount = ext.transient_storage().meter().current().amount;
+		let limit = ext.transient_storage().meter().current().limit;


maybe

let &MeterEntry { amount, limit } = ext.transient_storage().meter().current();

pgherveou · 2024-07-08T13:14:11Z

substrate/frame/contracts/src/wasm/runtime.rs

+	// cost_write!(name, a, b, c) -> T::WeightInfo::name(a, b, c).saturating_add(T::WeightInfo::rollback_transient_storage())
+	// .saturating_add(T::WeightInfo::set_transient_storage_full().saturating_sub(T::WeightInfo::set_transient_storage_empty())


Not clear to me why the cost is full - empty, can we explain the reasoning here

full - empty represents the weight of adding an item to a full BTreeMap. All transient storage host function benchmarks are done with an empty transient storage (BTreeMap), so this represents the worst-case scenario.

follow up question, should we refund the cost of rollback_transient_storage in the happy path since you pre-charge it

still not clear to me why we need to add the weight of
T::WeightInfo::set_transient_storage_full() - T::WeightInfo::set_transient_storage_empty()
when we do a ClearTransientStorage for example

The clear_transient_storage function modifies the storage. The benchmark is currently done for one item in the BTreeMap, but we would like to consider the worst-case scenario, where the modification is done on a fully filled BTreeMap.

I am treating it as a constant cost, similar to DBWrite, which is 100µs.

Make sense then, we should add that to the comment to make it clearer

pgherveou

lgtm as well, added a few more nits and a question

xermicus

LGTM

substrate/frame/contracts/uapi/src/host.rs

athei · 2024-07-08T16:26:23Z

Can you do one additional thing in this PR: Since we are bumping the major version of the uapi crate it would be a good time to also seal the HostFn trait. The only reason we have to do a major bump is because we forgot to prevent downstream crates to implement it.

paritytech-cicd-pr · 2024-07-09T08:34:06Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6660740

smiasojed · 2024-07-09T10:11:12Z

bot bench substrate-pallet --pallet=pallet_contracts

command-bot · 2024-07-09T10:11:17Z

@smiasojed https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6662227 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 1-3a0acf30-2f24-4ad3-9b64-1421719f53f6 to cancel this command or bot cancel to cancel all commands in this pull request.

…nto HEAD

…=dev --target_dir=substrate --pallet=pallet_contracts

command-bot · 2024-07-09T11:02:16Z

@smiasojed Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6662227 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6662227/artifacts/download.

* master: add elastic scaling MVP guide (#4663) Send PeerViewChange with high priority (#4755) [ci] Update forklift in CI image (#5032) Adjust base value for statement-distribution regression tests (#5028) [pallet_contracts] Add support for transient storage in contracts host functions (#4566) [1 / 5] Optimize logic for gossiping assignments (#4848) Remove `pallet-getter` usage from pallet-session (#4972) command-action: added scoped permissions to the github tokens (#5016) net/litep2p: Propagate ValuePut events to the network backend (#5018) rpc: add back rpc logger (#4952) Updated substrate-relay version for tests (#5017) Remove most all usage of `sp-std` (#5010) Use sp_runtime::traits::BadOrigin (#5011)

…t functions (paritytech#4566) Introduce transient storage, which behaves identically to regular storage but is kept only in memory and discarded after every transaction. This functionality is similar to the `TSTORE` and `TLOAD` operations used in Ethereum. The following new host functions have been introduced: `get_transient_storage` `set_transient_storage` `take_transient_storage` `clear_transient_storage` `contains_transient_storage` Note: These functions are declared as `unstable` and thus are not activated. --------- Co-authored-by: command-bot <> Co-authored-by: PG Herveou <pgherveou@gmail.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* master: (125 commits) add elastic scaling MVP guide (#4663) Send PeerViewChange with high priority (#4755) [ci] Update forklift in CI image (#5032) Adjust base value for statement-distribution regression tests (#5028) [pallet_contracts] Add support for transient storage in contracts host functions (#4566) [1 / 5] Optimize logic for gossiping assignments (#4848) Remove `pallet-getter` usage from pallet-session (#4972) command-action: added scoped permissions to the github tokens (#5016) net/litep2p: Propagate ValuePut events to the network backend (#5018) rpc: add back rpc logger (#4952) Updated substrate-relay version for tests (#5017) Remove most all usage of `sp-std` (#5010) Use sp_runtime::traits::BadOrigin (#5011) network/tx: Ban peers with tx that fail to decode (#5002) Try State Hook for Bounties (#4563) [statement-distribution] Add metrics for distributed statements in V2 (#4554) added sync command (#4818) Bridges V2 refactoring backport and `pallet_bridge_messages` simplifications (#4935) xcm-executor: Improve logging (#4996) Remove usage of `sp-std` on templates (#5001) ...

…t functions (paritytech#4566) Introduce transient storage, which behaves identically to regular storage but is kept only in memory and discarded after every transaction. This functionality is similar to the `TSTORE` and `TLOAD` operations used in Ethereum. The following new host functions have been introduced: `get_transient_storage` `set_transient_storage` `take_transient_storage` `clear_transient_storage` `contains_transient_storage` Note: These functions are declared as `unstable` and thus are not activated. --------- Co-authored-by: command-bot <> Co-authored-by: PG Herveou <pgherveou@gmail.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* master: (130 commits) add elastic scaling MVP guide (#4663) Send PeerViewChange with high priority (#4755) [ci] Update forklift in CI image (#5032) Adjust base value for statement-distribution regression tests (#5028) [pallet_contracts] Add support for transient storage in contracts host functions (#4566) [1 / 5] Optimize logic for gossiping assignments (#4848) Remove `pallet-getter` usage from pallet-session (#4972) command-action: added scoped permissions to the github tokens (#5016) net/litep2p: Propagate ValuePut events to the network backend (#5018) rpc: add back rpc logger (#4952) Updated substrate-relay version for tests (#5017) Remove most all usage of `sp-std` (#5010) Use sp_runtime::traits::BadOrigin (#5011) network/tx: Ban peers with tx that fail to decode (#5002) Try State Hook for Bounties (#4563) [statement-distribution] Add metrics for distributed statements in V2 (#4554) added sync command (#4818) Bridges V2 refactoring backport and `pallet_bridge_messages` simplifications (#4935) xcm-executor: Improve logging (#4996) Remove usage of `sp-std` on templates (#5001) ...

smiasojed added 5 commits May 23, 2024 11:59

Add transient storage

1732628

Add unit tests

6cda150

Merge remote-tracking branch 'origin/master' into sm/contracts-tstore

6670170

Add storage limit

826e073

Limit storage per contract

daf9ad1

smiasojed added the T7-smart_contracts This PR/Issue is related to smart contracts. label May 24, 2024

smiasojed added 6 commits May 29, 2024 10:50

Refactored

d4a29be

Add terminate function to transient storage

53ec270

Add unit tests

33f6c2d

Refactored to single BtreeMap

199edaa

Add comments

e683b72

Cleanup

1c3c844

smiasojed requested a review from ascjones June 4, 2024 08:07

smiasojed added 3 commits June 4, 2024 10:35

Fix unit tests

18cd9ad

Merge remote-tracking branch 'origin/master' into sm/contracts-tstore

d5736d5

Fix tests after master merge

58d87dc

athei reviewed Jun 4, 2024

View reviewed changes

smiasojed added 3 commits June 5, 2024 09:45

Rename storage err

29f1305

Benchmarking test

63aaaaf

Fix unit tests

78960e0

command-bot added 2 commits June 18, 2024 13:21

Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…

f09b80b

…nto HEAD

".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime…

6d63e58

…=dev --target_dir=substrate --pallet=pallet_contracts

Max storage elements set to 2000

6f713ae

command-bot added 2 commits June 18, 2024 18:41

Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…

d6e1c43

…nto HEAD

".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime…

8c11871

…=dev --target_dir=substrate --pallet=pallet_contracts

athei approved these changes Jul 8, 2024

View reviewed changes

pgherveou reviewed Jul 8, 2024

View reviewed changes

pgherveou approved these changes Jul 8, 2024

View reviewed changes

xermicus approved these changes Jul 8, 2024

View reviewed changes

substrate/frame/contracts/uapi/src/host.rs Show resolved Hide resolved

smiasojed added 2 commits July 8, 2024 23:00

Marked functions as unstable

03f10f9

Sealed HostFn trait

5c59336

Refactor benchmarks

8d36584

command-bot added 2 commits July 9, 2024 10:13

Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…

4db9c95

…nto HEAD

".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime…

f2cfb76

…=dev --target_dir=substrate --pallet=pallet_contracts

smiasojed added 2 commits July 10, 2024 14:53

Merge remote-tracking branch 'origin/master' into sm/contracts-tstore

3a8c2d2

Comment added

45603b8

smiasojed added this pull request to the merge queue Jul 16, 2024

Merged via the queue into master with commit eac2a22 Jul 16, 2024
155 of 161 checks passed

smiasojed deleted the sm/contracts-tstore branch July 16, 2024 11:40

This was referenced Aug 21, 2024

Update polkadot-sdk from v1.11.0 to stable2407 moondance-labs/tanssi#659

Open

Update polkadot-sdk from v1.11.0 to stable2407 moonbeam-foundation/moonbeam#2912

Open

chungquantin mentioned this pull request Aug 23, 2024

chore: upgrade to stable2407 r0gue-io/pop-node#234

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pallet_contracts] Add support for transient storage in contracts host functions #4566

[pallet_contracts] Add support for transient storage in contracts host functions #4566

smiasojed commented May 24, 2024 •

edited

Loading

smiasojed commented Jun 18, 2024

command-bot bot commented Jun 18, 2024 •

edited

Loading

command-bot bot commented Jun 18, 2024

smiasojed commented Jun 18, 2024

command-bot bot commented Jun 18, 2024 •

edited

Loading

pgherveou Jul 8, 2024

smiasojed Jul 9, 2024

pgherveou Jul 8, 2024 •

edited

Loading

pgherveou Jul 8, 2024

smiasojed Jul 9, 2024

pgherveou Jul 9, 2024

pgherveou Jul 9, 2024

smiasojed Jul 9, 2024

pgherveou Jul 10, 2024

smiasojed Jul 10, 2024

pgherveou left a comment

xermicus left a comment

athei commented Jul 8, 2024

paritytech-cicd-pr commented Jul 9, 2024

smiasojed commented Jul 9, 2024

command-bot bot commented Jul 9, 2024 •

edited

Loading

command-bot bot commented Jul 9, 2024

		let amount = ext.transient_storage().meter().current().amount;
		let limit = ext.transient_storage().meter().current().limit;

		// cost_write!(name, a, b, c) -> T::WeightInfo::name(a, b, c).saturating_add(T::WeightInfo::rollback_transient_storage())
		// .saturating_add(T::WeightInfo::set_transient_storage_full().saturating_sub(T::WeightInfo::set_transient_storage_empty())

[pallet_contracts] Add support for transient storage in contracts host functions #4566

[pallet_contracts] Add support for transient storage in contracts host functions #4566

Conversation

smiasojed commented May 24, 2024 • edited Loading

smiasojed commented Jun 18, 2024

command-bot bot commented Jun 18, 2024 • edited Loading

command-bot bot commented Jun 18, 2024

smiasojed commented Jun 18, 2024

command-bot bot commented Jun 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgherveou Jul 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgherveou left a comment

Choose a reason for hiding this comment

xermicus left a comment

Choose a reason for hiding this comment

athei commented Jul 8, 2024

paritytech-cicd-pr commented Jul 9, 2024

smiasojed commented Jul 9, 2024

command-bot bot commented Jul 9, 2024 • edited Loading

command-bot bot commented Jul 9, 2024

smiasojed commented May 24, 2024 •

edited

Loading

command-bot bot commented Jun 18, 2024 •

edited

Loading

command-bot bot commented Jun 18, 2024 •

edited

Loading

pgherveou Jul 8, 2024 •

edited

Loading

command-bot bot commented Jul 9, 2024 •

edited

Loading