frame_support::storage: Add StorageStreamIter

[bkchr]

This pr adds the StorageStreamIter trait. This trait is currently only implemented for StorageValue which store "SCALE container type" e.g. Vec, BTreeMap etc. A SCALE container type is a type that follows the following encoding structure Compact<u32>(len) ++ #( item.encode() )*. The streaming iterator is a way to decode container values using an almost constant memory usage in contrast to decoding the entire value once into memory. The memory of a runtime is constrained, especially when it comes to a single allocation. This can be used for stuff like decoding events in an offchain worker or other huge data. It could probably also used for on chain operations, but will may require some more performance optimizations.

Internally this works by using sp_io::storage::read to read chunks from the state as requested by the decoder. As each call to read is a host function call there is some internal cache that currently stores 2048 bytes. This reduces the number of host calls and improves the performance of the iterator.

[ggwpez]

We should also document that the value should not be modified as long as at least one Stream iter operates on it.

[koute]

We could still make the errors and the code a little nicer, but functionally looks good to me.

[koute]

Could reduce one level of indentation by doing an early exit.

if self.read >= self.length {
    return None;
}

[koute]

We could simplify this a little bit with split_at_mut, e.g.

let (out_already_read, out_remaining) = into.split_at_mut(self.buffer.len() - self.buffer_pos);
out_already_read.copy_from_slice(&self.buffer[self.buffer_pos..]);

And then later:

		if let Some(length_minus_offset) =
			sp_io::storage::read(&self.key, &mut out_remaining, self.offset)
		{
			if length_minus_offset as usize < out_remaining.len() {
				return Err("Not enough data to fill the buffer".into())
			}

			self.ensure_total_length_did_not_change(length_minus_offset)?;
			self.offset += out_remaining.len() as u32;

			Ok(())
		} else {

[bkchr]

bot merge

[paritytech-processbot]

Waiting for commit status.

[paritytech-processbot]

Merge cancelled due to error. Error: Statuses failed for a0302c5

[bkchr]

bot merge

[paritytech-processbot]

Waiting for commit status.

[paritytech-processbot]

Merge cancelled due to error. Error: Statuses failed for 10629aa

[bkchr]

bot merge

[paritytech-processbot]

Waiting for commit status.

[Polkadot-Forum]

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-release-analysis-v0-9-37/1736/1