Runtime: Accept full chain for para on multiple cores (elastic scaling)

[eskimor]

We accept N candidates for N scheduled cores, if those candidates form a chain back to the last included one.

[sandreim]

It is not clear if this one refers to scheduler to assign multiple cores per para in same RCB

[alindima]

or to accepting multiple parablocks in paras_inherent?

[eskimor]

This is about accepting multiple backed candidates in paras_inherent, if there are multiple cores for that para.

[eskimor]

Scheduler should already be fine.

[alindima]

currently, the check for whether the backed candidate is a descendant of the current para head is done in: <inclusion::Pallet<T>>::process_candidates.

for this to work with multiple candidates, I think we'll need to change the format of the inherent data to contain the parent_head_hash, so that we're able to check the dependency chain and also check the PersistedValidationData hash against the one in the candidate receipt

[sandreim]

I actually also need to change the ParachainsInherentData format due to a change of BackedCandidate primitive. Currently I am struggling to figure out how to implement this change only for Rococo runtime.

[eskimor]

for this to work with multiple candidates, I think we'll need to change the format of the inherent data to contain the parent_head_hash, so that we're able to check the dependency chain and also check the PersistedValidationData hash against the one in the candidate receipt

I don't think this is necessary. It seems right now we check dependency chains via PersistedValidationData hash here. We should be able to make persisted validation data for each provided candidate similar to here and then put the candidates in a table based on their persisted validation data hashes. Then we just need to lookup the hash provided in the descriptor to find the parent.

[eskimor]

I actually also need to change the ParachainsInherentData format due to a change of BackedCandidate primitive. Currently I am struggling to figure out how to implement this change only for Rococo runtime.

There is a hacky half-baked solution for this. If we establish the contract that provided BackedCandidates are provided in order of cores, then with all para cores occupied we are already golden - we know the first candidate for a para will be the lowest core for that para and so on.

Things get more tricky if there are holes and e.g. the candidate for core 1 of that para is missing, then we are off by one. We could get around this by inserting dummy BackedCandidates for those missing entries. Adding these dummy candidates, apart from being hacky, is also not backwards compatible though.

[eskimor]

Another possible hack 😬 : We could change the format of validator_indices so the last bits in the field identify the core (if there are multiple). E.g. let say we want to support up to 4 cores per para for now, we could use 2 bits here.

[burdges]

We'll require the candidate recipets appear backed on the relay chain in order?

[sandreim]

We'll require the candidate recipets appear backed on the relay chain in order?

That would be an optimization I guess, otherwise we should be able to establish ordering based on the parent heads.

[sandreim]

Another possible hack 😬 : We could change the format of validator_indices so the last bits in the field identify the core (if there are multiple). E.g. let say we want to support up to 4 cores per para for now, we could use 2 bits here.

I think this is by far the least intrusive method but there will still be breakage as current nodes rely on the bitfield len to identify the backing group size in client/runtime code. This will also make it subtle opt in interface. We'd probably have to introduce a temporary node side feature or runtime API to enable this only on Rococo.