Prysm rpc: Get local header #14409

terencechain · 2024-09-02T02:37:38Z

This PR adds a Prysm RPC endpoint to retrieve the local header for the proposer to sign. The getLocalHeader call uses the existing getLocalPayloadFromEngine method and the input slot and proposer index to obtain the payload from the execution client.

This PR does not include caching the payload, we should implement caching once the end-to-end process is more defined, so we have a clearer understanding of what the cache's API will look like.

This implements a helper to get the ptc committee from a state. It uses the cached beacon committees if possible It also implements a helper to compute the largest power of two of a uint64 and a helper to test for nil payload attestation messages

* Add ePBS to db

* Modify `get_ptc` function to follow the Python spec * Assign PTC members from the beginning of beacon committee array

* Add `remove_flag` and its unit test * Add a test case trying to remove a flag that is not set

* Ensure EPBS state getters and setters check versions * Rename to LatestExecutionPayloadHeaderEPBS * Add minimal beacon state

* Add `IndexedPayloadAttestation` container * Add `GetPayloadAttestingIndices` and its unit test * Add `GetIndexedPayloadAttestation` and its unit test * Add `is_valid_indexed_payload_attestation` and its unit test * Create a smaller set of validators for faster unit test * Pass context to `GetPayloadTimelinessCommittee` * Iterate `ValidatorsReadOnly` instead of copying all validators

* Allow nodes with and without payload in forkchoice This PR takes care of adding nodes to forkchoice that may or may not have a corresponding payload. The rationale is as follows - The node structure is kept almost the same as today. - A zero payload hash is considered as if the node was empty (except for the tree root) - When inserting a node we check what the right parent node would be depending on whether the parent had a payload or not. - For pre-epbs forks all nodes are full, no logic changes except a new steps to gather the parent hash that is needed for block insertion. This PR had to change some core consensus types and interfaces. - It removed the ROBlockEPBS interface and added the corresponding ePBS fields to the ReadOnlyBeaconBlockBody - It moved the setters and getters to epbs dedicated files. It also added a checker for `IsParentFull` on forkchoice that simply checks for the parent hash of the parent node. * review

* Read only payload attestation message with verifier * Payload attestation tests (#14242) * Payload attestation in verification package * Feedback #1 --------- Co-authored-by: Md Amaan <114795592+Redidacove@users.noreply.github.com>

) * Add `execution_payload` and `payload_attestation_message` topics * Set `SourcePubkey` to 48 bytes long * Add randomly populated `PayloadAttestationMessage` object * Add tests for `execution_payload` and `payload_attestation_message` topics

* test-added * nil check fix * randomized inputs * hardcoded inputs * suggestions applied * minor-typo fixed * deleted

Adds the processing of execution payload envelope Corrects the protos for attestations and slashings in Electra versions Adds generators of full blocks for Electra

* Payload Attestation Sync package changes * With verifier * change idx back to uint64 * subscribe to topic * add back error --------- Co-authored-by: terence tsao <terence@prysmaticlabs.com>

* Add getter for payload attestation cache * Check against status * Feedback #1

* Enable validator client to sign execution payload envelope * Update comment Co-authored-by: JihoonSong <jihoonsong@users.noreply.github.com> --------- Co-authored-by: JihoonSong <jihoonsong@users.noreply.github.com>

* Correct requirement list of EnvelopeVerifier * Add unit tests of ExecutionPayloadEnvelope verification

* Signed execution payload header for sync * Use RO state * SignedExecutionPayloadHeader by hash and root * Fix execution headers cache

* Add protos for ePBS except state * Add ePBS beacon state proto * ePBS configuration constants * Helper for Payload Attestation Signing (#13901) * Add ePBS stuff to consensus-types: block * Add testing utility methods to return randomly populated ePBS objects * Add ePBS to state (#13926) * Implement get_ptc This implements a helper to get the ptc committee from a state. It uses the cached beacon committees if possible It also implements a helper to compute the largest power of two of a uint64 and a helper to test for nil payload attestation messages * Add EPBS slashing params * Add ePBS to db (#13971) * Add ePBS to db * Fix GetPayloadTimelinessCommittee to return correct PTC size (#14012) * Change gwei math to primitives package for ePBS state * use Keymanager() in validator client * Add PTC assignment support for Duty endpoint (#14032) * Enable validator client to submit payload attestation message (#14064) * Remove inclusion list from epbs (#14188) * Modify `get_ptc` function to follow the Python spec (#14256) * Modify `get_ptc` function to follow the Python spec * Assign PTC members from the beginning of beacon committee array * Add `remove_flag` and its unit test (#14260) * Add `remove_flag` and its unit test * Add a test case trying to remove a flag that is not set * Ensure epbs state getters & setters check versions (#14276) * Ensure EPBS state getters and setters check versions * Rename to LatestExecutionPayloadHeaderEPBS * Add minimal beacon state * Use slot for latest message in forkchoice (#14279) * Add payload attestation helper functions (#14258) * Add `IndexedPayloadAttestation` container * Add `GetPayloadAttestingIndices` and its unit test * Add `GetIndexedPayloadAttestation` and its unit test * Add `is_valid_indexed_payload_attestation` and its unit test * Create a smaller set of validators for faster unit test * Pass context to `GetPayloadTimelinessCommittee` * Iterate `ValidatorsReadOnly` instead of copying all validators * Use BeaconCommittees helper to get the ptc (#14286) * Allow nodes with and without payload in forkchoice (#14288) * Allow nodes with and without payload in forkchoice This PR takes care of adding nodes to forkchoice that may or may not have a corresponding payload. The rationale is as follows - The node structure is kept almost the same as today. - A zero payload hash is considered as if the node was empty (except for the tree root) - When inserting a node we check what the right parent node would be depending on whether the parent had a payload or not. - For pre-epbs forks all nodes are full, no logic changes except a new steps to gather the parent hash that is needed for block insertion. This PR had to change some core consensus types and interfaces. - It removed the ROBlockEPBS interface and added the corresponding ePBS fields to the ReadOnlyBeaconBlockBody - It moved the setters and getters to epbs dedicated files. It also added a checker for `IsParentFull` on forkchoice that simply checks for the parent hash of the parent node. * review * Read only payload attestation message with Verifier (#14222) * Read only payload attestation message with verifier * Payload attestation tests (#14242) * Payload attestation in verification package * Feedback #1 --------- Co-authored-by: Md Amaan <114795592+Redidacove@users.noreply.github.com> * Broadcast signed execution payload header to peer (#14300) * Add `execution_payload` and `payload_attestation_message` topics (#14304) * Add `execution_payload` and `payload_attestation_message` topics * Set `SourcePubkey` to 48 bytes long * Add randomly populated `PayloadAttestationMessage` object * Add tests for `execution_payload` and `payload_attestation_message` topics * Indexed paylaod attestation test (#14299) * test-added * nil check fix * randomized inputs * hardcoded inputs * suggestions applied * minor-typo fixed * deleted * Process Execution Payload Envelope in Chain Service (#14295) Adds the processing of execution payload envelope Corrects the protos for attestations and slashings in Electra versions Adds generators of full blocks for Electra * Implement OnPayloadAttestationMessage; add setters and required fields * Update ptcVote from byte to primitives; update functions * Update fields in struct * fix lint error: unnecessary conversion * fix lint error: unnecessary conversion * organize code; move ptcVote to node from store * add check to avoid computation if the payload boost is already applied * cleanup * update code to loop over bitfield * fix deepsource errors * use preset value instead of .len() * Payload Attestation Sync package changes (#13989) * Payload Attestation Sync package changes * With verifier * change idx back to uint64 * subscribe to topic * add back error --------- Co-authored-by: terence tsao <terence@prysmaticlabs.com> * change the name of the function; Add comprehensive comments * Add comprehensive unit tests for updateVotes function * fix unused var * fix nits * only updateBoosts onDemand * handle potential equivocation * Add protos for ePBS except state * Add ePBS beacon state proto * ePBS configuration constants * Helper for Payload Attestation Signing (#13901) * Add ePBS stuff to consensus-types: block * Add testing utility methods to return randomly populated ePBS objects * Add ePBS to state (#13926) * Implement get_ptc This implements a helper to get the ptc committee from a state. It uses the cached beacon committees if possible It also implements a helper to compute the largest power of two of a uint64 and a helper to test for nil payload attestation messages * Add EPBS slashing params * Add ePBS to db (#13971) * Add ePBS to db * Fix GetPayloadTimelinessCommittee to return correct PTC size (#14012) * Change gwei math to primitives package for ePBS state * use Keymanager() in validator client * Add PTC assignment support for Duty endpoint (#14032) * Enable validator client to submit payload attestation message (#14064) * Remove inclusion list from epbs (#14188) * Modify `get_ptc` function to follow the Python spec (#14256) * Modify `get_ptc` function to follow the Python spec * Assign PTC members from the beginning of beacon committee array * Add `remove_flag` and its unit test (#14260) * Add `remove_flag` and its unit test * Add a test case trying to remove a flag that is not set * Ensure epbs state getters & setters check versions (#14276) * Ensure EPBS state getters and setters check versions * Rename to LatestExecutionPayloadHeaderEPBS * Add minimal beacon state * Use slot for latest message in forkchoice (#14279) * Add payload attestation helper functions (#14258) * Add `IndexedPayloadAttestation` container * Add `GetPayloadAttestingIndices` and its unit test * Add `GetIndexedPayloadAttestation` and its unit test * Add `is_valid_indexed_payload_attestation` and its unit test * Create a smaller set of validators for faster unit test * Pass context to `GetPayloadTimelinessCommittee` * Iterate `ValidatorsReadOnly` instead of copying all validators * Use BeaconCommittees helper to get the ptc (#14286) * Allow nodes with and without payload in forkchoice (#14288) * Allow nodes with and without payload in forkchoice This PR takes care of adding nodes to forkchoice that may or may not have a corresponding payload. The rationale is as follows - The node structure is kept almost the same as today. - A zero payload hash is considered as if the node was empty (except for the tree root) - When inserting a node we check what the right parent node would be depending on whether the parent had a payload or not. - For pre-epbs forks all nodes are full, no logic changes except a new steps to gather the parent hash that is needed for block insertion. This PR had to change some core consensus types and interfaces. - It removed the ROBlockEPBS interface and added the corresponding ePBS fields to the ReadOnlyBeaconBlockBody - It moved the setters and getters to epbs dedicated files. It also added a checker for `IsParentFull` on forkchoice that simply checks for the parent hash of the parent node. * review * Read only payload attestation message with Verifier (#14222) * Read only payload attestation message with verifier * Payload attestation tests (#14242) * Payload attestation in verification package * Feedback #1 --------- Co-authored-by: Md Amaan <114795592+Redidacove@users.noreply.github.com> * Broadcast signed execution payload header to peer (#14300) * Add `execution_payload` and `payload_attestation_message` topics (#14304) * Add `execution_payload` and `payload_attestation_message` topics * Set `SourcePubkey` to 48 bytes long * Add randomly populated `PayloadAttestationMessage` object * Add tests for `execution_payload` and `payload_attestation_message` topics * Indexed paylaod attestation test (#14299) * test-added * nil check fix * randomized inputs * hardcoded inputs * suggestions applied * minor-typo fixed * deleted * Process Execution Payload Envelope in Chain Service (#14295) Adds the processing of execution payload envelope Corrects the protos for attestations and slashings in Electra versions Adds generators of full blocks for Electra * Payload Attestation Sync package changes (#13989) * Payload Attestation Sync package changes * With verifier * change idx back to uint64 * subscribe to topic * add back error --------- Co-authored-by: terence tsao <terence@prysmaticlabs.com> * Add getter for payload attestation cache (#14328) * Add getter for payload attestation cache * Check against status * Feedback #1 * Initialize payload att message verfier in sync (#14323) * Enable validator client to sign execution header (#14333) * Enable validator client to sign execution header * Update proto/prysm/v1alpha1/validator-client/keymanager.proto --------- Co-authored-by: Potuz <potuz@prysmaticlabs.com> * Process withdrawal (#14297) * process_withdrawal_fn and isParentfull test * suggestions applied * minor change * removed * lint * lint fix * removed Latestheader * test added with nil error * tests passing * IsParentNode Test added * lint * fix test * updated godoc * fix in godoc * comment removed * fixed braces * removed var * removed var * Update beacon-chain/core/blocks/withdrawals.go * Update beacon-chain/core/blocks/withdrawals_test.go * gazelle * test added and removed previous changes in Testprocesswithdrawal * added check for nil state * decrease chromatic complexity --------- Co-authored-by: Potuz <potuz@potuz.net> Co-authored-by: Potuz <potuz@prysmaticlabs.com> * Update tests * undo changes * UpdateVote independent of timing; updatePayloadboosts if the block is early * fix deep source * remove timinng checks; move to onDemand payloadBoosts; update testCases for the changes * update errors and returns; update test cases for the changes * fix deep source errors * reduce cyclomatic complexity * gazelle fix * remove a comment Co-authored-by: Potuz <potuz@potuz.net> * fix duplication in beacon-chain config * fix config tests * fix number of config params --------- Co-authored-by: Potuz <potuz@prysmaticlabs.com> Co-authored-by: terence <terence@prysmaticlabs.com> Co-authored-by: JihoonSong <jihoonsong@users.noreply.github.com> Co-authored-by: Md Amaan <114795592+Redidacove@users.noreply.github.com> Co-authored-by: Potuz <potuz@potuz.net>

* Handle incoming ptc attestation messages in the chain package * fix double import * remove unused error

potuz · 2024-09-02T12:53:41Z

beacon-chain/rpc/prysm/v1alpha1/validator/proposer_epbs.go

+		return nil, status.Errorf(codes.FailedPrecondition, "EPBS fork has not occurred yet")
+	}
+
+	st, parentRoot, err := vs.getParentState(ctx, slot)


This is not good enough, this path will get the parent state from the blockroot without caring for full vs empty. I am not sure how to proceed, while this would be outside of the scope of this PR, I feel like merging this right now may turn out to surface subtle bugs later.

potuz · 2024-09-02T12:57:17Z

beacon-chain/rpc/prysm/v1alpha1/validator/proposer_epbs.go

+	}
+
+	proposerIndex := req.ProposerIndex
+	localPayload, err := vs.getLocalPayloadFromEngine(ctx, st, parentRoot, slot, proposerIndex)


This function is still not compatible with ePBS, it doesn't handle the fork

Add wait until PTC duty

potuz and others added 30 commits August 29, 2024 14:46

Add protos for ePBS except state

7804234

Add ePBS beacon state proto

5e5c9a9

ePBS configuration constants

c85f4cb

Helper for Payload Attestation Signing (#13901)

c3083d7

Add ePBS stuff to consensus-types: block

2aa88fa

Add testing utility methods to return randomly populated ePBS objects

fef173b

Add ePBS to state (#13926)

fea68b1

Implement get_ptc

8a96f2d

This implements a helper to get the ptc committee from a state. It uses the cached beacon committees if possible It also implements a helper to compute the largest power of two of a uint64 and a helper to test for nil payload attestation messages

Add EPBS slashing params

99bbdaa

Add ePBS to db (#13971)

c98053f

* Add ePBS to db

Fix GetPayloadTimelinessCommittee to return correct PTC size (#14012)

8515a08

Change gwei math to primitives package for ePBS state

abbb9e4

use Keymanager() in validator client

4b873a9

Add PTC assignment support for Duty endpoint (#14032)

fdf25bf

Enable validator client to submit payload attestation message (#14064)

de53d52

Remove inclusion list from epbs (#14188)

97c326a

Modify get_ptc function to follow the Python spec (#14256)

9c5d980

* Modify `get_ptc` function to follow the Python spec * Assign PTC members from the beginning of beacon committee array

Add remove_flag and its unit test (#14260)

5b4d79c

* Add `remove_flag` and its unit test * Add a test case trying to remove a flag that is not set

Ensure epbs state getters & setters check versions (#14276)

55c917e

* Ensure EPBS state getters and setters check versions * Rename to LatestExecutionPayloadHeaderEPBS * Add minimal beacon state

Use slot for latest message in forkchoice (#14279)

bddbb6a

Use BeaconCommittees helper to get the ptc (#14286)

15afb23

Broadcast signed execution payload header to peer (#14300)

3c245f2

Indexed paylaod attestation test (#14299)

383e5da

* test-added * nil check fix * randomized inputs * hardcoded inputs * suggestions applied * minor-typo fixed * deleted

Process Execution Payload Envelope in Chain Service (#14295)

c2eb0c0

Adds the processing of execution payload envelope Corrects the protos for attestations and slashings in Electra versions Adds generators of full blocks for Electra

Payload Attestation Sync package changes (#13989)

8bc5992

* Payload Attestation Sync package changes * With verifier * change idx back to uint64 * subscribe to topic * add back error --------- Co-authored-by: terence tsao <terence@prysmaticlabs.com>

Add getter for payload attestation cache (#14328)

b7a043c

* Add getter for payload attestation cache * Check against status * Feedback #1

potuz and others added 10 commits August 29, 2024 14:50

Sync changes to process execution payload envelopes

e67fb11

Add unit tests of ExecutionPayloadEnvelope verification (#14373)

59598fc

* Correct requirement list of EnvelopeVerifier * Add unit tests of ExecutionPayloadEnvelope verification

Refactor currentlySyncingPayload cache (#14350)

2c351f7

Signed execution payload header for sync (#14363)

6152818

* Signed execution payload header for sync * Use RO state * SignedExecutionPayloadHeader by hash and root * Fix execution headers cache

Add ePBS fork schedule to config (#14383)

29ee00a

Receive ptc message (#14394)

5ba9654

* Handle incoming ptc attestation messages in the chain package * fix double import * remove unused error

Prysm rpc: Submit payload attestation data (#14381)

3bf00b0

Remove Changelog workflow

a21d4c2

terencechain requested a review from a team as a code owner September 2, 2024 02:37

terencechain requested review from kasey, rauljordan and nisdas and removed request for a team September 2, 2024 02:37

terencechain mentioned this pull request Sep 2, 2024

Prysm rpc: Get local header #14378

Closed

potuz requested changes Sep 2, 2024

View reviewed changes

terencechain added the ePBS EIP-7732 label Sep 2, 2024

Prysm rpc: submit execution payload envelope (#14395)

b4cf2e4

terencechain force-pushed the get-local-header-1 branch from e37fe64 to 9ec23e3 Compare September 3, 2024 14:52

terencechain added the Blocked Blocked by research or external factors label Sep 3, 2024

terencechain force-pushed the get-local-header-1 branch from 9ec23e3 to 3471b7f Compare September 3, 2024 18:00

terencechain and others added 5 commits September 5, 2024 12:33

Add wait until PTC duty helper function (#14419)

25de893

Add wait until PTC duty

Add GetPTCVote helpers (#14420)

b2f47ca

Handle execution payload insertion in forkchoice (#14422)

674196e

Prysm rpc: Get payload attestation data (#14380)

fdccbb5

Prysm rpc: Get local header

e88900f

terencechain force-pushed the get-local-header-1 branch from 3471b7f to e88900f Compare September 10, 2024 21:54

potuz force-pushed the epbs branch from fdccbb5 to 74d5b79 Compare September 13, 2024 11:18

potuz requested a review from prestonvanloon as a code owner September 13, 2024 11:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prysm rpc: Get local header #14409

Prysm rpc: Get local header #14409

terencechain commented Sep 2, 2024

potuz Sep 2, 2024

potuz Sep 2, 2024

Prysm rpc: Get local header #14409

Are you sure you want to change the base?

Prysm rpc: Get local header #14409

Conversation

terencechain commented Sep 2, 2024

potuz Sep 2, 2024

Choose a reason for hiding this comment

potuz Sep 2, 2024

Choose a reason for hiding this comment