ethcore: add clique engine #9981

5chdn · 2018-11-28T10:05:10Z

ethcore/engine: clique engine
json/spec: clique apis
ethcore/ethereum: rinkeby config
ethcore/ethereum: goerli config (cross-client)
ethcore/ethereum: kotti config (classic testnet)

Fixes for broken spec code, sync speedup

fix broken sync

Merge updates downstream

merge upstream updates downstream

some preliminary cleanup

ethcore: fix goerli chain spec

ethcore/src/block.rs

ethcore/src/engines/clique/signer_snapshot.rs

ethcore/src/engines/clique/mod.rs

ethcore/src/engines/mod.rs

ethcore/src/engines/clique/mod.rs

…5-backport-downstream

backport downstream

ethcore/src/engines/clique/mod.rs

HCastano · 2019-03-25T10:29:45Z

ethcore/src/engines/clique/mod.rs

+				let elapsed = backfill_start.elapsed();
+				trace!(target: "engine",
+						"Back-filling succeed, took {} ms.",
+						// replace with Duration::as_millis after rust 1.33


I thought we'd resolved this, or am I mistaken?

Never fixed but I don't think that the trivial thingy is worth the effort of requiring Rust 1.33 to compile there are still some platforms (no rustup) that can't be updated for every release

I can fix it if you don't agree!

No, it's fine we can leave it as is. I think we should at least track it with an issue though so it doesn't get forgotten in the land of immortal TODOs

HCastano · 2019-03-25T10:53:32Z

ethcore/src/engines/clique/mod.rs

+		}
+
+		// Don't waste time checking blocks from the future
+		{


I don't think we need these brackets here

ethcore/src/engines/clique/mod.rs

ethcore/src/engines/mod.rs

fix(nits)

niklasad1 · 2019-03-26T09:58:27Z

ethcore/src/engines/clique/step_service.rs

+					thread::sleep(Duration::from_millis(2000));
+				}
+				trace!(target: "miner", "StepService: shut down.");
+			}).expect("Unable to launch thread.");


The API could be better here.

Ideally, we should return Result here instead of using expect if the thread creation fails.

I don't want to address it in this PR, file an issue?

fix(CliqueStepService): replace `RwLock` with `AtomicBool`

fix(clique): use `Duration::as_millis`

Clean up some Clique documentation

sorpaas

I specifically reviewed changes to existing code (only around 34 lines!) and cannot find anything that might break our other engines. So I think this should be in a state where we can merge!

sorpaas · 2019-03-26T22:16:56Z

ethcore/src/block.rs

@@ -405,15 +404,20 @@ impl LockedBlock {
 	/// Provide a valid seal in order to turn this into a `SealedBlock`.
 	///
 	/// NOTE: This does not check the validity of `seal` with the engine.
-	pub fn seal(self, engine: &EthEngine, seal: Vec<Bytes>) -> Result<SealedBlock, BlockError> {
-		let expected_seal_fields = engine.seal_fields(&self.block.header);


In case this confuses anyone -- LockedBlock implements a Deref to ExecutedBlock so both with block. and without are valid.

* fix broken sync * correct seal fields * ethcore: fix comment * parity: remove duplicate params * clique: fix whitespaces * ethcore: fix goerli chain spec * refactor signer_snapshot into pending/finalized state * move close_block_extra_data after seal is applied * refactor most of the logic into the signer_snapshot * clique: refactor locking logic out of the consensus engine interface * Fix jsonspec and add an unittest * Replace space with tabs * Unbroke sync * Fix broken sync * 1/2 state tracking without votes * 2/2 implement vote tracking * ci: use travis for goerli * ci: setup a clique network * ci: sync a görli node * add clique deploy script * ci: fix paths in clique deploy script * ci: use docker compose * ci: fix travis job names * ci: fix build deps * ci: massively reduce tests * Revert "ci: massively reduce tests" This reverts commit 6369f0b. * ci: run cargo test directly * ci: separate build and test stages * ci: cache rust installation * ci: simplify ci stages * ci: make clique deploy script executable * ci: shutdown goerli sync after 20min * ci: remove slow sync stage * ci: use timeout to finish jobs * ci: fix build path * ci: use absolute paths to end this confusion * ci: add geth and parity to path * ci: be more verbose * ci: allow for more relaxed caching timeout * ci: update repositories for custom ppa * ci: fix typo in file name * ci: fix docker compose file * ci: add ethkey to docker * ci: make sure deploy script is up to date with upstream * ci: stop docker container after certain time * ci: force superuser to update permissions on docker files * ci: reduce run time of script to ~30 min * ci: remove duplicate caching in travis * remove trace statements * clique: add more validation involving the recent signer list * ethcore: enable constantinople for rinkeby * ethcore: fix whitespaces in rinkeby spec * ethcore: reformat goerli.json * Revert "ci: remove duplicate caching in travis" This reverts commit a562838. * tmp commit * another tmp commit * it builds! * add sealing capabilities * add seal_header hook to allow separation of block seal/importing code paths * clique: remove populate_from_parent. * add panic * make turn delay random * initialize OpenBlock properly in 'enact' * misc: remove duplicate lines * misc: fix license headers * misc: convert spaces to tabs * misc: fix tabs * Update Cargo.toml * Update Cargo.toml * Update Cargo.toml * clique: ensure validator restores state before trying to seal * clique: make 'state' return an Error. Make some error messages more clear * Fix compile error after rebase & toolchain upgrade * fix a bunch of import warnings * Refactor code * Fix permissions * Refactoring syncing * Implement full validator checks * Refactor util functions to seperate file * mining 1 * ethcore: add chainspec for kotti * ethcore: rename pre-goerli configs * ethcore: load kotti chain spec * cli: add kotti to params * Implement working local sealing * making sealing & syncing work together * Relax timestamp checking * ethcore: prepare for the real goerli to launch * Implement NOTURN wiggle properly & cleanupnup warnings * Implement vote casting * Update docs & skip signing if no signer * Optimize step-service interval * Record state on local sealed block * Fix script filemode * Cleaning up codebase * restore enact trace logging * Delete clique.sh and move sync.sh * remove travis.yml * Remove dead code * Cleanup compile warning * address review comments * adding more comments and removing unwrap() * ci: remove sync script * Address review comments * fix compile error * adding better debugging for timing * Implement an dedicated thread for sealing timing * fix(add helper for timestamp overflows) (#10330) * fix(add helper timestamp overflows) * fix(simplify code) * fix(make helper private) * snap: official image / test (#10168) * official image / test * fix / test * bit more necromancy * fix paths * add source bin/df /test * add source bin/df /test2 * something w paths /test * something w paths /test * add source-type /test * show paths /test * copy plugin /test * plugin -> nil * install rhash * no questions while installing rhash * publish snap only for release * fix(docker): fix not receives SIGINT (#10059) * fix(docker): fix not receives SIGINT * fix: update with reviews * update with review * update * update * Don't add discovery initiators to the node table (#10305) * Don't add discovery initiators to the node table * Use enums for tracking state of the nodes in discovery * Dont try to ping ourselves * Fix minor nits * Update timeouts when observing an outdated node * Extracted update_bucket_record from update_node * Fixed typo * Fix two final nits from @todr * change docker image based on debian instead of ubuntu due to the chan… (#10336) * change docker image based on debian instead of ubuntu due to the changes of the build container * role back docker build image and docker deploy image to ubuntu:xenial based (#10338) * Bundle protocol and packet_id together in chain sync (#10315) Define a new `enum` where devp2p subprotocol packet ids (currently eth and par) are defined. Additionally provide functionality to query id value and protocol of a given id object. * snap: prefix version and populate candidate channel (#10343) * snap: populate candidate releases with beta snaps to avoid stale channel * snap: prefix version with v* * addressing review comments * engine: fix copyright header * scripts: restore permissions on sign command * ethcore: enforce tabs * ethcore: enforce tabs * ethcore: enforce tabs * addressing comments * addressing comments * addressing more comments * addressing more comments * addressing more comments * addressing more comments * addressing more comments * json-spec: fix clique epoch to non-zero u64 * ci: enable travis for parity goerli * ci: don't separate build and test step * ci: don't run c++ tests on travis * ci: simplify cargo test to squeeze into travis timeout * ci: don't run tests on travis at all * style(fixes) * fix(add tests) * fix(recent_signer bug) * fix(complete all tests) * fix(nits) * fix(simplify asserts) * fix(cliqueState): simplify code * fix(nits) * docs(comments what's need to fixed) * fix(revert unintended changes) * fix(tests) * fix(logs): voting logs * fix(readability + more logs) * fix(sync) * docs(add missing licens header) * fix(log): info! -> trace! * docs(fix nits) + fix(remove assert) * perf(use counter instead of vec) * fix(remove needless block in match) * fix(faulty comment) * grumbles(docs for tests) * fix(nits) * fix(revert_vote): only remove vote when votes == 0 * fix(vote counter): checked arithmetics * fix(simplify tests) * fix(nits) * fix(clique): err types * fix(clique utils): make use of errors * fix(cleanup nits) * fix(clique sealing): don't read state no signer * fix(replace Vec<Signers> with BTreeSet<Signers>) * fix(tests): BTreeSet and more generic helpers * fix(nits) * fix(ethcore_block_seal): remove needless `Box` * fix(faulty log): info -> trace * fix(checked SystemTime): prevent SystemTime panics * style(chain cfg): space after `:` * style(fn enact): fix whitespace * docs(clique): StepService * docs(nit): fix faulty comment * docs(fix typo) * style(fix bad indentation) * fix(bad regex match) * grumble(on_seal_block): make `&mut` to avoid clone * docs(on_seal_block): fix faulty documentation * Delete .travis.yml * docs: remove eth hf references in spec * Update client.rs * fix(nits) * fix(clique step): `RwLock` -> `AtomicBool` * fix(clique): use `Duration::as_millis` * Clean up some Clique documentation Co-authored-by: soc1c <soc1c@users.noreply.github.com> Co-authored-by: HCastano <HCastano@users.noreply.github.com> Co-authored-by: niklasad1 <niklasad1@users.noreply.github.com> Co-authored-by: jwasinger <jwasinger@users.noreply.github.com> Co-authored-by: ChainSafe <ChainSafe@users.noreply.github.com> Co-authored-by: thefallentree <thefallentree@users.noreply.github.com> Co-authored-by: 5chdn <5chdn@users.noreply.github.com>

soc1c · 2019-03-27T13:25:00Z

aa8487c is the actual merge commit on master (amended with co-authors)

#10514) * ethcore: add clique engine (#9981) * fix broken sync * correct seal fields * ethcore: fix comment * parity: remove duplicate params * clique: fix whitespaces * ethcore: fix goerli chain spec * refactor signer_snapshot into pending/finalized state * move close_block_extra_data after seal is applied * refactor most of the logic into the signer_snapshot * clique: refactor locking logic out of the consensus engine interface * Fix jsonspec and add an unittest * Replace space with tabs * Unbroke sync * Fix broken sync * 1/2 state tracking without votes * 2/2 implement vote tracking * ci: use travis for goerli * ci: setup a clique network * ci: sync a görli node * add clique deploy script * ci: fix paths in clique deploy script * ci: use docker compose * ci: fix travis job names * ci: fix build deps * ci: massively reduce tests * Revert "ci: massively reduce tests" This reverts commit 6369f0b. * ci: run cargo test directly * ci: separate build and test stages * ci: cache rust installation * ci: simplify ci stages * ci: make clique deploy script executable * ci: shutdown goerli sync after 20min * ci: remove slow sync stage * ci: use timeout to finish jobs * ci: fix build path * ci: use absolute paths to end this confusion * ci: add geth and parity to path * ci: be more verbose * ci: allow for more relaxed caching timeout * ci: update repositories for custom ppa * ci: fix typo in file name * ci: fix docker compose file * ci: add ethkey to docker * ci: make sure deploy script is up to date with upstream * ci: stop docker container after certain time * ci: force superuser to update permissions on docker files * ci: reduce run time of script to ~30 min * ci: remove duplicate caching in travis * remove trace statements * clique: add more validation involving the recent signer list * ethcore: enable constantinople for rinkeby * ethcore: fix whitespaces in rinkeby spec * ethcore: reformat goerli.json * Revert "ci: remove duplicate caching in travis" This reverts commit a562838. * tmp commit * another tmp commit * it builds! * add sealing capabilities * add seal_header hook to allow separation of block seal/importing code paths * clique: remove populate_from_parent. * add panic * make turn delay random * initialize OpenBlock properly in 'enact' * misc: remove duplicate lines * misc: fix license headers * misc: convert spaces to tabs * misc: fix tabs * Update Cargo.toml * Update Cargo.toml * Update Cargo.toml * clique: ensure validator restores state before trying to seal * clique: make 'state' return an Error. Make some error messages more clear * Fix compile error after rebase & toolchain upgrade * fix a bunch of import warnings * Refactor code * Fix permissions * Refactoring syncing * Implement full validator checks * Refactor util functions to seperate file * mining 1 * ethcore: add chainspec for kotti * ethcore: rename pre-goerli configs * ethcore: load kotti chain spec * cli: add kotti to params * Implement working local sealing * making sealing & syncing work together * Relax timestamp checking * ethcore: prepare for the real goerli to launch * Implement NOTURN wiggle properly & cleanupnup warnings * Implement vote casting * Update docs & skip signing if no signer * Optimize step-service interval * Record state on local sealed block * Fix script filemode * Cleaning up codebase * restore enact trace logging * Delete clique.sh and move sync.sh * remove travis.yml * Remove dead code * Cleanup compile warning * address review comments * adding more comments and removing unwrap() * ci: remove sync script * Address review comments * fix compile error * adding better debugging for timing * Implement an dedicated thread for sealing timing * fix(add helper for timestamp overflows) (#10330) * fix(add helper timestamp overflows) * fix(simplify code) * fix(make helper private) * snap: official image / test (#10168) * official image / test * fix / test * bit more necromancy * fix paths * add source bin/df /test * add source bin/df /test2 * something w paths /test * something w paths /test * add source-type /test * show paths /test * copy plugin /test * plugin -> nil * install rhash * no questions while installing rhash * publish snap only for release * fix(docker): fix not receives SIGINT (#10059) * fix(docker): fix not receives SIGINT * fix: update with reviews * update with review * update * update * Don't add discovery initiators to the node table (#10305) * Don't add discovery initiators to the node table * Use enums for tracking state of the nodes in discovery * Dont try to ping ourselves * Fix minor nits * Update timeouts when observing an outdated node * Extracted update_bucket_record from update_node * Fixed typo * Fix two final nits from @todr * change docker image based on debian instead of ubuntu due to the chan… (#10336) * change docker image based on debian instead of ubuntu due to the changes of the build container * role back docker build image and docker deploy image to ubuntu:xenial based (#10338) * Bundle protocol and packet_id together in chain sync (#10315) Define a new `enum` where devp2p subprotocol packet ids (currently eth and par) are defined. Additionally provide functionality to query id value and protocol of a given id object. * snap: prefix version and populate candidate channel (#10343) * snap: populate candidate releases with beta snaps to avoid stale channel * snap: prefix version with v* * addressing review comments * engine: fix copyright header * scripts: restore permissions on sign command * ethcore: enforce tabs * ethcore: enforce tabs * ethcore: enforce tabs * addressing comments * addressing comments * addressing more comments * addressing more comments * addressing more comments * addressing more comments * addressing more comments * json-spec: fix clique epoch to non-zero u64 * ci: enable travis for parity goerli * ci: don't separate build and test step * ci: don't run c++ tests on travis * ci: simplify cargo test to squeeze into travis timeout * ci: don't run tests on travis at all * style(fixes) * fix(add tests) * fix(recent_signer bug) * fix(complete all tests) * fix(nits) * fix(simplify asserts) * fix(cliqueState): simplify code * fix(nits) * docs(comments what's need to fixed) * fix(revert unintended changes) * fix(tests) * fix(logs): voting logs * fix(readability + more logs) * fix(sync) * docs(add missing licens header) * fix(log): info! -> trace! * docs(fix nits) + fix(remove assert) * perf(use counter instead of vec) * fix(remove needless block in match) * fix(faulty comment) * grumbles(docs for tests) * fix(nits) * fix(revert_vote): only remove vote when votes == 0 * fix(vote counter): checked arithmetics * fix(simplify tests) * fix(nits) * fix(clique): err types * fix(clique utils): make use of errors * fix(cleanup nits) * fix(clique sealing): don't read state no signer * fix(replace Vec<Signers> with BTreeSet<Signers>) * fix(tests): BTreeSet and more generic helpers * fix(nits) * fix(ethcore_block_seal): remove needless `Box` * fix(faulty log): info -> trace * fix(checked SystemTime): prevent SystemTime panics * style(chain cfg): space after `:` * style(fn enact): fix whitespace * docs(clique): StepService * docs(nit): fix faulty comment * docs(fix typo) * style(fix bad indentation) * fix(bad regex match) * grumble(on_seal_block): make `&mut` to avoid clone * docs(on_seal_block): fix faulty documentation * Delete .travis.yml * docs: remove eth hf references in spec * Update client.rs * fix(nits) * fix(clique step): `RwLock` -> `AtomicBool` * fix(clique): use `Duration::as_millis` * Clean up some Clique documentation * Add trace information to eth_estimateGas (#10519) * Add trace information to eth_estimateGas * replace unwrap better version * change vm::Error formatter to more user-friendly * remove extra error format * use map_or instead sequence of map/unwrap_or * fix(light/on_demand): extract as a trait * fix(grumble): OnDemand remove needless trait bound

* master: (48 commits) ethcore: remove eth social and easthub chain configs (#10531) Initial support sccache for windows build (#10520) fix(light): make `OnDemand` generic instead of using the concrete type (#10514) private-tx: replace error_chain (#10510) Add trace information to eth_estimateGas (#10519) ethcore: add clique engine (#9981) verbose flag for cpp tests (#10524) Add a more realistic Batch test (#10511) docs: add changelogs for 2.3.{6,7,8} and 2.4.{1,2,3} (#10494) fix(rpc): fix a bunch of clippy lints (#10493) fix Sha3/keccak256 hash calculation for binaries (#10509) Add additional request tests (#10503) whisper/cli: add p2p port and ip parameters (#10057) fix(time-utils): add missing license (#10497) fix(extract `timestamp_checked_add` as lib) (#10383) fix(rpc): lint `unused_extern_crates` + fix warns (#10489) fix win&mac build (#10486) Сaching through docker volume (#10477) OpenBlock::new take IntoIterator instead of mutable ref to Iterator (#10480) simplify block module and usage (#10479) ...

thefallentree and others added 5 commits November 19, 2018 09:08

Merge pull request #18 from goerli/clique-voting

6c0c97c

Fixes for broken spec code, sync speedup

fix broken sync

4fc037c

Merge pull request #19 from goerli/sync-fix

cca282e

fix broken sync

Merge pull request #20 from paritytech/master

5514e1a

Merge updates downstream

correct seal fields

efc326d

5chdn added A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. M4-core ⛓ Core client code / Rust. labels Nov 28, 2018

5chdn added this to the 2.3 milestone Nov 28, 2018

5chdn added 7 commits November 28, 2018 11:12

ethcore: fix comment

e7233f3

parity: remove duplicate params

3059f5a

clique: fix whitespaces

cf2ad32

Merge pull request #23 from paritytech/master

b045e13

merge upstream updates downstream

Merge pull request #22 from goerli/a5-config-fixes

cbf224d

some preliminary cleanup

ethcore: fix goerli chain spec

d492700

Merge pull request #24 from goerli/a5-config-goerlispec

93869f6

ethcore: fix goerli chain spec

5chdn added the B7-releasenotes 📜 Changes should be mentioned in the release notes of the next minor version release. label Nov 28, 2018

sorpaas reviewed Nov 29, 2018

View reviewed changes

jwasinger and others added 13 commits November 29, 2018 10:11

refactor signer_snapshot into pending/finalized state

2430e66

Merge branch 'master' of github.com:paritytech/parity-ethereum into a…

afc7aac

…5-backport-downstream

Merge pull request #25 from goerli/a5-backport-downstream

c9e651e

backport downstream

move close_block_extra_data after seal is applied

314fe35

refactor most of the logic into the signer_snapshot

d6ff43d

clique: refactor locking logic out of the consensus engine interface

9347601

Merge #21 "Voting logic from jwasinger. Fix up build.

ca62dde

Fix jsonspec and add an unittest

320fd9a

Replace space with tabs

af0cc3d

Merge remote-tracking branch 'upstream/master'

3a67edf

Unbroke sync

b7bf2d1

Merge remote-tracking branch 'upstream/master'

f807ff5

Fix broken sync

0802cac

soc1c requested review from niklasad1 and HCastano March 23, 2019 10:28

HCastano reviewed Mar 25, 2019

View reviewed changes

ethcore/src/engines/clique/mod.rs Outdated Show resolved Hide resolved

HCastano reviewed Mar 25, 2019

View reviewed changes

ethcore/src/engines/clique/mod.rs Outdated Show resolved Hide resolved

HCastano reviewed Mar 25, 2019

View reviewed changes

ethcore/src/engines/clique/mod.rs Outdated Show resolved Hide resolved

HCastano reviewed Mar 25, 2019

View reviewed changes

ethcore/src/engines/mod.rs Outdated Show resolved Hide resolved

5chdn mentioned this pull request Mar 25, 2019

Parity with rinkeby testnet #6045

Closed

niklasad1 and others added 2 commits March 25, 2019 14:47

fix(nits)

0502915

Merge pull request #82 from niklasad1/na-cleanup

0b02823

fix(nits)

soc1c mentioned this pull request Mar 25, 2019

Clean up chain specs and network configurations #10522

Closed

fix(clique step): RwLock -> AtomicBool

e39abfe

niklasad1 reviewed Mar 26, 2019

View reviewed changes

soc1c and others added 5 commits March 26, 2019 11:51

Merge pull request #83 from niklasad1/StepService-AtomicBool

5dccb6b

fix(CliqueStepService): replace `RwLock` with `AtomicBool`

fix(clique): use Duration::as_millis

f384261

Clean up some Clique documentation

70c0617

Merge pull request #84 from niklasad1/na-fix-duration-asmillis

7b5c361

fix(clique): use `Duration::as_millis`

Merge pull request #85 from HCastano/hc-fix-clique-docs

b2d1b3b

Clean up some Clique documentation

sorpaas approved these changes Mar 26, 2019

View reviewed changes

sorpaas merged commit 67e331b into openethereum:master Mar 26, 2019

soc1c mentioned this pull request Mar 27, 2019

restore authorship for clique engine #10530

Closed

soc1c added the B1-patch-beta 🕷🕷 label Mar 27, 2019

0xsarvesh mentioned this pull request May 14, 2019

User should be able to choose the client OpenST/mosaic-chains#22

Closed

gumb0 mentioned this pull request Jun 20, 2019

Clique PoA implementation plan ethereum/aleth#5522

Open

niklasad1 mentioned this pull request Feb 14, 2020

Include the seal when populating the header for a new block #11475

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ethcore: add clique engine #9981

ethcore: add clique engine #9981

5chdn commented Nov 28, 2018 •

edited

Loading

HCastano Mar 25, 2019

niklasad1 Mar 25, 2019

niklasad1 Mar 25, 2019

HCastano Mar 26, 2019

HCastano Mar 25, 2019

niklasad1 Mar 26, 2019 •

edited

Loading

sorpaas left a comment

sorpaas Mar 26, 2019

soc1c commented Mar 27, 2019

ethcore: add clique engine #9981

ethcore: add clique engine #9981

Conversation

5chdn commented Nov 28, 2018 • edited Loading

HCastano Mar 25, 2019

Choose a reason for hiding this comment

niklasad1 Mar 25, 2019

Choose a reason for hiding this comment

niklasad1 Mar 25, 2019

Choose a reason for hiding this comment

HCastano Mar 26, 2019

Choose a reason for hiding this comment

HCastano Mar 25, 2019

Choose a reason for hiding this comment

niklasad1 Mar 26, 2019 • edited Loading

Choose a reason for hiding this comment

sorpaas left a comment

Choose a reason for hiding this comment

sorpaas Mar 26, 2019

Choose a reason for hiding this comment

soc1c commented Mar 27, 2019

5chdn commented Nov 28, 2018 •

edited

Loading

niklasad1 Mar 26, 2019 •

edited

Loading