[Consensus] Follower engine message queue

[durkmurder]

https://github.com/dapperlabs/flow-go/issues/6173

Context

This PR implements message queues and message processing on dedicated worker for follower engine. It also updates FollowerLoop to use component.ComponentManager instead of SingleRunner.
This PR is required to implement https://github.com/dapperlabs/flow-go/issues/6424 because follower engine and compliance engine need to implement the same interface.

Note on updating follower engine. This PR introduces message queues, proper implementation of component.Component interface and standalone startup of follower loop. Other changes will be handled in separate PR(like extracting interface for compliance engine).

Originally I didn't plan to implement standalone startup as part of this PR but observing that some nodes already start follower loop separately I've decided to implement this for other nodes otherwise it was resulting in double Start which is a panic.

Notable changes

• Updated follower engine and follower loop to use ComponentManager
• Updated FollowerLoop to work with buffered queues
• Removed SingleRunner
• Updated test suites

[codecov-commenter]

Codecov Report

Merging #3479 (b988ad5) into feature/active-pacemaker (c02eefd) will decrease coverage by 11.56%.
The diff coverage is 0.00%.

❗ Current head b988ad5 differs from pull request most recent head c6a89c8. Consider uploading reports for the commit c6a89c8 to get more accurate results

@@                      Coverage Diff                      @@
##           feature/active-pacemaker    #3479       +/-   ##
=============================================================
- Coverage                     55.75%   44.18%   -11.57%     
=============================================================
  Files                           769      228      -541     
  Lines                         71245    25244    -46001     
=============================================================
- Hits                          39723    11155    -28568     
+ Misses                        28311    13070    -15241     
+ Partials                       3211     1019     -2192     

Flag	Coverage Δ
unittests	44.18% <0.00%> (-11.57%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
cmd/execution_builder.go	0.00% <0.00%> (ø)
cmd/verification_builder.go	0.00% <0.00%> (ø)
fvm/environment/contract_updater.go	66.88% <0.00%> (-0.66%)	⬇️
ledger/complete/wal/checkpointer.go	63.10% <0.00%> (-0.40%)	⬇️
engine/common/follower/engine.go
...rk/p2p/unicast/ratelimit/bandwidth_rate_limiter.go
engine/execution/convert.go
storage/badger/operation/computation_result.go
engine/access/rest/request/get_events.go
module/mempool/herocache/backdata/heropool/pool.go
... and 535 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[durkmurder]

Originally I didn't plan to implement standalone startup as part of this PR but observing that some nodes already start follower loop separately I've decided to implement this for other nodes otherwise it was resulting in double Start which is a panic.

[durkmurder]

Previously we didn't start consensus committee it kinda worked in integration tests but generally speaking we would be missing some events without it

[AlexHentschel]

🎉

[AlexHentschel]

should we add a termination check here? Like this:

flow-go/engine/collection/compliance/engine.go

Lines 171 to 175 in 00033d2

	select {
	case <-doneSignal:
	return nil
	default:
	}

Maybe best to append do PR #3512

[durkmurder]

bors merge

[bors]

Build succeeded:

• Integration Tests (make -C integration access-tests)
• Integration Tests (make -C integration bft-tests)
• Integration Tests (make -C integration collection-tests)
• Integration Tests (make -C integration consensus-tests)
• Integration Tests (make -C integration epochs-tests)
• Integration Tests (make -C integration execution-tests)
• Integration Tests (make -C integration ghost-tests)
• Integration Tests (make -C integration mvp-tests)
• Integration Tests (make -C integration network-tests)
• Integration Tests (make -C integration verification-tests)
• Lint (./)
• Lint (./crypto/)
• Lint (./integration/)
• Unit Tests (access)
• Unit Tests (admin)
• Unit Tests (cmd)
• Unit Tests (consensus)
• Unit Tests (engine)
• Unit Tests (fvm)
• Unit Tests (ledger)
• Unit Tests (module)
• Unit Tests (network)
• Unit Tests (others)
• Unit Tests (utils)