[CI] Add github action to build boot-tools on demand #3787

peterargue · 2023-01-07T00:36:21Z

The PR adds a github action workflow that can be run on demand to build the bootstrap, transit and util tools, then upload them to GCS

codecov-commenter · 2023-01-07T00:42:23Z

Codecov Report

Merging #3787 (d1ceb3b) into master (3745401) will increase coverage by 1.82%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3787      +/-   ##
==========================================
+ Coverage   53.61%   55.44%   +1.82%     
==========================================
  Files         548      768     +220     
  Lines       49302    70277   +20975     
==========================================
+ Hits        26432    38963   +12531     
- Misses      20629    28135    +7506     
- Partials     2241     3179     +938

Flag	Coverage Δ
unittests	`55.44% <ø> (+1.82%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
fvm/environment/value_store.go	`55.23% <0.00%> (-2.80%)`	⬇️
fvm/transactionVerifier.go	`82.32% <0.00%> (-2.13%)`	⬇️
fvm/derived/derived_block_data.go	`30.55% <0.00%> (-1.80%)`	⬇️
fvm/context.go	`53.47% <0.00%> (-1.47%)`	⬇️
consensus/hotstuff/eventloop/event_loop.go	`73.46% <0.00%> (-1.37%)`	⬇️
fvm/transactionSequenceNum.go	`70.00% <0.00%> (-1.16%)`	⬇️
fvm/transactionInvoker.go	`80.51% <0.00%> (-0.19%)`	⬇️
fvm/environment/contract_updater.go	`66.77% <0.00%> (-0.11%)`	⬇️
module/trace/noop.go	`0.00% <0.00%> (ø)`
module/trace/trace.go	`0.00% <0.00%> (ø)`
... and 234 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

zhangchiqing · 2023-01-09T17:41:25Z

.github/workflows/tools.yml

+  docker-push:
+    name: Push to container registry


I don't think we are pushing containers to registry, maybe we can rename it to build-boot-tools?

zhangchiqing · 2023-01-09T17:42:25Z

.github/workflows/tools.yml

+    - name: Set up Google Cloud SDK
+      uses: google-github-actions/setup-gcloud@v1
+      with:
+        project_id: flow # TODO: is this correct?


cc @sjonpaulbrown to confirm

This is the correct project id.

sjonpaulbrown · 2023-01-09T18:03:13Z

.github/workflows/tools.yml

+        mkdir boot-tools
+        mv bootstrap util transit boot-tools/
+        tar -czf boot-tools.tar ./boot-tools/
+        gsutil cp boot-tools.tar gs://flow-genesis-bootstrap/boot-tools.tar


Could we update this to conditionally support test builds that are uploaded with an artifact name that uses the ref rather than boot-tools.tar?

If we also address the comment on the inputs, we could essentially use the following to conditionally set the name. This would allow us to kick of on-demand builds that would allow us to build the boot tools for testing purposes.

ARTIFACT_NAME=$([ "$INPUT_PRODBUILD" = "1" ] && echo "boot-tools.tar" || echo "$(git rev-parse --short HEAD).tar") gsutil cp boot-tools.tar gs://flow-genesis-bootstrap/$ARTIFACT_NAME

how about it uploads to a tag specific directory by default, and there's an option to promote it to the official image
e.g.

upload to tools/v0.x.x/boot-tools.tar (or tools/1234abc/boot-tools.tar if that's what was passed to tag)

if "promote" is set, also upload to ./boot-tools.tar

then the spork tooling could always pull from the versioned file, and the node operator docs could continue to point to the current location

That sounds like a good solution to me.

sjonpaulbrown · 2023-01-09T18:03:17Z

.github/workflows/tools.yml

+    inputs: 
+      tag:
+        description: 'Tagged commit to build tools against'
+        required: true
+        type: string


In conjunction with my other comment, this additional input would allow us to conditionally build production builds as well as test builds that we could use before publishing the production tools.

Suggested change

inputs:

tag:

description: 'Tagged commit to build tools against'

required: true

type: string

inputs:

tag:

description: 'Tagged commit to build tools against'

required: true

type: string

prodbuild:

description: 'Execute production artifact build. Defaults to test build that will create artifact with short commit'

type: string

default: 0

peterargue · 2023-01-10T22:32:02Z

.github/workflows/tools.yml

+    - id: auth
+      uses: google-github-actions/auth@v1
+      with:
+        credentials_json: ${{ secrets.GCR_SERVICE_KEY }} # TODO: we need a new key to allow uploads


@sjonpaulbrown @gomisha could one of your help create and add a new secret for this usecase?

peterargue · 2023-01-11T20:08:19Z

bors merge

bors · 2023-01-11T20:26:44Z

Build succeeded:

3787: [CI] Add github action to build boot-tools on demand r=peterargue a=peterargue The PR adds a github action workflow that can be run on demand to build the `bootstrap`, `transit` and `util` tools, then upload them to GCS Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>

[CI] Add github action to build boot-tools on demand

e6bc1f3

peterargue requested a review from gomisha as a code owner January 7, 2023 00:36

peterargue requested review from sjonpaulbrown and zhangchiqing January 7, 2023 00:49

zhangchiqing approved these changes Jan 9, 2023

View reviewed changes

sjonpaulbrown reviewed Jan 9, 2023

View reviewed changes

add optional promote feature, upload util separately

306fa8d

peterargue requested a review from sjonpaulbrown January 10, 2023 22:30

peterargue commented Jan 10, 2023

View reviewed changes

cleanup commands

d1ceb3b

gomisha approved these changes Jan 11, 2023

View reviewed changes

bors bot merged commit 65d67fd into master Jan 11, 2023

bors bot deleted the petera/ci-build-tools branch January 11, 2023 20:26

This was referenced Jan 12, 2023

[CI] Add github action to build boot-tools on demand - v0.29-no-pacemaker backport #3806

Merged

[CI] Add github action to build boot-tools on demand - v0.29 backport #3807

Merged

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Add github action to build boot-tools on demand #3787

[CI] Add github action to build boot-tools on demand #3787

peterargue commented Jan 7, 2023

codecov-commenter commented Jan 7, 2023 •

edited

Loading

zhangchiqing Jan 9, 2023

zhangchiqing Jan 9, 2023

sjonpaulbrown Jan 9, 2023

sjonpaulbrown Jan 9, 2023 •

edited

Loading

peterargue Jan 10, 2023 •

edited

Loading

sjonpaulbrown Jan 11, 2023

sjonpaulbrown Jan 9, 2023 •

edited

Loading

peterargue Jan 10, 2023

peterargue commented Jan 11, 2023

bors bot commented Jan 11, 2023

[CI] Add github action to build boot-tools on demand #3787

[CI] Add github action to build boot-tools on demand #3787

Conversation

peterargue commented Jan 7, 2023

codecov-commenter commented Jan 7, 2023 • edited Loading

Codecov Report

zhangchiqing Jan 9, 2023

Choose a reason for hiding this comment

zhangchiqing Jan 9, 2023

Choose a reason for hiding this comment

sjonpaulbrown Jan 9, 2023

Choose a reason for hiding this comment

sjonpaulbrown Jan 9, 2023 • edited Loading

Choose a reason for hiding this comment

peterargue Jan 10, 2023 • edited Loading

Choose a reason for hiding this comment

sjonpaulbrown Jan 11, 2023

Choose a reason for hiding this comment

sjonpaulbrown Jan 9, 2023 • edited Loading

Choose a reason for hiding this comment

peterargue Jan 10, 2023

Choose a reason for hiding this comment

peterargue commented Jan 11, 2023

bors bot commented Jan 11, 2023

codecov-commenter commented Jan 7, 2023 •

edited

Loading

sjonpaulbrown Jan 9, 2023 •

edited

Loading

peterargue Jan 10, 2023 •

edited

Loading

sjonpaulbrown Jan 9, 2023 •

edited

Loading