New cd images (ADX support and ARM64) #5254

tarakby · 2024-01-19T13:22:45Z

Update the node software process after the latest crypto module changes:

the crypto module has non-portable code that requires proper cross compilation when building for a different target
the crypto module uses CPU's ADX instructions. If the target CPU does not support ADX, such code should be disabled.

In this PR:

fix a bug in a previous PR (commit) where the empty netgo tag was removed. This fixes the build without netgo.
rename the build that uses the local script arch and ADX detection to "native build". These builds assume the image is built and run on the same machine type (arch and cpu properties). This is the case of CI tests and integration tests for instance.
add new images to build production images with and without ADX code:
- with ADX code: can only run on a machine with ADX support, this is the default case, and it offers optimal performance.
- without ADX code: uses portable code and can run on all machines. The build requires adding a CGO flag, passed as an argument to Dockerfile. It offers slower performance.
add a new image for arm64 cross build. The build requires specifying the cross-compiler in CC. CC is passed to the Dockerfile as an argument.
add all new images build to cd.yml and add a step to install the cross-compilation tools (works only on Debian)
update builds.yml to use the new commands.
image names in Makefile use IMAGE_TAG only (which is derived from the short commits in most cases)

These are the images built in cd.yml (assumed to run on amd64):

with ADX code
with ADX code and without netgo.
without ADX code and without netgo.
arm64 images
( without ADX code and with netgo was skipped for now but can be added if needed. With the current PR, if we disable ADX, netgo has to be disabled too. This assumes that there are no machines that cannot run images with netgo disabled)
builds.yml can generate the same images but the arm64 ones, because of the limitation of number of input boxes in githhub workflows.

…hout adx

codecov-commenter · 2024-01-19T13:30:33Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (f0b7f2a) 55.61% compared to head (12d4b6c) 55.58%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5254      +/-   ##
==========================================
- Coverage   55.61%   55.58%   -0.03%     
==========================================
  Files        1002     1002              
  Lines       96600    96600              
==========================================
- Hits        53720    53695      -25     
- Misses      38820    38844      +24     
- Partials     4060     4061       +1

Flag	Coverage Δ
unittests	`55.58% <ø> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tarakby · 2024-01-19T23:08:16Z

.github/workflows/cd.yml

@@ -29,11 +29,17 @@ jobs:
    - name: Authenticate docker with gcloud
      run: |
        gcloud auth configure-docker
+    - name: Install Tools
+      run: make install-cross-build-tools


needed for the arm cross-build

.github/workflows/ci.yml

tarakby · 2024-01-19T23:18:36Z

Makefile

+	apt-get -y install apt-utils gcc-aarch64-linux-gnu
+
+.PHONY: docker-native-build-collection
+docker-native-build-collection:


native builds detect whether the current machine supports ADX using crypto_adx_flag.mk

tarakby · 2024-01-19T23:21:40Z

Makefile

-docker-build-collection-without-netgo:
-	docker build -f cmd/Dockerfile --build-arg TARGET=./cmd/collection --build-arg COMMIT=$(COMMIT)  --build-arg VERSION=$(IMAGE_TAG_NO_NETGO) --build-arg GOARCH=$(GOARCH) --build-arg CGO_FLAG=$(CRYPTO_FLAG) --target production \
+.PHONY: docker-build-collection-with-adx
+docker-build-collection-with-adx:


Optimal case (with ADX and with netgo) - this is called in CD.

Note that the image has the same "tag" as the native builds. Native builds (locally or CI) are not run in CD.

so the difference between this an docker-native-build-collection is that native allows overriding ADX while this doesn't?

This target hardcodes the arguments for a specific target (ideally I would want to hardcode GOARCH too, but left it in case we want to call it on ARM machines too).

docker-native-build-... set the arguments based on the host (detect ADX support, set ARCH)

For better clarity and consistency, I hardcoded amd64 for production targets that are not natively built and not meant for arm d3b99bd (though I can revert it if needed)

Makefile

peterargue · 2024-01-22T14:14:38Z

Makefile

-docker-build-collection-without-netgo:
-	docker build -f cmd/Dockerfile --build-arg TARGET=./cmd/collection --build-arg COMMIT=$(COMMIT)  --build-arg VERSION=$(IMAGE_TAG_NO_NETGO) --build-arg GOARCH=$(GOARCH) --build-arg CGO_FLAG=$(CRYPTO_FLAG) --target production \
+.PHONY: docker-build-collection-with-adx
+docker-build-collection-with-adx:


so the difference between this an docker-native-build-collection is that native allows overriding ADX while this doesn't?

Makefile

sjonpaulbrown · 2024-01-22T19:49:50Z

@tarakby, using this branch, could you please run all of the workflows being modified so that we can verify the execution?

Also, as a general comment, I think we will need to update the following lines in the build workflow to use the new make targets

sjonpaulbrown · 2024-01-22T19:58:20Z

If the targets are changing, the following workflow will also need to be updated to align with the target changes.

This will need to be updated separately, but this will break BN2 until the workflow is updated.

Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>

This reverts commit d6e2c25.

.github/workflows/builds.yml

peterargue

Thanks for adding support for ARM builds and unravelling our complex mess of a makefile.

I'm a little confused by all make target names, but I think the changes are correct, so I don't want to hold this up this PR over naming.

We should circle back and refactor the build system soon. It's become pretty complex.

This reverts commit 4f6602d.

tarakby · 2024-01-24T01:17:30Z

@sjonpaulbrown, the PR is now ready.

cd.yml was tested in this workflow
builds.yml was tested in this workflow

tarakby · 2024-01-24T01:19:22Z

I'm a little confused by all make target names, but I think the changes are correct, so I don't want to hold this up this PR over naming.
We should circle back and refactor the build system soon. It's become pretty complex.

Agree, the build system could be made simpler.

sjonpaulbrown · 2024-01-25T15:18:17Z

Makefile

@@ -19,7 +19,9 @@ ifeq (${IMAGE_TAG},)
 IMAGE_TAG := ${SHORT_COMMIT}
 endif

-IMAGE_TAG_NO_NETGO := $(IMAGE_TAG)-without-netgo


Altering this tag name could impact partner automation. We will want to work with @vishalchangrani to ensure that the partners who rely on this know the image tag is changing.

tarakby added 3 commits January 19, 2024 12:08

remove CGO_FLAG var to simplify - add comments about native build

8fa38f2

rename targets made for native build

695db73

add non native builds with and without adx, without netgo is made wit…

264ad3f

…hout adx

tarakby added 6 commits January 19, 2024 18:58

fix bug

d77fed3

revert accidentally removed TAGS for without netgo images

666f9bb

bump optimization when building without adx

ba333ab

add arm build and push targets - add them to cd

d783ed6

rename -build-native- to -native-build- to match -cross-build

3a767a8

add install tools step in cdd

cc258a3

tarakby marked this pull request as ready for review January 19, 2024 22:49

tarakby changed the title ~~Tarak/new cd images~~ New cd images (ADX support and ARM64) Jan 19, 2024

tarakby requested review from peterargue, Kay-Zee and sjonpaulbrown January 19, 2024 22:50

tarakby commented Jan 19, 2024

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

tarakby commented Jan 19, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

fix redundant target

ee14067

peterargue reviewed Jan 22, 2024

View reviewed changes

update image name and extend verion tests

de0c98c

tarakby requested a review from ramtinms as a code owner January 22, 2024 17:29

peterargue reviewed Jan 22, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

tarakby and others added 3 commits January 22, 2024 21:20

limit target to Debian

0324c18

Update Makefile

62542c7

Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>

add new images to builds.yml

5742af3

tarakby added 3 commits January 23, 2024 13:23

add env cadence deploy key in CD

d6e2c25

Revert "add env cadence deploy key in CD"

b050309

This reverts commit d6e2c25.

expand install-cross-build-tools to Linux

f47a25f

peterargue reviewed Jan 23, 2024

View reviewed changes

.github/workflows/builds.yml Outdated Show resolved Hide resolved

peterargue reviewed Jan 23, 2024

View reviewed changes

.github/workflows/builds.yml Outdated Show resolved Hide resolved

peterargue approved these changes Jan 23, 2024

View reviewed changes

use root for apt-get

03f8c61

tarakby force-pushed the tarak/new-cd-images branch from 4d163bc to 03f8c61 Compare January 23, 2024 18:24

move cross-compiler install inside docker image

ec8c7ae

tarakby requested a review from AlexHentschel as a code owner January 23, 2024 19:12

fix arm64 arch instead of arm

8d1ad22

tarakby force-pushed the tarak/new-cd-images branch from 242d37e to 8d1ad22 Compare January 23, 2024 19:37

tarakby added 5 commits January 23, 2024 20:47

make amd64 with adx the default build - add arm64 build

f1e015e

merge all alternative builds

a395109

Revert "tmp: cd runs on pushes"

cf672b3

This reverts commit 4f6602d.

fix quotes in yml

46f58cb

update make call

ee83f78

tarakby force-pushed the tarak/new-cd-images branch from 3aed998 to ee83f78 Compare January 24, 2024 01:04

sjonpaulbrown approved these changes Jan 25, 2024

View reviewed changes

Merge branch 'master' into tarak/new-cd-images

12d4b6c

tarakby enabled auto-merge January 25, 2024 17:47

tarakby added this pull request to the merge queue Jan 25, 2024

Merged via the queue into master with commit 5006941 Jan 25, 2024
50 of 51 checks passed

tarakby deleted the tarak/new-cd-images branch January 25, 2024 18:20

tarakby restored the tarak/new-cd-images branch February 7, 2024 14:42

tarakby mentioned this pull request Feb 7, 2024

Backport PR 5254 #5359

Merged

2 tasks

tarakby deleted the tarak/new-cd-images branch April 5, 2024 20:51

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New cd images (ADX support and ARM64) #5254

New cd images (ADX support and ARM64) #5254

tarakby commented Jan 19, 2024 •

edited

Loading

codecov-commenter commented Jan 19, 2024 •

edited

Loading

tarakby Jan 19, 2024

tarakby Jan 19, 2024 •

edited

Loading

tarakby Jan 19, 2024

peterargue Jan 22, 2024

tarakby Jan 22, 2024

tarakby Jan 22, 2024

peterargue Jan 22, 2024

sjonpaulbrown commented Jan 22, 2024

sjonpaulbrown commented Jan 22, 2024

peterargue left a comment

tarakby commented Jan 24, 2024

tarakby commented Jan 24, 2024

sjonpaulbrown Jan 25, 2024

New cd images (ADX support and ARM64) #5254

New cd images (ADX support and ARM64) #5254

Conversation

tarakby commented Jan 19, 2024 • edited Loading

codecov-commenter commented Jan 19, 2024 • edited Loading

Codecov Report

tarakby Jan 19, 2024

Choose a reason for hiding this comment

tarakby Jan 19, 2024 • edited Loading

Choose a reason for hiding this comment

tarakby Jan 19, 2024

Choose a reason for hiding this comment

peterargue Jan 22, 2024

Choose a reason for hiding this comment

tarakby Jan 22, 2024

Choose a reason for hiding this comment

tarakby Jan 22, 2024

Choose a reason for hiding this comment

peterargue Jan 22, 2024

Choose a reason for hiding this comment

sjonpaulbrown commented Jan 22, 2024

sjonpaulbrown commented Jan 22, 2024

peterargue left a comment

Choose a reason for hiding this comment

tarakby commented Jan 24, 2024

tarakby commented Jan 24, 2024

sjonpaulbrown Jan 25, 2024

Choose a reason for hiding this comment

tarakby commented Jan 19, 2024 •

edited

Loading

codecov-commenter commented Jan 19, 2024 •

edited

Loading

tarakby Jan 19, 2024 •

edited

Loading