chore(ci): use unified Gitlab pipeline for APM libraries #9930

randomanderson · 2024-07-24T21:26:55Z

What does this PR do?
Converts the Gitlab workflow to use the unified pipeline.

By using the unified pipeline across all APM libraries, there is less maintenance hassle, fewer mistakes, and a more uniform creation of artifacts.

Additionally, this change creates OCI packages and lib inject images for every dd-trace-py build. Previously, only released versions to pypi were built and tested.

The changes can be summarized as follows:

Removing github actions that are now taken over by the unified pipeline
Removing Gitlab jobs taken over by the unified pipeline
Adding a job to download artifacts from the github build
Merging the download artifacts with dependencies from pypi

.gitlab-ci.yml is best reviewed in its final form rather than as a diff.

How to test the change?
The changes were tested manually. CI/CD workflows are notoriously hard to test. Final products the final lib-init image was run through automated onboarding tests. dd-trace-py doesn't have automated tests for OCI images

Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

This reverts commit b1d3d41.

# Conflicts: # .gitlab-ci.yml

github-actions · 2024-07-24T21:27:22Z

CODEOWNERS have been resolved as:

.gitlab/download-dependency-wheels.sh                                   @DataDog/python-guild @DataDog/apm-core-python
.gitlab/download-wheels-from-gh-actions.sh                              @DataDog/python-guild @DataDog/apm-core-python
.gitlab/prepare-oci-package.sh                                          @DataDog/python-guild @DataDog/apm-core-python
.github/workflows/build_deploy.yml                                      @DataDog/python-guild @DataDog/apm-core-python
.gitlab-ci.yml                                                          @DataDog/apm-core-python
lib-injection/dl_wheels.py                                              @DataDog/apm-core-python

pr-commenter · 2024-07-24T22:33:52Z

Benchmarks

Benchmark execution time: 2024-07-25 16:09:51

Comparing candidate commit 0553e25 in PR branch landerson/one-pipeline with baseline commit 2de06b2 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 214 metrics, 2 unstable metrics.

brettlangdon

very minor, non-blocking, nits, but otherwise lgtm

thanks!

.gitlab/prepare-oci-package.sh

lib-injection/dl_wheels.py

brettlangdon · 2024-07-25T08:40:14Z

you'll need to run hatch run lint:fmt, seems lib-injection/dl_wheels.py is failing linting.

**What does this PR do?** Converts the Gitlab workflow to use the unified pipeline. By using the unified pipeline across all APM libraries, there is less maintenance hassle, fewer mistakes, and a more uniform creation of artifacts. Additionally, this change creates OCI packages and lib inject images for every `dd-trace-py` build. Previously, only released versions to pypi were built and tested. The changes can be summarized as follows: * Removing github actions that are now taken over by the unified pipeline * Removing Gitlab jobs taken over by the unified pipeline * Adding a job to download artifacts from the github build * Merging the download artifacts with dependencies from pypi `.gitlab-ci.yml` is best reviewed in its final form rather than as a diff. **How to test the change?** The changes were tested manually. CI/CD workflows are notoriously hard to test. Final products the final lib-init image was run through automated onboarding tests. `dd-trace-py` doesn't have automated tests for OCI images ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

**What does this PR do?** Converts the Gitlab workflow to use the unified pipeline. By using the unified pipeline across all APM libraries, there is less maintenance hassle, fewer mistakes, and a more uniform creation of artifacts. Additionally, this change creates OCI packages and lib inject images for every `dd-trace-py` build. Previously, only released versions to pypi were built and tested. The changes can be summarized as follows: * Removing github actions that are now taken over by the unified pipeline * Removing Gitlab jobs taken over by the unified pipeline * Adding a job to download artifacts from the github build * Merging the download artifacts with dependencies from pypi `.gitlab-ci.yml` is best reviewed in its final form rather than as a diff. **How to test the change?** The changes were tested manually. CI/CD workflows are notoriously hard to test. Final products the final lib-init image was run through automated onboarding tests. `dd-trace-py` doesn't have automated tests for OCI images - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

randomanderson added 30 commits June 27, 2024 12:25

First cut

17350b4

add github download step

a2e7456

remove hardcoded id

dfbf2f9

hardcode commit for testing

3fce84c

fix variant list

e126e96

remove +x to get debug output

04e6b09

use a different token

b31f2b5

fix path to wheels

3e4941d

assign python version from wheel

24927f8

move things to shared pipeline

28981c4

disable installer tests for python because there are no scenarios

dbf415b

remove hardcoded commit id

78f4a44

wait for artifacts to appear

39db334

build workflows on every push

13f360a

don't build lib inject image as part of workflow

815f23d

change intervals

a489c39

retry downloading wheels

b1d3d41

Revert "retry downloading wheels"

bb21d98

This reverts commit b1d3d41.

Merge remote-tracking branch 'origin/main' into landerson/one-pipeline

79ceffd

# Conflicts: # .gitlab-ci.yml

Add some logging

a5b7a18

create folder to store packages

39eeaa8

Download dependency wheels

eb76247

add arch

fa9fff6

c/p error

f1cbe05

fix path, hardcode commit sha for now

7785220

skip rust check for now

e9af621

ignore wheel redownloads

2603599

flush print statements explicitly

66aecd6

Use matrix to download dependencies

b1ad3ab

use new 3.8 mirror

b8deb38

brettlangdon approved these changes Jul 25, 2024

View reviewed changes

.gitlab/prepare-oci-package.sh Outdated Show resolved Hide resolved

lib-injection/dl_wheels.py Outdated Show resolved Hide resolved

brettlangdon changed the title ~~CI: Use unified Gitlab pipeline for APM libraries~~ chore(ci): use unified Gitlab pipeline for APM libraries Jul 25, 2024

brettlangdon changed the title ~~chore(ci): use unified Gitlab pipeline for APM libraries~~ chore(ci): use unified Gitlab pipeline for APM libraries Jul 25, 2024

formatting, PR feedback

0553e25

randomanderson enabled auto-merge (squash) July 25, 2024 15:36

gnufede approved these changes Jul 26, 2024

View reviewed changes

Merge branch 'main' into landerson/one-pipeline

57b3518

randomanderson merged commit 879a733 into main Jul 26, 2024
94 of 95 checks passed

randomanderson deleted the landerson/one-pipeline branch July 26, 2024 07:37

emmettbutler added backport 2.9 backport 2.10 backport 2.11 backport 2.12 labels Sep 5, 2024

emmettbutler restored the landerson/one-pipeline branch September 5, 2024 15:00

emmettbutler added backport 2.9 backport 2.10 backport 2.11 backport 2.12 and removed backport 2.9 backport 2.10 backport 2.11 backport 2.12 labels Sep 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(ci): use unified Gitlab pipeline for APM libraries #9930

chore(ci): use unified Gitlab pipeline for APM libraries #9930

randomanderson commented Jul 24, 2024 •

edited by brettlangdon

Loading

github-actions bot commented Jul 24, 2024

pr-commenter bot commented Jul 24, 2024 •

edited

Loading

brettlangdon left a comment

brettlangdon commented Jul 25, 2024

chore(ci): use unified Gitlab pipeline for APM libraries #9930

chore(ci): use unified Gitlab pipeline for APM libraries #9930

Conversation

randomanderson commented Jul 24, 2024 • edited by brettlangdon Loading

Checklist

Reviewer Checklist

github-actions bot commented Jul 24, 2024

pr-commenter bot commented Jul 24, 2024 • edited Loading

Benchmarks

brettlangdon left a comment

Choose a reason for hiding this comment

brettlangdon commented Jul 25, 2024

randomanderson commented Jul 24, 2024 •

edited by brettlangdon

Loading

pr-commenter bot commented Jul 24, 2024 •

edited

Loading