Propose namespaced tasks #404

michaelsauter · 2022-01-14T08:11:07Z

This proposal aims to adress the following issues:

Sidecars are barely usable (Support sidecars #135)
Resource Consumption is not configurable from pipeline (cluster tasks do not define resource limits #372)

Let's try out a different format this time. Instead of discussing in an issue, this approach allows to discuss one specific proposal PR-style (allowing to comment inline etc.)

Looking forward to feedback of any kind!

[ci skip]

gerardcl

I like this approach, meanwhile we keep checking what is providing tekton in future.
I see more pros than cons 👍
There would be more ownership, which is good!

docs/proposals/local-tasks.adoc

[ci skip Co-authored-by: gerardcl <gerardcl@gmail.com>

oalyman · 2022-01-14T10:54:37Z

I am not exited tbh.
But as long as there is no support from the tekton side we don't have that much options of which this seems to be the best / most flexible one.
So I am OK with the approach in hope for a better solution out of tekton itself.

henrjk · 2022-01-14T15:01:58Z

I suppose if one adjusted the "user installation" so that one would have a subtree (or perhaps submodule) of the entire ods-pipeline repo at <project-cd>/ods-pipeline one could then install the two helm charts of what is currently in deploy/cd-namespace and deploy/cd-tasks which would be derived from deploy/central.
Now you have all that is needed to also test pipeline changes locally. I suppose it might also be possible to contribute changes back.
Feels a bit weird so, or?

michaelsauter · 2022-01-17T11:05:12Z

Based on an off-screen discussion with @henrjk following his comment, I am inclined to update my proposal with @henrjk's suggestion. In effect, he proposes to go one step further and abolish a central installation altogether. This has the following advantages:

simpler to understand as there is no admin/user distinction
simpler to install as no collaboration with a cluster admin is required at all
images are then controlled by users, making them easier to modify

Disadvantage is obviously that every project needs to build images themselves. However the images build quickly and reliably at the moment. We could also further explore to get rid of building images in the cluster at all, and simply provide images from a central registry like Docker Hub. Technically, there are only two blockers right now: one is the baked in sonar edition (easily solved, see #410) and the other is the aqua scanner. This one could be handled either as a special case or we could start downloading the scanner in the pipeline run (potentially caching it in the PVC to avoid re-download).

@kuebler @gerardcl @oalyman since you already approved, I would like to hear your thoughts on expanding the scope of the PR even further

gerardcl · 2022-01-17T12:11:19Z

I also see the point and agree, I like it. Things to take into consideration:

is this still ensuring a compliance/controlled installation from an enterprise point of view? I mean, is the option of making the installation central still valid/possible, so a team in a company can provide support in a scalable way?
I believe then the other services we rely on are still in a central installation or we consider then them as just service dependencies one can decice where to offer them from? (aka nexus, sonarqube,..)

michaelsauter · 2022-01-17T14:22:12Z

is this still ensuring a compliance/controlled installation from an enterprise point of view?

Validation of tooling can (and I believe should) happen at the project level (qualifying a specific use case vs. a generic one).

I mean, is the option of making the installation central still valid/possible, so a team in a company can provide support in a scalable way?

The proposal right now removes this option. We could think about making it optional though.

I believe then the other services we rely on are still in a central installation or we consider then them as just service dependencies one can decice where to offer them from?

Correct, they are part of ODS core. The proposal does not affect how we see them.

kuebler · 2022-01-17T15:37:55Z

Really like the extended scope, in particular w.r.t. Docker Hub, we need to consider their rate limiting, though.

michaelsauter · 2022-01-17T16:23:39Z

in particular w.r.t. Docker Hub, we need to consider their rate limiting, though.

Two ideas come to mind:

GitHub also provides a registry we could use as an alternative.
OpenShift image streams can pull through from Docker Hub. I am not sure how that affects rate limiting but it could help.

docs/proposals/local-tasks.adoc

oalyman · 2022-01-18T10:45:30Z

@michaelsauter I like it. Especially that it puts some pressure on the topic to pre build images as I already proposed.

[ci skip] Co-authored-by: Vincent Elcrin <vincent.elcrin@gmail.com>

michaelsauter · 2022-01-24T08:13:50Z

So I've put some more thought into this and want to suggest the following now.

I'm going to expand the scope to remove the admin installation completely. For the images, I entertained a few options but have now settled on an approach that we have already tried and tested with ODS core, and I believe is the best we can do for now.

I'll call the approach the "wrapper technique". The idea is simple: built an image in GitHub, but wrap this image in a BuildConfig/ImageStream in OpenShift. For example, say we build the ods-sonar image in GitHub and push it to the Docker Hub. Further, we have an ods-sonar ImageStream in each project. That image stream is filled from a BuildConfig, which defines an inline Dockerfile. For ods-sonar, that is simply FROM index.docker.io/opendevstackorg/ods-sonar.

While the wrapper images comes with some conceptual overhead, the advantages easily outweigh this in my opinion:

it allows to inject cluster env variables. In my comment above (Propose namespaced tasks #404 (comment)) I missed that we also need to inject proxy env variables (HTTP_PROXY and friends). We could also do this differently, but it is quite convenient to have that baked into the image.
it makes it very trivial to adjust the dockerfile for users, allowing them to install more packages for example if they need to.
it allows to download the (password protected) aqua scanner in the ods-buildah image
it ensures we do not run into pull limits imposed by Docker Hub or GitHub registry
it removes external dependencies: once the installation is complete, all images are present locally and do not depend on an external registry to be present
pulling images in pipeline runs should be quicker compared to a situation where pulling involves remote registries
for regulated use cases, it is good to know that the project-local images are under our control and cannot change after installation unless explicitly triggered (as opposed to images pulled from Docker Hub which could change without notice)
it allows a completely project-local installation without requiring people to build the same image over and over again.

@kuebler @gerardcl @oalyman @henrjk Would love to get one more round of feedback :)

gerardcl · 2022-01-24T09:26:02Z

hi! makes sense to me! but! how and where is the deployment of, for example, ods-sonar?
I still see the need of:
1- having a central installation of ods services or clear options defined (I see three indeed: central, project relative, cluster external)
2- having project ownership of tasks but I would avoid having to maintain ods services (aka nexus, sonar) at project level

michaelsauter · 2022-01-24T09:35:33Z

where is the deployment of, for example, ods-sonar?

I assume you talk about a SonarQube server? This, along with other services, is provided by a central ODS core installation. Projects do not need to provide/maintain their own services (SQ, Nexus, Bitbucket, Aqua), although they could if they wanted to.

The interfaces of ODS Pipeline are described here: https://github.com/opendevstack/ods-pipeline/blob/master/docs/design/software-architecture.adoc#interfaces.

We could further clarify the exact requirements of this central ODS core installation but I am OK with the current state for now.

gerardcl · 2022-01-24T09:52:01Z

so then this will still exist https://github.com/opendevstack/ods-pipeline/tree/master/deploy/central, right? but only with the https://github.com/opendevstack/ods-pipeline/tree/master/deploy/central/images-chart bc related to central installation?

Makefile

deploy/ods-pipeline/Chart.yaml

deploy/ods-pipeline/charts/images/templates/bc-ods-buildah.yaml

…proposal # Conflicts: # deploy/central/tasks-chart/templates/task-ods-build-typescript-with-sidecar.yaml # docs/tasks/ods-build-typescript-with-sidecar.adoc

michaelsauter · 2022-03-10T13:38:13Z

docs/design/relationship-shared-library.adoc

-* Pipeline users cannot specify task resources. This was possible in Jenkins and also used by many users. See issue https://github.com/opendevstack/ods-pipeline/issues/195. Currently support is not even on the Tekton roadmap. Only workaround: multiple tasks or high defaults. If that does not work, users must create their own copies of the tasks.
-* Pipeline users cannot specify sidecars. This was possible in Jenkins and also used by many users (e.g. to spin up a database for testing). See issue https://github.com/opendevstack/ods-pipeline/issues/135. Currently support is not even on the Tekton roadmap. Only workaround: multiple tasks. If that does not work (e.g. you need more than one sidecar), users must create their own copies of the tasks.
+* Pipeline users cannot specify task resources dynamically without changing the task at hand. This was possible in Jenkins and also used by many users. See issue https://github.com/opendevstack/ods-pipeline/issues/195. Currently, support is not even on the Tekton roadmap. Only workaround: multiple tasks or high defaults.
+* Pipeline users cannot specify sidecars dynamically without changing the task at hand. This was possible in Jenkins and also used by many users (e.g. to spin up a database for testing). See issue https://github.com/opendevstack/ods-pipeline/issues/135. Currently, support is not even on the Tekton roadmap. Only workaround: multiple tasks.


Thanks for the updates to this file!

I think we can take out the first two limitations: specifying resources and sidecars is possible with this PR :)

We were thinking about removing those points as well, but after a small discussion decided to add that this is indeed possible but only on a project-wide scope by changing the task itself, not on a per-repo scope as it was possible with a Jenkinsfile.

But we are also happy to remove both of those bullets if you think that this is confusing.

Yes I think the new solution is good enough to be a solution for now. We should remove this as it isn't a "blocker" any longer.

I would love to have both pain points mentioned in the ADR though, and in the discussion we should link to the issues / Tekton TEPs.

henrjk

Awesome. Looks good to me.

I believe some of the task yml's will need updating when refreshed to the latest master. Update: This may only happen if some other PRs are merged first, but presumably git will make this also obvious.

deploy/ods-pipeline/charts/setup/values.yaml

gerardcl

this is awesome work here! LGTM! 🚀

docs/design/relationship-shared-library.adoc

gerardcl · 2022-03-10T18:50:10Z

docs/design/relationship-shared-library.adoc


 Users will still be able to consume quickstarters via the provisioning app.

-Authoring a new quickstarter means potentially creating a new task (say a Rust task). This task would then need to be installed centrally as a ClusterTask like the others. I do not know yet how a less official task could be shared easily ... each namespace would need to install that task separately.
+Authoring a new quickstarter means potentially creating a new task (say a Rust task). We do not know yet how a less official task could be shared easily ... each namespace would need to install that task separately.


michaelsauter

Really awesome!

Thanks also for all the fixes of my typos etc 👍

I made a few smaller comments but then we are good to merge!

README.md

deploy/ods-pipeline/values.kind.yaml

deploy/ods-pipeline/values.yaml

docs/adr/20220308-namespaced-installation.md

docs/authoring-tasks.adoc

docs/design/software-design-specification.adoc

michaelsauter

👍

I'd approve but since you continued in "my" PR I can't approve it now ;)

Propose local tasks

79680e8

[ci skip]

michaelsauter requested review from oalyman, kuebler, gerardcl, henrjk and henninggross January 14, 2022 08:11

michaelsauter self-assigned this Jan 14, 2022

michaelsauter mentioned this pull request Jan 14, 2022

Using sidecars is very limited #405

Closed

michaelsauter modified the milestone: 0.3.0 Jan 14, 2022

gerardcl approved these changes Jan 14, 2022

View reviewed changes

docs/proposals/local-tasks.adoc Outdated Show resolved Hide resolved

Update docs/proposals/local-tasks.adoc

a4c7767

[ci skip Co-authored-by: gerardcl <gerardcl@gmail.com>

oalyman approved these changes Jan 14, 2022

View reviewed changes

kuebler approved these changes Jan 17, 2022

View reviewed changes

velcrin reviewed Jan 18, 2022

View reviewed changes

docs/proposals/local-tasks.adoc Outdated Show resolved Hide resolved

docs/proposals/local-tasks.adoc Outdated Show resolved Hide resolved

docs/proposals/local-tasks.adoc Outdated Show resolved Hide resolved

michaelsauter and others added 2 commits January 18, 2022 13:21

Update docs/proposals/local-tasks.adoc

fc20384

[ci skip] Co-authored-by: Vincent Elcrin <vincent.elcrin@gmail.com>

Update docs/proposals/local-tasks.adoc

3ae2183

[ci skip] Co-authored-by: Vincent Elcrin <vincent.elcrin@gmail.com>

oalyman approved these changes Jan 18, 2022

View reviewed changes

kuebler added 2 commits March 10, 2022 08:19

Associate secret values with correct subcharts

0891018

Enable images chart by default

50a3a11

michaelsauter commented Mar 10, 2022

View reviewed changes

Makefile Outdated Show resolved Hide resolved

michaelsauter commented Mar 10, 2022

View reviewed changes

deploy/ods-pipeline/Chart.yaml Show resolved Hide resolved

michaelsauter commented Mar 10, 2022

View reviewed changes

deploy/ods-pipeline/charts/images/templates/bc-ods-buildah.yaml Show resolved Hide resolved

kuebler added 2 commits March 10, 2022 14:32

Adjust documentation

7ecae6d

Merge remote-tracking branch 'origin/master' into task/task-location-…

5e353aa

…proposal # Conflicts: # deploy/central/tasks-chart/templates/task-ods-build-typescript-with-sidecar.yaml # docs/tasks/ods-build-typescript-with-sidecar.adoc

michaelsauter commented Mar 10, 2022

View reviewed changes

Add change to CHANGELOG

1bb6ecc

henrjk approved these changes Mar 10, 2022

View reviewed changes

deploy/ods-pipeline/charts/setup/values.yaml Outdated Show resolved Hide resolved

gerardcl approved these changes Mar 10, 2022

View reviewed changes

michaelsauter commented Mar 11, 2022

View reviewed changes

kuebler added 10 commits March 14, 2022 11:16

Fix type in comment

d73faac

Remove user guide heading and move contributer's guide

8f5ca3e

Fix indentation

f0dca6f

Mention centrally provided images in ADR

43e7b9f

Remove taskRef.kind hint from task authoring guide

e2faa42

Fix version of node UBI image

553e02b

Refer to PR from ADR

129a726

Remove unused options in values.yaml

c582005

Fix typo in sidecar example

f621f43

Add hint how to use public images in kind cluster

c91067d

michaelsauter commented Mar 14, 2022

View reviewed changes

michaelsauter merged commit e972e7f into master Mar 14, 2022

michaelsauter deleted the task/task-location-proposal branch March 14, 2022 15:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Propose namespaced tasks #404

Propose namespaced tasks #404

michaelsauter commented Jan 14, 2022

gerardcl left a comment

oalyman commented Jan 14, 2022

henrjk commented Jan 14, 2022

michaelsauter commented Jan 17, 2022

gerardcl commented Jan 17, 2022

michaelsauter commented Jan 17, 2022 •

edited

Loading

kuebler commented Jan 17, 2022

michaelsauter commented Jan 17, 2022

oalyman commented Jan 18, 2022

michaelsauter commented Jan 24, 2022 •

edited

Loading

gerardcl commented Jan 24, 2022

michaelsauter commented Jan 24, 2022 •

edited

Loading

gerardcl commented Jan 24, 2022

michaelsauter Mar 10, 2022

henninggross Mar 10, 2022

michaelsauter Mar 10, 2022

henrjk left a comment •

edited

Loading

gerardcl left a comment

gerardcl Mar 10, 2022

michaelsauter left a comment

michaelsauter left a comment

Propose namespaced tasks #404

Propose namespaced tasks #404

Conversation

michaelsauter commented Jan 14, 2022

gerardcl left a comment

Choose a reason for hiding this comment

oalyman commented Jan 14, 2022

henrjk commented Jan 14, 2022

michaelsauter commented Jan 17, 2022

gerardcl commented Jan 17, 2022

michaelsauter commented Jan 17, 2022 • edited Loading

kuebler commented Jan 17, 2022

michaelsauter commented Jan 17, 2022

oalyman commented Jan 18, 2022

michaelsauter commented Jan 24, 2022 • edited Loading

gerardcl commented Jan 24, 2022

michaelsauter commented Jan 24, 2022 • edited Loading

gerardcl commented Jan 24, 2022

michaelsauter Mar 10, 2022

Choose a reason for hiding this comment

henninggross Mar 10, 2022

Choose a reason for hiding this comment

michaelsauter Mar 10, 2022

Choose a reason for hiding this comment

henrjk left a comment • edited Loading

Choose a reason for hiding this comment

gerardcl left a comment

Choose a reason for hiding this comment

gerardcl Mar 10, 2022

Choose a reason for hiding this comment

michaelsauter left a comment

Choose a reason for hiding this comment

michaelsauter left a comment

Choose a reason for hiding this comment

michaelsauter commented Jan 17, 2022 •

edited

Loading

michaelsauter commented Jan 24, 2022 •

edited

Loading

michaelsauter commented Jan 24, 2022 •

edited

Loading

henrjk left a comment •

edited

Loading