Pipeline level sidecar

[bobcatfish]

Feature request

In addition to letting Tasks specify sidecars https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#specifying-sidecars, which are containers that run alongside a Tasks's steps and the lifecycle of which is managed by the Tekton controller, we allow Pipelines to specify sidecars as well.

We could implement these as pods, which the Tekton controller starts when execution of the Pipeline starts and stops when the Pipeline completes (probably after the finally Tasks? not sure). It probably doesn't make sense to specify them as Tasks b/c Tasks are designed to run to completion and this would run as long as a Pipeline runs.

If the pod stopped with an error, we'd need to decide if we want to restart it or fail the Pipeline (maybe make that configurable?).

Use case

In tektoncd/catalog#408 I'm creating Tasks to lease and return Boskos resources. A regular heartbeat needs to be sent to Boskos while the resource is being used; if Boskos doesn't get a heartbeat it will assume the resource is no longer in use, clean it up and return it to its pool.

I've implemented this by having the Task that leases the resource create a pod and the Task the returns the resource stops the pod.

Other use cases

A few more use cases were mentioned in the API working group:

• Using “sidecar” to update github
• Use cases: external dependencies or dependency stubs during integration testing (run a DB, messaging platform or else)
• Use case: (pipeline-run-local) docker registry with pipeline run scope

[bobcatfish]

• @chhsia0

[vdemeester]

As of today, this can be achieve by using a Task — that would do the deploy in the running cluster, … and clean-up in a finally task. There might some rbac issues as this means the service account running the task needs to have some rights on the cluster.
Overall it seems "fine" to me but I have the same though as with #2925 — how required it is ? how useful it is compared to the "potential" complexity it adds to the API ? (not saying it is complex, just that it adds another feature, etc…)

Using “sidecar” to update github

I feel this "should" be handled by integration like commit-status-tracker.

Use cases: external dependencies or dependency stubs during integration testing (run a DB, messaging platform or else)

This is indeed a use case that we have / will have with the hub (cc @sthaha).

Use case: (pipeline-run-local) docker registry with pipeline run scope

I am not sure I got this one. The idea would be to make sure a docker registry runs when the pipeline runs ? (so same as the registry sidecar we have in the catalog repository (cc @chmouel).

[bobcatfish]

• @skaegi

[dlorenc]

Another crazy idea: make pipeline-level heartbeats a first-class thing - maybe through cloudevents.

[skaegi]

re: docker / registry sidecar Use Case...

One thing we see frequently is in response to a git commit, a single pipeline is run that splits into into separate Tasks used to both build sub-component images and sanity/unit test them concurrently. A later join Task decides if everything is in order and if so finally tags and pushes the images to a final pre-product image registry.

We see this done with a Task-level sidecar docker daemon and a remote image registry. It might be good to have one dedicated Sidecar/Task to run a docker daemon the other Tasks share and use a pipeline scope image registry until ready to do the final push.

--

[chmouel]

I think a pipeline level sidecar object make sense for those use cases.

Ideally it would be if we are able to specify a service attached to it so other tasks/pods can access it easily.
This is in context of the DB or docker registry use case.

[dlorenc]

If we go ahead with this, some questions to think through:

• Do we run this as a pod? Or a replicaset for restarts/scaling?
• Do we wait for it to be "ready" before the pipeline starts? How do we know when it's "ready"?
• How do we make it accessible to the pipelinerun? And hopefully only that pipelinerun.
• How much customization do we allow on the created resources? Full Replicaset/service templates?

[imjasonh]

These are useful questions.

I think the level of customization required might make it easier to just recommend that people start a Service at the beginning of a Pipeline and tear it down in a finally.

[dlorenc]

Yeah I think the only downside to that pattern is then that the Tasks at the start and cleanup have to run with enough RBAC to create the required k8s objects...

[bobcatfish]

I wonder if the feature that would work would be less about being a "sidecar" and more about tying the lifecycle of to a Pipeline - e.g. a kubernetes resource you "create" at the beginning of a pipeline and "delete" at the end?

[dlorenc]

That makes sense - stuff using "generateName" could be easily kubectl created repeatedly. We'd need to think of a way to plumb the address/info of that object into the pipeline though. Maybe just make the gvk and name/namespace available via params?

[coryrc]

For the boskos leasing, Pipeline-level sidecar is a bit too wide... could we define a sidecar for a subset of Tasks? Effectively, once all of N listed Tasks are complete, Task X should be killed.

A simpler primitive is to allow an anti-dependency. Today if Task 2 requires input from Task 1, it doesn't start executing until Task 1 is complete. Anti-dependency would be if Task 2 has anti-dependency on Task 1, it would start normally but be killed when Task 1 is complete.

For Boskos heartbeat, there'd be 3 Tasks: acquire, heartbeat, and release
heartbeat depends on acquire
heartbeat anti-depends on release

Once acquire is complete, heartbeat would start. Once release completes, heartbeat is terminated.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[vdemeester]

/remove-lifecycle stale

[bobcatfish]

I feel like we haven't had much interest in this one so I'd be happy to close it and re-open it if needed some day in the future @vdemeester

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.