Skip to content

Commit

Permalink
WhenExpressions in Finally Tasks
Browse files Browse the repository at this point in the history 
Users can guard execution of `Tasks` using `WhenExpressions`, but that
is currently not supported in `Finally Tasks`.

This change adds support for `WhenExpressions` in `Finally Tasks` not
only to provide efficient guarded execution but also to improve the
reusability of `Tasks` in `Finally`. The proposal is described further in
the [WhenExpressions in Finally Tasks TEP](https://github.com/tektoncd/community/blob/master/teps/0045-whenexpressions-in-finally-tasks.md).

Given we've recently added support for `Results` and `Status` in
`Finally Tasks`, this is an opportune time to enable `WhenExpressions`
in `Finally Tasks`.
  • Loading branch information
@jerop
jerop committed Jan 31, 2021
1 parent c8c80ed commit 13d88a2
Show file tree
Hide file tree
Showing 9 changed files with 408 additions and 85 deletions.
147 changes: 138 additions & 9 deletions docs/pipelines.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,27 +6,47 @@ weight: 3
-->
# Pipelines

- [Overview](#pipelines)
- [Configuring a `Pipeline`](#configuring-a-pipeline)
- [Pipelines](#pipelines)
- [Overview](#overview)
- [Configuring a `Pipeline`](#configuring-a-pipeline)
- [Specifying `Resources`](#specifying-resources)
- [Specifying `Workspaces`](#specifying-workspaces)
- [Specifying `Parameters`](#specifying-parameters)
- [Adding `Tasks` to the `Pipeline`](#adding-tasks-to-the-pipeline)
- [Using Tekton Bundles](#tekton-bundles)
- [Tekton Bundles](#tekton-bundles)
- [Using the `from` parameter](#using-the-from-parameter)
- [Using the `runAfter` parameter](#using-the-runafter-parameter)
- [Using the `retries` parameter](#using-the-retries-parameter)
- [Guard `Task` execution using `When Expressions`](#guard-task-execution-using-whenexpressions)
- [Guard `Task` execution using `WhenExpressions`](#guard-task-execution-using-whenexpressions)
- [Guard `Task` execution using `Conditions`](#guard-task-execution-using-conditions)
- [Configuring the failure timeout](#configuring-the-failure-timeout)
- [Using variable substitution](#using-variable-substitution)
- [Using `Results`](#using-results)
- [Passing one Task's `Results` into the `Parameters` of another](#passing-one-tasks-results-into-the-parameters-of-another)
- [Passing one Task's `Results` into the `Parameters` or `WhenExpressions` of another](#passing-one-tasks-results-into-the-parameters-or-whenexpressions-of-another)
- [Emitting `Results` from a `Pipeline`](#emitting-results-from-a-pipeline)
- [Configuring the `Task` execution order](#configuring-the-task-execution-order)
- [Adding a description](#adding-a-description)
- [Adding `Finally` to the `Pipeline`](#adding-finally-to-the-pipeline)
- [Specifying `Workspaces` in Final Tasks](#specifying-workspaces-in-final-tasks)
- [Specifying `Parameters` in Final Tasks](#specifying-parameters-in-final-tasks)
- [Consuming `Task` execution results in `finally`](#consuming-task-execution-results-in-finally)
- [`PipelineRun` Status with `finally`](#pipelinerun-status-with-finally)
- [Using Execution `Status` of `pipelineTask`](#using-execution-status-of-pipelinetask)
- [Guard `Finally Task` execution using `WhenExpressions`](#guard-finally-task-execution-using-whenexpressions)
- [`WhenExpressions` using `Parameters` in `Finally Tasks`](#whenexpressions-using-parameters-in-finally-tasks)
- [`WhenExpressions` using `Results` in `Finally Tasks`](#whenexpressions-using-results-in-finally-tasks)
- [`WhenExpressions` using `Execution Status` of `PipelineTask` in `Finally Tasks`](#whenexpressions-using-execution-status-of-pipelinetask-in-finally-tasks)
- [Known Limitations](#known-limitations)
- [Specifying `Resources` in Final Tasks](#specifying-resources-in-final-tasks)
- [Cannot configure the Final Task execution order](#cannot-configure-the-final-task-execution-order)
- [Cannot specify execution `Conditions` in Final Tasks](#cannot-specify-execution-conditions-in-final-tasks)
- [Cannot configure `Pipeline` result with `finally`](#cannot-configure-pipeline-result-with-finally)
- [Using Custom Tasks](#using-custom-tasks)
- [Specifying the target Custom Task](#specifying-the-target-custom-task)
- [Specifying parameters](#specifying-parameters-1)
- [Specifying workspaces](#specifying-workspaces-1)
- [Using `Results`](#using-results-1)
- [Limitations](#limitations)
- [Code examples](#code-examples)

## Overview
Expand Down Expand Up @@ -788,7 +808,7 @@ spec:
value: "someURL"
```

#### Consuming `Task` execution results in `finally`
### Consuming `Task` execution results in `finally`

Final tasks can be configured to consume `Results` of `PipelineTask` from the `tasks` section:

Expand Down Expand Up @@ -880,10 +900,119 @@ This kind of variable can have any one of the values from the following table:

For an end-to-end example, see [`status` in a `PipelineRun`](../examples/v1beta1/pipelineruns/pipelinerun-task-execution-status.yaml).

### Guard `Finally Task` execution using `WhenExpressions`

Similar to `Tasks`, `Finally Tasks` can be guarded using [`WhenExpressions`](#guard-task-execution-using-whenexpressions)
that operate on static inputs or variables. Like in `Tasks`, `WhenExpressions` in `Finally Tasks` can operate on
`Parameters` and `Results`. Unlike in `Tasks`, `WhenExpressions` in `Finally Tasks` can also operate on the [`Execution
Status`](#using-execution-status-of-pipelinetask) of `Tasks`.

#### `WhenExpressions` using `Parameters` in `Finally Tasks`

`WhenExpressions` in `Finally Tasks` can utilize `Parameters` as demonstrated using [`golang-build`](https://github.com/tektoncd/catalog/tree/master/task/golang-build/0.1)
and [`send-to-channel-slack`](https://github.com/tektoncd/catalog/tree/master/task/send-to-channel-slack/0.1) Catalog
`Tasks`:

```yaml
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: pipelinerun-
spec:
pipelineSpec:
params:
- name: enable-notifications
type: string
description: a boolean indicating whether the notifications should be sent
tasks:
- name: golang-build
taskRef:
name: golang-build
# […]
finally:
- name: notify-build-failure # executed only when build task fails and notifications are enabled
when:
- input: $(tasks.golang-build.status)
operator: in
values: ["Failed"]
- input: $(params.enable-notifications)
operator: in
values: ["true"]
taskRef:
name: send-to-slack-channel
# […]
params:
- name: enable-notifications
value: true
```

#### `WhenExpressions` using `Results` in `Finally Tasks`

`WhenExpressions` in `Finally Tasks` can utilize `Results`, as demonstrated using [`boskos-acquire`](https://github.com/tektoncd/catalog/tree/master/task/boskos-acquire/0.1)
and [`boskos-release`](https://github.com/tektoncd/catalog/tree/master/task/boskos-release/0.1) Catalog `Tasks`:

```yaml
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: pipelinerun-
spec:
pipelineSpec:
tasks:
- name: boskos-acquire
taskRef:
name: boskos-acquire
- name: use-resource
# […]
finally:
- name: boskos-release # executed only when leased resource is phonetic-project
when:
- input: $(tasks.boskos-acquire.results.leased-resource)
operator: in
values: ["phonetic-project"]
taskRef:
name: boskos-release
# […]
```

If the `WhenExpressions` in a `Finally Task` use `Results` from a skipped or failed non-finally `Tasks`, then the
`Finally Task` would also be skipped and be included in the list of `Skipped Tasks` in the `Status`, [similarly to when using
`Results` in other parts of the `Finally Task`](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#consuming-task-execution-results-in-finally).

#### `WhenExpressions` using `Execution Status` of `PipelineTask` in `Finally Tasks`

`WhenExpressions` in `Finally Tasks` can utilize [`Execution Status` of `PipelineTasks`](#using-execution-status-of-pipelinetask),
as as demonstrated using [`golang-build`](https://github.com/tektoncd/catalog/tree/master/task/golang-build/0.1) and
[`send-to-channel-slack`](https://github.com/tektoncd/catalog/tree/master/task/send-to-channel-slack/0.1) Catalog `Tasks`:

```yaml
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: pipelinerun-
spec:
pipelineSpec:
tasks:
- name: golang-build
taskRef:
name: golang-build
# […]
finally:
- name: notify-build-failure # executed only when build task fails
when:
- input: $(tasks.golang-build.status)
operator: in
values: ["Failed"]
taskRef:
name: send-to-slack-channel
# […]
```

For an end-to-end example, see [PipelineRun with WhenExpressions](../examples/v1beta1/pipelineruns/pipelinerun-with-when-expressions.yaml).

### Known Limitations

### Specifying `Resources` in Final Tasks
#### Specifying `Resources` in Final Tasks

Similar to `tasks`, you can use [PipelineResources](#specifying-resources) as inputs and outputs for
final tasks in the Pipeline. The only difference here is, final tasks with an input resource can not have a `from` clause
Expand Down Expand Up @@ -914,13 +1043,13 @@ spec:
- tests
```

### Cannot configure the Final Task execution order
#### Cannot configure the Final Task execution order

It's not possible to configure or modify the execution order of the final tasks. Unlike `Tasks` in a `Pipeline`,
all final tasks run simultaneously and start executing once all `PipelineTasks` under `tasks` have settled which means
no `runAfter` can be specified in final tasks.

### Cannot specify execution `Conditions` in Final Tasks
#### Cannot specify execution `Conditions` in Final Tasks

`Tasks` in a `Pipeline` can be configured to run only if some conditions are satisfied using `conditions`. But the
final tasks are guaranteed to be executed after all `PipelineTasks` therefore no `conditions` can be specified in
Expand Down
42 changes: 41 additions & 1 deletion examples/v1beta1/pipelineruns/pipelinerun-with-when-expressions.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,47 @@ spec:
image: ubuntu
script: exit 1
finally:
- name: do-something-finally
- name: finally-task-should-be-skipped-1 # when expression using execution status, evaluates to false
when:
- input: "$(tasks.echo-file-exists.status)"
operator: in
values: ["Failure"]
taskSpec:
steps:
- name: echo
image: ubuntu
script: exit 1
- name: finally-task-should-be-skipped-2 # when expression using task result, evaluates to false
when:
- input: "$(tasks.check-file.results.exists)"
operator: in
values: ["missing"]
taskSpec:
steps:
- name: echo
image: ubuntu
script: exit 1
- name: finally-task-should-be-skipped-3 # when expression using parameter, evaluates to false
when:
- input: "$(params.path)"
operator: notin
values: ["README.md"]
taskSpec:
steps:
- name: echo
image: ubuntu
script: exit 1
- name: finally-task-should-be-executed # when expression using execution status, param and results
when:
- input: "$(tasks.echo-file-exists.status)"
operator: in
values: ["Succeeded"]
- input: "$(tasks.check-file.results.exists)"
operator: in
values: ["yes"]
- input: "$(params.path)"
operator: in
values: ["README.md"]
taskSpec:
steps:
- name: echo
Expand Down
52 changes: 31 additions & 21 deletions pkg/apis/pipeline/v1beta1/pipeline_validation.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ func (ps *PipelineSpec) Validate(ctx context.Context) (errs *apis.FieldError) {
errs = errs.Also(validatePipelineResults(ps.Results))
errs = errs.Also(validateTasksAndFinallySection(ps))
errs = errs.Also(validateFinalTasks(ps.Tasks, ps.Finally))
errs = errs.Also(validateWhenExpressions(ps.Tasks))
errs = errs.Also(validateWhenExpressions(ps.Tasks, ps.Finally))
return errs
}

Expand Down Expand Up @@ -328,22 +328,32 @@ func validateExecutionStatusVariablesInFinally(tasks []PipelineTask, finally []P
ptNames := PipelineTaskList(tasks).Names()
for idx, t := range finally {
for _, param := range t.Params {
// retrieve a list of substitution expression from a param
if ps, ok := GetVarSubstitutionExpressionsForParam(param); ok {
// validate tasks.pipelineTask.status if this expression is not a result reference
if !LooksLikeContainsResultRefs(ps) {
for _, p := range ps {
// check if it contains context variable accessing execution status - $(tasks.taskname.status)
if containsExecutionStatusRef(p) {
// strip tasks. and .status from tasks.taskname.status to further verify task name
pt := strings.TrimSuffix(strings.TrimPrefix(p, "tasks."), ".status")
// report an error if the task name does not exist in the list of dag tasks
if !ptNames.Has(pt) {
errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("pipeline task %s is not defined in the pipeline", pt),
"value").ViaFieldKey("params", param.Name).ViaFieldIndex("finally", idx))
}
}
}
if expressions, ok := GetVarSubstitutionExpressionsForParam(param); ok {
errs = errs.Also(validateExecutionStatusVariablesExpressions(expressions, ptNames, "value").ViaFieldKey(
"params", param.Name).ViaFieldIndex("finally", idx))
}
}
for i, we := range t.WhenExpressions {
if expressions, ok := we.GetVarSubstitutionExpressions(); ok {
errs = errs.Also(validateExecutionStatusVariablesExpressions(expressions, ptNames, "").ViaFieldIndex(
"when", i).ViaFieldIndex("finally", idx))
}
}
}
return errs
}

func validateExecutionStatusVariablesExpressions(expressions []string, ptNames sets.String, fieldPath string) (errs *apis.FieldError) {
// validate tasks.pipelineTask.status if this expression is not a result reference
if !LooksLikeContainsResultRefs(expressions) {
for _, expression := range expressions {
// check if it contains context variable accessing execution status - $(tasks.taskname.status)
if containsExecutionStatusRef(expression) {
// strip tasks. and .status from tasks.taskname.status to further verify task name
pt := strings.TrimSuffix(strings.TrimPrefix(expression, "tasks."), ".status")
// report an error if the task name does not exist in the list of dag tasks
if !ptNames.Has(pt) {
errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("pipeline task %s is not defined in the pipeline", pt), fieldPath))
}
}
}
Expand Down Expand Up @@ -428,9 +438,6 @@ func validateFinalTasks(tasks []PipelineTask, finalTasks []PipelineTask) *apis.F
if len(f.Conditions) != 0 {
return apis.ErrInvalidValue(fmt.Sprintf("no conditions allowed under spec.finally, final task %s has conditions specified", f.Name), "").ViaFieldIndex("finally", idx)
}
if len(f.WhenExpressions) != 0 {
return apis.ErrInvalidValue(fmt.Sprintf("no when expressions allowed under spec.finally, final task %s has when expressions specified", f.Name), "").ViaFieldIndex("finally", idx)
}
}

ts := PipelineTaskList(tasks).Names()
Expand Down Expand Up @@ -486,11 +493,14 @@ func validateTasksInputFrom(tasks []PipelineTask) (errs *apis.FieldError) {
return errs
}

func validateWhenExpressions(tasks []PipelineTask) (errs *apis.FieldError) {
func validateWhenExpressions(tasks []PipelineTask, finalTasks []PipelineTask) (errs *apis.FieldError) {
for i, t := range tasks {
errs = errs.Also(validateOneOfWhenExpressionsOrConditions(t).ViaFieldIndex("tasks", i))
errs = errs.Also(t.WhenExpressions.validate().ViaFieldIndex("tasks", i))
}
for i, t := range finalTasks {
errs = errs.Also(t.WhenExpressions.validate().ViaFieldIndex("finally", i))
}
return errs
}

Expand Down
Loading

0 comments on commit 13d88a2

Please sign in to comment.