Support Task-level Resources Requirements: Part #1

Required fields and related webhook validations are added to support a user to configure the computing resources per Task/Pod which significantly reduces the over-asked amount configured by the step-level.
austinzhao-go · May 25, 2022 · ae9ee06 · ae9ee06
1 parent 4a93b62
commit ae9ee06
Show file tree

Hide file tree

Showing 12 changed files with 638 additions and 7 deletions.
diff --git a/docs/compute-resources.md b/docs/compute-resources.md
@@ -52,6 +52,251 @@ requirements for `Step`containers, they must be treated as if they are running i
 Tekton adjusts `Step` resource requirements to comply with [LimitRanges](#limitrange-support).
 [ResourceQuotas](#resourcequota-support) are not currently supported.
 
+Kubernetes interprets `Pod` resources requirements by taking the value summed up by each container's resources requirements. It leads Tekton to hoard resources computed by all `Steps`(containers), which can only be used on each sequentially executed `Step`. This observation forms the interest to support a `Task`/`Pod` level resources requirements, which naturally matches Kubernetes computing logic and can help users save significantly on requested resources.
+
+`Task`-level resources requirements can be configured in `Task.spec.resources` or `PipelineRun.spec.taskRunSpecs.resources`.
+
+e.g.
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Task
+metadata:
+  name: foo 
+spec:
+  steps:
+    - name: foo1
+    - name: foo2
+  resources:
+    requests:
+      cpu: 1 
+    limits:
+      cpu: 2
+```
+
+### Applying Resources Requirements
+
+**Requests**
+`Task`(`Pod`) level resources requests will be applied only on one `Step`(container) to acquire the same effective request for the `Task`. (An exception with `LimitRange` will be explained in the below section.)
+
+**Limits**
+Different from requests, `Task` level resources limits will be applied on each `Step` as a non-limits-configured `Step` will have higher limits than configured ones by Kubernetes. Although the summed-up limits will lead to a value higher than the desired for `Task`, the effective limits will not be used for scheduling a `Pod` but applied in container runtime. So during any time of the `Task`, this limit will be enforced on each `Step`.
+
+### Interaction with Step-level Resources Requirements
+
+When Tekton now supports configuring step-level resources requirements, from `Task.spec.step.resources`, `Task.spec.stepTemplate.resources`, `TaskRun.spec.stepOverrides.resources`, and `PipelineRun.spec.taskRunSpecs.stepOverrides.resources`, either the task-level or step-level will be allowed to configure resources requirements, therefore avoiding possible conflicts.
+
+The following configurations will be rejected during the webhook validation:
+
+Case #1: `Task.spec.step.resources` and `Task.spec.resources`
+
+```yaml
+kind: Task
+spec:
+  steps:
+    - name: foo
+      requests:
+        cpu: 1
+  resources:
+    requests:
+      cpu: 2
+```
+
+Case #2: `Task.spec.stepTemplate.resources` and `Task.spec.resources`
+
+```yaml
+kind: Task
+spec:
+  steps:
+    - name: foo
+  stepTemplate:
+    requests:
+      cpu: 1
+  resources:
+    requests:
+      cpu: 2
+```
+
+Case #3: `TaskRun.spec.stepOverrides.resources` and `TaskRun.spec.taskSpec.resources`
+
+```yaml
+kind: TaskRun
+spec:
+  stepOverrides:
+    - name: foo
+      resources:
+        requests:
+          cpu: 1
+  taskSpec:
+    resources:
+      requests:
+        cpu: 2  
+```
+
+Case #4: `PipelineRun.spec.taskRunSpecs.stepOverrides.resources` and `PipelineRun.spec.taskRunSpecs.resources`
+
+```yaml
+kind: PipelineRun
+spec:
+  taskRunSpecs:
+    - pipelineTaskName: foo
+      stepOverrides:
+        - name: foo 
+          resources:
+            requests:
+              cpu: 1 
+      resources:
+        requests:
+          cpu: 2 
+```
+
+### Interaction with Runtime Overwriting
+
+As runtime specifications are based on an execution context, Tekton allows users to overwrite certain statically configured fields in the runtime. For task-level resources requirements, there are `TaskRun.spec.taskSpec.resources` and `PipelineRun.spec.taskRunSpecs.resources`.
+
+e.g.
+
+```yaml
+kind: Task
+metadata:
+  name: foo
+spec:
+  resources:
+    requests:
+      cpu: 1
+---
+kind: TaskRun
+spec:
+  taskRef:
+    name: foo
+  taskSpec:
+    resources:
+      requests:
+        cpu: 2
+```
+
+The final task cpu resources request will be 2 by the overwriting.
+
+### Interaction with LimitRanges
+
+Min or max resources requirements can be configured by `LimitRange`. For a min request on certain containers, the task-level request can subtract this amount and assign the rest to one `Step`(container) as stated before. Furthermore, if a container has a value more than the max limit after being set with task-level requests, the request amount will be spread out to each container (the number will be rounded to ignore decimals).
+
+Case #1: Applying Min
+
+```yaml
+kind: LimitRange
+spec:
+  limits:
+    - min:
+        cpu: 200m
+---
+kind: Task
+spec:
+  steps:
+    - name: step1 # applied with min
+    - name: step2 # applied with min
+    - name: step3 # not applied
+  resources:
+    requests:
+      cpu: 1
+```
+
+| Step name | CPU request |
+| --------- | ----------- |
+| step1     | 800m        |
+| step2     | 200m        |
+| step3     | N/A         |
+
+Here the 800m on step1 comes from `200m + (1 - 200m * 2)`.
+
+Case #2: Applying Max
+
+```yaml
+kind: LimitRange
+spec:
+  limits:
+    - max:
+        cpu: 800m
+---
+kind: Task
+spec:
+  steps:
+    - name: step1 # applied with max
+    - name: step2 # applied with max
+    - name: step3 # not applied
+  resources:
+    requests:
+      cpu: 1
+```
+
+| Step name | CPU request |
+| --------- | ----------- |
+| step1     | 333m        |
+| step2     | 333m        |
+| step3     | 333m        |
+
+Here the 333m comes from `1 / 3` with number rounding to leave decimals out.
+
+Case #3: Applying Min and Max
+
+```yaml
+kind: LimitRange
+spec:
+  limits:
+    - min:
+        cpu: 200m
+    - max:
+        cpu: 700m
+---
+kind: Task
+spec:
+  steps:
+    - name: step1 # applied with min and max
+    - name: step2 # applied with min and max
+    - name: step3 # not applied
+  resources:
+    requests:
+      cpu: 1
+```
+
+| Step name | CPU request |
+| --------- | ----------- |
+| step1     | 400m        |
+| step2     | 400m        |
+| step3     | 200m        |  
+
+Here the 400m comes from the min `200` + the spread out `(1 - 200m * 2) / 3`.
+
+### Interaction with Sidecar Resources Requirements
+
+A `Sidecar` container will run with sequentially executed `Steps` in parallel, which leads to a summed up resources requirements for the `Task`(`Pod`) from the 2 parts. In order to avoid possible conflicts on limits, resources requirements will be configured separately for `Steps` and `Sidecar`.
+
+e.g.
+
+```yaml
+kind: Task
+spec:
+  steps:
+    - name: step1
+    - name: step2
+  sidecars:
+    - name: sidecar1
+      resources:
+        requests:
+          cpu: 750m
+        limits:
+          cpu: 1
+  resources:
+    requests:
+      cpu: 2
+```
+
+| Step/Sidecar name | CPU request | CPU limit |
+| ----------------- | ----------- | --------- |
+| step1             | 2           | N/A       |
+| step2             | N/A         | N/A       |
+| sidecar1          | 750m        | 1         |
+
 ## LimitRange Support
 
 Kubernetes allows users to configure [LimitRanges]((https://kubernetes.io/docs/concepts/policy/limit-range/)),

diff --git a/pkg/apis/pipeline/v1beta1/openapi_generated.go b/pkg/apis/pipeline/v1beta1/openapi_generated.go
diff --git a/pkg/apis/pipeline/v1beta1/pipelinerun_types.go b/pkg/apis/pipeline/v1beta1/pipelinerun_types.go
@@ -20,6 +20,7 @@ import (
 	"context"
 	"time"
 
+	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 
 	"github.com/tektoncd/pipeline/pkg/apis/config"
@@ -600,6 +601,11 @@ type PipelineTaskRunSpec struct {
 	StepOverrides []TaskRunStepOverride `json:"stepOverrides,omitempty"`
 	// +listType=atomic
 	SidecarOverrides []TaskRunSidecarOverride `json:"sidecarOverrides,omitempty"`
+	// Compute Resources required by the Task/TaskRun (Pod).
+	// Cannot be updated.
+	// More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
+	// +optional
+	Resources corev1.ResourceRequirements `json:"resources,omitempty" protobuf:"bytes,8,opt,name=resources"`
 }
 
 // GetTaskRunSpec returns the task specific spec for a given

diff --git a/pkg/apis/pipeline/v1beta1/pipelinerun_validation.go b/pkg/apis/pipeline/v1beta1/pipelinerun_validation.go
@@ -195,6 +195,9 @@ func validateTaskRunSpec(ctx context.Context, trs PipelineTaskRunSpec) (errs *ap
 		if trs.SidecarOverrides != nil {
 			errs = errs.Also(validateSidecarOverrides(trs.SidecarOverrides).ViaField("sidecarOverrides"))
 		}
+		if trs.Resources.Size() > 0 {
+			errs = errs.Also(validateResourcesRequirements(trs).ViaField("resources"))
+		}
 	} else {
 		if trs.StepOverrides != nil {
 			errs = errs.Also(apis.ErrDisallowedFields("stepOverrides"))
@@ -205,3 +208,15 @@ func validateTaskRunSpec(ctx context.Context, trs PipelineTaskRunSpec) (errs *ap
 	}
 	return errs
 }
+
+// validateResourcesRequirements() validates if only step-level or task-level resources requirements are configured
+func validateResourcesRequirements(trs PipelineTaskRunSpec) (errs *apis.FieldError) {
+	for _, taskRunStepOverride := range trs.StepOverrides {
+		if taskRunStepOverride.Resources.Size() > 0 {
+			return &apis.FieldError{
+				Message: "PipelineTaskRunSpec can't be configured with both step-level(stepOverrides.resources) and task-level(taskRunSpecs.resources) resources requirements",
+			}
+		}
+	}
+	return nil
+}