Skip to content

Commit

Permalink
Initial design ideas for Source (#39)
Browse files Browse the repository at this point in the history 
* initial design ideas for source

* changes to implement Resource concept

* remove extra function in resource interface

* Update JSON struct tag to pass go vet

* changes to make tests work

* minor readme changes

* remove resource ref, update configs and samples

* Add resources as CRD

* Rename task inputs & outputs
  • Loading branch information
@nader-ziada @knative-prow-robot
nader-ziada authored and knative-prow-robot committed Sep 20, 2018
1 parent 801db1d commit 1c4deea
Show file tree
Hide file tree
Showing 32 changed files with 897 additions and 604 deletions.
21 changes: 16 additions & 5 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ The CRDs involved are:
* [PipelineParams](#pipelineparams)
* [TaskRun](#taskrun)
* [PipelineRun](#pipelinerun)
* [Resources](#resources)

High level details of this design:

Expand All @@ -41,6 +42,7 @@ High level details of this design:
easily is powerful (e.g. see failures easily, dig into logs, e.g. like
[the Jenkins test analyzer plugin](https://wiki.jenkins.io/display/JENKINS/Test+Results+Analyzer+Plugin))
* [Tasks](#tasks) can depend on artifacts, output and parameters created by other tasks.
* [Resources](#resources) are the artifacts used as inputs and outputs of TaskRuns.

## Task

Expand All @@ -57,9 +59,7 @@ with additional input types and clearly defined outputs.

`Pipeline` describes a graph of [Tasks](#task) to execute. It defines the DAG
and expresses how all inputs (including [PipelineParams](#pipelineparams) and outputs
from previous `Tasks`) feed into each `Task`. It allows for fan in and fan out, and
ordering can be expressed explicitly using `prev` and `next`, or it can be inferred
from a `Task’s` inputs.
from previous `Tasks`) feed into each `Task`.

Dependencies between parameters or inputs/outputs are expressed as references to k8s objects.

Expand All @@ -70,9 +70,7 @@ can be invoked with many different instances of `PipelineParams`, which can allo
for scenarios such as running against PRs and against a user’s personal setup.
`PipelineParams` can control:

* What **sources** the `Pipeline` runs against
* Which **serviceAccount** to use (provided to all tasks)
* What **artifact** stores are used (e.g. Docker registries)
* Where **results** are stored (e.g. in GCS)

## TaskRun
Expand Down Expand Up @@ -129,3 +127,16 @@ completes (or fails).
When the `PipelineRun` has completed, the `taskRuns` field will contain
references to all `TaskRuns` which were executed and their next and
previous `TaskRuns`.

### Resources

`Resources` in a pipelines are the set of objects that are going to be used
as inputs and outputs of a `TaskRun`.

* `Resources` is created directly in a pipeline configuration and bound
to `TaskRun` as an input and/or output source.
* The (optional) `passedConstraint` key on an `input source` defines a set of previous task names.
* When the `passedConstraint` key is specified on an input source, only the version of
the resource that passed through the defined list of tasks is used.
* The `passedConstraint` allows for `Tasks` to fan in and fan out, and ordering can be expressed explicitly
using this key since a task needing a resource from a another task would have to run after.
71 changes: 38 additions & 33 deletions config/crds/pipeline_v1beta1_pipeline.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,40 +22,60 @@ spec:
type: object
spec:
properties:
resources:
items:
properties:
name:
type: string
resourceRef:
properties:
apiVersion:
type: string
name:
type: string
required:
- name
type: object
required:
- name
- resourceRef
type: object
type: array
tasks:
items:
properties:
artifactStoreBindings:
inputSourceBindings:
items:
properties:
storeKey:
name:
type: string
storeName:
passedConstraints:
items:
type: string
type: array
sourceKey:
type: string
required:
- storeName
- storeKey
- name
- sourceKey
type: object
type: array
name:
type: string
nextTasks:
items:
type: string
type: array
paramBindings:
outputSourceBindings:
items:
properties:
inputName:
type: string
taskName:
name:
type: string
taskOutputName:
passedConstraints:
items:
type: string
type: array
sourceKey:
type: string
required:
- inputName
- taskName
- taskOutputName
- name
- sourceKey
type: object
type: array
params:
Expand All @@ -70,22 +90,6 @@ spec:
- value
type: object
type: array
prevTasks:
items:
type: string
type: array
sourceBindings:
items:
properties:
inputName:
type: string
sourceKey:
type: string
required:
- inputName
- sourceKey
type: object
type: array
taskRef:
properties:
apiVersion:
Expand All @@ -102,6 +106,7 @@ spec:
type: array
required:
- tasks
- resources
type: object
status:
type: object
Expand Down
39 changes: 0 additions & 39 deletions config/crds/pipeline_v1beta1_pipelineparams.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,21 +22,6 @@ spec:
type: object
spec:
properties:
artifactStores:
items:
properties:
name:
type: string
type:
type: string
url:
type: string
required:
- name
- type
- url
type: object
type: array
results:
properties:
logs:
Expand Down Expand Up @@ -84,32 +69,8 @@ spec:
type: object
serviceAccount:
type: string
sources:
items:
properties:
branch:
type: string
commit:
type: string
name:
type: string
serviceAccount:
type: string
type:
type: string
url:
type: string
required:
- name
- type
- url
- branch
type: object
type: array
required:
- serviceAccount
- sources
- artifactStores
- results
type: object
status:
Expand Down
19 changes: 19 additions & 0 deletions config/crds/pipeline_v1beta1_pipelinerun.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,25 @@ spec:
- lastTransitionTime
type: object
type: array
resourceVersion:
items:
properties:
resourceRef:
properties:
apiVersion:
type: string
name:
type: string
required:
- name
type: object
version:
type: string
required:
- resourceRef
- version
type: object
type: array
taskRuns:
items:
properties:
Expand Down
61 changes: 61 additions & 0 deletions config/crds/pipeline_v1beta1_resource.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
creationTimestamp: null
labels:
controller-tools.k8s.io: "1.0"
name: resources.pipeline.knative.dev
spec:
group: pipeline.knative.dev
names:
kind: Resource
plural: resources
scope: Namespaced
validation:
openAPIV3Schema:
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
properties:
resources:
items:
properties:
name:
type: string
params:
items:
properties:
name:
type: string
value:
type: string
required:
- name
- value
type: object
type: array
type:
type: string
required:
- name
- type
- params
type: object
type: array
required:
- resources
type: object
status:
type: object
type: object
version: v1beta1
status:
acceptedNames:
kind: ""
plural: ""
conditions: null
34 changes: 24 additions & 10 deletions config/crds/pipeline_v1beta1_task.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -59,38 +59,52 @@ spec:
properties:
name:
type: string
type:
value:
type: string
required:
- name
- type
- value
type: object
type: array
sources:
resources:
items:
properties:
name:
type: string
resourceRef:
properties:
apiVersion:
type: string
name:
type: string
required:
- name
type: object
required:
- name
- resourceRef
type: object
type: array
type: object
outputs:
properties:
artifacts:
resources:
items:
properties:
name:
type: string
storeKey:
type: string
type:
type: string
resourceRef:
properties:
apiVersion:
type: string
name:
type: string
required:
- name
type: object
required:
- name
- type
- storeKey
- resourceRef
type: object
type: array
results:
Expand Down
Loading

0 comments on commit 1c4deea

Please sign in to comment.