Pipelineruns with failed task with result fails on PipelineValidationFailed reason, hiding the failure

[vdemeester]

Given the following PipelineRun:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: test-results
spec:
  pipelineSpec:
    results:
      - name: abc
        value: "$(tasks.xxx.results.abc)"
    tasks:
    - name: xxx
      taskSpec:
        results:
          - name: abc
        steps:
        - name: update-sa
          image: bash:latest
          script: |
            echo 'test' >  $(results.abc.path)
            exit 1

Expected Behavior

When a Task "with a results" in a PipelineRun fails, the PipelineRun result in a PipelineValidationFailed instead of saying that the TaskRun failed.

Actual Behavior

When a Task "with a results" in a PipelineRun fails, the PipelineRun result in a the Pipeline failing because the Task failed.

Steps to Reproduce the Problem

1. Apply the PipelineRun above
2. See the failure

Additional Info

The result was written even 👼🏼 So technically, it has the result.

• Tekton Pipeline version: v0.45.0 (and below)

/kind bug
cc @pritidesai @jerop I think it "might be" by design, but it is confusing.

[pritidesai]

This is not by design, its a bug 😞

The pipelineRun must not fail with pipelineValidationFailed. If the taskRun fails, the results are not initialized in taskRun status even if the termination message contains that particular result. We have a PR to support initializing results irrespective of the taskRun status - #5750

@Yongxuanzhang has a fix for similar error in case of skipped tasks. Please help fix this issue if its not already fixed.

[pritidesai]

I ran into the same issue while creating a release for v0.46.0 when the publish-images failed and dependent tasks report-bucket and publish-* tasks were declared as skipped. These are not skipped tasks but they were not executed because of a failure of a parent task. These should be counted towards incomplete tasks.

"conditions": [
    {
      "lastTransitionTime": "2023-03-16T21:22:54Z",
      "message": "invalid pipelineresults [release-file release-file-no-tag], the referred results don't exist",
      "reason": "PipelineValidationFailed",
      "status": "False",
      "type": "Succeeded"
    }
  ],

 tkn --context dogfooding pr describe release-pipeline-v0-46-0-1
Name:           release-pipeline-v0-46-0-1
Namespace:      default
Pipeline Ref:   pipeline-release
Labels:
 tekton.dev/pipeline=pipeline-release
Annotations:
 chains.tekton.dev/signed=true

🌡️  Status

STARTED        DURATION   STATUS
18 hours ago   33m47s     Failed(PipelineValidationFailed)

💌 Message

invalid pipelineresults [release-file release-file-no-tag], the referred results don't exist
TaskRun(s) cancelled: release-pipeline-v0-46-0-1-publish-images

⏱  Timeouts
 Pipeline:   1h0m0s

⚓ Params

 NAME                   VALUE
 ∙ gitRevision          70a2d714dcbaefb952924edf6df0c217aff80548
 ∙ serviceAccountPath   release.json
 ∙ versionTag           v0.46.0
 ∙ releaseBucket        gs://tekton-releases/pipeline

📂 Workspaces

 NAME               SUB PATH   WORKSPACE BINDING
 ∙ release-secret   ---        Secret (secret=release-secret)
 ∙ workarea         ---        VolumeClaimTemplate

🗂  Taskruns

 NAME                                          TASK NAME        STARTED        DURATION   STATUS
 ∙ release-pipeline-v0-46-0-1-publish-images   publish-images   18 hours ago   24m25s     Failed
 ∙ release-pipeline-v0-46-0-1-unit-tests       unit-tests       18 hours ago   7m42s      Succeeded
 ∙ release-pipeline-v0-46-0-1-build            build            18 hours ago   5m15s      Succeeded
 ∙ release-pipeline-v0-46-0-1-precheck         precheck         18 hours ago   1m16s      Succeeded
 ∙ release-pipeline-v0-46-0-1-git-clone        git-clone        18 hours ago   23s        Succeeded

⏭️  Skipped Tasks

 NAME
 ∙ publish-to-bucket
 ∙ publish-to-bucket-latest
 ∙ report-bucket

pipeline/tekton/release-pipeline.yaml

Lines 47 to 52 in 086e76a

	- name: release-file
	description: the URL of the release file
	value: $(tasks.report-bucket.results.release)
	- name: release-file-no-tag
	description: the URL of the release file
	value: $(tasks.report-bucket.results.release-no-tag)

[pritidesai]

This might help troubleshoot this issue:

  "skippedTasks": [
    {
      "name": "publish-to-bucket",
      "reason": "PipelineRun was stopping"
    },
    {
      "name": "publish-to-bucket-latest",
      "reason": "PipelineRun was stopping",
      "whenExpressions": [
        {
          "input": "true",
          "operator": "in",
          "values": [
            "true"
          ]
        }
      ]
    },
    {
      "name": "report-bucket",
      "reason": "PipelineRun was stopping"
    }
  ],

[pritidesai]

We are running 0.44 in dogfooding:

tkn --context dogfooding version
Client version: 0.29.1
Chains version: v0.14.0
Pipeline version: v0.44.0
Triggers version: v0.22.1
Dashboard version: v0.32.0

[Yongxuanzhang]

/assign

[Yongxuanzhang]

The direct cause of this issue is the change in this PR #5060
It allows the false condition pipelinerun to apply results(where we also do the validation).
So if a task failed, the pipelinerun status is false, the validation will update the status again which makes the status confusing to users.

We should handle these cases together:

1. Taskrun failed but emit results: emit task results to pipeline results
2. Taskrun not succeeded (failed or skipped) and doesn't emit results: pipeline task results are omitted and don't validate the results (pipelinerun shouldn't fail because of the results validation failure)
3. Taskrun succeeded and emit results: validate the results references.
4. Taskrun succeeded but doesn't emit results: fail validation