Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validation of pipelines with hundreds of tasks and links fails on default webhook timeout (10s) #5420

Closed
rafalbigaj opened this issue Sep 2, 2022 · 2 comments
Closed

Validation of pipelines with hundreds of tasks and links fails on default webhook timeout (10s) #5420

rafalbigaj opened this issue Sep 2, 2022 · 2 comments
Labels
area/performance Issues or PRs that are related to performance aspects. kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.

Comments

@rafalbigaj
Copy link
Contributor

rafalbigaj commented Sep 2, 2022 •
edited
Loading

Expected Behavior

Pipeline DAG validation should work efficiently for hundreds of tasks and links between them. Tekton Pipelines webhook performance should neither be impacted by DAG validation nor fail with default timeout (10s).

Actual Behavior

PipelineRun with hundreds of tasks and links creation fails with:

failed calling webhook "validation.webhook.pipeline.tekton.dev":
failed to call webhook: 
  Post "https://tekton-pipelines-webhook.tekton-pipelines.svc:443/resource-validation?timeout=10s": 
  context deadline exceeded

Steps to Reproduce the Problem

  1. Create a YAML file with PipelineRun containing hundreds of tasks and links like this one: pipeline.hundreds-of-tasks.yaml
  2. Apply the YAML file on Tekton cluster
  3. tekton-pipelines-webhook fails with timeout

Additional Info

  • Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:36:49Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.9+IKS", GitCommit:"1dc6043da4c077ebf5fa7a44d98695719495ec79", GitTreeState:"clean", BuildDate:"2022-07-14T04:23:53Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"}
  • Tekton Pipeline version:
 v0.38.3
@rafalbigaj rafalbigaj added the kind/bug Categorizes issue or PR as related to a bug. label Sep 2, 2022
@rafalbigaj rafalbigaj changed the title Validation of pipelines with hundreds of tasks and links fau Validation of pipelines with hundreds of tasks and links fails on default webhook timeout (10s) Sep 2, 2022
rafalbigaj added a commit to rafalbigaj/tekton-pipelines that referenced this issue Sep 2, 2022
@rafalbigaj
Improve DAG validation for pipelines with hundreds of tasks
18a6a2d 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at tektoncd#5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
@rafalbigaj rafalbigaj mentioned this issue Sep 2, 2022
Merged
5 tasks
rafalbigaj added a commit to rafalbigaj/tekton-pipelines that referenced this issue Sep 2, 2022
@rafalbigaj
Improve DAG validation for pipelines with hundreds of tasks
171d89a 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at tektoncd#5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
rafalbigaj added a commit to rafalbigaj/tekton-pipelines that referenced this issue Sep 2, 2022
@rafalbigaj
Improve DAG validation for pipelines with hundreds of tasks
db0c6ad 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at tektoncd#5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
rafalbigaj added a commit to rafalbigaj/tekton-pipelines that referenced this issue Sep 2, 2022
@rafalbigaj
Improve DAG validation for pipelines with hundreds of tasks
cf2f32d 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at tektoncd#5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
rafalbigaj added a commit to rafalbigaj/tekton-pipelines that referenced this issue Sep 2, 2022
@rafalbigaj
Improve DAG validation for pipelines with hundreds of tasks
6895508 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at tektoncd#5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
@pritidesai pritidesai added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Sep 2, 2022
tekton-robot pushed a commit that referenced this issue Sep 5, 2022
@rafalbigaj @tekton-robot
Improve DAG validation for pipelines with hundreds of tasks
7b00bd2 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at #5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
@vdemeester
Copy link
Member

@rafalbigaj I think this is fixed by #5421 isn't it ?

@rafalbigaj
Copy link
Contributor Author

@vdemeester indeed. Thanks

tekton-robot pushed a commit to tekton-robot/pipeline that referenced this issue Sep 6, 2022
@rafalbigaj
Improve DAG validation for pipelines with hundreds of tasks
81a79ad 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at tektoncd#5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
tekton-robot pushed a commit to tekton-robot/pipeline that referenced this issue Sep 6, 2022
@rafalbigaj
Improve DAG validation for pipelines with hundreds of tasks
cb58191 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at tektoncd#5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
tekton-robot pushed a commit to tekton-robot/pipeline that referenced this issue Sep 6, 2022
@rafalbigaj
Improve DAG validation for pipelines with hundreds of tasks
21478ff 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at tektoncd#5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
rafalbigaj added a commit to rafalbigaj/tekton-pipelines that referenced this issue Sep 6, 2022
@rafalbigaj
Improve DAG validation for pipelines with hundreds of tasks - v1beta1…
1b72250 
… tests

Add pipeline validation tests for `v1beta1`.

Related to: tektoncd#5420
rafalbigaj added a commit to rafalbigaj/tekton-pipelines that referenced this issue Sep 6, 2022
@rafalbigaj
Improve DAG validation for pipelines with hundreds of tasks - v1beta1…
fbf79c0 
… tests

Add pipeline validation tests for `v1beta1`.

Related to: tektoncd#5420
tekton-robot pushed a commit that referenced this issue Sep 6, 2022
@rafalbigaj @tekton-robot
Improve DAG validation for pipelines with hundreds of tasks - v1beta1…
9b1afed 
… tests

Add pipeline validation tests for `v1beta1`.

Related to: #5420
tekton-robot pushed a commit that referenced this issue Sep 7, 2022
@rafalbigaj @tekton-robot
Improve DAG validation for pipelines with hundreds of tasks
ce9c916 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at #5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
tekton-robot pushed a commit that referenced this issue Sep 7, 2022
@rafalbigaj @tekton-robot
Improve DAG validation for pipelines with hundreds of tasks
059fdce 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at #5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
tekton-robot pushed a commit that referenced this issue Sep 7, 2022
@rafalbigaj @tekton-robot
Improve DAG validation for pipelines with hundreds of tasks
3171a67 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at #5420 is a root cause of poor validation webhook performance, which fails on default timeout (10s).
@pritidesai pritidesai mentioned this issue Sep 7, 2022
Open
@pritidesai pritidesai added the area/performance Issues or PRs that are related to performance aspects. label Sep 7, 2022
afrittoli pushed a commit to afrittoli/pipeline that referenced this issue Sep 9, 2022
@rafalbigaj @afrittoli
Improve DAG validation for pipelines with hundreds of tasks
eeb0760 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at tektoncd#5420
is a root cause of poor validation webhook performance, which fails on default timeout (10s).
afrittoli pushed a commit to afrittoli/pipeline that referenced this issue Sep 9, 2022
@rafalbigaj @afrittoli
Improve DAG validation for pipelines with hundreds of tasks - v1beta1…
ec69272 
… tests

Add pipeline validation tests for `v1beta1`.

Related to: tektoncd#5420
tekton-robot pushed a commit that referenced this issue Sep 9, 2022
@rafalbigaj @tekton-robot
Improve DAG validation for pipelines with hundreds of tasks
666cc8f 
DAG validation rewritten using Kahn's algorithm to find cycles in task dependencies.

Original implementation, as pointed at #5420
is a root cause of poor validation webhook performance, which fails on default timeout (10s).
tekton-robot pushed a commit that referenced this issue Sep 9, 2022
@rafalbigaj @tekton-robot
Improve DAG validation for pipelines with hundreds of tasks - v1beta1…
edbca30 
… tests

Add pipeline validation tests for `v1beta1`.

Related to: #5420
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/performance Issues or PRs that are related to performance aspects. kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vdemeester @pritidesai @rafalbigaj