Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

revert: "fix(cli): cannot hotswap ECS task definitions containing certain intrinsics" #27358

Merged
merged 4 commits into from
Oct 12, 2023

Conversation

tmokmss
Copy link
Contributor

@tmokmss tmokmss commented Sep 29, 2023

Closes #27343

From v2.93.0 (#26404), we hotswap ECS task definition in the following steps:

  1. Calculate patch from old/new CFn template
  2. Apply the patch to the task definition fetched from describeTaskDefinition API
  3. register the new task definition and update services

The root cause of the issue #27343 is the order of containerDefinitions array is somehow shuffled when deployed to ECS, so the patch calculated from CFn template becomes invalid.

For example, when the containerDefinitions in a CFn template is like below:

    "ContainerDefinitions": [
     {
      "Name": "main",
      "Image": "imageA"
     },
     {
      "Name": "sidecar",
      "Image": "imageB"
     }
    ],

the deployed task definition can sometimes become like this:

    "ContainerDefinitions": [
     {
      "Name": "sidecar",
      "Image": "imageB"
     },
     {
      "Name": "main",
      "Image": "imageA"
     }
    ],

This makes a patch calculated from CFn template diff completely invalid. We can sort both CFn template and the response of describeTaskDefinition API in a deterministic order, but it is still unreliable because there can be more arrays whose order will be shuffled. The response of describeTaskDefinition has many array fields, and it is not documented if they may be shuffled or not.

I guess we should completely abandon this approach, because it cannot be reliable enough. I have an idea for more reliable approach, but at least it should be reverted asap as it's breaking the ECS hotswap feature.

I'm really sorry for me not being aware with this behavior 🙏


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@aws-cdk-automation aws-cdk-automation requested a review from a team September 29, 2023 16:54
@github-actions github-actions bot added bug This issue is a bug. p2 admired-contributor [Pilot] contributed between 13-24 PRs to the CDK labels Sep 29, 2023
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@aws-cdk-automation aws-cdk-automation added the pr/needs-cli-test-run This PR needs CLI tests run against it. label Sep 29, 2023
@tmokmss tmokmss changed the title Revert "fix(cli): cannot hotswap ECS task definitions containing certain intrinsics fix(cli): ECS hotswap not working properly on multiple container definitions Sep 29, 2023
@akbisw
Copy link

akbisw commented Oct 4, 2023

🤞 looking forward to this

@tmokmss
Copy link
Contributor Author

tmokmss commented Oct 10, 2023

Exemption Request: We already have CLI integ tests.

@aws-cdk-automation aws-cdk-automation added pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback. pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. labels Oct 10, 2023
@vinayak-kukreja vinayak-kukreja self-assigned this Oct 11, 2023
@vinayak-kukreja vinayak-kukreja added the pr-linter/exempt-integ-test The PR linter will not require integ test changes label Oct 11, 2023
@vinayak-kukreja vinayak-kukreja temporarily deployed to test-pipeline October 11, 2023 23:12 — with GitHub Actions Inactive
@vinayak-kukreja
Copy link
Contributor

I'm really sorry for me not being aware with this behavior 🙏

Hey @tmokmss, no worries, thank you for working on this. Really appreciate your effort. :)

@vinayak-kukreja
Copy link
Contributor

Could you tell me why this is not named with a revert? For instance,

revert "fix(cli): cannot hotswap ECS task definitions containing certain intrinsics"

@vinayak-kukreja
Copy link
Contributor

I have initiated a CLI Test run in pipeline for this PR. Will keep track of it.

@tmokmss
Copy link
Contributor Author

tmokmss commented Oct 12, 2023

@vinayak-kukreja Thanks!

Could you tell me why this is not named with a revert?

I just didn't know the naming rule :) Please feel free to change the title

@vinayak-kukreja vinayak-kukreja changed the title fix(cli): ECS hotswap not working properly on multiple container definitions revert: "fix(cli): cannot hotswap ECS task definitions containing certain intrinsics" Oct 12, 2023
@github-actions github-actions bot added p1 and removed p2 labels Oct 12, 2023
@vinayak-kukreja vinayak-kukreja temporarily deployed to test-pipeline October 12, 2023 17:13 — with GitHub Actions Inactive
@aws-cdk-automation aws-cdk-automation added pr/needs-maintainer-review This PR needs a review from a Core Team Member and removed pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. labels Oct 12, 2023
@aws-cdk-automation
Copy link
Collaborator

➡️ PR build request submitted to test-main-pipeline ⬅️

A maintainer must now check the pipeline and add the pr-linter/cli-integ-tested label once the pipeline succeeds.

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 492a691
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@vinayak-kukreja vinayak-kukreja added the pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested label Oct 12, 2023
@aws-cdk-automation aws-cdk-automation dismissed their stale review October 12, 2023 21:32

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@aws-cdk-automation aws-cdk-automation removed the pr/needs-cli-test-run This PR needs CLI tests run against it. label Oct 12, 2023
@mergify
Copy link
Contributor

mergify bot commented Oct 12, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit 48d7726 into aws:main Oct 12, 2023
22 of 25 checks passed
@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Oct 12, 2023
@tmokmss tmokmss deleted the revert_ecs_hotswap_patch branch October 13, 2023 00:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
admired-contributor [Pilot] contributed between 13-24 PRs to the CDK bug This issue is a bug. p1 pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested pr-linter/exempt-integ-test The PR linter will not require integ test changes pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

aws-cdk/hotswap: ecs hotswap not updating image in new task definition
4 participants