-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWS ECR build permissions issue with Kaniko v1.19.0
& v1.19.1
(found using v1.19.0-debug
& v1.19.1-debug
)
#2882
Comments
JFYI we started having issues pushing to ECR a few hours ago and using gcr.io/kaniko-project/executor:v1.18.0-debug works for us. error:
was using this image before: gcr.io/kaniko-project/executor:debug |
Added a thumbs up but wanted to also say seeing the same exact thing with the latest version of debug and rolling back to 1.18 works fine |
Okay well dang guys, if it's THAT bad, to me that's a sign they need to pull that image immediately and re-tag the 1.18 release ASAP. |
Thanks for flagging this @AndrewFarley, are you able to post the debug logs for a failed run w/ kaniko using
|
I have reverted the |
I've also added the below env vars to the job and they made no difference in my case:
|
Based on the above list and the verbose debug this feels like it's probably one of the AWS dependency updates that caused this, meaning that this problem might be an upstream issue in one of these bumps above. If someone has the time, might be good to rebuild kaniko debug but reverting these above commits and see if it makes a difference. Sorry I don't have the time to do that. :( |
Just noticed this on our builds too. Gitlab CI runners running on a AWS EKS Cluster. Reverting to 1.18 solved the issue. This setup has been working for us for well over a year.
(Account ID and repo redacted) |
v1.19.0
(found using v1.19.0-debug
)
v1.19.0
(found using v1.19.0-debug
)v1.19.0
(found using v1.19.0-debug
)
@dennis-helm-sp I believe this is only affecting AWS ECR users atm from this thread, if someone in the thread encountered an issue w/ |
We have the same issue when using latest |
Maybe related to aws/aws-sdk-go-v2#2370 If it is, the solution is to upgrade all AWS SDK dependencies at once: |
@pdecat thank you for tagging the issue aws/aws-sdk-go-v2#2370 (comment), I believe this is the root cause here. |
This should now be fixed with the release of Closing |
It seems that v1.19.1 still has the same issue. |
Seeing the same issue as @cm3lindsay, pinning 1.18.0 is our workaround as well. |
v1.19.0
(found using v1.19.0-debug
)v1.19.0
& v1.19.1
(found using v1.19.0-debug
& v1.19.1-debug
)
@aaron-prindle, |
Thank you @ChristopherKlinge, I will ping the thread here for test assistance in trying to fix this in another patch release. Additionally kaniko has per PR merge image builds (closest thing to nightly) available @ the labels w/ the git commit sha - see example below. I'll add the commit sha to test with in the thread here to validate the fix once it is submitted. gcr.io/kaniko-project/executor:<git-commit-sha> |
I've managed to reproduce the issue with a local build of the current
And also ensured v1.18.0 works in the same context:
|
I believe the issue is with the |
Can confirm that, will submit a PR ASAP. |
…esolve issues with AWS ECR authentication (resolves GoogleContainerTools#2882) Signed-off-by: Patrick Decat <pdecat@gmail.com>
Submitted #2908 |
…esolve issues with AWS ECR authentication (resolves GoogleContainerTools#2882) As mentioned in aws/aws-sdk-go-v2#2370, AWS SDK for Go v2 releases after 2023/11/15 broke compatibility with all previous releases. Signed-off-by: Patrick Decat <pdecat@gmail.com>
…esolve issues with AWS ECR authentication As mentioned in aws/aws-sdk-go-v2#2370, AWS SDK for Go v2 releases after 2023/11/15 broke compatibility with all previous releases. Resolves GoogleContainerTools#2882 Signed-off-by: Patrick Decat <pdecat@gmail.com>
…esolve issues with AWS ECR authentication (#2908) As mentioned in aws/aws-sdk-go-v2#2370, AWS SDK for Go v2 releases after 2023/11/15 broke compatibility with all previous releases. Resolves #2882 Signed-off-by: Patrick Decat <pdecat@gmail.com>
Thank you @pdecat for the fix PR here which is now merged. Our CI/CD build with the fix can be pulled from the following locations: NOTE: a946b82 is the git SHA for kaniko that has the fix PR included
@ChristopherKlinge + other folks in the thread - can you attempt to use the CI/CD image(s) linked above and reply here validating/invalidating that kaniko w/ this fix PR resolves the previously seen AWS ECR auth regression? Thanks! |
@aaron-prindle We've had an open dependency update to v1.19.0 and then to v1.19.1, which has been broken due to this in our CI. Seeing your request above, I moved that over to the |
@aaron-prindle I can confirm that the `a946b82f22240eb8e3f7e73aaf0e592a323fa466-debug` works on GitLab CI that pushes to AWS ECR. |
|
Actual behavior
I noticed that the 1.19 build was just promoted for the
debug
tag in gcr and that since this occurred all our builds have been failing. We've had a build process working for years now ondebug
without more version locking than that. However, it appears 1.19 breaks something. Here's the command and log...Expected behavior
That same exact command run on the same exact runner in the same exact instance in the same exact environment when run with any version of debug released in the last two years including 1.18 works perfectly. The setup here is we are using an AWS IAM instance role on an EC2 instance which is running a "gitlab runner" to perform our Kaniko builds. I don't want to use "docker-in-docker" for security reasons so am using Kaniko so we can do Docker builds from within' docker. Gitlab runner is configured to run it's executor as docker images, thus the kaniko image above. The way I fixed this issue is I version pinned to use
gcr.io/kaniko-project/executor:v1.18.0-debug
instead of simplygcr.io/kaniko-project/executor:debug
. To confirm the issue, I temporarily pinned using the v1.19.0-debug image and the issue re-surfaced. I do not know enough about Kaniko to gather more debug data, if needed please provide me recommendation for command(s)/args to add to provide more logs to you about the possibly underlying issue.Note: I searched through the issues and couldn't find a duplicate, sorry if there is one!
The text was updated successfully, but these errors were encountered: