Error when updating all packages

[tyrken]

When using the latest AMI (amazon-eks-node-v25) as input to a further Packer run for customisations, one step does a mass yum update -y ... this today started erroring (worked 10 days ago). Repeating that command on a mid-erroring packer build run showed the detail:

...
Running transaction
  Updating   : python-urllib3-1.24.1-1.amzn2.0.1.noarch                                                                                        1/2 
Error unpacking rpm package python-urllib3-1.24.1-1.amzn2.0.1.noarch
error: unpacking of archive failed on file /usr/lib/python2.7/site-packages/urllib3/packages/ssl_match_hostname: cpio: rename
  Verifying  : python-urllib3-1.24.1-1.amzn2.0.1.noarch                                                                                        1/2 
python-urllib3-1.10.2-3.amzn2.0.1.noarch was supposed to be removed but is not!
  Verifying  : python-urllib3-1.10.2-3.amzn2.0.1.noarch                                                                                        2/2 

Failed:
  python-urllib3.noarch 0:1.10.2-3.amzn2.0.1                               python-urllib3.noarch 0:1.24.1-1.amzn2.0.1                              

Complete!

This looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1187057

Given I can see this repo does include a "sudo pip install" at

amazon-eks-ami/install-worker.sh

Line 42 in c1934bb

sudo pip install --upgrade awscli

I'm guessing that is what happened. The output of pip freeze shows urllib3 1.24.1, whereas yum thinks that 1.10.2 is still installed.

What you expected to happen:
Given AWS provides awscli as a yum package, can you switch to installing that and python-pip3 via yum instead of messing with get-pip.py and sudo pip.

Anything else we need to know?:
Issuing the RH suggested workaround of sudo pip uninstall urllib3 seems to get things going for now.

Environment:

• AWS Region: eu-west-1
• Instance Type(s): m4.large
• EKS Platform version : n/a
• Kubernetes version : n/a
• AMI Version: amazon-eks-node-v25 (ami-00c3b2d35bddd4f5c)
• Kernel (e.g. uname -a): Linux ip-192-168-201-184.eu-west-1.compute.internal 4.14.72-73.55.amzn2.x86_64 Template is missing source_ami_id in the variables section #1 SMP Thu Sep 27 23:37:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Release information (run cat /tmp/release on a node): No such file!

[ac-hibbert]

I have this problem too with the latest AMI, I can probably work around this for now, just wandering when this will be run by AWS to create a new public AMI?

[brandond]

It's like nobody even tests these things.

[tylernisonoff]

@brandond this is an open-source project...if you're going to be that rude about it, submit a PR instead.

[brandond]

@tylernisonoff I can't exactly submit a PR to have AWS publish new non-broken AMIs though. All I can do is whine to my TAM, point at our giant bill, and hope that they do better next time.

[tylernisonoff]

great, I'm sure you can find ways to not contribute to open-source toxicity while you're waiting then.

[micahhausler]

Hi all, we'll be releasing an updated AMI next week or two with a fix for this.

We do test our AMIs, but our testing up to this point hasn't involved installing additional RPM packages after AMI build. We do a yum update at AMI build time, and don't re-run it at instance start in order to produce an AMI with predictable packages, instead of whatever is available at the time you boot your AMI.

As always, if you need something changed in the AMI and can't or don't want to wait for a release from us, you are free to build and run your own AMIs.

Locking this conversation