Bad error message if instance profile assume role policy is wrong

[rvandegrift]

We had an instance profile with a misconfigured role - the assume role policy didn't permit ec2.amazonaws.com. awscli gets very confusing:

ubuntu@i-7b4dc7eb:~$ aws --region us-east-1 ec2 describe-instances

'AccessKeyId'
ubuntu@i-7b4dc7eb:~$

It'd be nice if awscli could return the helpful error message from the meta-data API:

ubuntu@i-7b4dc7eb:~$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/testing-node-role
{
  "Code" : "AssumeRoleUnauthorizedAccess",
  "Message" : "EC2 cannot assume the role testing-node-role.  Please see documentation at http://docs.amazonwebservices.com/IAM/latest/UserGuide/RolesTroubleshooting.html.",
  "LastUpdated" : "2016-07-11T18:09:39Z"
}

Thanks,
Ross

[kyleknap]

I can confirm this. Here is my debug logs:

$ aws s3 ls --debug
2016-07-12 19:30:37,299 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.10.33 Python/2.7.10 Linux/4.4.11-23.53.amzn1.x86_64 botocore/1.4.23
2016-07-12 19:30:37,299 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug']
2016-07-12 19:30:37,300 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7fefe98bec80>
2016-07-12 19:30:37,300 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fefe9c73f50>
2016-07-12 19:30:37,300 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2016-07-12 19:30:37,300 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7fefe98c4578>
2016-07-12 19:30:37,301 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,301 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ls: calling handler <function add_waiters at 0x7fefe98c4578>
2016-07-12 19:30:37,302 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,302 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,302 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fefe98c94d0>
2016-07-12 19:30:37,302 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,302 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,303 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fefe98c94d0>
2016-07-12 19:30:37,303 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2016-07-12 19:30:37,307 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2016-07-12 19:30:37,308 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 23
2016-07-12 19:30:37,310 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2016-07-12 19:30:37,311 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/DataPipelineDefaultRole HTTP/1.1" 200 267
2016-07-12 19:30:37,312 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/awscli/clidriver.py", line 186, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/commands.py", line 190, in __call__
    parsed_globals)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/subcommands.py", line 438, in _run_main
    super(ListCommand, self)._run_main(parsed_args, parsed_globals)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/subcommands.py", line 424, in _run_main
    parsed_globals.verify_ssl)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/subcommands.py", line 417, in get_client
    config=config)
  File "/usr/lib/python2.7/dist-packages/botocore/session.py", line 808, in create_client
    credentials = self.get_credentials()
  File "/usr/lib/python2.7/dist-packages/botocore/session.py", line 442, in get_credentials
    'credential_provider').load_credentials()
  File "/usr/lib/python2.7/dist-packages/botocore/credentials.py", line 1017, in load_credentials
    creds = provider.load()
  File "/usr/lib/python2.7/dist-packages/botocore/credentials.py", line 473, in load
    metadata = fetcher.retrieve_iam_role_credentials()
  File "/usr/lib/python2.7/dist-packages/botocore/utils.py", line 198, in retrieve_iam_role_credentials
    'access_key': data[role_name]['AccessKeyId'],
KeyError: 'AccessKeyId'
2016-07-12 19:30:37,493 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

'AccessKeyId'

Looks like we should check if there is an error message if the access or secret access keys are missing.

[rsalmond]

Based on the above I would suggest this bug should be raised against botocore rather than this repo, since that's the component which cycles through various AWS auth methods trying and then ultimately failing to get access. awscli is just relaying the exception.

[jobwat]

For the next guy coming here with the same error and no clue what's happening:
For me the badly setup IAM Role part was the "Trust entities".
I had rds.amazonaws.com instead of ec2.amazonaws.com
Good luck! :)

[srvvr]

thanks @jobwat - that saved my butt today!!!

[rdonkin]

Since this is such a poor error, here's how to get the underlying error - use your IAM role name in the curl command from this AWS troubleshooting section. In my case, the EC2 service (ec2.amazonaws.com) was not authorized as a principal allowed to assume the IAM role.

[diepes]

I was just getting a non-descript hex string back
'c3f484259f5e6992259fee1633847ca7128c'

After adding --debug, found

Traceback (most recent call last):
  File "botocore/credentials.py", line 297, in __getitem__
PermissionError: [Errno 13] Permission denied: '/home/<user>/.aws/cli/cache/c3f4849f5e625992259fee1633847ca7128c.json'

During handling of the above exception, another exception occurred:

...
KeyError: 'c3f484259f5e6992259fee1633847ca7128c'
2020-12-04 09:28:39,143 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
...

[tim-finnigan]

It looks like this is the current error message you would see:

An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.

@rvandegrift do you think that message is sufficient or did you have any other feedback?

[rvandegrift]

@tim-finnigan No - it might be better than before, but it's not correct. That message would lead me to look at the IAM policies assigned to the instance profile role instead of the role's assume role policy.

@rsalmond's point in #2060 (comment) seems right. botocore should detect that the instance profile failed due to this misconfiguration, and raise a more useful exception. Then, awscli can handle that exception however it does to provide the feedback to the user.

[tim-finnigan]

Hi @rvandegrift, thanks for your feedback. I created an instance profile and received the following error when testing this:
aws sts assume-role --role-arn "arn:aws:iam::<masked>:role/test-role" --role-session-name AWSCLI-Session

An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::<masked>:assumed-role/test-role/AWSCLI-Session is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<masked>:role/test-role

Does that error message provide the clarity you were looking for? Please let me know if I misunderstood your workflow in this example.

[rvandegrift]

@tim-finnigan that's not quite the right scenario. You have a correctly configured IAM profile, but the attached policies don't permit the instance to call sts:AssumeRole. This issue is about misconfigured IAM profiles.

I just reproduced this. The current behavior is better - it doesn't crash. But it's just reported as a generic failure to load credentials:

admin@ip-172-31-11-44:~$ aws sts get-caller-identity
Unable to locate credentials. You can configure credentials by running "aws configure".

Ideally, the error from the metadata service could be passed to the user, since it pinpoints the problem exactly:

$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/wooo
{
  "Code" : "AssumeRoleUnauthorizedAccess",
  "Message" : "EC2 cannot assume the role wooo.  Please see documentation at https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html#troubleshoot_iam-ec2_errors-info-doc.",
  "LastUpdated" : "2021-11-11T23:12:38Z"
}

Here's a full walk-through to reproduce this. You'll need to substitute appropriate values for your subnet, security groups, and key name when creating the instance. NB: the EC2 console won't let you launch an instance with this IAM role, you need to use the cli.

First, create the IAM role and instance profile:

$ cat arpd.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "lambda.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
$ aws  iam create-role --role-name wooo --assume-role-policy-document file://arpd.json
{
    "Role": {
        "Path": "/",
        "RoleName": "wooo",
        "RoleId": "AROAUFLHAGW3CX7LLDI3C",
        "Arn": "arn:aws:iam::012345678901:role/wooo",
        "CreateDate": "2021-11-11T23:06:03Z",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": [
                            "lambda.amazonaws.com"
                        ]
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}
$ aws  iam create-instance-profile --instance-profile-name wooo
{
    "InstanceProfile": {
        "Path": "/",
        "InstanceProfileName": "wooo",
        "InstanceProfileId": "AIPAUFLHAGW3NRHUFGGCH",
        "Arn": "arn:aws:iam::012345678901:instance-profile/wooo",
        "CreateDate": "2021-11-11T23:07:11Z",
        "Roles": []
    }
}
$ aws  iam add-role-to-instance-profile --instance-profile-name wooo --role-name wooo

Note that the assume role policy does not permit ec2.amazonaws.com - that's the scenario in this issue.

Second, launch an instance using this profile:

$ aws  ec2 run-instances \
    --region us-west-2 \
    --image-id ami-0c510e75579d98979 \
    --instance-type t4g.nano \
    --subnet-id subnet-2c5bcd71 \
    --iam-instance-profile Name=wooo \
    --block-device-mappings 'DeviceName=/dev/xvda,Ebs={VolumeType=gp2,VolumeSize=8}' \
    --security-group-ids sg-0dd88689cc8361feb sg-34613a0a \
    --key-name user-ross
{
    "Groups": [],
    "Instances": [
        {
            "AmiLaunchIndex": 0,
            "ImageId": "ami-0c510e75579d98979",
            "InstanceId": "i-0b8dd3d02584ba40f",
            ...

Wait a bit for a public IP to be assigned... then login, and:

$ ssh admin@54.213.84.155
admin@ip-172-31-11-44:~$ aws sts get-caller-identity
Unable to locate credentials. You can configure credentials by running "aws configure".
admin@ip-172-31-11-44:~$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/wooo
{
  "Code" : "AssumeRoleUnauthorizedAccess",
  "Message" : "EC2 cannot assume the role wooo.  Please see documentation at https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html#troubleshoot_iam-ec2_errors-info-doc.",
  "LastUpdated" : "2021-11-11T23:12:38Z"
}

[tim-finnigan]

@rvandegrift thanks for providing that walk-through, I was able to reproduce the behavior you’re seeing. But I don’t think this has to do with the role:

Unable to locate credentials. You can configure credentials by running "aws configure".

In this premium support article on troubleshooting CLI issues with EC2 it says to “Verify that the AWS CLI is installed and configured correctly.”

If you configure your CLI do you still get an error when running aws sts get-caller-identity?

[rvandegrift]

@tim-finnigan If there's no config, awscli falls back to boto3's credential discovery process. Among other things, this checks the EC2 metadata service. When an instance has an IAM instance profile assigned, the metadata service tries to provide temporary credentials for the associated IAM role. EC2 calls STS on your behalf and provides the creds at the URL mentioned above - hence why the assume role policy must permit ec2.amazonaws.com. If everything is setup correctly, awscli will automatically use those credentials. For more on this functionality, see [1].

In the misconfiguration that this issue is about, EC2 is unable to call STS because the assume role policy does not permit it. The metadata service's error (now [2], but my first post has an older link that's also helpful) explains how to fix the issue - but awscli does not pass that information back to the user.

To verify all of this, follow the above but change the assume role policy on the IAM role to permit ec2.amazonaws.com. Your cli will work with no configuration (you'll need to specify a region for some services).

boto3's credential discovery is documented at [3]. Nowadays, most (all?) of the official SDKs support a similar mechanism so most software that interacts with AWS can run without any manual configuration.

[1] - https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html
[2] - https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html#troubleshoot_iam-ec2_no-keys
[3] - https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials

[tim-finnigan]

Thanks @rvandegrift for the explanation and sorry for the confusion on my end! I understand what you’re saying now and could reproduce that. I can see the need for a clearer error message in this scenario.