How can you check that an identity has been correctly applied?

[dhirschfeld]

I've got a container which I have assigned an aadpodidentity label to match an AzureIdentity identity object in my cluster which references a Managed Identity which has been assigned permission to access a database.

The pod has the label, the identity has the correct access, however the pod can't connect to the database.

From within the pod, how can I check that that the Azure identity really is the one implied by the pod label?

i.e. is there some sort of whoami tool for Azure Active Directory which I can use to verify the container identity is actually correct?

[aramase]

I've got a container which I have assigned an aadpodidentity label to match an AzureIdentity identity object in my cluster which references a Managed Identity which has been assigned permission to access a database.

The label required on the pod is aadpodidbinding. If that's not the label set, then MIC will not assign the identity for the pod.

From within the pod, how can I check that that the Azure identity really is the one implied by the pod label?

You can check the status of the AzureAssignedIdentity if using pod identity in standard mode (MIC and NMI deployed). The AzureAssignedIdentity name format is <pod name>-<namespace>-<azure identity name>. The status field in AzureAssignedIdentity will be set to Assigned if the identity is assigned. If there is no AzureAssignedIdentity for the pod, then the label defined in the pod spec is incorrect.

You can also check the token requests in NMI pod that'll log the redacted clientID for which token is being requested.

[dhirschfeld]

Thanks @aramase that's exactly what I was looking for!

Unfortunately, the identity appears to have been assigned correctly so my bug hunting continues down another avenue... :/

[dhirschfeld]

I tried:

az login --identity --allow-no-subscriptions --debug

• https://azure.github.io/aad-pod-identity/docs/troubleshooting/#run-a-pod-to-validate-your-identity-setup

...and I'm seeing an exception getting the token:

urllib3.connectionpool: Starting new HTTP connection (1): 169.254.169.254:80
urllib3.connectionpool: http://169.254.169.254:80 "GET /metadata/identity/oauth2/token?resource=https%3A%2F%2Fmanagement.core.windows.net%2F&api-version=2018-02-01 HTTP/1.1" 400 168
msrestazure.azure_active_directory: MSI: Retrieving a token from http://169.254.169.254/metadata/identity/oauth2/token, with payload {'resource': 'https://management.core.windows.net/', 'api-version': '2018-02-01'}
cli.azure.cli.core.adal_authentication: throw requests.exceptions.HTTPError when doing MSIAuthentication: 
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/adal_authentication.py", line 167, in set_token
    super(MSIAuthenticationWrapper, self).set_token()
  File "/opt/az/lib/python3.6/site-packages/msrestazure/azure_active_directory.py", line 600, in set_token
    token_entry = self._vm_msi.get_token(self.resource)
  File "/opt/az/lib/python3.6/site-packages/msrestazure/azure_active_directory.py", line 649, in get_token
    token_entry = self._retrieve_token_from_imds_with_retry(resource)
  File "/opt/az/lib/python3.6/site-packages/msrestazure/azure_active_directory.py", line 704, in _retrieve_token_from_imds_with_retry
    raise HTTPError(request=result.request, response=result.raw)
requests.exceptions.HTTPError

cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception:
cli.azure.cli.core.util: Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/adal_authentication.py", line 167, in set_token
    super(MSIAuthenticationWrapper, self).set_token()
  File "/opt/az/lib/python3.6/site-packages/msrestazure/azure_active_directory.py", line 600, in set_token
    token_entry = self._vm_msi.get_token(self.resource)
  File "/opt/az/lib/python3.6/site-packages/msrestazure/azure_active_directory.py", line 649, in get_token
    token_entry = self._retrieve_token_from_imds_with_retry(resource)
  File "/opt/az/lib/python3.6/site-packages/msrestazure/azure_active_directory.py", line 704, in _retrieve_token_from_imds_with_retry
    raise HTTPError(request=result.request, response=result.raw)
requests.exceptions.HTTPError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 657, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 720, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 691, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/profile/custom.py", line 138, in login
    return profile.find_subscriptions_in_vm_with_msi(username, allow_no_subscriptions)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 355, in find_subscriptions_in_vm_with_msi
    msi_creds = MSIAuthenticationWrapper(resource=resource)
  File "/opt/az/lib/python3.6/site-packages/msrestazure/azure_active_directory.py", line 592, in __init__
    self.set_token()
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/adal_authentication.py", line 179, in set_token
    .format(err.response.status, err.response.reason))
azure.cli.core.azclierror.AzureResponseError: Failed to connect to MSI. Please make sure MSI is configured correctly.
Get Token request returned http error: 400, reason: Bad Request

cli.azure.cli.core.azclierror: Failed to connect to MSI. Please make sure MSI is configured correctly.
Get Token request returned http error: 400, reason: Bad Request
az_command_data_logger: Failed to connect to MSI. Please make sure MSI is configured correctly.
Get Token request returned http error: 400, reason: Bad Request

[dhirschfeld]

In my case there is no NMI pod on the same node - perhaps because the node has a taint.

☝️ could that be the problem?

[dhirschfeld]

Yup, that was the problem.