{Core} Quick patch for `managed_by_tenants` missing from response #17509

jiasli · 2021-03-30T06:37:55Z

Update: The rollback of ARM has completed at 2021-03-31 08:00 UTC. This quick patch is not longer needed.

⚠ WARNING

This PR is a workaround for an ongoing ARM incident. The ARM service team is investigating with highest priority. We will close this PR once ARM service is fixed.

Symptom

az login fails with

Invalid profile is used for cloud 'AzureCloud'. To configure the cloud profile, 
run `az cloud set --name AzureCloud --profile <profile>(e.g. 2019-03-01-hybrid)`. 
For more information about using Azure CLI with Azure Stack, see 
https://docs.microsoft.com/azure-stack/user/azure-stack-version-profiles-azurecli2

or (Azure CLI versions smaller than 2.15.0)

azure/cli/core/_profile.py, ln 276, in _normalize_properties
    subscription_dict[_MANAGED_BY_TENANTS] = [{_TENANT_ID: t.tenant_id} for t in s.managed_by_tenants]
TypeError: 'NoneType' object is not iterable

Root Cause Analysis

Azure CLI started using Subscriptions - List API /subscriptions?api-version=2019-06-01 since Feb 2020 for public cloud AzureCloud ([Core][Profile] Support lighthouse multi-tenant subscription #11886). This call is made during az login to fetch the subscriptions.
Azure Stack returns the response for /subscriptions?api-version=2019-06-01 with older API version 2016-06-01 format without managedByTenants property ([Resources] Response from Azure Stack doesn't comply with the spec subscriptions.json azure-rest-api-specs#9567).
Az CLI checks this explicitly and shows the above error message to the user implying "Invalid profile is used for Azure Stack cloud and the profile is set to latest, you need to change the API profile to something like 2019-03-01-hybrid" ({Core} az login: Capture managed_by_tenants error #15550).
Due to the ARM incident, now in public cloud AzureCloud, the /subscriptions API's response omits managedByTenants in some regions. As a consequence, the lack of managedByTenants
1. breaks Azure Lighthouse functionality
2. makes Azure CLI think the Azure Stack issue ([Resources] Response from Azure Stack doesn't comply with the spec subscriptions.json azure-rest-api-specs#9567) is hit, so the user is presented with the above error message implying you are logged in to Azure Stack. This causes az login to fail.

Changes in this PR

This PR provides a quick patch which bypasses the check on managedByTenants and thus bypasses the ARM service incident.

Apply the quick patch

The following commands replaces the _profile.py file of Azure CLI installed on local machine or Azure DevOps agent:

# Ubuntu, `sudo` may be needed
curl https://raw.githubusercontent.com/Azure/azure-cli/fix-managed_by_tenants/src/azure-cli-core/azure/cli/core/_profile.py --output /opt/az/lib/python3.6/site-packages/azure/cli/core/_profile.py

# CentOS, `sudo` may be needed
curl https://raw.githubusercontent.com/Azure/azure-cli/fix-managed_by_tenants/src/azure-cli-core/azure/cli/core/_profile.py --output /usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/_profile.py

# Windows, "Run as administrator" may be needed
curl https://raw.githubusercontent.com/Azure/azure-cli/fix-managed_by_tenants/src/azure-cli-core/azure/cli/core/_profile.py --output "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\core\_profile.py"

Remove the quick patch

After the ARM incident is fixed, the patch needs to be removed.

Azure DevOps agent: Remove the above line from the script.
Linux: The patch will be overwritten automatically when the next Azure CLI version is installed. You may also reinstall the current Azure CLI to get the patch removed.

Windows ("Run as administrator" may be needed):

del "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\core\_profile.py"

Additional information

Microsoft internal tracking ticket for the ARM incident: IcM 234164333

yonzhan

LGTM

jiasli · 2021-03-30T07:21:24Z

src/azure-cli-core/azure/cli/core/_profile.py

-                                       "For more information about using Azure CLI with Azure Stack, see "
-                                       "https://docs.microsoft.com/azure-stack/user/azure-stack-version-profiles-azurecli2"
-                                       .format(cloud_name=self.cli_ctx.cloud.name))
+                if getattr(s, 'managed_by_tenants', None) is not None:


If managed_by_tenants doesn't exist, getattr returns None and this if evaluates to False

If managed_by_tenants is [], getattr returns [] and this if evaluates to True, so [] is preserved

2 years later, I feel it was lucky that we didn't merge this PR.

There are 3 conditions:

# Condition 1: managedByTenants missing { } # Condition 2: managedByTenants is null { "managedByTenants": null } # Condition 3: managedByTenants is empty list [] { "managedByTenants": [] }

This statement doesn't take condition 2 "managedByTenants": null into consideration. Even though this has never been observed, if ARM returns "managedByTenants": null, Azure CLI output will omit managedByTenants.

Also, for condition 1, the absence of managedByTenants from ARM's Subscriptions - List REST API will propagate all the way to Azure CLI output, causing breaking change.

yufeih · 2021-03-30T07:40:17Z

When will this fix be patched to azure pipeline? We have a bunch of AzureCLI@2 task failures blocking our release pipeline.

pahlawanUtara · 2021-03-30T07:47:31Z

Thank you so much to the product group for this patch!

giacris82 · 2021-03-30T16:42:43Z

Sorry but if I apply the work around on windows it doesn't work. Which version of az cli i need for this workaround to work?

jiasli · 2021-03-30T23:59:57Z

@giacris82, the patch is for Azure CLI installed with MSI.

How did you install Azure CLI? Could you check the installation location?

# PowerShell
Get-Command az

# CMD
where az

kuzhao · 2021-03-31T05:53:13Z

Are we sure this one works out 100%? Below is what we saw on Ubuntu after the quick patch:

az login --service-principal --username **** --password **** --tenant ****
The command failed with an unexpected error. Here is the traceback:

cannot import name 'BasicTokenCredential'
Traceback (most recent call last):
  File "/usr/lib64/az/lib/python3.6/site-packages/knack/cli.py", line 215, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 654, in execute
    raise ex
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 718, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 711, in _run_job
    six.reraise(*sys.exc_info())
  File "/usr/lib64/az/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 688, in _run_job
    result = cmd_copy(params)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 325, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 545, in default_command_handler
    return op(**command_args)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/command_modules/profile/custom.py", line 150, in login
    use_cert_sn_issuer=use_cert_sn_issuer)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 209, in find_subscriptions_on_login
    username, sp_auth, tenant, self._ad_resource_uri)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 923, in find_from_service_principal_id
    result = self._find_using_specific_tenant(tenant, token_entry[_ACCESS_TOKEN])
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 1014, in _find_using_specific_tenant
    from azure.cli.core.adal_authentication import BasicTokenCredential
ImportError: cannot import name 'BasicTokenCredential'

jiasli · 2021-03-31T05:59:18Z

@kuzhao, BasicTokenCredential is introduced in Azure CLI 2.16.0 (#15711), so you should be running an Azure CLI older than that.

The patch is only guaranteed to work with the latest Azure CLI 2.21.0. Please update to the latest Azure CLI and apply the patch.

giacris82 · 2021-03-31T07:49:06Z

@kuzhao, BasicTokenCredential is introduced in Azure CLI 2.16.0 (#15711), so you should be running an Azure CLI older than that.

The patch is only guaranteed to work with the latest Azure CLI 2.21.0. Please update to the latest Azure CLI and apply the patch.

It wass not clear

Quick patch for managed_by_tenants missing from response

5a34f30

yonzhan self-requested a review March 30, 2021 06:49

yonzhan assigned jiasli Mar 30, 2021

yonzhan added this to the S185 milestone Mar 30, 2021

yonzhan approved these changes Mar 30, 2021

View reviewed changes

keep []

8ba86f9

jiasli commented Mar 30, 2021

View reviewed changes

yufeih mentioned this pull request Mar 30, 2021

Apply az login patch dotnet/docfx#7251

Closed

timja mentioned this pull request Mar 30, 2021

Hotfix Azure CLI hmcts/cnp-aks-client#16

Merged

This was referenced Mar 30, 2021

Errors when az login #17504

Closed

Error using az login to Azure Portal #17514

Closed

az login: Invalid profile error for non-AzureStack cloud #17513

Closed

fengzhou-msft mentioned this pull request Mar 30, 2021

az aks get-credentials does not completed due to The command requires the extension aks-preview #17497

Closed

fengzhou-msft approved these changes Mar 30, 2021

View reviewed changes

egodigitus mentioned this pull request Mar 30, 2021

error az login #17515

Closed

jiasli marked this pull request as ready for review March 31, 2021 07:42

jiasli requested review from evelyn-ys, houk-ms, jsntcy and Juliehzl as code owners March 31, 2021 07:42

jiasli mentioned this pull request Mar 31, 2021

{Core} Remove the check on managed_by_tenants #17526

Merged

jiasli marked this pull request as draft March 31, 2021 09:31

jiasli closed this Mar 31, 2021

jiasli deleted the fix-managed_by_tenants branch April 8, 2021 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{Core} Quick patch for `managed_by_tenants` missing from response #17509

{Core} Quick patch for `managed_by_tenants` missing from response #17509

jiasli commented Mar 30, 2021 •

edited

Loading

yonzhan left a comment

jiasli Mar 30, 2021

jiasli Jan 30, 2023 •

edited

Loading

yufeih commented Mar 30, 2021 •

edited

Loading

pahlawanUtara commented Mar 30, 2021

giacris82 commented Mar 30, 2021

jiasli commented Mar 30, 2021

kuzhao commented Mar 31, 2021

jiasli commented Mar 31, 2021 •

edited

Loading

giacris82 commented Mar 31, 2021

{Core} Quick patch for managed_by_tenants missing from response #17509

{Core} Quick patch for managed_by_tenants missing from response #17509

Conversation

jiasli commented Mar 30, 2021 • edited Loading

Update: The rollback of ARM has completed at 2021-03-31 08:00 UTC. This quick patch is not longer needed.

⚠ WARNING

Symptom

Root Cause Analysis

Changes in this PR

Apply the quick patch

Remove the quick patch

Additional information

yonzhan left a comment

Choose a reason for hiding this comment

jiasli Mar 30, 2021

Choose a reason for hiding this comment

jiasli Jan 30, 2023 • edited Loading

Choose a reason for hiding this comment

yufeih commented Mar 30, 2021 • edited Loading

pahlawanUtara commented Mar 30, 2021

giacris82 commented Mar 30, 2021

jiasli commented Mar 30, 2021

kuzhao commented Mar 31, 2021

jiasli commented Mar 31, 2021 • edited Loading

giacris82 commented Mar 31, 2021

{Core} Quick patch for `managed_by_tenants` missing from response #17509

{Core} Quick patch for `managed_by_tenants` missing from response #17509

jiasli commented Mar 30, 2021 •

edited

Loading

jiasli Jan 30, 2023 •

edited

Loading

yufeih commented Mar 30, 2021 •

edited

Loading

jiasli commented Mar 31, 2021 •

edited

Loading