[ARM] Migrate resource to track2 SDK #17783

zhoxing-ms · 2021-04-20T09:41:42Z

Description

This PR upgrades the Python SDK version of azure-mgmt-resource from 12.1.0 to 16.1.0

Here are the main changes for resource migration to track2:

LRO operations are renamed to begin_xxx_xxx. For example in track1, the name is create_or_update; while in track2, the name is begin_create_or_update.
When api-version is less than 2019-10-01, the parameters of all deployment operations are changed from DeploymentProperties to Deployment, which is consistent with the SDK after version 2019-10-01.
The objects of input parameters for some methods are further encapsulated:

begin_what_if(): DeploymentWhatIfProperties --> DeploymentWhatIf
begin_what_if_at_subscription_scope()/...: DeploymentWhatIfProperties --> ScopedDeploymentWhatIf
begin_export_template(): flattened parameters --> ExportTemplateRequest
begin_move_resources(): flattened parameters --> ResourcesMoveInfo
links_client.create_or_update(): ResourceLinkProperties --> ResourceLink
tags.create_or_update_at_scope(): flattened parameters --> TagsResource
tags.update_at_scope(): flattened parameters --> TagsPatchResource

In method get() of ResourcesOperations, some parameters are passed in through **kwargs instead
CloudError and HttpOperationError are replaced by HttpResponseError
In track 2, the logic corresponding to the class name of Codegen generating operations changes, so DeploymentOperations becomes DeploymentOperationsOperations
For more details, please refer to this issue: remove double "operations" in operation group names autorest.python#910 (comment)
config in management client becomes a private variable _config.
In method policy_definitions.create_or_update_at_management_group(), the positions of parameters management_groupand parameters are reversed.
generate_client_request_id in ResourceManagementClientConfiguration no longer needs, SDK automatically add x-ms-request-id now through the RequestIdPolicy
The accept_language is no longer supported in ResourceManagementClientConfiguration, so the fixed value en-US in track 2 is hard coded.
The pipeline and policy in management client migrated from msrest to azure-core. And when doing ARM deployment, the pipeline needs to add custom policy JsonCTemplatePolicy() to support JSON templates.
get_by_id() and get() of ResourcesOperations no longer support parameter raw, but passes in a callback method cls instead. So the custom callback function add_response_body() is used to realize the logic of include_response_body.
The resource action lacks generic POST operation in Swagger, so for the time being, AzureOperationPoller of msrestis still used for low-cost migration. And the request way is changed to pipeline.run in track 2.
Before version 2019-08-01, get_by_id() and get() of ResourcesOperations lacked parameter api-version. The problem was solved by modifying the Swagger to regenerate a new Python SDK. PR link: Swagger PR
Fix some live tests due to the changes in track2.

Known commands that are not covered by test

appservice domain create
appservice ase create

Migration for extension
PR link: Azure/azure-cli-extensions#3355

Live Test
result link

TODO

Refactor the temporary code in the subsequent PR Refactor the temporary code in migration track 2 of resource module #18023

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.

This checklist is used to make sure that common guidelines for a pull request are followed.

The PR title and description has followed the guideline in Submitting Pull Requests.
I adhere to the Command Guidelines.
I adhere to the Error Handling Guidelines.

yonzhan · 2021-04-20T11:01:31Z

ARM

jiasli · 2021-04-22T02:27:19Z

src/azure-cli/azure/cli/command_modules/resource/custom.py

-            ],
-            sender=PipelineRequestsHTTPSender(RequestsHTTPSender(smc.config))
+        from azure.core.pipeline import Pipeline
+        smc._client._pipeline._impl_policies.append(JsonCTemplatePolicy())


Consider using per_call_policies introduced by Azure/azure-sdk-for-python#17340.

It requires azure-core 1.13.0 which is bumped by #17671.

jiasli · 2021-04-22T02:29:45Z

src/azure-cli/azure/cli/command_modules/resource/custom.py

@@ -3188,34 +3195,35 @@ def invoke_action(self, action, request_body):
        # Construct headers
        header_parameters = {}
        header_parameters['Content-Type'] = 'application/json; charset=utf-8'
-        if self.rcf.resources.config.generate_client_request_id:
-            header_parameters['x-ms-client-request-id'] = str(uuid.uuid4())


_prepare_client_kwargs_track2 from core already configures x-ms-client-request-id.

azure-cli/src/azure-cli-core/azure/cli/core/commands/client_factory.py

Lines 156 to 158 in 039fa21

# Prepare x-ms-client-request-id header, used by RequestIdPolicy

if 'x-ms-client-request-id' in cli_ctx.data['headers']:

client_kwargs['request_id'] = cli_ctx.data['headers']['x-ms-client-request-id']

jiasli · 2021-04-22T02:57:11Z

Migrate for extension

I would suggest we provide a method for users to revert back to old versions of Azure CLI because these are not small changes and may break extensions, especially those not in azure-cli-extensions (#13653).

jiasli · 2021-04-22T03:20:50Z

Please kindly put this PR on hold until #17671 and #17812 are merged so that they can be tested together.

Update: ✔ #17671 and #17812 are merged. Please merge from dev.

src/azure-cli/azure/cli/command_modules/acs/custom.py

kairu-ms · 2021-04-23T03:05:21Z

src/azure-cli/azure/cli/command_modules/network/custom.py

+    Deployment = cmd.get_models('Deployment', resource_type=ResourceType.MGMT_RESOURCE_RESOURCES)
+    deployment = Deployment(properties=properties)


Can we get the Deployment model from SDK which api-version less than 2019-10-01

Yes, because in track 2, the SDK before version 2019-10-01 has added this model
Related code: code link

kairu-ms · 2021-04-23T03:07:04Z

src/azure-cli/azure/cli/command_modules/network/custom.py

+    Deployment = cmd.get_models('Deployment', resource_type=ResourceType.MGMT_RESOURCE_RESOURCES)
+    deployment = Deployment(properties=properties)


Same concern as above.

qwordy

Approve vm part

zhoxing-ms force-pushed the resource_track2 branch from 383c07b to cbaa422 Compare April 20, 2021 09:47

yonzhan assigned zhoxing-ms Apr 20, 2021

yonzhan requested review from Juliehzl and jiasli April 20, 2021 11:01

yonzhan added this to the S186 milestone Apr 20, 2021

yonzhan requested review from jsntcy, evelyn-ys, msyyc, fengzhou-msft and kairu-ms April 20, 2021 11:02

zhoxing-ms force-pushed the resource_track2 branch 3 times, most recently from fe95df2 to 782fff3 Compare April 21, 2021 06:27

zhoxing-ms marked this pull request as ready for review April 21, 2021 14:57

zhoxing-ms requested review from houk-ms, qwordy and yungezz as code owners April 21, 2021 14:57

jiasli reviewed Apr 22, 2021

View reviewed changes

zhoxing-ms force-pushed the resource_track2 branch from 2bed04b to 344efa0 Compare April 22, 2021 15:23

Juliehzl reviewed Apr 23, 2021

View reviewed changes

src/azure-cli/azure/cli/command_modules/acs/custom.py Show resolved Hide resolved

kairu-ms reviewed Apr 23, 2021

View reviewed changes

fengzhou-msft approved these changes Apr 23, 2021

View reviewed changes

zhoxing-ms force-pushed the resource_track2 branch 2 times, most recently from b3efbbb to e247f5d Compare April 25, 2021 03:19

zhoxing-ms mentioned this pull request Apr 27, 2021

[ARM] az ts : Add UiFormDefinition file support to TemplateSpecs for GA (05/04) #17869

Merged

migrate resource to track 2

4c096c6

zhoxing-ms force-pushed the resource_track2 branch from 256d1dd to 4c096c6 Compare April 29, 2021 09:08

zhoxing-ms requested a review from zqingqing1 as a code owner April 29, 2021 09:08

Add temporary work around to add custom polcies

d763e8b

yonzhan modified the milestones: S186, S187 May 1, 2021

Merge from dev

5b7040d

kairu-ms approved these changes May 10, 2021

View reviewed changes

jiasli approved these changes May 10, 2021

View reviewed changes

zhoxing-ms mentioned this pull request May 10, 2021

Add compatible logic for the track 2 migration of resource Azure/azure-cli-extensions#3355

Merged

3 tasks

houk-ms approved these changes May 11, 2021

View reviewed changes

zhoxing-ms mentioned this pull request May 11, 2021

Bump azure-mgmt-resource to Track 2 15.0.0 #15530

Closed

evelyn-ys approved these changes May 11, 2021

View reviewed changes

qwordy approved these changes May 11, 2021

View reviewed changes

jsntcy approved these changes May 11, 2021

View reviewed changes

Supplement missing cases from live test

8a1576d

Juliehzl approved these changes May 12, 2021

View reviewed changes

zhoxing-ms merged commit 4a5d742 into Azure:dev May 12, 2021

jiasli deleted the resource_track2 branch May 12, 2021 09:02

jiasli mentioned this pull request May 12, 2021

{Core} Stop using vendored azure-mgmt-resource SDK #18047

Merged

jiasli mentioned this pull request Jun 2, 2021

az container create fails with AttributeError: 'ResourcesOperations' object has no attribute 'create_or_update' #18276

Closed

jiasli mentioned this pull request Jul 14, 2021

TypeError: __init__() got an unexpected keyword argument 'user_agent' in CI pipeline command-az extension add -n azure-cli-ml #18626

Closed

jiasli mentioned this pull request Sep 14, 2021

Cannot use the azure-mgmt-resourcegraph python library on the CLI #19528

Closed

zhoxing-ms mentioned this pull request Oct 26, 2021

appservice-kube extension Azure/azure-cli-extensions#3934

Merged

3 tasks

zhoxing-ms mentioned this pull request Dec 15, 2021

{Core} Remove the try-catch logic of CloudError in get_default_location_from_resource_group() #20739

Merged

3 tasks

zhoxing-ms mentioned this pull request Jan 11, 2022

[ARM] Bump version of azure-mgmt-managementgroups from 0.2.0 to 1.0.0 (Track 2 SDK) and add new sub commands for management-group #20942

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ARM] Migrate resource to track2 SDK #17783

[ARM] Migrate resource to track2 SDK #17783

zhoxing-ms commented Apr 20, 2021 •

edited

Loading

yonzhan commented Apr 20, 2021

jiasli Apr 22, 2021 •

edited

Loading

jiasli Apr 22, 2021

jiasli commented Apr 22, 2021

jiasli commented Apr 22, 2021 •

edited

Loading

kairu-ms Apr 23, 2021

zhoxing-ms Apr 23, 2021 •

edited

Loading

kairu-ms Apr 23, 2021

qwordy left a comment

	# Prepare x-ms-client-request-id header, used by RequestIdPolicy
	if 'x-ms-client-request-id' in cli_ctx.data['headers']:
	client_kwargs['request_id'] = cli_ctx.data['headers']['x-ms-client-request-id']

		Deployment = cmd.get_models('Deployment', resource_type=ResourceType.MGMT_RESOURCE_RESOURCES)
		deployment = Deployment(properties=properties)

[ARM] Migrate resource to track2 SDK #17783

[ARM] Migrate resource to track2 SDK #17783

Conversation

zhoxing-ms commented Apr 20, 2021 • edited Loading

yonzhan commented Apr 20, 2021

jiasli Apr 22, 2021 • edited Loading

Choose a reason for hiding this comment

jiasli Apr 22, 2021

Choose a reason for hiding this comment

jiasli commented Apr 22, 2021

jiasli commented Apr 22, 2021 • edited Loading

kairu-ms Apr 23, 2021

Choose a reason for hiding this comment

zhoxing-ms Apr 23, 2021 • edited Loading

Choose a reason for hiding this comment

kairu-ms Apr 23, 2021

Choose a reason for hiding this comment

qwordy left a comment

Choose a reason for hiding this comment

zhoxing-ms commented Apr 20, 2021 •

edited

Loading

jiasli Apr 22, 2021 •

edited

Loading

jiasli commented Apr 22, 2021 •

edited

Loading

zhoxing-ms Apr 23, 2021 •

edited

Loading