-
Notifications
You must be signed in to change notification settings - Fork 5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* add apitestErrorCode doc * add armRPC doc * add roundTripInconsistentProperty doc * small fix * update * update signalR armTemplate * update yaml * new file * add testScenario with armTemplate * update doc * update readme.md * update doc * update NOTE section * add signalRCreateOrUpdate example file * udpate doc * add generate test scenario section * update doc * add serviceFacbric test scenario file * update managedClusters.yaml * update generateABasicTestScenario.md * update features doc * add run api test gif
- Loading branch information
Showing
22 changed files
with
430 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| <!-- | ||
| Copyright (c) 2021 Microsoft Corporation | ||
| This software is released under the MIT License. | ||
| https://opensource.org/licenses/MIT | ||
| --> | ||
|
|
||
| # API testing error code | ||
|
|
||
| OAV api testing define some rules and check whether actual service response match with example. Example is very important for downstream SDK test code generation, code samples and docs. The goal to detect example quality issues and check service behaviors consistent with example. | ||
|
|
||
| ## Rule descriptions | ||
|
|
||
| #### INCORRECT_PROVISIONING_STATE | ||
|
|
||
| If service return 200 status code,it means the operation is succeed and finished. So the provisioning state should be one of the terminal states ["succeeded", "failed", "canceled", "ready", "created", "deleted"]. | ||
|
|
||
| The provisioning state is very important for downstream terraform or cli to manage resource status. For more details about provisioning state, please refer to this [ARM RPC provisioning state](https://github.com/Azure/azure-resource-manager-rpc/blob/master/v1.0/Addendum.md#provisioningstate-property) | ||
|
|
||
| #### RESPONSE_MISSING_VALUE | ||
|
|
||
| **Error message**: The response value is missing. Path: {}. Expected: {}. Actual: undefined | ||
|
|
||
| The example has defined response value, but actually the server doesn't return that value. | ||
|
|
||
| Example: | ||
|
|
||
| ```diff | ||
| { | ||
| "properties":{ | ||
| "targetType":"blobNfs", | ||
| "junctions":[ | ||
| { | ||
| "namespacePath":"/blobnfs" | ||
| } | ||
| ], | ||
| "blobNfs":{ | ||
| "target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs", | ||
| - "usageModel":"WRITE_WORKLOAD_15" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| #### RESPONSE_ADDITIONAL_VALUE | ||
|
|
||
| **Error message** Return additional response value. Path: {}. Expected: undefined. Actual: {} | ||
|
|
||
| The example doesn't define the response value, but service actually return this value. | ||
|
|
||
| Example: | ||
|
|
||
| ```diff | ||
| { | ||
| "properties":{ | ||
| "targetType":"blobNfs", | ||
| "junctions":[ | ||
| { | ||
| "namespacePath":"/blobnfs" | ||
| } | ||
| ], | ||
| "blobNfs":{ | ||
| "target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs", | ||
| "usageModel":"WRITE_WORKLOAD_15" | ||
| + "enableFeature": true | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| #### RESPONSE_INCONSISTENT_VALUE | ||
|
|
||
| **Error message** The actual response value is different from example. Path: {}. Expected: {}. Actual: {} | ||
|
|
||
| The service returned value is different from example value. | ||
|
|
||
| Example: | ||
|
|
||
| ```diff | ||
| { | ||
| "properties":{ | ||
| "targetType":"blobNfs", | ||
| "junctions":[ | ||
| { | ||
| "namespacePath":"/blobnfs" | ||
| } | ||
| ], | ||
| "blobNfs":{ | ||
| "target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs", | ||
| - "usageModel":"WRITE_WORKLOAD_15" | ||
| + "usageModel":"WORK_LOAD_14" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| #### ROUNDTRIP_INCONSISTENT_PROPERTY | ||
|
|
||
| **Error message** The property's value in the response is different from what was set in the request. Path: {}. Request: {}. Response: {} | ||
|
|
||
| Example: The sku in request parameters is `default`, but actual return is `standard`. | ||
|
|
||
| ```diff | ||
| { | ||
| "parameters":{ | ||
| "properties":{ | ||
| "name":"myService", | ||
| "SKU":"default" | ||
| } | ||
| }, | ||
| "responses":{ | ||
| "200":{ | ||
| "properties":{ | ||
| "name":"myService", | ||
| + "SKU":"standard" | ||
| - "SKU":"default" | ||
|
|
||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| ``` |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # Generate a basic test scenario file | ||
|
|
||
| ## Prerequisite | ||
|
|
||
| We use `oav` tools to generate basic test scenario. `oav` analyze swagger file and use swagger example as test scenario steps. | ||
|
|
||
| ## Introduction | ||
|
|
||
| `oav` support rule based test scenario file generation. We use this command to generate test scenario file. | ||
|
|
||
| `oav generate-static-test-scenario --readme <readme> --tag <tag> --rules <generated-rules>` | ||
|
|
||
| - readme: swagger readme file. | ||
| - tag: which tag to generate. oav will analyze swagger file under the tag and generate test scenario. | ||
|
|
||
| Example: | ||
|
|
||
|  | ||
|
|
||
| This command will load and analyze swagger and generate a basic test scenario file. | ||
|
|
||
| Result: the output contains two files | ||
|
|
||
| - test-scenarios/signalR.yaml: The test scenario file. | ||
| - readme.test.md: The entry for SDK test generation | ||
|
|
||
| The generated test scenario file: The generated test scenario file contains two steps. Create signalR and delete it. It's a basic test scenario and developer can add more step based on the basic test scenario file. | ||
|
|
||
| ``` | ||
| scope: ResourceGroup | ||
| testScenarios: | ||
| - description: Microsoft.SignalRService/signalR SignalR_CreateOrUpdate | ||
| steps: | ||
| - step: SignalR_CreateOrUpdate | ||
| exampleFile: ../examples/SignalR_CreateOrUpdate.json | ||
| - step: SignalR_Delete | ||
| exampleFile: ../examples/SignalR_Delete.json | ||
| ``` | ||
|
|
||
| ## Reference | ||
|
|
||
| - [oav](https://github.com/Azure/oav/tree/develop) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
104 changes: 104 additions & 0 deletions
104
documentation/test-scenario/testScenarioWithARMTemplate.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| # Test scenario integrate with armTemplate | ||
|
|
||
| ## Background | ||
|
|
||
| In some cases, we need to do some more complex operations before creating a resource. For example, | ||
|
|
||
| - Create Azure SignalR service with a global unique name. Using ARMTemplate to generate a random unique string. | ||
| - Create VM with a storage account. Using ARMTemplate to provision storage account and passing the storage account resourceId as VM creation parameter. | ||
|
|
||
| ## Examples | ||
|
|
||
| Here is an example about `generate unique resource name for signalR service` | ||
|
|
||
| #### Generate unique resource name | ||
|
|
||
| We use `armTemplate output` to overwrite `resourceName` variable and following `createResource` step will use this variable. Below is generate unique name armTemplate. This armTemplate output `resourceName` variables, so test scenario following step will using the output variable. | ||
|
|
||
| ```json | ||
| { | ||
| "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", | ||
| "contentVersion": "1.0.0.0", | ||
| "parameters": { | ||
| "resourceNamePrefix": { | ||
| "type": "string", | ||
| "defaultValue": "signalr-" | ||
| } | ||
| }, | ||
| "variables": { | ||
| "resourceName": "[concat(parameters('resourceNamePrefix'), uniqueString(resourceGroup().id))]" | ||
| }, | ||
| "resources": [], | ||
| "outputs": { | ||
| "resourceName": { | ||
| "type": "string", | ||
| "value": "[variables('resourceName')]" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| After we have this armTemplate, we could define current test scenario file. We defined `resourceName` variable globally. `./generate_unique_string.json` is armTemplate. | ||
|
|
||
| `SignalR_CreateOrUpdate.json` | ||
|
|
||
| ```json | ||
| { | ||
| "parameters": { | ||
| "parameters": { | ||
| "tag": { | ||
| "key1": "tag1" | ||
| }, | ||
| "properties":{ | ||
| ... | ||
| } | ||
| }, | ||
| "api-version": "2020-07-01-preview", | ||
| "subscriptionId": "00000000-0000-0000-0000-000000000000", | ||
| "resourceGroupName": "myResourceGroup", | ||
| "resourceName": "mySignalRService123xx" | ||
| }, | ||
| "responses": { | ||
| "200": { | ||
| "body": { | ||
| ... | ||
| } | ||
| }, | ||
| "201": { | ||
| "body": { | ||
| ... | ||
| } | ||
| }, | ||
| "202": {} | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| > NOTE: the example file `../examples/SignalR_CreateOrUpdate` has the same parameter name `resourceName`. So it will be automatically overwrite in runner. | ||
| ```yaml | ||
| scope: ResourceGroup | ||
| variables: | ||
| resourceName: "" | ||
| testScenarios: | ||
| - description: Microsoft.SignalRService/signalR CRUD | ||
| steps: | ||
| - step: Generate_Unique_string | ||
| armTemplateDeployment: ./generate_unique_string.json | ||
| - step: SignalR_checknameAvailability | ||
| exampleFile: ../examples/SignalR_CheckNameAvailability.json | ||
| - step: SignalR_CreateOrUpdate | ||
| exampleFile: ../examples/SignalR_CreateOrUpdate.json | ||
| - step: SignalR_ListKey | ||
| exampleFile: ../examples/SignalR_ListKeys.json | ||
| - step: SignalR_Delete | ||
| exampleFile: ../examples/SignalR_Delete.json | ||
| ``` | ||
| **Result**: | ||
|  | ||
| ## Reference | ||
| - [ARMTemplate deployment script](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-script-template) |
2 changes: 1 addition & 1 deletion
2
...e-manager/Microsoft.ServiceBus/preview/2021-01-01-preview/examples/SBOperations_List.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| { | ||
| "parameters": { | ||
| "api-version": "2021-01-01-preview" | ||
| "api-version": "2017-04-01" | ||
| }, | ||
| "responses": { | ||
| "200": { | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
14 changes: 14 additions & 0 deletions
14
...er/Microsoft.ServiceFabric/preview/2020-01-01-preview/test-scenarios/managedClusters.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| scope: ResourceGroup | ||
| testScenarios: | ||
| - description: >- | ||
| Microsoft.ServiceFabric/managedClusters Put a cluster with minimum | ||
| parameters | ||
| steps: | ||
| - step: Put a cluster with minimum parameters | ||
| exampleFile: ../examples/ManagedClusterPutOperation_example_min.json | ||
| - step: Delete a cluster | ||
| exampleFile: ../examples/ManagedClusterDeleteOperation_example.json | ||
| - step: Put a cluster with maximum parameters | ||
| exampleFile: ../examples/ManagedClusterPutOperation_example_max.json | ||
| - step: Delete a cluster | ||
| exampleFile: ../examples/ManagedClusterDeleteOperation_example.json |
12 changes: 12 additions & 0 deletions
12
...c/resource-manager/Microsoft.ServiceFabric/stable/2020-03-01/test-scenarios/clusters.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| scope: ResourceGroup | ||
| testScenarios: | ||
| - description: Microsoft.ServiceFabric/clusters Put a cluster with minimum parameters | ||
| steps: | ||
| - step: Put a cluster with minimum parameters | ||
| exampleFile: ../examples/ClusterPutOperation_example_min.json | ||
| - step: Delete a cluster | ||
| exampleFile: ../examples/ClusterDeleteOperation_example.json | ||
| - step: Put a cluster with maximum parameters | ||
| exampleFile: ../examples/ClusterPutOperation_example_max.json | ||
| - step: Delete a cluster | ||
| exampleFile: ../examples/ClusterDeleteOperation_example.json |
10 changes: 10 additions & 0 deletions
10
specification/servicefabric/resource-manager/readme.test.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
|
|
||
| ### Tag: package-2020-03 | ||
|
|
||
| These settings apply only when `--tag=package-2020-03` is specified on the command line. | ||
|
|
||
| ```yaml $(tag) == 'package-2020-03' | ||
| test-resources: | ||
| - test: Microsoft.ServiceFabric/stable/2020-03-01/test-scenarios/clusters.yaml | ||
| - test: Microsoft.ServiceFabric/preview/2020-01-01-preview/test-scenarios/managedClusters.yaml | ||
| ``` |
8 changes: 8 additions & 0 deletions
8
...e-manager/Microsoft.SignalRService/preview/2018-03-01-preview/test-scenarios/SignalR.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| scope: ResourceGroup | ||
| testScenarios: | ||
| - description: Microsoft.SignalRService/SignalR CreateOrUpdate | ||
| steps: | ||
| - step: CreateOrUpdate | ||
| exampleFile: ../examples/CreateOrUpdate.json | ||
| - step: Delete | ||
| exampleFile: ../examples/Delete.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.