Investigate enabling idempotency for the core MLZ deployment

[brooke-hamilton]

Benefit/Result/Outcome

So that an IT Admin can manage ongoing MLZ configuration changes from Bicep code.

Description

Deploying MLZ a second time without modifications results in a deployment error because at least one resource provider tries to destroy and re-create resources rather than performing an update.

For example, a second deployment of MLZ to the same subscription with no changes in parameters or Bicep code results in this error:

{
    "status": "Failed",
    "error": {
        "code": "InUseSubnetCannotBeDeleted",
        "message": "Subnet <subnet ID> is in use by /subscriptions/<subscription id>/resourceGroups/<resource group name>/providers/Microsoft.Network/networkInterfaces/<network interface name>/ipConfigurations/<private endpoint name> and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet.",
        "details": []
    }
}

The goal of the spike is to determine if we can make simple changes to the structure of the Bicep code to enable idempotent deployments.

**Work on this spike is limited to: 16 hours

Possible Outcomes

The spike will result in either one of the following outcomes (not both):

• Simple changes are made to the Bicep code that enable idempotent deployments.
• We know that Support standalone deployment of the network hub and tiers 0, 1, 2 #658 and other work (to be defined) will be required to enable idempotent deployments.

[brooke-hamilton]

Note: some ARM idempotency issues may be related to calculated names of the deployments. If the names change then ARM sees it as a new resource.

[lisamurphy-msft]

Can very likely be implemented in bicep using conditional resource deployment; ie:

@allowed([
  'new'
  'existing'
])
param newOrExisting string = 'new'

resource sa 'Microsoft.Storage/storageAccounts@2019-06-01' = if (newOrExisting == 'new') {
  name: storageAccountName

https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/conditional-resource-deployment#new-or-existing-resource

[glennmusa]

This error showed up again from a customer implementation who is looking to use MLZ Bicep as the "root" stamp and deploy on top of it with new resources:

{
    ...
    "status": "Failed",
    "error": {
        "code": "InUseSubnetCannotBeDeleted",
        "message": "Subnet <subnet ID> is in use by /subscriptions/<subscription id>/resourceGroups/<resource group name>/providers/Microsoft.Network/networkInterfaces/<network interface name>/ipConfigurations/<private endpoint name> and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet.",
        "details": []
    }
}

+@mikedzikowski

[colemanjrNIH]

This is the resource that caused this error for my team. Relevant doc we found.
My best guess is that this is defined this way to avoid a circular dependency with the Firewall Private IP for the Subnet route table. Hard coding the Firewall Private IP is a workaround that lets this child resource be pulled into the parent resource definition.

missionlz/src/bicep/modules/hubNetwork.bicep

Lines 154 to 172 in f261f1d

	resource subnet 'Microsoft.Network/virtualNetworks/subnets@2021-02-01' = {
	name: '${virtualNetworkName}/${subnetName}'
	properties: {
	addressPrefix: subnetAddressPrefix
	networkSecurityGroup: {
	id: networkSecurityGroup.outputs.id
	}
	routeTable: {
	id: routeTable.outputs.id
	}
	serviceEndpoints: subnetServiceEndpoints
	privateEndpointNetworkPolicies: 'Disabled'
	privateLinkServiceNetworkPolicies: 'Enabled'
	}
	dependsOn: [
	virtualNetwork
	firewall
	]
	}

[brooke-hamilton]

Please consider the concerns in #700 as part of this work.

[brooke-hamilton]

Closing due to consensus that some resource providers will not be idempotent, like networking and keyvault.