(SURE-2565) Support for managed identity in AKS

[kkaempf]

See also rancher/rancher#27559

We got an issue when using Rancher to provision Windows nodes using Managed identities for Azure resources (formerly known as Managed Service Identity - MSI).

Rancher provisioning script only handles cases where we authenticate using service principal and will not work properly with managed identity.

Relevant code: https://github.com/rancher/rke-tools/blob/a23ff70c7a1ae0b8ec5c91bc56d51b0ad9f541ad/windows/cloud-provider.psm1#L70

Currently, we have to patch the file in runtime to use az login --identity instead of az login --service-principal

Steps to reproduce (least amount of steps as possible):

Create an AKS cluster without specifying any Service Principal

Result:

An AKS cluster is created using the --enable-managed-identity flag.

Other details that may be helpful:

The feature is stable in AKS: https://docs.microsoft.com/en-us/azure/aks/use-managed-identity

Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.4.4
Installation option (single install/HA): single container

The UI presents mandatory fields to enter the SP for AKS; I propose to add a ratio button where the user can select to create a MI instead.

[cpinjani]

Validated on build: v2.8-7113f094007e3105a906b414fe18ee370c77bb5e-head

As stated in issue description and reported version v2.4 - "The UI presents mandatory fields to enter the SP for AKS; I propose to add a ratio button where the user can select to create a MI instead."

Rancher v2.4:

However, we still use service principal for AKS clusters (not managed identity)
prerequisites-in-microsoft-azure

@kkaempf @davidstauffer Please review this, as support for it is not added yet.

[cpinjani]

Moving to Backlog, cc: @mjura

[yiannistri]

Upon investigation, it seems that the aks-operator already creates clusters using managed identities. After a cluster gets provisioned from Rancher, the following command can be used to determine what type of managed identity the cluster is using (source):

az aks show --name myAKSCluster --resource-group myResourceGroup --query identity.type --output tsv

It is also indicated in the Azure Portal by accessing the Managed Identities service. There should be a managed identity associated with the cluster's node pool.

There are two types of managed identities (source): system-assigned and user-assigned. The CRD used by the aks-operator allows users to create clusters with either type, see below for examples.

Cluster using system-assigned managed identity

apiVersion: aks.cattle.io/v1
kind: AKSClusterConfig
metadata:
  name: myAKSCluster
  namespace: cattle-global-data
spec:
  azureCredentialSecret:  cattle-global-data:cc-abcde
  clusterName: myAKSCluster
  dnsPrefix: myAKSCluster
  managedIdentity: true # also true if field is omitted and aks-operator detects a tenantId
  imported: false
  kubernetesVersion: 1.30.3
  linuxAdminUsername: azureuser
  loadBalancerSku: Standard
  networkPlugin: kubenet
  nodePools:
  - availabilityZones:
    - "1"
    count: 1
    enableAutoScaling: false
    maxPods: 110
    mode: System
    name: agentpool
    orchestratorVersion: 1.30.3
    osDiskSizeGB: 30
    osDiskType: Managed
    osType: Linux
    vmSize: Standard_B4ms
  privateCluster: false
  resourceLocation: eastus
  resourceGroup: myResourceGroup

Cluster using user-assigned managed identity

apiVersion: aks.cattle.io/v1
kind: AKSClusterConfig
metadata:
  name: myAKSCluster
  namespace: cattle-global-data
spec:
  azureCredentialSecret:  cattle-global-data:cc-abcde
  clusterName: myAKSCluster
  dnsPrefix: myAKSCluster
  managedIdentity: true
  userAssignedIdentity: '/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/myResourceGroup/providers/Microsoft.ManagedIdentity/userAssignedIdentities/identity1'
  imported: false
  kubernetesVersion: 1.30.3
  linuxAdminUsername: azureuser
  loadBalancerSku: Standard
  networkPlugin: kubenet
  nodePools:
  - availabilityZones:
    - "1"
    count: 1
    enableAutoScaling: false
    maxPods: 110
    mode: System
    name: agentpool
    orchestratorVersion: 1.30.3
    osDiskSizeGB: 30
    osDiskType: Managed
    osType: Linux
    vmSize: Standard_B4ms
  privateCluster: false
  resourceLocation: eastus
  resourceGroup: myResourceGroup

Note that when using user-assigned identity, the identity needs to have been created prior to creating the cluster. Also these fields are used during cluster creation only.

While looking into this, I came across an issue with user-assigned managed identities not being associated correctly to node pools, at least the association is not shown in the Azure portal as it is shown for system-assigned managed identities. Not sure if we prefer to close this one and open a new issue though cc @kkaempf @cpinjani

Turns out spec.managedIdentity needs to be set to true when using user-assigned identities, which makes sense. I think this can be closed now since it works as expected wdyt @kkaempf @cpinjani ?

[cpinjani]

Validated on v2.9.0, agreed with @yiannistri's assessment, however to use feature it would require UI changes.
Since currently ability to provide managed identity is available only while creating private AKS cluster.