A guide on howto create a redundant storage pool using GlusterFS and Azure Shared Disks
In this article I'll show you how to create a redundtant storage pool using GlusterFS and Azure Shared Disks. GlusterFS is a network-attached storage filesystem that allows you to pool storage resources of multiple machines. Azure shared disks is a new feature for Azure managed disks that allows you to attach a managed disk to multiple virtual machines (VMs) simultaneously. Please note that enabling shared disks is only available to a subset of disk types. Currently only ultra disks and premium SSDs can enable shared disks. Check if the VM type you are planning to use support ultra or premium disks.
Our setup will consist in:
- An Azure Resource Group containing the resources
- An Azure VNET and a Subnet
- An Availability Set into a Proximity Placement Group
- 2 Linux VMs (Ubuntu 18.04)
- 2 Public IPs (one for each VM)
- 2 Network Security Groups (1 per VM Network Interface Card)
- A Shared Data Disk attached to the both VMs
I'll be using the Azure Cloud Shell once is fully integrated to Azure and with all modules I need already installed. Make sure to swhich to PowerShell:
- Create SSH key pair
- Create a resource group
- Create virtual network resources
- Create a subnet configuration
- Create a virtual network
- Create a public IP address for the VM01
- Create a public IP address for the VM02
- Create an inbound network security group rule for port 22
- Create a network security group for the VM01
- Create a network security group for the VM02
- Create a virtual network card for VM01 and associate with public IP address and NSG
- Create a virtual network card for VM02 and associate with public IP address and NSG
- Create availability set for the virtual machines
- Create the first virtual machine (myVM01)
- Create the second virtual machine (myVM02)
- Create a Shared Data Disk
- Create a proximity placement group
- Configure the Disk on Linux VM01
- Configure the Disk on Linux VM02
- Install GlusterFS on Linux VM01
- Install GlusterFS on Linux VM02
- Configure GlusterFS on Linx VM01
- Configure GlusterFS on Linx VM02
- Test
ssh-keygen -t rsa -b 4096
New-AzResourceGroup -Name "myResourceGroup" -Location "EastUS"
$subnetConfig = New-AzVirtualNetworkSubnetConfig `
-Name "mySubnet" `
-AddressPrefix 192.168.1.0/24
$vnet = New-AzVirtualNetwork `
-ResourceGroupName "myResourceGroup" `
-Location "EastUS" `
-Name "myVNET" `
-AddressPrefix 192.168.0.0/16 `
-Subnet $subnetConfig
$pip01 = New-AzPublicIpAddress `
-ResourceGroupName "myResourceGroup" `
-Location "EastUS" `
-AllocationMethod Static `
-IdleTimeoutInMinutes 4 `
-Name "mypublicip01"
$pip02 = New-AzPublicIpAddress `
-ResourceGroupName "myResourceGroup" `
-Location "EastUS" `
-AllocationMethod Static `
-IdleTimeoutInMinutes 4 `
-Name "mypublicip02"
$nsgRuleSSH = New-AzNetworkSecurityRuleConfig `
-Name "myNetworkSecurityGroupRuleSSH" `
-Protocol "Tcp" `
-Direction "Inbound" `
-Priority 1000 `
-SourceAddressPrefix * `
-SourcePortRange * `
-DestinationAddressPrefix * `
-DestinationPortRange 22 `
-Access "Allow"
$nsg = New-AzNetworkSecurityGroup `
-ResourceGroupName "myResourceGroup" `
-Location "EastUS" `
-Name "myNetworkSecurityGroup01" `
-SecurityRules $nsgRuleSSH
$nsg = New-AzNetworkSecurityGroup `
-ResourceGroupName "myResourceGroup" `
-Location "EastUS" `
-Name "myNetworkSecurityGroup02" `
-SecurityRules $nsgRuleSSH
$nic01 = New-AzNetworkInterface `
-Name "myNic01" `
-ResourceGroupName "myResourceGroup" `
-Location "EastUS" `
-SubnetId $vnet.Subnets[0].Id `
-PublicIpAddressId $pip01.Id `
-NetworkSecurityGroupId $nsg.Id
$nic02 = New-AzNetworkInterface `
-Name "myNic02" `
-ResourceGroupName "myResourceGroup" `
-Location "EastUS" `
-SubnetId $vnet.Subnets[0].Id `
-PublicIpAddressId $pip02.Id `
-NetworkSecurityGroupId $nsg.Id
$set = @{
Name = 'myAvSet'
ResourceGroupName = 'myResourceGroup'
Location = 'eastus'
Sku = 'Aligned'
PlatformFaultDomainCount = '2'
PlatformUpdateDomainCount = '2'
}
$avs = New-AzAvailabilitySet @set
$securePassword = ConvertTo-SecureString ' ' -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential ("azureuser", $securePassword)
$vmConfig = New-AzVMConfig `
-AvailabilitySetId $avs.Id `
-VMName "myVM01" `
-VMSize "Standard_D4s_v3" | `
Set-AzVMOperatingSystem `
-Linux `
-ComputerName "myVM01" `
-Credential $cred `
-DisablePasswordAuthentication | `
Set-AzVMSourceImage `
-PublisherName "Canonical" `
-Offer "UbuntuServer" `
-Skus "18.04-LTS" `
-Version "latest" | `
Add-AzVMNetworkInterface `
-Id $nic01.Id
$sshPublicKey = cat ~/.ssh/id_rsa.pub
Add-AzVMSshPublicKey `
-VM $vmconfig `
-KeyData $sshPublicKey `
-Path "/home/azureuser/.ssh/authorized_keys"
New-AzVM `
-ResourceGroupName "myResourceGroup" `
-Location eastus -VM $vmConfig
$securePassword = ConvertTo-SecureString ' ' -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential ("azureuser", $securePassword)
$vmConfig = New-AzVMConfig `
-AvailabilitySetId $avs.Id `
-VMName "myVM02" `
-VMSize "Standard_D4s_v3" | `
Set-AzVMOperatingSystem `
-Linux `
-ComputerName "myVM02" `
-Credential $cred `
-DisablePasswordAuthentication | `
Set-AzVMSourceImage `
-PublisherName "Canonical" `
-Offer "UbuntuServer" `
-Skus "18.04-LTS" `
-Version "latest" | `
Add-AzVMNetworkInterface `
-Id $nic02.Id
$sshPublicKey = cat ~/.ssh/id_rsa.pub
Add-AzVMSshPublicKey `
-VM $vmconfig `
-KeyData $sshPublicKey `
-Path "/home/azureuser/.ssh/authorized_keys"
New-AzVM `
-ResourceGroupName "myResourceGroup" `
-Location eastus -VM $vmConfig
$dataDiskConfig = New-AzDiskConfig -Location 'EastUS' -DiskSizeGB 1024 -AccountType Premium_LRS -CreateOption Empty -MaxSharesCount 2
New-AzDisk -ResourceGroupName 'myResourceGroup' -DiskName 'mySharedDisk' -Disk $dataDiskConfig
$dataDisk = Get-AzDisk -ResourceGroupName "myResourceGroup" -DiskName "mySharedDisk"
$VirtualMachine = Get-AzVM -ResourceGroupName "myResourceGroup" -Name "myVM01"
Add-AzVMDataDisk -VM $VirtualMachine -Name "mySharedDisk" -CreateOption Attach -ManagedDiskId $dataDisk.Id -Lun 0
update-AzVm -VM $VirtualMachine -ResourceGroupName "myResourceGroup"
$dataDisk = Get-AzDisk -ResourceGroupName "myResourceGroup" -DiskName "mySharedDisk"
$VirtualMachine = Get-AzVM -ResourceGroupName "myResourceGroup" -Name "myVM02"
Add-AzVMDataDisk -VM $VirtualMachine -Name "mySharedDisk" -CreateOption Attach -ManagedDiskId $dataDisk.Id -Lun 0
update-AzVm -VM $VirtualMachine -ResourceGroupName "myResourceGroup"
$ppg = New-AzProximityPlacementGroup -Location "EastUS" -Name "myPPG" -ResourceGroupName "myResourceGroup" -ProximityPlacementGroupType Standard
$resourceGroup = "myResourceGroup"
$avSetName = "myAvSet"
$avSet = Get-AzAvailabilitySet -ResourceGroupName $resourceGroup -Name $avSetName
$vmIds = $avSet.VirtualMachinesReferences
foreach ($vmId in $vmIDs){
$string = $vmID.Id.Split("/")
$vmName = $string[8]
Stop-AzVM -ResourceGroupName $resourceGroup -Name $vmName -Force
}
$ppg = Get-AzProximityPlacementGroup -ResourceGroupName myResourceGroup -Name myPPG
Update-AzAvailabilitySet -AvailabilitySet $avSet -ProximityPlacementGroupId $ppg.Id
foreach ($vmId in $vmIDs){
$string = $vmID.Id.Split("/")
$vmName = $string[8]
Start-AzVM -ResourceGroupName $resourceGroup -Name $vmName
}
ssh azureuser@13.82.29.9
lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"
sudo parted /dev/sdb --script mklabel gpt mkpart xfspart xfs 0% 100%
sudo mkfs.xfs /dev/sdb1
sudo partprobe /dev/sdb1
sudo mkdir /datadrive
sudo mount /dev/sdb1 /datadrive
sudo blkid
The ouput should be something similar to:
/dev/sdc1: LABEL="cloudimg-rootfs" UUID="5a9997c3-aafd-46e9-954c-781f2b11fb68" TYPE="ext4" PARTUUID="cbc2fcb7-e40a-4fec-a370-51888c246f12"
/dev/sdc15: LABEL="UEFI" UUID="2FBA-C33A" TYPE="vfat" PARTUUID="53fbf8ed-db79-4c52-8e42-78dbf30ff35c"
/dev/sda1: UUID="c62479eb-7c96-49a1-adef-4371d27509e6" TYPE="ext4" PARTUUID="a5bb6861-01"
/dev/sdb1: UUID="f0b4e401-e9dc-472e-b9ca-3fa06a5b2e22" TYPE="xfs" PARTLABEL="xfspart" PARTUUID="af3ca4e5-cb38-4856-8791-bd6b650ba1b3"
/dev/sdc14: PARTUUID="de01bd39-4bfe-4bc8-aff7-986e694f7972"
sudo nano /etc/fstab
use the UUID value for the /dev/sdb1 device. Change by the UUID from your case and add the following at the end of the file:
UUID=f0b4e401-e9dc-472e-b9ca-3fa06a5b2e22 /datadrive xfs defaults,nofail 1 2
ssh azureuser@40.114.24.217
lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"
As the disk already was partitioned on the VM01, we can skip this step now.
sudo mkdir /datadrive
sudo mount /dev/sda1 /datadrive
sudo blkid
The ouput should be something similar to:
/dev/sdb1: LABEL="cloudimg-rootfs" UUID="5a9997c3-aafd-46e9-954c-781f2b11fb68" TYPE="ext4" PARTUUID="cbc2fcb7-e40a-4fec-a370-51888c246f12"
/dev/sdb15: LABEL="UEFI" UUID="2FBA-C33A" TYPE="vfat" PARTUUID="53fbf8ed-db79-4c52-8e42-78dbf30ff35c"
/dev/sdc1: UUID="d1b59101-225e-48f4-8373-4f1a92a81607" TYPE="ext4" PARTUUID="b0218b4e-01"
/dev/sda1: UUID="f0b4e401-e9dc-472e-b9ca-3fa06a5b2e22" TYPE="xfs" PARTLABEL="xfspart" PARTUUID="dda03810-f1f9-45a5-9613-08e9b5e89a32"
/dev/sdb14: PARTUUID="de01bd39-4bfe-4bc8-aff7-986e694f7972"
sudo nano /etc/fstab
use the UUID value for the /dev/sda1 device. Change by the UUID from your case and add the following at the end of the file:
UUID=f0b4e401-e9dc-472e-b9ca-3fa06a5b2e22 /datadrive xfs defaults,nofail 1 2
Please note that in my case the IPs 192.168.1.4 and 192.168.1.5 are the private ip's from VM01 and VM02. Add those configuration on the /etc/hosts
sudo nano /etc/hosts
192.168.1.4 gluster1.local gluster1
192.168.1.5 gluster2.local gluster2
sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository ppa:gluster/glusterfs-7
sudo apt update
sudo apt install glusterfs-server
sudo systemctl status glusterd.service
Please note that the IPs 192.168.1.4 and 192.168.1.5 are the private ip's from VM01 and VM02. Add those configuration on the /etc/hosts
sudo nano /etc/hosts
192.168.1.4 gluster1.local gluster1
192.168.1.5 gluster2.local gluster2
sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository ppa:gluster/glusterfs-7
sudo apt update
sudo apt install glusterfs-server
sudo systemctl status glusterd.service
sudo gluster peer probe gluster2
sudo gluster peer status
sudo gluster volume create sharedvolume replica 2 gluster1.local:/datadrive gluster2.local:/datadrive force
sudo gluster volume start sharedvolume
sudo gluster volume status
sudo apt install glusterfs-client
sudo mkdir /gluster-storage
sudo nano /etc/fstab
Add the following at the end of the file
gluster1.local:sharedvolume /gluster-storage glusterfs defaults,_netdev 0 0
sudo mount -a
sudo gluster peer probe gluster1
sudo gluster peer status
sudo gluster volume status
sudo apt install glusterfs-client
sudo mkdir /gluster-storage
sudo nano /etc/fstab
Add the following at the end of the file
gluster2.local:sharedvolume /gluster-storage glusterfs defaults,_netdev 0 0
sudo mount -a
In one of the nodes, go to /gluster-storage and create some files:
ssh azureuser@myVM01
azureuser@myVM01:~# sudo touch /gluster-storage/file{1..10}
Then go to the another node and check those files:
ssh azureuser@myVM02
azureuser@myVM02:~# ls -l /gluster-storage
total 0
-rw-r--r-- 1 root root 0 Apr 1 19:48 file1
-rw-r--r-- 1 root root 0 Apr 1 19:48 file10
-rw-r--r-- 1 root root 0 Apr 1 19:48 file2
-rw-r--r-- 1 root root 0 Apr 1 19:48 file3
-rw-r--r-- 1 root root 0 Apr 1 19:48 file4
-rw-r--r-- 1 root root 0 Apr 1 19:48 file5
-rw-r--r-- 1 root root 0 Apr 1 19:48 file6
-rw-r--r-- 1 root root 0 Apr 1 19:48 file7
-rw-r--r-- 1 root root 0 Apr 1 19:48 file8
-rw-r--r-- 1 root root 0 Apr 1 19:48 file9
Now execute a shutdown on myVM02.
azureuser@myVM02:~# sudo init 0
Connection to 40.114.24.217 closed by remote host.
Connection to 40.114.24.217 closed.
Access myVM01 and you notice that you still with access to the files:
azureuser@myVM01:~$ ls -l /gluster-storage/
total 0
-rw-r--r-- 1 root root 0 Apr 1 19:48 file1
-rw-r--r-- 1 root root 0 Apr 1 19:48 file10
-rw-r--r-- 1 root root 0 Apr 1 19:48 file2
-rw-r--r-- 1 root root 0 Apr 1 19:48 file3
-rw-r--r-- 1 root root 0 Apr 1 19:48 file4
-rw-r--r-- 1 root root 0 Apr 1 19:48 file5
-rw-r--r-- 1 root root 0 Apr 1 19:48 file6
-rw-r--r-- 1 root root 0 Apr 1 19:48 file7
-rw-r--r-- 1 root root 0 Apr 1 19:48 file8
-rw-r--r-- 1 root root 0 Apr 1 19:48 file9
Now let's create some new files.
azureuser@myVM01:~$ sudo touch /gluster-storage/new-file{1..10}
azureuser@myVM01:~$ sudo ls -l /gluster-storage/
total 0
-rw-r--r-- 1 root root 0 Apr 1 19:48 file1
-rw-r--r-- 1 root root 0 Apr 1 19:48 file10
-rw-r--r-- 1 root root 0 Apr 1 19:48 file2
-rw-r--r-- 1 root root 0 Apr 1 19:48 file3
-rw-r--r-- 1 root root 0 Apr 1 19:48 file4
-rw-r--r-- 1 root root 0 Apr 1 19:48 file5
-rw-r--r-- 1 root root 0 Apr 1 19:48 file6
-rw-r--r-- 1 root root 0 Apr 1 19:48 file7
-rw-r--r-- 1 root root 0 Apr 1 19:48 file8
-rw-r--r-- 1 root root 0 Apr 1 19:48 file9
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file1
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file10
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file2
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file3
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file4
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file5
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file6
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file7
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file8
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file9
Then just turn on the myVM02 and you will be able the see all files syncronized on myVM02:
azureuser@myVM02:~$ ls -l /gluster-storage/
total 0
-rw-r--r-- 1 root root 0 Apr 1 19:48 file1
-rw-r--r-- 1 root root 0 Apr 1 19:48 file10
-rw-r--r-- 1 root root 0 Apr 1 19:48 file2
-rw-r--r-- 1 root root 0 Apr 1 19:48 file3
-rw-r--r-- 1 root root 0 Apr 1 19:48 file4
-rw-r--r-- 1 root root 0 Apr 1 19:48 file5
-rw-r--r-- 1 root root 0 Apr 1 19:48 file6
-rw-r--r-- 1 root root 0 Apr 1 19:48 file7
-rw-r--r-- 1 root root 0 Apr 1 19:48 file8
-rw-r--r-- 1 root root 0 Apr 1 19:48 file9
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file1
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file10
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file2
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file3
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file4
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file5
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file6
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file7
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file8
-rw-r--r-- 1 root root 0 Apr 1 20:00 new-file9
As you can see the files was in sync and without any kind of data loss even in the case of one of the nodes was offline.