Increase PostgreSQL Database storage

The current configuration

The 1G storage is currectly mounted to /var/lib/pgsql/data in postgresql pod. This is where the PostgreSQL database is located.

Storage:

postgresql:  Bound to volume app-pv-152-1g	1 GiB  

Postgresql deployment configuration:

    spec:
	template:
		spec:
		  volumes:
			- name: postgresql-data
			  persistentVolumeClaim:
				claimName: postgresql
	containers:
		  volumeMounts:
			- name: postgresql-data
			  mountPath: /var/lib/pgsql/data				

Increase database storage

Step1: send out notifications to end users if necessary
scale down the application gwell's pod to 0 so that client can't access the application

Step2: backup current database
Use oc command to login OpenShift
Switch to project moe-gwells-
Run command 'oc get pods' to get all pods and find the active PostgreSQL pod, it should be named like postgresql-42-t01rv
Run command 'oc rsync :/var/lib/pgsql/data . This command should copy everything under /var/lib/pgsql/data to local directory
Run command 'oc rsh ' to rsh to the pod and verify if the data has been all copied to load directory, especially the hidden files if there are any.

Step3: create a new 2GiB storage named as postgresql2, Openshift will allocate 5G as the maximum size

postgresql2:  Bound to volume <new storage name>	5GiB			  

Step4: update postgresql deployment configuration to mount current storage claim to postgresql-data-old and mount the new storage claim to postgresql-data:

    spec:
	template:
		spec:
		  volumes:
			- name: postgresql-data
			  persistentVolumeClaim:
				claimName: postgresql
			- name: postgresql-data-new
			  persistentVolumeClaim:
				claimName: postgresql2
	containers:
		  volumeMounts:
			- name: postgresql-data
			  mountPath: /var/lib/pgsql/data				
			- name: postgresql-data-new
			  mountPath: /var/lib/pgsql/data-new

Step5: verify if a new deployment for postgresql has happened, manually trigger one if not. After the new pod gets created, the both data and data-new folders should be found under /var/lib/pgsql directory. The data-new folder is mapped to the new storage and it should have no any content.

Step6: find the new pod, the login to it and copy everything under /var/lib/pgsql/data to /var/lib/pgsql/data-new

Step7: update postgresql deployment configuration as below:

    spec:
	template:
		spec:
		  volumes:
			- name: postgresql-data
			  persistentVolumeClaim:
				claimName: postgresql2
	containers:
		  volumeMounts:
			- name: postgresql-data
			  mountPath: /var/lib/pgsql/data				

Step8: verify if a new deployment for postgresql has happened, manually trigger one if not

Step9: verify if application works properly

Step10: remember to remove the previous storage to release the persistent volume

Backout Plan

As the 1G storage postgresql still stays, in case of any failure, just need to restore to the original configuration which can refer to the section on the top. Remember to run the redeployment after all the the configuration is restored. The local backup copied at step2 can be used as source for the restore as well in case the above backout process encountered any issues.

Further consideration

A site maintenance page may need to be displayed before we run the storage increase process.