From a930b80561c04e4e763a7c225b7b6fb618cd4374 Mon Sep 17 00:00:00 2001 From: ChrisBAshton Date: Fri, 12 Jul 2024 15:38:23 +0100 Subject: [PATCH] Replicate data from staging, not integration Since the env sync was ported over to govuk-helm-charts, not all apps have backups saved to s3 on integration (e.g. note the lack of `backup` operation [here](https://github.com/alphagov/govuk-helm-charts/blob/cae5e10baf0f67423bbebf66454da737da787d37/charts/db-backup/values.yaml#L767-L773)). This means that the data replication scripts fail, as they're looking for a bucket that does not exist. In contrast, all data is backed up to s3 on staging, so it should be safe to pull from there by default. The only downside is that integration access is given to devs by default, but that access to staging and production required Production Admin access. This is something that can hopefully be ironed out in the move to using govuk-helm-charts for local development, over the coming months. For now it seems better to fix the script for prod-admin users than to have it broken for everyone. NB I've also removed the unnecessary duplication of replication examples where one used a readonly role and one used a poweruser role. We should use principle of least privilege, so only show the readonly example. --- bin/replicate-mongodb.sh | 2 +- bin/replicate-mysql.sh | 2 +- bin/replicate-postgresql.sh | 2 +- docs/how-tos.md | 8 ++------ 4 files changed, 5 insertions(+), 9 deletions(-) diff --git a/bin/replicate-mongodb.sh b/bin/replicate-mongodb.sh index 9e7bd1a2..65b0e292 100755 --- a/bin/replicate-mongodb.sh +++ b/bin/replicate-mongodb.sh @@ -33,7 +33,7 @@ case "$app" in ;; esac -folder="govuk-integration-database-backups/$instance" +folder="govuk-staging-database-backups/$instance" archive_dir="${replication_dir}/mongodb" archive_file="${database}.gz" diff --git a/bin/replicate-mysql.sh b/bin/replicate-mysql.sh index 162fbec2..5e7dcb1b 100755 --- a/bin/replicate-mysql.sh +++ b/bin/replicate-mysql.sh @@ -25,7 +25,7 @@ app="${1//_/-}" replication_dir="${GOVUK_DOCKER_REPLICATION_DIR:-${GOVUK_DOCKER_DIR:-${GOVUK_ROOT_DIR:-$HOME/govuk}/govuk-docker}/replication}" -bucket="govuk-integration-database-backups" +bucket="govuk-staging-database-backups" archive_dir="${replication_dir}/mysql" archive_file="${app//-/_}_production.dump.gz" archive_path="${archive_dir}/${archive_file}" diff --git a/bin/replicate-postgresql.sh b/bin/replicate-postgresql.sh index a12599ec..56cb221c 100755 --- a/bin/replicate-postgresql.sh +++ b/bin/replicate-postgresql.sh @@ -39,7 +39,7 @@ app="${1//_/-}" replication_dir="${GOVUK_DOCKER_REPLICATION_DIR:-${GOVUK_DOCKER_DIR:-${GOVUK_ROOT_DIR:-$HOME/govuk}/govuk-docker}/replication}" -bucket="govuk-integration-database-backups" +bucket="govuk-staging-database-backups" archive_dir="${replication_dir}/postgresql" archive_file="${app//-/_}_production.dump.gz" diff --git a/docs/how-tos.md b/docs/how-tos.md index 3c6e0374..3a64b807 100644 --- a/docs/how-tos.md +++ b/docs/how-tos.md @@ -36,7 +36,7 @@ gem "govuk_publishing_components", path: "../govuk_publishing_components" ## How to: replicate data locally -There may be times when a full database is required locally. The following scripts in the `bin` directory allow replicating data from integration: +There may be times when a full database is required locally. The following scripts in the `bin` directory allow replicating data from staging: - `replicate-elasticsearch.sh` - `replicate-mongodb.sh APP-NAME` @@ -46,11 +46,7 @@ There may be times when a full database is required locally. The following scri You will need to assume-role into AWS using the [gds-cli](https://docs.publishing.service.gov.uk/manual/access-aws-console.html) before running the scripts. For example, to replicate data for Content Publisher, run: ``` -# as an AWS PowerUser... -gds aws govuk-integration-poweruser --assume-role-ttl 180m ./bin/replicate-postgresql.sh content-publisher - -# as an AWS User... -gds aws govuk-integration-readonly --assume-role-ttl 180m ./bin/replicate-postgresql.sh content-publisher +gds aws govuk-staging-readonly --assume-role-ttl 180m ./bin/replicate-postgresql.sh content-publisher ``` All the scripts, other than `replicate-elasticsearch.sh`, take the name of the app to replicate data for.