Dev Guide

The purpose of this guide is to walk you through running the App both on localhost and deployment to staging (test environment) and production. Our aim is to pre-empt any obvious questions as much as possible, but if we have skipped a step or you are scratching your head wondering how to do/run something, please open an issue and we will gladly/promptly answer and improve this guide: https://github.com/club-soda/club-soda-guide/issues

Setup on `localhost`

In order to get the project setup and running on your localhost you will need to have the following installed:

Elixir and Phoenix see: github.com/dwyl/learn-phoenix-framework
PostgreSQL see: github.com/dwyl/learn-postgresql
Node.js and Elm see: github.com/dwyl/learn-elm#how

Run the Following Commands

Clone the git repository:

git clone git@github.com:club-soda/club-soda-guide.git && cd club-soda-guide

Environment Variables

In order to run the application on your localhost you will need to have a a few environment variables defined.

Rather than listing them here in the docs, we use Heroku as our "single source of truth" for Environment Variables.

Note: if you are new to (or "rusty" on) Environment Variables please see: github.com/dwyl/learn-environment-variables And/or if you need a primer on Heroku, please see: github.com/dwyl/learn-heroku

To get the latest environment variables you will need access to the club-soda-guide-staging Heroku app.

First visit the "Settings" tab for your Heroku App: https://dashboard.heroku.com/apps/club-soda-guide-staging/settings

Click on the "Reveal Config Vars" button:

Then Run this script in your web browser's developer console:

var keys = document.getElementsByClassName('config-var-key');
var vals = document.getElementsByClassName('config-var-value');
var vars = '';
for (var i = 0; i < keys.length - 1; i++) {
  var key = keys[i].value
  if (key && key !== 'DATABASE_URL' && key !== 'HEROKU_POSTGRESQL') {
    var index = (i == 0) ? 0 : i * 2; // cause there are two values for every key ... 🙄
    vars = vars + 'export ' + keys[i].value + '=' + vals[index].value + '\n';
  }
}
console.log(vars);

More detail/insight on the history of this script, see: github.com/club-soda/club-soda-guide/issues/512

You should see something like this:

Copy the output from the browser console and paste it into your .env file.

export AWS_ACCESS_KEY_ID=AKIA****************
export AWS_S3_BUCKET=bucket-name
export AWS_S3_REGION=eu-west-1
export AWS_SECRET_ACCESS_KEY=****************
export DATABASE_URL=postgres://databasename@compute.amazonaws.com:5432/password
export ENCRYPTION_KEYS='****************='
export GOOGLE_MAPS_API_KEY=****************
export HEROKU_POSTGRESQL=postgres://databasename@compute.amazonaws.com:5432/password
export IMPORT_FILES_DIR=temp
export SECRET_KEY_BASE=****************
export SENDERS_EMAIL=hello@example.com
export SES_PORT=25
export SES_SERVER=email-smtp.eu-west-1.amazonaws.com
export SITE_URL=https://www.example.com
export SMTP_PASSWORD=****************
export SMTP_USERNAME=AKIA****************
export URL=your-app.herokuapp.com

Obviously the real environment variables don't contain any **** characters ... this is just a sanitised example.

Once you have saved the .env file, run the following command in your terminal:

source .env

Install Dependencies

Install the Elixir dependencies:

mix deps.get

Create the database:

mix ecto.create

Create the database schema (tables):

mix ecto.migrate

Start the App

Run the application on your localhost:

mix phx.server

Visit http://localhost:4000 in your web browser. You should expect to see:

You will notice that the app looks quite "bare" ... this is because it's a "content" web app so without content it will look empty.

You have two options for getting content:

Get the latest data from Heroku (real content data is good for testing UI).
Run the migration scripts to insert "seed" data (much less data).

Using Real Data

There are many instances where having "real" data on localhost is useful for UI/UX debugging. For those cases the easiest/fastest thing to do is grab a fresh backup from Heroku and "restore" it to your local Postgres.

These are the steps you will need to follow:

Visit: https://dashboard.heroku.com/apps/club-soda-guide/resources

Click on the "Heroku Postgres" link:

That will take you to the Datastores page: https://data.heroku.com/datastores/a870534e-1277-4b9a-a487-0f10d551b7f3

Click on the "Durability" menu item:

Scroll to the bottom of the page until you see the list of backups:

Click on the "Create Manual Backup" button:

You will see a "processing" message:

When the "created" column changes to "a minute ago":

Click the "Download" button:

Get the download link from your browser downloads e.g: chrome://downloads/

The link is super long because it contains auth token/credentials:

https://xfrtu.s3.amazonaws.com/7562f7d3-503a-4d40-a450-42dffc34f523/2019-04-05T08%3A04%3A29Z/8b390ead-83d7-4ed8-9161-97bc50263842?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ5HNUZMBKBNNOSYQ%2F20190405%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190405T090047Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=54ac21e88e3219f93c64efc8

Data Privacy/Security Note: For you infosec conscious people out there, in addition to changing the signature on this link so it's invalid, all Heroku database backup download links have a 10 min expiry, so even if the link was valid, it has long since expired. 🤓

Using the command format: curl "http://[url]" > production.dump construct your curl command, e.g:

Run the command in your terminal:

curl "https://xfrtu.s3.amazonaws.com/7562f7d3-503a-4d40-a450-42dffc34f523/2019-04-05T08%3A04%3A29Z/8b390ead-83d7-4ed8-9161-97bc50263842?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ5HNUZMBKBNNOSYQ%2F20190405%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190405T090047Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=54ac21e88e3219f93c64efc8" > production.dump

You should output similar to the following:

Now run the pg_restore command:

pg_restore --verbose --clean --no-acl --no-owner -h localhost -d cs_guide_dev production.dump

Given that the --verbose flag was used, you should see a bunch of output in terminal:

That's a good sign it worked ... Open the database in your chosen GUI/CLI and you should see: Postico:

DBeaver:

Thanks to @wrburgess for his helpful gist: https://gist.github.com/wrburgess/5528649 ❤️

Importing Data

The fastest way to on-board a lot of venues into the Guide is to import their data (with permission) in bulk. This section takes you through the steps to add a spreadsheet of data into the Guide App's Database.

1. Backup and Restore the Database

Before importing any new data, ensure that you have made a backup of the production database. Follow the instructions in: dev-guide.md#using-real-data to backup, download and import the production data.

Having the latest production data on your localhost is essential to testing the import process.

2. Download the Data as CSV

In order to import the data, we need to have it in a useable format. Most of our imports are stored in Google Drive as spreadsheets.

Open the Google Drive file, click "Download as >" and select "Comma-separated values":

3. Move the `.csv` file to the `./temp` directory and Re-name it.

The file you download from Google Sheets will be named something human-friendly like Bermondsey-Pub-Company - Sheet1.csv in our case:

Move the .csv file you just downloaded to the /temp directory and re-name it to something machine readable e.g: temp/bermondsey_pub_company.csv. Removing spaces and using underscores in the filename is necessary for Elixir to understand it.

4. Review the Column Headers

When you open the .csv file, the first row/line are the "headers" (or column names). The spreadsheet will have human-readable names e.g:

Copy that first line of the file and past it into your text editor.

Venue Name,Parent company,Street Address,City,Post Code,Telephone,Venue Type,
Email,Description,Website,Facebook,Twitter,Instagram,
Low/No Alcohol Drink 1,Low/No Alcohol Drink 2,Low/No Alcohol Drink 3,
Low/No Alcohol Drink 4,Low/No Alcohol Drink 5,Low/No Alcohol Drink 6,
Low/No Alcohol Drink 7,Low/No Alcohol Drink 8,Low/No Alcohol Drink 9,
Low/No Alcohol Drink 10,Low/No Alcohol Drink 11,Low/No Alcohol Drink 12,
Low/No Alcohol Drink 13,Low/No Alcohol Drink 14,Low/No Alcohol Drink 15,
Low/No Alcohol Drink 16,Low/No Alcohol Drink 17,Low/No Alcohol Drink 18

Note: we have split the header into multiple lines for legibility. A valid CSV file headers are always on a single line.

5. Map the CSV Column Headers to Database Fields

In order to import the data, we need to go through the csv file column headers and map them to the database column names.

The following is the list of venues table fields This list is separated by spaces because that's how our Elixir venue importer script new_venues.exs needs them.

venue_name parent_company address city postcode phone_number venue_types
email description website facebook twitter instagram
drink_1 drink_2 drink_3 drink_4 drink_5 drink_6
drink_7 drink_8 drink_9 drink_10 drink_11 drink_12
drink_13 drink_14 drink_15 drink_16 drink_17 drink_18

Note: this needs to be a single line of code when you paste it into the priv/repo/new_venues.exs below; we have split it into 5 lines to aid legibility. Also, some .csv files do not contain all the fields in the database (venues) table e.g: num_cocktails is not present in the case of Berbomondsey.

6. Add line to `script_helpers.exs`

Open the priv/repo/script_helpers.exs file and find the venues/0 function. map. The venues/0 function returns a map which is a key-value lookup of the chains of venues we have imported and the corresponding field names for the csv files.

Add the following line to the map:

bermondsey_pub_company:
  ~w(venue_name parent_company address city postcode phone_number venue_types email description website facebook twitter instagram drink_1 drink_2 drink_3 drink_4 drink_5 drink_6 drink_7 drink_8 drink_9 drink_10 drink_11 drink_12 drink_13 drink_14 drink_15 drink_16 drink_17 drink_18)a,

bermondsey_pub_company corresponds to the csv file name and the contents of the ~w()a is the sigil for a "word list"; in our case the list of database column headers. the list of words should be on a single line. (some horizontal scrolling may be required ...)

Save the file and prepare to run a script!

7. Run the `new_venues.exs` import script

Before attempting to run the new_venues.exs script, ensure that you have a IMPORT_FILES_DIR key-value in your .env file and that you have the environment variable set by running source .env. If you don't yet have an .env file, GOTO: dev-guide.md#environment-variables

Open your database GUI of choice and confirm the number of rows for the venues table. e.g:

Run the new_venues.exs script with the following command:

mix run priv/repo/new_venues.exs

After successfully running the script, refresh the DB and you should see the increase in the number of records. e.g: 7486

7486 - 7428 = 58. 58 corresponds to the number of rows in the .csv file (minus the header row)

Confirm the Venue Managers (Users) are Associated with the Venue

Open your PostgreSQL GUI and visit the venue_users table.

venue_users before running the new_venues.exs script: 310 rows

venue_users after running the new_venues.exs script: 367 rows

This is consistent with the data we just imported because there are 58 rows but one of the rows does not have an email address so we only expect 57 new entries in the venue_users table:

Visit: http://localhost:4000/admin/users

Newly imported venue admin users viewable in Admin UI:

8. Test Importing Venues on Staging Environment

Visit: https://club-soda-guide-staging.herokuapp.com/admin/users

BEFORE:

Tasks to run the script and insert data into staging:

get DATABASE_URL environment variable from staging: https://dashboard.heroku.com/apps/club-soda-guide-staging/settings
add DATABASE_URL to .env file on localhost
MIX_ENV=prod mix run priv/repo/new_venues.exs

AFTER:

9. Test Importing Venues on PRODUCTION Environment

get DATABASE_URL environment variable from prod: https://dashboard.heroku.com/apps/club-soda-guide/settings
add DATABASE_URL to .env file on localhost
repeat for ENCRYPTION_KEYS environment variable for prod
MIX_ENV=prod mix run priv/repo/new_venues.exs

10. Repeat steps 1 - 9 for a New file

craft_union.csv

Headers:

Venue Name,Street Address,City/Town,Post Code,Venue Type,Parent Company,Website,Facebook,Twitter,Low/No Alcohol Drink 1,Low/No Alcohol Drink 2,Low/No Alcohol Drink 3,Low/No Alcohol Drink 4,Low/No Alcohol Drink 5,Low/No Alcohol Drink 6,Low/No Alcohol Drink 7,Low/No Alcohol Drink 8,Low/No Alcohol Drink 9,Low/No Alcohol Drink 10,Low/No Alcohol Drink 11,Low/No Alcohol Drink 12,Low/No Alcohol Drink 13,Low/No Alcohol Drink 14,Low/No Alcohol Drink 15,Low/No Alcohol Drink 16,Low/No Alcohol Drink 17,Low/No Alcohol Drink 18,Low/No Alcohol Drink 19,Low/No Alcohol Drink 20,Low/No Alcohol Drink 21,Low/No Alcohol Drink 22,Low/No Alcohol Drink 23,Low/No Alcohol Drink 24,Low/No Alcohol Drink 25

Field order:

venue_name address city postcode venue_types parent_company website facebook twitter drink_1 drink_2 drink_3 drink_4 drink_5 drink_6 drink_7 drink_8 drink_9

@venues entry (code to be added to new_venues.exs):

craft_union:
  ~w(venue_name address city postcode venue_types parent_company website facebook twitter drink_1 drink_2 drink_3 drink_4 drink_5 drink_6 drink_7 drink_8 drink_9)a,

IMPORTANT: remember to comment out prod/staging DATABASE_URL before running the script! (ensure you're running it on localhost!!)

run:

source .env
mix run priv/repo/new_venues.exs

Before: (latest pubs are bermondsey...)

After:

7784 - 7486 = 298

The Craft Union Spreadsheet (and corresponding CSV file) has 298 rows (discounting the header row)

11. Creating Venue Admins for existing venues

In some cases you may have had to run a script to import venues that did not have an email addresses for the venue manager. If this has ever happened then you will have venues that do not have venue managers associated with them.

In these cases you may be asked to create users for these venues at a later date. This is where add_emails_to_existing_venues.exs comes in. This script works in the same way as the above so you can follow steps 1-6 from above. Steps 7-9 are almost the same as above, the only difference is that you will need to replace all the mix run priv/repo/new_venues.exs calls with mix run priv/repo/add_emails_to_existing_venues.exs. That is the only difference!

The script will check to see if a venue exists in our database. If it does, it will then check to see if the email from the csv file belongs to an existing user. If so, it associates that user with the existing venue. If the user doesn't exist, it creates a new user and add the association.

In some cases, you may be asked to run this script but the csv will have a venue that doesn't exist in our database. In these cases the script will still run as expected but it will return a list of errors containing the list of venue names that do not exist in our database.

Superseded

The following docs are superseded and only kept here for reference.

The existing data is imported through our seeds file.

The environment variable IMPORT_FILES_DIR should be the path to the directory containing the csv files (For example, if those files are hosted on AWS S3, it would be the path of the S3 bucket).

On Heroku these files are stored in the /temp directory. So we do the same on localhost, store them in club-soda-guide/temp.

The files should be named correctly such that the format of the file matches the function that will be calling it. (That is, the brands file should be brands.csv, drinks drinks.csv and the venues venues_1.csv, venues_2.csv or venues_3.csv, depending on which format it is. These should be named correctly already, and as this import is only intended to be done once, shouldn't need to be changed. This documentation is just here as a guide if this import function ever needs to be extended.)

If you do need to import more venues, use this script as a guide, but you will most likely have to create a new one based on the format of the csv file.

After new venues have been imported, you may have to run mix run priv/repo/update_cs_score.exs if any drinks were attached to venues as part of the upload process.

Update Club Soda scores for venues

After new venues have been imported, you may have to run mix run priv/repo/update_cs_score.exs if any drinks were attached to venues as part of the upload process. This script will update all incorrect cs_score values.

Creating Admin Users

To create an admin user, open iex with iex -S mix.

From here do:

iex> %CsGuide.Accounts.User{} |> CsGuide.Accounts.User.changeset(%{email: "", password: "", verified: NaiveDateTime.utc_now(), role: "site_admin"}) |> CsGuide.Accounts.User.insert()

remember to define the email address and password, e.g:

%CsGuide.Accounts.User{} |> CsGuide.Accounts.User.changeset(%{email: "nelson@dwyl.io", password: "BeExcellentToEachOther", verified: NaiveDateTime.utc_now(), role: "site_admin"}) |> CsGuide.Accounts.User.insert()

Filling in the empty strings as necessary.

Adding latitude and longitude values to the database

Latitude and longitude values are being used to calculate the distance from a user to a venue. To add these values to your venues in the database run the command...

mix run priv/repo/add_lat_long_to_venue.exs

Deployment

The master branch is automatically deployed to staging. When it's time to deploy to production follow these steps:

Adding environment variables

If you have added any environment variables to the project since it was last deployed that are needed for the it to work you will need to add them to heroku as well. To do this click the Settings tab on the heroku dashboard.

Once here, click the Reveal Config Vars...

and enter the key, value pairs into the input boxes provided...

After entering the info, click Add to add the variable to heroku.

Backup the current database

From the heroku dashboard, go to the Resources tab, then click Heroku Postgres.

This will take you to the database dashboard. From here, got to the Durability tab, then click Create Manual Backup.

Deploy `master` branch

Back on the Heroku dashboard, go to the Deploy tab.

Scroll down to the bottom section: Manual Deploy. Make sure the branch is set to master, then click Deploy Branch.

A few minutes later, you should be notified that the deployment is complete.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dev-guide.md

dev-guide.md

Dev Guide

Setup on `localhost`

Run the Following Commands

Environment Variables

Install Dependencies

Start the App

Using Real Data

Importing Data

1. Backup and Restore the Database

2. Download the Data as CSV

3. Move the `.csv` file to the `./temp` directory and Re-name it.

4. Review the Column Headers

5. Map the CSV Column Headers to Database Fields

6. Add line to `script_helpers.exs`

7. Run the `new_venues.exs` import script

Confirm the Venue Managers (Users) are Associated with the Venue

8. Test Importing Venues on Staging Environment

Tasks to run the script and insert data into staging:

9. Test Importing Venues on PRODUCTION Environment

10. Repeat steps 1 - 9 for a New file

11. Creating Venue Admins for existing venues

Superseded

Update Club Soda scores for venues

Creating Admin Users

Adding latitude and longitude values to the database

Deployment

Adding environment variables

Backup the current database

Deploy `master` branch

Files

dev-guide.md

Latest commit

History

dev-guide.md

File metadata and controls

Dev Guide

Setup on localhost

Run the Following Commands

Environment Variables

Install Dependencies

Start the App

Using Real Data

Importing Data

1. Backup and Restore the Database

2. Download the Data as CSV

3. Move the .csv file to the ./temp directory and Re-name it.

4. Review the Column Headers

5. Map the CSV Column Headers to Database Fields

6. Add line to script_helpers.exs

7. Run the new_venues.exs import script

Confirm the Venue Managers (Users) are Associated with the Venue

8. Test Importing Venues on Staging Environment

Tasks to run the script and insert data into staging:

9. Test Importing Venues on PRODUCTION Environment

10. Repeat steps 1 - 9 for a New file

11. Creating Venue Admins for existing venues

Superseded

Update Club Soda scores for venues

Creating Admin Users

Adding latitude and longitude values to the database

Deployment

Adding environment variables

Backup the current database

Deploy master branch

Setup on `localhost`

3. Move the `.csv` file to the `./temp` directory and Re-name it.

6. Add line to `script_helpers.exs`

7. Run the `new_venues.exs` import script

Deploy `master` branch