[Task] should support basic (manual) autoscaling group with load balancer

[OriHoch]

Spark server seems overloaded and crashes, we should support autoscaling and load balancer to support fast recovery and server redundancy

This issue details the most basic requirement to improve stability in minimal time

overview of expected outcome

• AMI with support for getting Spark version along with the .env file in userdata
• Launch configuration with latest Spark version and a copy of the current Spark .env file
• Load balancer
• update deployment documentation - for this stage, will stop using the spark slack command, instead, will have to manually duplicate launch configuration, modify release, increase number of servers in scaling group, then decrease

[OriHoch]

seems we already have a load balancer - it's just not in use yet.. @amotenko - could you please link to the issue where this load balancer was added? Can I use this load balancer? Is someone else working on this?
Waiting for your reply before I continue to work.. It seems there is someone else working on similar issue

[OriHoch]

load balancer is called "spark-production" - seems that it's configured correctly with SSL (different certificate from production server though) - and it contains the spark-production instance
it's just that DNS is not pointing to this load balancer so it's not in use

[OriHoch]

ok, I'm going to create a new load balancer "the-real-spark-production" and use that..
see #328 regarding the old "spark-production" load balancer

[OriHoch]

• create new security groups:
  • the-real-spark-production-load-balancer - (open for HTTP and HTTPS)
  • the-real-spark-production-instances (open for SSH, open for HTTP only from the load balancer, open for port 82 for the load balancer - to redirect http to https)
• create AMI from spark-production instance (Without reboot) - spark-production-2017-04-22 ami-aed9dfc8
• launch instance from that AMI
  • ssh key-pair: spark-production
• ssh and modify configurations:
  • ensure nginx and midburn-spark don't start on startup - sudo systemctl disable midburn-spark nginx
  • modify nginx - remove ssl, listen on port 80 for web-app, listen on port 82 for redirect to https
    • sudo rm /etc/nginx/sites-enabled/default
    • sudo nano /etc/nginx/sites-enabled/midburn-spark - see comment below
  • modify /opt/spark/deploy.sh - to enable using it from cloud-init - nano /opt/spark/deploy.sh - see comment below
  • create template for the userdata - see comment below
• update documentation
  • migrations should run before deployment - need to start an instance, then ssh and ./knex migrate:latest manually
• create new AMI for autoscale - spark-production-autoscale-2017-04-22
• create launch configuration - spark-production-v2.2.2
• create auto scaling group - spark-production

[OriHoch]

userdata template

#!/usr/bin/env bash
echo "
DEPLOYMENT_PACKAGE_URL="slack package url for release to deploy"

-- REST OF SPARK ENV HERE --
" > /opt/spark/.env
sudo -u ubuntu /opt/spark/deploy.sh

/etc/nginx/sites-enabled/midburn-spark

server {
    listen 80 default_server;
    server_name _;
    location / {
        proxy_pass http://127.0.0.1:3000;
    }
}
server {
    listen 82;
    return 301 https://spark.midburn.org$request_uri;
}

/opt/spark/deploy.sh

#!/usr/bin/env bash

set -e  # exit on errors

LOG_FILE="/var/log/spark-deploy.log"

log() {
    echo `date "+%F %T "`${1} | tee -a $LOG_FILE
}

logger -stspark-deploy "started spark deployment, see log file: ${LOG_FILE}"

log "Setting up the environment..."
source /opt/spark/.nvm/nvm.sh
source /opt/spark/.env

if [ "${DEPLOYMENT_PACKAGE_URL}" == "" ]; then
    log "no DEPLOYMENT_PACKAGE_URL"
    exit 1
fi

log "Downloading package from ${DEPLOYMENT_PACKAGE_URL}"
curl -H "Authorization: Bearer ${SLACK_API_TOKEN}" -g "${DEPLOYMENT_PACKAGE_URL}" -o- > /opt/spark/package.tar.gz

log "Extracting..."
rm -rf /opt/spark/latest
mkdir -p /opt/spark/latest
pushd /opt/spark/latest
    tar xzf /opt/spark/package.tar.gz 2>&1 | tee -a $LOG_FILE
popd

log "Copying configurations..."
cp /opt/spark/.env /opt/spark/latest/.env

log "Setting up the deployment directory..."
pushd /opt/spark/latest
    nvm install 2>&1 | tee -a $LOG_FILE
    nvm use 2>&1 | tee -a $LOG_FILE
    ln -s node_modules/.bin/knex knex
    # disabled npm run deploy - as it produces errors at the moment (doesn't seem to be needed anyway)
    # npm run deploy 2>&1 | tee -a $LOG_FILE
    # migrations also produce errors at the moment, should be run manually for now
    # ./knex migrate:latest 2>&1 | tee -a $LOG_FILE
popd

log "starting the services"
sudo service midburn-spark start
sudo service midburn-spark status 2>&1 | tee -a $LOG_FILE
sudo service nginx start
sudo service nginx status 2>&1 | tee -a $LOG_FILE
log "DONE"

[OriHoch]

done, see v2.2.3