Enable Consul Connect on terraform-aws-consul

examples/example-with-consul-connect/README.md

@@ -0,0 +1,33 @@
+# Consul Cluster with Connect service mesh
+
+This folder shows an example of Terraform code that uses the [run-consul module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/consul-cluster) to deploy
+a [Consul](https://www.consul.io/) cluster in [AWS](https://aws.amazon.com/) with the Consul Connect Service Mesh turned on. The cluster consists of three Services with
+side-proxies and Intentions that enable secure service mesh connections.
+
+You will need to create an [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
+that has Consul installed, which you can do using the [consul-ami example](https://github.com/hashicorp/terraform-aws-consul/tree/master/examples/consul-ami)). Note that to keep
+this example simple, both the server ASG and client ASG are running the exact same AMI. In real-world usage, you'd
+probably have multiple client ASGs, and each of those ASGs would run a different AMI that has the Consul agent
+installed alongside your apps.
+
+For more info on how the Consul cluster works, check out the [consul-cluster](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/consul-cluster) documentation.
+
+
+
+## Quick start
+
+To deploy a Consul Cluster:
+
+1. `git clone` this repo to your computer.
+1. Optional: build a Consul AMI. See the [consul-ami example](https://github.com/hashicorp/terraform-aws-consul/tree/master/examples/consul-ami) documentation for instructions. Make sure to
+   note down the ID of the AMI.
+1. Install [Terraform](https://www.terraform.io/).
+1. Open `variables.tf`, set the environment variables specified at the top of the file, and fill in any other variables that
+   don't have a default. If you built a custom AMI, put the AMI ID into the `ami_id` variable. Otherwise, one of our
+   public example AMIs will be used by default. These AMIs are great for learning/experimenting, but are NOT
+   recommended for production use.
+1. Run `terraform init`.
+1. Run `terraform apply`.
+1. Run the [consul-examples-helper.sh script](https://github.com/hashicorp/terraform-aws-consul/tree/master/examples/consul-examples-helper/consul-examples-helper.sh) to
+   print out the IP addresses of the Consul servers and some example commands you can run to interact with the cluster:
+   `../consul-examples-helper/consul-examples-helper.sh`.

examples/example-with-consul-connect/main.tf

@@ -0,0 +1,176 @@
+# ---------------------------------------------------------------------------------------------------------------------
+# DEPLOY A CONSUL CLUSTER IN AWS
+# These templates show an example of how to use the consul-cluster module to deploy Consul in AWS. We deploy two Auto
+# Scaling Groups (ASGs): one with a small number of Consul server nodes and one with a larger number of Consul client
+# nodes. Note that these templates assume that the AMI you provide via the ami_id input variable is built from
+# the examples/consul-ami/consul.json Packer template.
+# ---------------------------------------------------------------------------------------------------------------------
+
+# ----------------------------------------------------------------------------------------------------------------------
+# REQUIRE A SPECIFIC TERRAFORM VERSION OR HIGHER
+# This module has been updated with 0.12 syntax, which means it is no longer compatible with any versions below 0.12.
+# ----------------------------------------------------------------------------------------------------------------------
+terraform {
+  required_version = ">= 0.12"
+}
+
+# ---------------------------------------------------------------------------------------------------------------------
+# AUTOMATICALLY LOOK UP THE LATEST PRE-BUILT AMI
+# This repo contains a CircleCI job that automatically builds and publishes the latest AMI by building the Packer
+# template at /examples/consul-ami upon every new release. The Terraform data source below automatically looks up the
+# latest AMI so that a simple "terraform apply" will just work without the user needing to manually build an AMI and
+# fill in the right value.
+#
+# !! WARNING !! These exmaple AMIs are meant only convenience when initially testing this repo. Do NOT use these example
+# AMIs in a production setting because it is important that you consciously think through the configuration you want
+# in your own production AMI.
+#
+# NOTE: This Terraform data source must return at least one AMI result or the entire template will fail. See
+# /_ci/publish-amis-in-new-account.md for more information.
+# ---------------------------------------------------------------------------------------------------------------------
+data "aws_ami" "consul" {
+  most_recent = true
+
+  # If we change the AWS Account in which test are run, update this value.
+  owners = ["562637147889"]
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  filter {
+    name   = "is-public"
+    values = ["true"]
+  }
+
+  filter {
+    name   = "name"
+    values = ["consul-ubuntu-*"]
+  }
+}
+
+# ---------------------------------------------------------------------------------------------------------------------
+# DEPLOY THE CONSUL SERVER NODES
+# ---------------------------------------------------------------------------------------------------------------------
+
+module "consul_servers" {
+  # When using these modules in your own templates, you will need to use a Git URL with a ref attribute that pins you
+  # to a specific version of the modules, such as the following example:
+  # source = "git::git@github.com:hashicorp/terraform-aws-consul.git//modules/consul-cluster?ref=v0.0.1"
+  source = "../../modules/consul-cluster"
+
+  cluster_name  = "${var.cluster_name}-server"
+  cluster_size  = var.num_servers
+  instance_type = "t2.micro"
+  spot_price    = var.spot_price
+
+  # The EC2 Instances will use these tags to automatically discover each other and form a cluster
+  cluster_tag_key   = var.cluster_tag_key
+  cluster_tag_value = var.cluster_name
+
+  ami_id    = "${var.ami_id == null ? data.aws_ami.consul.image_id : var.ami_id}"
+  user_data = "${data.template_file.user_data_server.rendered}"
+
+  vpc_id     = data.aws_vpc.default.id
+  subnet_ids = data.aws_subnet_ids.default.ids
+
+  # To make testing easier, we allow Consul and SSH requests from any IP address here but in a production
+  # deployment, we strongly recommend you limit this to the IP address ranges of known, trusted servers inside your VPC.
+  allowed_ssh_cidr_blocks = ["0.0.0.0/0"]
+
+  allowed_inbound_cidr_blocks = ["0.0.0.0/0"]
+  ssh_key_name                = var.ssh_key_name
+
+  tags = [
+    {
+      key                 = "Environment"
+      value               = "development"
+      propagate_at_launch = true
+    }
+  ]
+}
+
+# ---------------------------------------------------------------------------------------------------------------------
+# THE USER DATA SCRIPT THAT WILL RUN ON EACH CONSUL SERVER EC2 INSTANCE WHEN IT'S BOOTING
+# This script will configure and start Consul
+# ---------------------------------------------------------------------------------------------------------------------
+
+data "template_file" "user_data_server" {
+  template = file("${path.module}/user-data-server.sh")
+
+
+  vars = {
+    cluster_tag_key   = var.cluster_tag_key
+    cluster_tag_value = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------------------------------------------------
+# DEPLOY THE CONSUL CLIENT NODES
+# Note that you do not have to use the consul-cluster module to deploy your clients. We do so simply because it
+# provides a convenient way to deploy an Auto Scaling Group with the necessary IAM and security group permissions for
+# Consul, but feel free to deploy those clients however you choose (e.g. a single EC2 Instance, a Docker cluster, etc).
+# ---------------------------------------------------------------------------------------------------------------------
+
+module "consul_clients" {
+  # When using these modules in your own templates, you will need to use a Git URL with a ref attribute that pins you
+  # to a specific version of the modules, such as the following example:
+  # source = "git::git@github.com:hashicorp/terraform-aws-consul.git//modules/consul-cluster?ref=v0.0.1"
+  source = "../../modules/consul-cluster"
+
+  cluster_name  = "${var.cluster_name}-client"
+  cluster_size  = var.num_clients
+  instance_type = "t2.micro"
+  spot_price    = var.spot_price
+
+  cluster_tag_key   = "consul-clients"
+  cluster_tag_value = var.cluster_name
+
+  ami_id    = "${var.ami_id == null ? data.aws_ami.consul.image_id : var.ami_id}"
+  user_data = "${data.template_file.user_data_client.rendered}"
+
+  vpc_id     = data.aws_vpc.default.id
+  subnet_ids = data.aws_subnet_ids.default.ids
+
+  # To make testing easier, we allow Consul and SSH requests from any IP address here but in a production
+  # deployment, we strongly recommend you limit this to the IP address ranges of known, trusted servers inside your VPC.
+  allowed_ssh_cidr_blocks = ["0.0.0.0/0"]
+
+  allowed_inbound_cidr_blocks = ["0.0.0.0/0"]
+  ssh_key_name                = var.ssh_key_name
+}
+
+# ---------------------------------------------------------------------------------------------------------------------
+# THE USER DATA SCRIPT THAT WILL RUN ON EACH CONSUL CLIENT EC2 INSTANCE WHEN IT'S BOOTING
+# This script will configure and start Consul
+# ---------------------------------------------------------------------------------------------------------------------
+
+data "template_file" "user_data_client" {
+  template = file("${path.module}/user-data-client.sh")
+
+
+  vars = {
+    cluster_tag_key   = var.cluster_tag_key
+    cluster_tag_value = var.cluster_name
+  }
+}
+
+# ---------------------------------------------------------------------------------------------------------------------
+# DEPLOY CONSUL IN THE DEFAULT VPC AND SUBNETS
+# Using the default VPC and subnets makes this example easy to run and test, but it means Consul is accessible from the
+# public Internet. For a production deployment, we strongly recommend deploying into a custom VPC with private subnets.
+# ---------------------------------------------------------------------------------------------------------------------
+
+data "aws_vpc" "default" {
+  default = var.vpc_id == null ? true : false
+  id      = "${var.vpc_id}"
+}
+
+data "aws_subnet_ids" "default" {
+  vpc_id = data.aws_vpc.default.id
+}
+
+data "aws_region" "current" {
+}
+

examples/example-with-consul-connect/outputs.tf

@@ -0,0 +1,60 @@
+output "num_servers" {
+  value = module.consul_servers.cluster_size
+}
+
+output "asg_name_servers" {
+  value = module.consul_servers.asg_name
+}
+
+output "launch_config_name_servers" {
+  value = module.consul_servers.launch_config_name
+}
+
+output "iam_role_arn_servers" {
+  value = module.consul_servers.iam_role_arn
+}
+
+output "iam_role_id_servers" {
+  value = module.consul_servers.iam_role_id
+}
+
+output "security_group_id_servers" {
+  value = module.consul_servers.security_group_id
+}
+
+output "num_clients" {
+  value = module.consul_clients.cluster_size
+}
+
+output "asg_name_clients" {
+  value = module.consul_clients.asg_name
+}
+
+output "launch_config_name_clients" {
+  value = module.consul_clients.launch_config_name
+}
+
+output "iam_role_arn_clients" {
+  value = module.consul_clients.iam_role_arn
+}
+
+output "iam_role_id_clients" {
+  value = module.consul_clients.iam_role_id
+}
+
+output "security_group_id_clients" {
+  value = module.consul_clients.security_group_id
+}
+
+output "aws_region" {
+  value = data.aws_region.current.name
+}
+
+output "consul_servers_cluster_tag_key" {
+  value = module.consul_servers.cluster_tag_key
+}
+
+output "consul_servers_cluster_tag_value" {
+  value = module.consul_servers.cluster_tag_value
+}
+

examples/example-with-consul-connect/user-data-client.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+# This script is meant to be run in the User Data of each EC2 Instance while it's booting. The script uses the
+# run-consul script to configure and start Consul in client mode. Note that this script assumes it's running in an AMI
+# built from the Packer template in examples/consul-ami/consul.json.
+
+set -e
+
+# Send the log output from this script to user-data.log, syslog, and the console
+# From: https://alestic.com/2010/12/ec2-user-data-output/
+exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
+
+# These variables are passed in via Terraform template interplation
+/opt/consul/bin/run-consul --client --cluster-tag-key "${cluster_tag_key}" --cluster-tag-value "${cluster_tag_value}"
+
+# You could add commands to boot your other apps here

examples/example-with-consul-connect/user-data-server.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# This script is meant to be run in the User Data of each EC2 Instance while it's booting. The script uses the
+# run-consul script to configure and start Consul in server mode. Note that this script assumes it's running in an AMI
+# built from the Packer template in examples/consul-ami/consul.json.
+
+set -e
+
+# Send the log output from this script to user-data.log, syslog, and the console
+# From: https://alestic.com/2010/12/ec2-user-data-output/
+exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
+
+# These variables are passed in via Terraform template interplation
+/opt/consul/bin/run-consul --server --cluster-tag-key "${cluster_tag_key}" --cluster-tag-value "${cluster_tag_value}" --enable-connect

examples/example-with-consul-connect/variables.tf

@@ -0,0 +1,61 @@
+# ---------------------------------------------------------------------------------------------------------------------
+# ENVIRONMENT VARIABLES
+# Define these secrets as environment variables
+# ---------------------------------------------------------------------------------------------------------------------
+
+# AWS_ACCESS_KEY_ID
+# AWS_SECRET_ACCESS_KEY
+# AWS_DEFAULT_REGION
+
+# ---------------------------------------------------------------------------------------------------------------------
+# OPTIONAL PARAMETERS
+# These parameters have reasonable defaults.
+# ---------------------------------------------------------------------------------------------------------------------
+
+variable "ami_id" {
+  description = "The ID of the AMI to run in the cluster. This should be an AMI built from the Packer template under examples/consul-ami/consul.json. To keep this example simple, we run the same AMI on both server and client nodes, but in real-world usage, your client nodes would also run your apps. If the default value is used, Terraform will look up the latest AMI build automatically."
+  type        = string
+  default     = "ami-0fce111d3bd9490e1"
+}
+
+variable "cluster_name" {
+  description = "What to name the Consul cluster and all of its associated resources"
+  type        = string
+  default     = "consul-example"
+}
+
+variable "num_servers" {
+  description = "The number of Consul server nodes to deploy. We strongly recommend using 3 or 5."
+  type        = number
+  default     = 3
+}
+
+variable "num_clients" {
+  description = "The number of Consul client nodes to deploy. You typically run the Consul client alongside your apps, so set this value to however many Instances make sense for your app code."
+  type        = number
+  default     = 6
+}
+
+variable "cluster_tag_key" {
+  description = "The tag the EC2 Instances will look for to automatically discover each other and form a cluster."
+  type        = string
+  default     = "consul-servers"
+}
+
+variable "ssh_key_name" {
+  description = "The name of an EC2 Key Pair that can be used to SSH to the EC2 Instances in this cluster. Set to an empty string to not associate a Key Pair."
+  type        = string
+  default     = "nt-trial"
+}
+
+variable "vpc_id" {
+  description = "The ID of the VPC in which the nodes will be deployed.  Uses default VPC if not supplied."
+  type        = string
+  default     = null
+}
+
+variable "spot_price" {
+  description = "The maximum hourly price to pay for EC2 Spot Instances."
+  type        = number
+  default     = null
+}

modules/run-consul/README.md

@@ -80,6 +80,8 @@ The `run-consul` script accepts the following arguments:
 * `ca-file-path` (optional): Path to the CA file used to verify outgoing connections. Must be specified with `enable-rpc-encryption`, `cert-file-path` and `key-file-path`.
 * `cert-file-path` (optional): Path to the certificate file used to verify incoming connections. Must be specified with `enable-rpc-encryption`, `ca-file-path`, and `key-file-path`.
 * `key-file-path` (optional): Path to the certificate key used to verify incoming connections. Must be specified with `enable-rpc-encryption`, `ca-file-path` and `cert-file-path`.
+* `enable-connect` (optional): If this flag is set, turn on Consul Connect when bootstrapping a cluster. To specify your own CA, specify an override config as outlined below.
+* `services-config-dir` (optional): Path to dir of one or many service configurations. Services can be further configured to set Consul Connect Proxies, Sidecar registrations & upstream service dependencies for a Service Mesh
 * `skip-consul-config` (optional): If this flag is set, don't generate a Consul configuration file. This is useful if
   you have a custom configuration file and don't want to use any of of the default settings from `run-consul`.
 
@@ -276,4 +278,4 @@ track other servers. A server is considered healthy when:
 There are Autopilot settings called [upgrade migrations](https://www.consul.io/docs/guides/autopilot.html#upgrade-migrations)
 that are useful when adding new members to the cluster either with newer configurations or using
 newer versions of Consul. These configurations manage how Consul will promote new servers and demote
-old ones. These settings, however, are only available at the Consul Enterprise version. 
+old ones. These settings, however, are only available at the Consul Enterprise version.