Mesos marathon Docker cluster setup on RHEL 7 with three masters

Consider myserver1,myserver3,myserver7 are masters and myserver4,myserver2 as salves

Setting up masters

on each node of master myserver1,myserver3,myserver7 Ip address of these servers are as follows respectively

Servername	ip address
myserver1	10.193.154.120
myserver3	10.193.154.122
myserver7	10.193.154.140

on each master server

• sudo rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
• sudo yum -y install mesos marathon
• sudo yum -y install mesosphere-zookeeper

myserver1

echo 1 > /var/lib/zookeeper/myid
echo '10.193.154.120' > /etc/mesos-master/hostname

myserver3

echo 3 > /var/lib/zookeeper/myid
echo '10.193.154.122' > /etc/mesos-master/hostname

myserver7

echo 7 > /var/lib/zookeeper/myid echo '10.193.154.140' > /etc/mesos-master/hostname

on each master server

• /etc/zookeeper/conf/zoo.cfg add this entry to the end of config file
  server.1=10.193.154.120:2888:3888
server.3=10.193.154.122:2888:3888
server.7=10.193.154.140:2888:3888

• sudo systemctl start zookeeper

• echo 'zk://10.193.154.120:2181,10.193.154.122:2181,10.193.154.140:2181/mesos' > /etc/mesos/zk

• echo 2 > /etc/mesos-master/quorum

• systemctl stop mesos-slave.service

• sudo service mesos-master restart

• sudo service marathon restart

setting up slaves

on each server

systemctl disable mesos-master myserver4,myserver2

• sudo rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
• sudo yum -y install mesos
• echo 'docker,mesos' > /etc/mesos-slave/containerizers
• echo '5mins' > /etc/mesos-slave/executor_registration_timeout
• echo 'zk://10.193.154.120:2181,10.193.154.122:2181,10.193.154.140:2181/mesos' > /etc/mesos/zk
• sudo service mesos-slave restart

Using master also as slaves

on each master

• systemctl enable mesos-master
• echo 'docker,mesos' > /etc/mesos-slave/containerizers
• sudo service mesos-slave restart

Automatic service Discovery of micro services / rest api's

Suppose you have a gateway server mygatewayserver from where you need to orchestrate your services/micro services

Install HAproxy

• rpm -Uvh haproxy-1.5.4-2.el7.x86_64.rpm # please find this rpm attached in this repo
• useradd --system haproxy
• systemctl start haproxy.service

setup mesos bridge

script that automatically loads haproxy.cfg when ever a new micro service is created

• wget https://raw.githubusercontent.com/mesosphere/marathon/master/bin/haproxy-marathon-bridge
• chmod +x haproxy-marathon-bridge
• ./haproxy-marathon-bridge install_cronjob
• To /etc/haproxy-marathon-bridge/marathons add following
  myserver1:8080
myserver3:8080
myserver7:8080

starting a new container

to any master say http://myserver7:8080/v2/apps make a post request

{
    "id": "hello",
    "container": {
        "type": "DOCKER",
        "docker": {
            "image": "cjonagam/helloworld",
            "network": "BRIDGE",
            "portMappings": [
                { "containerPort": 8080, "hostPort": 0, "servicePort": 8000, "protocol": "tcp" }
             ]
        }
    },
    "env":
       { "MSG": "chakri" },
    "healthChecks": [
    {
      "protocol": "HTTP",
      "portIndex": 0,
      "path": "/",
      "gracePeriodSeconds": 5,
      "intervalSeconds": 20,
      "maxConsecutiveFailures": 3
    }]
}

Testing the setup

Now if you make a http get request to mygatewayserver:8000 port you should see hello world
curl http://mygatewayserver:8000

Hello world

Trouble Shooting

Some times slaves do not pick up jobs

you may need to restart docker deamon
rm -f /tmp/mesos/meta/slaves