OpenShift PaaS Build July 2016
The notes and diagram below represents our planned deployment layout/architecture.
- The diagram is deliberately abstract, but is mainly intended to provide a reference point for discussion of the overall architectural topology/layout. It does, however, represent our current intention, and is our likely target, in the absence of informed advice to the contrary.
- Many of the items in the diagram and elsewhere in this page are “proposed”/draft and open to discussion/advice.
- We expect our eventual workloads to be diverse and range from small applications with a handful of users to moderately large applications with 1000s of concurrent users. We expect a variety of tech stacks from lightweight microservices to “modern” full-stack webapps (e.g. MEAN) to “traditional” webapps (e.g. JBoss Java + RDBMS). Our initial workloads will be towards modern webapps.
- This environment will be within its own dedicated VLAN
- Overall architecture is OpenShift HA cluster on VMWare cluster; clustered VMWare infrastructure will provide additional level of resiliency from OpenShift/Kubernetes
- VMWare affinity rules will be configured to ensure that each OpenShift Master VM host, OpenShift compute Node VM host, OpenShift infrastructure Node VM host, and etcd instance run on separate hardware hosts from each other
- Initial sizing (vCPUs + RAM) of VMs will meet or exceed minimum requirements as specified by Red Hat
- Storage is backed by SAN storage array
- Is the described architecture appropriate to achieve the objective of a reasonably performant, resilient and highly available environment in which application users and developers could expect to see no service interruptions as a result of normal operational procedures such as host patching, backups, host scaling/resizing or moderate levels of system/infrastructure failure?
- Is this environment radically over- or under-provisioned in any way?
- What are the implications/tradeoffs of the multi-tenant SDN plugin vs. standard?
- What are the considerations on different approaches to load balancing across the 2 application HA-Proxy routers? Round robin DNS vs. existing shared reverse proxy (F5, I believe)?
Host Inventory
| Internal Hostname | IP Address | Role | vCPUs | RAM |
|---|---|---|---|---|
| ociopf-e-001.dmz | Master #1 | 2 | 16GB | |
| ociopf-e002.dmz | Master #2 | 2 | 16GB | |
| ociopf-e003.dmz | Master #3 | 2 | 16GB | |
| ociopf-e004.dmz | Master Load Balancer | 2 | 16GB | |
| ociopf-e005.dmz | Infrastructure Node | 2 | 16GB | |
| ociopf-e006.dmz | Infrastructure Node | 2 | 16GB | |
| ociopf-e007.dmz | Node #1 | 2 | 64GB | |
| ociopf-e008.dmz | Node #2 | 2 | 64GB | |
| ociopf-e009.dmz | Node #3 | 2 | 64GB | |
| ociopf-e010.dmz | Node #4 | 2 | 64GB | |
| ociopf-e011.dmz | Node #5 | 2 | 64GB | |
| ociopf-e012.dmz | Node #6 | 2 | 64GB | |
| ociopf-e013.dmz | NFS Server | 2 | 16GB |
| Host | Mount point | Size | Tier | Purpose |
|---|---|---|---|---|
| ociopf-e-001.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e-001.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e-001.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e002.dmz | /var/lib/etcd | 50 | 3 | etcd storage |
| ociopf-e002.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e002.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e-001.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e002.dmz | /var/lib/etcd | 50 | 3 | etcd storage |
| ociopf-e003.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e003.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e003.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e003.dmz | /var/lib/etcd | 50 | 3 | etcd storage |
| ociopf-e005.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e005.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e005.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e005.dmz | /registry | 300 | 3 | OpenShift Docker Registry |
| ociopf-e006.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e006.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e006.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e006.dmz | /registry | 300 | 3 | OpenShift Docker Registry |
| ociopf-e007.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e007.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e007.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e008.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e008.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e008.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e009.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e009.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e009.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e010.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e010.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e010.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e011.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e011.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e011.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e012.dmz | /var/lib/docker | 300 | 3 | Docker container storage |
| ociopf-e012.dmz | docker_vg (logical volume) | 300 | 3 | Docker image cache |
| ociopf-e012.dmz | /var/log | 100 | 3 | System logs |
| ociopf-e013.dmz | various | 1000 | 2 | Application / database persistent storage |
Updated Storage Details (July 29)
| ociopf-d-101.dmz | | | |------------------|---------| | docker-pool | 119.88g | | docker-poolmeta | 44.00m | | var_lib_docker | 300.00g | | var_lib_etcd | 50.00g | | var_log | 100.00g | | ociopf-d-131.dmz | | | | docker-pool | 119.88g | | docker-poolmeta | 44.00m | | var_lib_docker | 300.00g | | var_lib_origin | 100.00g | | registry | 300.00g | | var_log | 100.00g | | ociopf-d-110.dmz | | | | docker-pool | 119.88g | | docker-poolmeta | 44.00m | | var_lib_docker | 300.00g | | var_log | 100.00g | | ociopf-d-133.dmz | | | | docker-pool | 119.88g | | docker-poolmeta | 44.00m | | var_lib_docker | 300.00g | | var_lib_origin | 100.00g | | registry | 300.00g | | var_log | 100.00g | | ociopf-d-102.dmz | | | | docker-pool | 119.88g | | docker-poolmeta | 44.00m | | var_lib_docker | 300.00g | | var_lib_etcd | 50.00g | | var_log | 100.00g | | ociopf-d-111.dmz | | | | docker-pool | 119.88g | | docker-poolmeta | 44.00m | | var_lib_docker | 300.00g | | var_log | 100.00g | | ociopf-d-132.dmz | | | | docker-pool | 119.88g | | docker-poolmeta | 44.00m | | var_lib_docker | 300.00g | | var_lib_origin | 100.00g | | registry | 300.00g | | var_log | 100.00g | | ociopf-d-135.dmz | | | | docker-pool | 119.88g | | docker-poolmeta | 44.00m | | var_lib_docker | 300.00g | | var_lib_origin | 100.00g | | registry | 300.00g | | var_log | 100.00g | | ociopf-d-100.dmz | | | | docker-pool | 119.88g | | var_lib_docker | 300.00g | | var_lib_etcd | 50.00g | | var_log | 100.00g | | ociopf-d-130.dmz | | | | docker-pool | 119.88g | | var_lib_docker | 300.00g | | var_lib_origin | 100.00g | | registry | 300.00g | | var_log | 100.00g | | ociopf-d-134.dmz | | | | docker-pool | 119.88g | | docker-poolmeta | 44.00m | | var_lib_docker | 300.00g | | var_lib_origin | 100.00g | | registry | 300.00g | | var_log | 100.00g |
TODO