Question: Multiple apps on one host sharing the same traefik instance

[jpdombrowski]

I have been experimenting w/ MRSK over the last week to see how many different deployment scenarios I could cover w/ it and so far it performed terrific.

One particulary scenario feels a bit ikky at this point though.

I deployed a Rails API and a Vue.js app (static files w/ Nginx in front) on a single host w/ lots of customization in the treafik config (endpoints for port 80 and 443, TLS termination w/ letsencrypt certificate generation, redirect from http to https, gzip compression) with no other proxy (load balancers or cdn) in front. Traefik was put into host networking mode to get real ips b/c we do not get proxy headers in this scenario, which sucks but a proxy / load balancer in front would solve this. All traefik config ended up in ONE of the two repos.

So far this actually works well w/ some gotchas.

• The repo NOT managing Traefik must never run mrsk deploy / mrsk traefik reboot.
• Instead you can run mrsk build create, mrsk registry login, mrsk redeploy to not touch the Traefik instance on the first deploy and mrsk redeploy on subsequent deploys.

Two solutions that came to mind are hard to maintain.

1. The Traefik config could be mirrored to both repos.
2. Alternatively one of the two apps could end up as an accessory, but that would mean to docker build in repo 1, then go to repo 2, update deploy.yml, and redeploy accessories from there.

Granted, this is problaby not your main concern w/ the work MRSK is doing for you at 37signals. In production scenarions multiple apps would probaby end up on different hosts and Traefik customization would be minimal b/c we would have load balancers or a CDN in front doing most of the work for us. BUT the Readme describes "to run multiple applications on the same server sharing the same Traefik instance and port" as a feature, Traefik is great at that and still every setup doing this should run into the issues described.

So question is, am I doing it wrong or stretching MRSK too far?

One possible solution I see is allowing a web: false or traefik: false on the web role which I understand is impossible right now.

The additional benefit of enabling this would be that cmd role / job workloads could be deployed w/ a different Dockerfile as well w/o ending up as an accessory.

Another way could be to configure different Dockerfiles per role and doing monorepos.

Thanks for your thoughts and the terrific work so far!

[dhh]

This does seem like probably stretching things too far. I'm still not convinced that we actually should have supported multiple apps on the same hosts. I think things become a lot simpler when you can ensure it's one app per host (even if that host is allowed accessories).

Are you deploying on your own hardware, is that why you don't want to separate apps by host?

[jpdombrowski]

Are you deploying on your own hardware, is that why you don't want to separate apps by host?

That's one of the potential scenarios I was thinking about.

My main concern is simplicity as well, but from another angle. Slicing bare metal into VMs (higher complexity) might be "the way", but sometimes isolation w/ containers could be "good enough".

I see situations where a single host deployment is fine for scenarios w/ low resource requirements in general or for developers who are testing things out and want to use MRSK to deploy rough dev stages to a single host while sticking to the tooling they know.

Another concern is toolchain consolidation. I love the idea of streamlining Kubernetes and capistrano deployments to one new imperative system with the full benefits of containerization, no matter if I want to scale a Rails app to multiple servers or deploy a Rust job queue worker fleet or a small-ish single host deployment for multiple irregular cron-type workloads. Right now cron jobs and workers have to live in the same code base as your main web-type application, which feels restrictive.

That being said I agree that no elegant solution to resolve the "where to put a shared Traefik config for multiple containers on a single host" problem is in clear sight right now. And falling back to capistrano to deploy containers when MRSK is out of its comfort zone might be a valid albeit less appealing alternative.

Would be interested to hear how you handle these types of scenarios or if you managed to resist to break smaller services / workers out of Rails 100%.

[jacoscaz]

Am I correct if I state that MRSK's implies a deployment model geared towards high(er) numbers of micro-servers (virtual or not), each running a single instance of a single app (excl. accessories) ?

At the other end of the spectrum would be a deployment model in which a limited number of beefy servers would host multiple instances of (potentially) multiple apps.

The descriptions of commands such as mrsk remove, stating Remove Traefik, app, accessories, and registry session from servers, imply that Traefik itself is bound to one specific application. Calling mrsk remove would, for example, entirely remove Traefik from the host, impacting all other co-hosted app instances.

[conorh]

Just to address this part:

I'm still not convinced that we actually should have supported multiple apps on the same hosts. I think things become a lot simpler when you can ensure it's one app per host (even if that host is allowed accessories).

We host many applications, perhaps 6 or more on the same server, the reason we do this is because the majority of those applications receive very little traffic. Instead of paying for 6 separate servers we pay for one server, with enough ram to allow for bursty traffic or other ram intensive operations when needed.

[jacoscaz]

After experimenting with MRSK some more - and loving it! - I do agree that an improvement in this regard would be to make the web role not mandatory, merely enforcing the presence of at least any one role. I have not (yet) taken a look at the codebase but this should be a relatively small change that would go a long way towards improving support for deploying multiple apps on the same server, especially when coupled with the ability to pass custom container options via the options dictionary.

[laptopmutia]

so you could do 1 vps multiple apps? how?

just by pointing to same IP?

[jacoscaz]

Given we can pass custom run options to Docker itself, hosting additional applications would simply be a matter of mapping to the correct ports. Any application deployed outside of the web role would loose the benefit of having Traefik in between to achieve zero-downtime deployments, yes, but that would IMHO be an acceptable compromise given how minimally invasive the required change would be (insofar as I can image, I haven't looked at the code yet).

[jacoscaz]

Furthermore, in most of the use cases I am familiar with we would only deploy one web application per server. All other applications would be non-web components of our systems.

[n-studio]

@jpdombrowski I'm also experimenting hosting several apps on a single server. How did you configure your traefik to handle the traffic for both your apps?

[n-studio]

For anyone interested, here is a gist of how I set up my dedicated server to host multiple apps: https://gist.github.com/n-studio/ac38cf85022dfebcc44b1cc8c6790301

[meglio]

Crucial to host multiple apps on a single server - for me too.

[tabishiqbal]

Came across this article that allows you to host multiple apps on a single server: https://www.erikminkel.com/2023/09/29/using-kamal-to-host-multiple-apps-on-a-single-server/

[tangit86]

That was pretty much it :)

[divyenduz]

How do you manage the traefik configuration in the two deploy files? Make it the same for both?
Let's say I want to expose traefik dashboard, so totally unrelated to my apps. Will I make the "exposing dashboard" part of traefik copied in both deploy.yml files or is there a better way?

[hrstoyanov]

A good feature to have, if you run Kamal on a home lab mini server, exploring a few startup MVPs, and do not want to pay the price for (full) Proxmox/VMware virtualization, where Docker/Podman container isolation is just enough.

[skojin]

Host severals staging apps on single server is what I need

[Matic]

I think it would be beneficial to support either:

• multiple roles from multiple images or
• having the option to provide a docker file for accessories, which would then build those accessories during the build stage.

It would also help if they would share the network.

Example:

• a next.js server that you can use for SSR + Frontend (role web), the image needs to be rebuilt on each new deploy
• a django/flask... as a backend (role backend), the image needs to be rebuilt on each new deploy
• accessory db with a prebuilt image
• accessory redis with a prebuilt image
• accessory chromadb with a custom Dockerfile in order to be able to modify some settings
  ... and multiple other options

Support for such a setup would enable unique ways to grow the project. You could start by running all on one node, then as the traffic grows you can easily move the roles and accessories around and scale on to multiple nodes.

[jpdombrowski]

All of the problems I described in my initial post have been solved in principle with the introduction of #580, which makes the traefik enabled web role optional.

We can now say:

primary_role: workers

servers:
  workers:
    traefik: false
    hosts:
      - my-host.example.com

This solves a lot.

We can now run standalone cmd role / job workloads and even put them on the same host with another kamal web deployment (and its traefik instance).

We could also deploy multiple web apps on the same host by managing traefik in only one of them or by pulling traefik out of kamal entirely and manage it on the host directly or even replace it with another reverse proxy of our own choosing.

EDIT: The last use cases are not ideal, b/c of healthchecks / zero-downtime issues but they should be considered "hacky" anyway.

[99linesofcode]

Definitely a fan of this being now possible. I feel Kamal is perfectly able to deploy some or all services to the same host without it turning into a jack of all trades master of non. Personally I love to be able to use a tool that scales with my needs. From small side projects running on a home server or cheap VPS to something that spans multiple containers and or cloud providers.

As it stands now it can deploy to the same host but in my case I get stuck routing traffic from a reverse proxy already running on the host to newly deployed application because there seems to be no way to tell Kamal to join an existing network. At least, if I want to use the container name and don't want to resort to manually overriding iptables.
The above doesn't seem to be true and is probably the result of me not thoroughly understanding Kamal yet.

That being said I feel like it would be great if we could use every docker and docker-compose configuration option with a few sprinkles of Kamal ontop. Passing in Dockerfiles to hosts/accessories or possibly handling the majority of configuration through docker compose.

[AlexHashgraph]

Were you able to route traffic from the existing reverse proxy to the newly deployed applications through kamal using the container name or hostname? i can't find a way to keep either one of these static between deployments

[miharekar]

While possible but hacky currently, it looks like this is coming with full support in Kamal 2: https://twitter.com/dhh/status/1766151697817313628

[jacoscaz]

@miharekar thank you for linking to that tweet - fantastic news!

[dehoej]

2.0.0 beta is released. Does anyone know if the feature has been implemented?

[miharekar]

Well it no longer uses Traefik, but in the readme of kamal proxy it says:

you can run multiple applications on the same server without port conflicts.

[dhh]

Kamal 2 will be out very soon and this is one of the two headline features (the other is auto SSL)!