feat: Extend rabbitmq chart with autocluster to support multiple rabbitmq instances

[alanmeadows]

See https://github.com/aweber/rabbitmq-autocluster

We should investigate leveraging the DNS backend, since this should be supported natively with a kubernetes service, as opposed to using etcd.

For a working example, see https://github.com/openstack/fuel-ccp-rabbitmq - however this example uses etcd and would prefer to validate the DNS approach works.

[DTadrzak]

One guy from Intel is working to improve rabbitmq chart.

[alanmeadows]

Great news @DTadrzak. Once we tackle rabbit and memcached, the resiliency will be significantly improved.

[intlabs]

@SamYaple may also have some good insight on this.

[ghost]

I use rabbitmq-clusterer. It works well. autoclusterer is great if you have consul/etcd or something that it supports already setup. clusterer doesn't have any external deps, but requirements better config management from a deployment tool.

Just for more info, rabbitmq 3.7.0 has most of the features from both of these tools baked in.

[DTadrzak]

AFAIK @ss7pro's solution is based on etcd.

[ghost]

then autoclusterer is a good choice, yes. In you situation, I would go with autoclusterer.

[janwillies]

We are running RabbitMQ with Kubernetes-based discovery and it's not great tbh. While I agree it's the cleanest approach there's lots of custom work to be done. You need to run an unreleased version of autocluster (HEAD basically). You need to hack-in a custom restart because autocluster doesn't like being started together with other plugins (or vice versa). Then the RabbitMQ cluster is happily partitioning on the first occasion and you need to handle that somehow. While it sounds easy on the Mirantis blog post, autocluster:cluster_health_check_report() is only available in their fork and tailored towards the etcd backend.

[alanmeadows]

@janwillies This is useful real-world information.

I am hearing that the the dream of a "native" DNS-backed autoclustering scenario is the part of this issue that should be put on ice for the moment.

With that said, would an alternative etcd based autocluster approach also require the same great care for standing it up and ensuring reliability? @SamYaple is saying hes had success there (etcd based autocluster) so I am trying to ensure we aren't going down a path that is contrary to the goal of resiliency.

I'm fine if native isn't quite there -- we will likely find other uses for an etcd cluster as well. We always have to strike a balance. The ideal is native, but not at the expense of functionality.

[ghost]

3.7.0 rabbitmq will solve alot of issues here. They pull in alot of autocluster code and make clustering easier as well.

As for the partitioning, we should absolutely be running with automatic partition recovery. With rabbitmq availability > lose of message

Food for thought

[v1k0d3n]

@SamYaple this is valuable input. we should look at:

rabbitmq/rabbitmq-server#486
and
rabbitmq/rabbitmq-server#988

and edit Charts according to this path forward.

[ghost]

Yea @v1k0d3n those features were pulled out of autocluster. The guy that wrote autocluster is a member of the rabbitmq team. Lots of synergy there.

[ss7pro]

Mirantis did significant amount of work in making sure that autocluster with etcd backend is bulletproof (race conditions, split brains, recovery, scale up/scale down). My PR #123 is 100% based on Mirantis work.

[v1k0d3n]

going to say that this is closed, however we need to do some further work to nail down the dependency on etcd. this takes us a bit off what we do today. this would normally be handled as part of the chart (as an init-container), rather than another chart.