Per-vhost connection limit

[michaelklishin]

See #498 for background. We already have a channel limit but this should perhaps be configured via a policy to be more flexible. Note that this means extending policies to vhosts, so using effort-medium as it can take more than a couple of days.

[michaelklishin]

The more I think about it, the more I'm considering adding a separate mechanism to policies for vhosts. It won't be conceptually different and might reuse rabbit_registry or even more but it has to be separate, otherwise we risk confusing the user (as policies that apply to queues and exchanges already have a logical connection with a vhost).

For example, this feature can be called "vhost limits" and used much like policies are but with a new rabbitmqctl command.

@dumbbell @hairyhum @camelpunch @vgkiziakis do you have an opinion on this?

[michaelklishin]

Some research suggests the above fits our existing runtime parameters framework very well.

[michaelklishin]

I'm going to work on a spike for this this week.

[michaelklishin]

New ctl command I'm working with:

rabbitmqctl set_vhost_limits -p a-vhost --max-connections [val] --max-queues [val]

alternatively it can be

rabbitmqctl set_vhost_limits -p a-vhost '{"max-connections": 1000, "max-queues": 10000}'

(so, as a JSON object) but I'm leaning towards the former.

HTTP API version will have to use JSON, of course, when we introduce it.

[michaelklishin]

After a discussion with the team we ended up with

rabbitmqctl set_vhost_limits -p a-vhost '{"max-connections": 1000, "max-queues": 10000}'

[michaelklishin]

Setting the limit(s) is in place. We are now looking into what's the best way to keep track of connections scoped per vhost across the entire cluster. The tricky part is nodes going down, being restarted, changing names, and so on.

[michaelklishin]

An update on this: naive connection tracking couples rabbit_common with rabbit via rabbit_reader's need to use Mnesia (via rabbit_connection_tracking). A better way to do this would be to introduce a gen_event handler that only listens to events on the node it is running on.

This is similar to how management stats DB keeps track of connections, except that here we don't want all nodes to perform updates, only the local one.

[michaelklishin]

Considering the scope of the changes, I'm moving this to 3.7.0 because our internal data store schema changes in a way that will prevent 3.6.0 nodes from accepting it from later 3.6.x releases.

[michaelklishin]

Connection limit is in place, I'm going to work on some (synthetic) benchmarks to see how efficient our queries are with a few millions rows. In case it's not efficient enough we may need to take a different approach and use separate tables with counters.

[michaelklishin]

I created a synthetic benchmark that loads all connections in one of 10 vhosts with 1m records in the table. It takes over 1 second to count connections that way. Not surprising but a data point. So we need to use ets:select_count/2 (benchmarking it next) or separate tables with counter fields.

[michaelklishin]

Expectedly counter columns produce < 100 microseconds responses even with 10m connections.

[michaelklishin]

Limit support is done, and two more related features are extracted into #627 and #628. What's left to cover in this issue:

• Integration testing
• Node unavailability handling