RabbitMQ¶
Introduction¶
The following information are mostly taken from a discussion we had on the mailing list. You can see this discussion here:
http://lists.openstack.org/pipermail/openstack-discuss/2020-August/thread.html#16362
The quorum queues / streams part has been added later, after the “Who framed RabbitMQ” presentation at OpenInfra Summit 2025:
https://www.youtube.com/watch?v=Uj4elX2OONw
and the following discussion:
Clustering or not clustering?¶
When deploying RabbitMQ, you have two possibility:
Deploy rabbit in a cluster
Deploy only one rabbit node
Deploying only one rabbit node can be seen as dangerous, mostly because if the node is down, your service is also down.
On the other hand, clustering rabbit has some downside that make it harder to configure / manage.
So, if your cluster is less reliable than a single node, the single node solution is better.
Moreover, as many OpenStack services are using RabbitMQ for internal communication (a.k.a. RPC), having a highly available rabbit solution is a must have.
If you choose the clustering mode, you should always keep an odd number of servers in the cluster (like 3 / 5 / 7, etc) to avoid split-brain issues.
One rabbit to rule them all?¶
You can also consider deploying rabbit in two ways:
one rabbit (cluster or not) for each OpenStack services
one big rabbit (cluster or not) for all OpenStack services
The recommendation is to split your rabbit in separate clusters for multiple reasons:
Reduce the impact when a rabbit cluster is down
Allow intervention on smaller part of infrastructure
Note: you can considerer splitting at least neutron and nova from other services, so you would end up with 3 clusters.
Note 2: neutron is the heaviest project for rabbit when using ovs. So keep in mind that the rabbit cluster for neutron should be your biggest.
Which version of rabbit should I run?¶
You should always try to consider running the latest version of rabbit.
We also know that rabbit before 3.8 may have some issues on clustering side, so you might consider running at least rabbitmq 3.8.x.
See https://groups.google.com/forum/#!newtopic/rabbitmq-users/rabbitmq-users/zFhmpHF2aWk
Quorum queues need rabbitmq 3.8+ and streams need rabbitmq 3.9+. Classic mirrored queues are removed in rabbitmq 4.0, so if you run 4.x you must use quorum queues or HA won’t be possible.
Rabbit config recommendation¶
Most of the config we explain in the following parts apply only on rabbitmq in clustering mode.
Service config¶
When running the rabbit software on a node, you can configure some parameters for it in:
/etc/rabbitmq/rabbitmq.config # (debian / ubuntu)
The most important configuration are the following
Erlang scheduler configuration¶
If you run multiple rabbitmq clusters it is common to run these clusters on the same 3 hosts. If you do this then you must ensure that all of the rabbitmq/erlang processes are not pinned to particular cores. This ensures that erlang does not compete over using the same cpu cores. To do this start erlang/the beam VM using -stbt u or -sbt u (you might need to replace - with + depending on how you start erlang).
For details see https://www.erlang.org/doc/man/erl.html#+sbt
net_ticktime and heartbeat¶
See https://www.rabbitmq.com/nettick.html and https://www.rabbitmq.com/heartbeats.html
A node is considered down after this time
This config mostly depends on the network you have between the nodes.
We consider that the default values are good.
disk / ram mode¶
https://www.rabbitmq.com/clustering.html#cluster-node-types
“In the vast majority of cases you want all your nodes to be disk nodes; RAM nodes are a special case that can be used to improve the performance clusters with high queue, exchange, or binding churn. RAM nodes do not provide higher message rates. When in doubt, use disk nodes only.”
So we recommend staying on disk node
OpenStack services config¶
Quorum queues and streams¶
This is now the recommended setup (oslo.messaging >= 14.7, i.e. 2024.1 Caracal), instead of classic mirrored queues.
Set the following in the [oslo_messaging_rabbit] section of every OpenStack
config file:
[oslo_messaging_rabbit]
rabbit_quorum_queue = True
rabbit_transient_quorum_queue = True
rabbit_transient_queues_ttl = 0
rabbit_qos_prefetch_count = 1
use_queue_manager = True
rabbit_stream_fanout = True
kombu_reconnect_splay = 2
[oslo_concurrency]
lock_path = /var/lib/<service>/tmp
rabbit_quorum_queue¶
Use quorum queues (x-queue-type: quorum) for RPC queues.
Quorum queues are replicated (Raft) and durable by nature, so
amqp_durable_queues is ignored and no ha-* policy is needed.
This conflicts with rabbit_ha_queues, which must be disabled.
rabbit_transient_quorum_queue¶
Also use quorum queues for transient queues (reply_ and fanout queues).
Mandatory when rabbit_stream_fanout is enabled.
use_queue_manager¶
By default, oslo.messaging uses a random uuid in reply_ and fanout queue
names. So each time a service restarts, new queues are created and the old
ones have to expire and be deleted by rabbit, which costs a lot at scale.
With the queue manager, queue names are built from hostname, processname
and a counter, so a restarted service reuses the same queues.
This is also highly recommended with rabbit_transient_quorum_queue, to
avoid exhausting erlang atoms over time.
The counter is kept in /dev/shm and protected by an oslo.concurrency
external lock, so [oslo_concurrency]/lock_path must be set in every
service using it. Otherwise, you will get an obscure error like:
oslo_config.cfg.RequiredOptError: <unprintable RequiredOptError object>
raised from oslo_messaging/_drivers/amqpdriver.py in _get_reply_q.
rabbit_stream_fanout¶
Use a single stream (x-queue-type: stream) per fanout exchange instead of
one fanout queue per consumer. Streams are persistent, replicated and
non-destructive, so this drastically reduces the number of queues to manage
(especially for neutron).
Requires rabbit_transient_quorum_queue = True and
rabbit_qos_prefetch_count > 0.
rabbit_qos_prefetch_count¶
The default is 0 (unlimited). We recommend 1.
A higher value (like 100) works most of the time, but we identified that some messages were lost during services restart (e.g. nova-conductor or neutron rpc server). With 1, services handle messages one by one, and nothing is prefetched (and lost) when a service stops.
It also gives a better dispatching of the messages: with a high prefetch, one worker can grab a batch of messages while the others are idle. With 1, each message goes to the next available worker.
This does not limit concurrency: there are still multiple workers
(rpc_workers) and multiple threads per worker (executor_thread_pool_size), and
RPC servers are usually oversized so there is no message queueing in rabbit
even with a very low prefetch.
Remember that the issue at scale has never been the message throughput (rabbit can handle thousands of messages per second, OpenStack does not need that) but the number of queues.
rabbit_transient_queues_ttl¶
The default is 1800 (30 minutes). We recommend 0. Yes, that sounds weird. Here is why.
This option sets x-expires on reply and fanout queues (and x-max-age
on fanout streams when rabbit_stream_fanout is enabled). With 0, nothing
is set by oslo.messaging and the retention is fully handled by the rabbit
policies (expires for quorum queues, size based retention for streams, see
Policies), so there is a single place to tune it.
This follows the RabbitMQ guidance: queue arguments set by the client
(x-expires, x-max-age, …) are fixed at declaration time, changing
them requires a config change, a service restart and a queue re-declaration,
while policies are designed for parameters that can change at runtime.
Moreover, when the same key is set by both, the client argument wins, so the
policy would be ignored for those queues.
See https://www.rabbitmq.com/docs/policies#policies-vs-the-optional-arguments-set-by-applications
With use_queue_manager, a restarted service reuses its previous queues, so
there is no more pile of expired queues to delete after a restart anyway.
kombu_reconnect_splay¶
Random extra delay (in seconds, 0 by default) added before reconnecting to rabbit. When a rabbit node goes down, all the services connected to it reconnect at the same time: a small splay spreads the reconnection storm.
rabbit_ha_queues and amqp_durable_queues¶
You may see these parameters in some configuration files, they were used with classic mirrored queues.
rabbit_ha_queues conflicts with rabbit_quorum_queue and must be
disabled. amqp_durable_queues is ignored, quorum queues are always durable.
Note that the type and durability of a queue cannot be changed AFTER the queue has been created. So when migrating from classic to quorum queues, you will have to delete your queues before OpenStack can recreate them with the correct type.
update_resources_interval¶
This is a nova settings: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.update_resources_interval
This option specifies how often the update_available_resource periodic task should run.
The default periodic interval is 60 seconds, which could be too often for large-scale.
The recommendation is to increase this value to something that would fit your needs, like 600 secs:
update_resources_interval = 600
Policies¶
RabbitMQ apply some policies on queues and exchanges. See here: https://www.rabbitmq.com/parameters.html
Remember that Rabbit can apply only one policy to a queue or an exchange. So you should avoid having multiples policies in your deployment, or if you do, try to avoid overlapping policies because you wont be able to predict which one is effective on a queue.
With quorum queues and streams (see Quorum queues and streams), replication is a
property of the queue type itself, so no ha-* policy is needed.
We only use two policies, one per queue type
(--apply-to quorum_queues / streams is available since rabbitmq 3.12):
rabbitmqctl set_policy --apply-to quorum_queues quorum-policy '^(?!amq\.).*' \
'{"expires": 86400000, "message-ttl": 600000}'
rabbitmqctl set_policy --apply-to streams stream-policy '.*_fanout.*' \
'{"max-length-bytes": 15000000, "stream-max-segment-size-bytes": 5000000}'
Time is in milliseconds as per https://www.rabbitmq.com/ttl.html
quorum-policy¶
Applies to all quorum queues except the ones starting with amq..
expires¶
Queue expiration period in milliseconds. By default there is no expires.
We recommend 86400000 (1 day): a queue without any consumer for 1 day will be automatically deleted.
Warning
Setting expires is important: without it, queues left behind (e.g. by
a renamed or removed node) are never deleted and keep piling up, until the
cluster is no longer able to handle the number of queues.
message-ttl¶
Message TTL in queues.
By default, no TTL.
We recommend to set it to 600000 (10 minutes).
This is supposed to be safe because most of the OpenStack elements are timeouting after 300 sec.
So a message not consumed in 10 minutes will be dropped from queues.
stream-policy¶
Applies to the fanout streams created by oslo.messaging when
rabbit_stream_fanout is enabled.
For streams, there is no message TTL nor queue expiration: retention is
size based. Age based retention (x-max-age) is set by oslo.messaging from
rabbit_transient_queues_ttl but we don’t recommend setting it anyway.
max-length-bytes¶
Caps the size of each fanout stream on disk (15MB here).
stream-max-segment-size-bytes¶
Segment file size (5MB here). Retention is applied per segment, so keep it
small compared to max-length-bytes.
