Repair a RabbitMQ node

Introduction

There may be times when a cloud’s RabbitMQ cluster experiences difficulties. When this happens one option to bringing the cluster back to a healthy state is to repair one of its nodes.

Important

This procedure will not result in cloud downtime providing that there is at least one functional RabbitMQ node present.

Advantageously, it will not trigger the restart of every client application (cloud service) that has a relation to the rabbitmq-server application.

Repair vs replacement

An alternative approach to repairing a node is to replace it. See operation Replace a RabbitMQ node.

Below is a comparison of the repair and replacement methods.

Method

Pro

Con

repair

less time required

less comprehensive

no service restarts

replace

comprehensive

more time required*

service restarts**

* especially if a new machine must be provisioned

** can be mitigated with the Deferred service events feature

Initial state of the cluster

For this example, the juju status rabbitmq-server command describes the initial state of the cluster:

Model      Controller       Cloud/Region      Version  SLA          Timestamp
openstack  maas-controller  maas-one/default  2.9.18   unsupported  23:18:34Z

App              Version  Status  Scale  Charm            Store       Channel  Rev  OS      Message
rabbitmq-server  3.8.2    active      3  rabbitmq-server  charmstore  stable   118  ubuntu  Unit is ready and clustered

Unit                Workload  Agent  Machine  Public address  Ports     Message
rabbitmq-server/0   error     idle   0/lxd/0  10.0.0.177      5672/tcp  ????????????????
rabbitmq-server/1*  active    idle   1/lxd/0  10.0.0.175      5672/tcp  Unit is ready and clustered
rabbitmq-server/2   active    idle   2/lxd/0  10.0.0.176      5672/tcp  Unit is ready and clustered

Machine  State    DNS         Inst id              Series  AZ       Message
0        started  10.0.0.172  node2                focal   default  Deployed
0/lxd/0  started  10.0.0.177  juju-64dabb-0-lxd-0  focal   default  Container started
1        started  10.0.0.173  node4                focal   default  Deployed
1/lxd/0  started  10.0.0.175  juju-64dabb-1-lxd-0  focal   default  Container started
2        started  10.0.0.174  node3                focal   default  Deployed
2/lxd/0  started  10.0.0.176  juju-64dabb-2-lxd-0  focal   default  Container started

In this example the unhealthy node is assumed to be rabbitmq-server/0. This is the node that is to be repaired.

The status report of the cluster shows that all three nodes are both present and running. From a healthy node (rabbitmq-server/2):

juju ssh rabbitmq-server/2 sudo rabbitmqctl cluster_status

Partial output:

Disk Nodes

rabbit@juju-64dabb-0-lxd-0
rabbit@juju-64dabb-1-lxd-0
rabbit@juju-64dabb-2-lxd-0

Running Nodes

rabbit@juju-64dabb-0-lxd-0
rabbit@juju-64dabb-1-lxd-0
rabbit@juju-64dabb-2-lxd-0

Pause the unhealthy node’s service

Pause the RabbitMQ service on the unhealthy node/unit:

juju run-action --wait rabbitmq-server/0 pause

Identify the unhealthy node’s hostname

The status report of the cluster should show that a node is no longer running.

From a healthy node:

juju ssh rabbitmq-server/2 sudo rabbitmqctl cluster_status

The cluster’s status output now includes:

Disk Nodes

rabbit@juju-64dabb-0-lxd-0
rabbit@juju-64dabb-1-lxd-0
rabbit@juju-64dabb-2-lxd-0

Running Nodes

rabbit@juju-64dabb-1-lxd-0
rabbit@juju-64dabb-2-lxd-0

The hostname of the unhealthy node is the one that is no longer running. In this example, it is rabbit@juju-64dabb-0-lxd-0.

Remove the unhealthy node from the cluster

Apply the forget-cluster-node action to a healthy node’s unit and refer to the unhealthy node by its now-known hostname. This removes the unhealthy node from the cluster:

juju run-action --wait rabbitmq-server/2 forget-cluster-node node=rabbit@juju-64dabb-0-lxd-0

The cluster’s status output should now include:

Disk Nodes

rabbit@juju-64dabb-1-lxd-0
rabbit@juju-64dabb-2-lxd-0

Running Nodes

rabbit@juju-64dabb-1-lxd-0
rabbit@juju-64dabb-2-lxd-0

Refresh the unhealthy node and rejoin it to the cluster

We refresh the unhealthy node and rejoin it to the cluster by invoking commands directly on the machine that is hosting the unhealthy node. Begin by connecting to it:

juju ssh rabbitmq-server/0
  1. Move the current database out of the way:

    Note

    Normally, a previously working node cannot join a cluster with its old database intact. It must first be removed or renamed.

    > sudo mv /var/lib/rabbitmq/{mnesia,mnesia.bak}
    
  2. Start the RabbitMQ service in standalone (unclustered) mode

    > sudo rabbitmq-server -detached
    
  3. Stop the RabbitMQ server but keep the Erlang VM running

    > sudo rabbitmqctl stop_app
    
  4. Rejoin the unhealthy node to the cluster

    Rejoin the unhealthy node to the cluster by referencing an existing cluster member:

    > sudo rabbitmqctl join_cluster rabbit@juju-64dabb-2-lxd-0
    

    The cluster’s status output should now include:

    Disk Nodes
    
    rabbit@juju-64dabb-0-lxd-0
    rabbit@juju-64dabb-1-lxd-0
    rabbit@juju-64dabb-2-lxd-0
    
    Running Nodes
    
    rabbit@juju-64dabb-0-lxd-0
    rabbit@juju-64dabb-2-lxd-0
    
  5. Start a refreshed RabbitMQ server

    > sudo rabbitmqctl start_app
    

    The cluster’s status output should now include:

    Disk Nodes
    
    rabbit@juju-64dabb-0-lxd-0
    rabbit@juju-64dabb-1-lxd-0
    rabbit@juju-64dabb-2-lxd-0
    
    Running Nodes
    
    rabbit@juju-64dabb-0-lxd-0
    rabbit@juju-64dabb-1-lxd-0
    rabbit@juju-64dabb-2-lxd-0
    

    The repaired node has rejoined the cluster and is now running.

  6. Stop the RabbitMQ service

    The RabbitMQ service must be stopped in the current environment in order for it to be managed by Juju:

    > sudo rabbitmqctl stop
    

Resume the repaired node’s service

Resume the RabbitMQ service on the repaired node/unit:

juju run-action --wait rabbitmq-server/0 resume

Verify model health

Verify the model’s health with the juju status rabbitmq-server command. Its partial output should look like:

App              Version  Status  Scale  Charm            Store       Channel  Rev  OS      Message
rabbitmq-server  3.8.2    active      3  rabbitmq-server  charmstore  stable   118  ubuntu  Unit is ready and clustered

Unit                Workload  Agent  Machine  Public address  Ports     Message
rabbitmq-server/0   active    idle   0/lxd/0  10.0.0.177      5672/tcp  Unit is ready and clustered
rabbitmq-server/1*  active    idle   1/lxd/0  10.0.0.175      5672/tcp  Unit is ready and clustered
rabbitmq-server/2   active    idle   2/lxd/0  10.0.0.176      5672/tcp  Unit is ready and clustered