RabbitMQ cluster recovery after a node outage: how insufficient monitoring resulted in a split-brain cluster

A client contacted us after an application started intermittently failing when writing to RabbitMQ Streams.

The unusual part was that RabbitMQ was not completely unavailable.

Some requests succeeded. Others failed with errors similar to:

1

Stream coordinator unavailable

RabbitMQ logs also contained repeated coordinator timeouts:

1
2

new_stream coordinator timeout on rabbitmq-slough-2
coordinator unavailable

From the application side, the failures appeared almost random.

All three RabbitMQ pods were also Running, so there was no obvious Kubernetes-level outage.

The real problem was deeper: the RabbitMQ nodes no longer agreed about cluster state, and the Stream coordinator had very different Raft histories on different nodes.

That changed the recovery question from:

Which RabbitMQ node is alive?

to:

Which nodes contain the newest consistent state that must be preserved?

The incident

The cluster consisted of three RabbitMQ nodes:

1
2
3

rabbitmq-slough-0
rabbitmq-slough-1
rabbitmq-slough-2

running RabbitMQ 4.1.3 with Erlang 27.3.4.

At Kubernetes level, everything initially looked acceptable:

1
2
3

rabbitmq-slough-0   Running   0 restarts
rabbitmq-slough-1   Running   0 restarts
rabbitmq-slough-2   Running   0 restarts

But a running pod only proves that the container is alive. It does not prove that the broker is still participating correctly in the RabbitMQ cluster.

When we checked RabbitMQ itself:

1

rabbitmqctl cluster_status

rabbitmq-slough-0 reported only itself as a cluster member.

There were no reported partitions or alarms from its point of view.

That was already suspicious: a node believing that it is a healthy one-node cluster is very different from the expected three-node cluster.

  flowchart LR
    APP[Application] --> S[RabbitMQ Service]

    S --> R0[rabbitmq-slough-0]
    S --> R1[rabbitmq-slough-1]
    S --> R2[rabbitmq-slough-2]

    R0 -. different cluster view .-> R1
    R1 <--> R2

    APP --> ERR[Intermittent<br/>Stream errors]

Cluster membership alone was not enough

Knowing that the cluster had split still did not tell us which state should survive recovery.

For RabbitMQ Streams, the important evidence was inside the Stream coordinator.

We inspected it on every node:

1

rabbitmqctl eval 'rabbit_stream_coordinator:status().'

Once the nodes could communicate sufficiently for comparison, the difference was clear.

This was the key evidence.

rabbitmq-slough-0 had a commit position of only:

1

3794

while both other nodes agreed on:

1

1537397

They also agreed on the newer Raft term:

1

24

and on the same snapshot:

1

1531904

rabbitmq-slough-1 was the leader and rabbitmq-slough-2 its follower.

  flowchart TB
    R0["rabbitmq-slough-0<br/>pre_vote<br/>term 23<br/>commit 3,794"]

    subgraph AUTH["Newest consistent state"]
        R1["rabbitmq-slough-1<br/>LEADER<br/>term 24<br/>commit 1,537,397"]
        R2["rabbitmq-slough-2<br/>FOLLOWER<br/>term 24<br/>commit 1,537,397"]

        R1 <--> R2
    end

    R0 -. divergent history .-> AUTH

Which state was authoritative?

The answer was not determined by pod age, Kubernetes health, or which broker happened to accept a client connection.

The strongest evidence was agreement between multiple Raft voters.

Two nodes independently reported:

• the same current term;
• the same commit position;
• the same snapshot;
• a valid leader/follower relationship.

The isolated node reported an dramatically older history and remained in pre_vote.

That made the decision straightforward:

1
2
3
4
5

rabbitmq-slough-1  \
                   > state to preserve
rabbitmq-slough-2  /

rabbitmq-slough-0  -> stale/divergent coordinator state

This distinction matters during recovery.

Commands such as reset, force_reset, force_boot, offline node removal, or recreating persistent storage can discard state or redefine which cluster metadata survives.

The safe sequence is therefore:

1. collect state from every node;
2. compare Raft term, commit index, role and snapshot;
3. determine which nodes agree on the newest history;
4. protect that state;
5. only then reconcile the stale node and restore normal cluster membership.

A cluster that successfully forms around the wrong state is still an unsuccessful recovery.

Why were only some writes failing?

The original application symptom now made sense.

The broker processes were alive, but RabbitMQ was no longer operating with one consistent Stream coordinator state.

Some operations could still complete through available paths.

Others required coordinator activity and failed with:

1

Stream coordinator unavailable

That is why the incident looked intermittent instead of like a complete RabbitMQ outage.

It also explains why ordinary infrastructure checks missed it.

Monitoring:

1
2
3

pod is Running
process exists
TCP port is open

was not enough.

The failure was in the distributed state of the RabbitMQ cluster.

The root cause: cluster degradation was not detected

Recovering the correct RabbitMQ state solved the immediate incident.

But that was not the complete fix.

A production three-node RabbitMQ cluster should not silently degrade into disconnected cluster views and remain that way until applications start returning errors.

This incident should have been detected much earlier.

  flowchart LR
    A[Inter-node connectivity fails] --> B[RabbitMQ cluster degrades]

    B --> C{Cluster alerting?}

    C -->|Yes| D[Alert]
    D --> E[Investigate before<br/>application impact]

    C -->|No| F[Failure remains unnoticed]
    F --> G[Coordinator problems]
    G --> H[Application write errors]

The missing control was monitoring of RabbitMQ cluster connectivity, not just Kubernetes pod availability.

Alert when a RabbitMQ node loses its peers

RabbitMQ exposes the state of Erlang distribution connections between nodes through Prometheus metrics.

For a fixed three-node cluster, each broker should normally see two peer connections in the up state.

An alert can therefore verify this invariant.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

groups:
  - name: rabbitmq-cluster
    rules:
      - alert: RabbitMQClusterPeerConnectivityDegraded
        expr: |
          (
            max by (namespace, pod) (
              rabbitmq_identity_info{job="rabbitmq"}
            )
            unless on (namespace, pod)
            (
              count by (namespace, pod) (
                erlang_vm_dist_node_state{job="rabbitmq"} == 3
              ) >= 2
            )
          ) == 1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "RabbitMQ cluster peer connectivity degraded"
          description: >-
            RabbitMQ node {{ $labels.pod }} does not see all expected
            cluster peers. A three-node cluster should have two active
            Erlang distribution peer links per node.

The exact labels depend on the Prometheus configuration, but the principle is more important than the particular query.

The alert should answer:

Can every RabbitMQ broker still communicate with all expected cluster peers?

rather than simply:

Are three RabbitMQ pods running?

For this environment, the monitored invariant becomes:

1
2
3

Expected RabbitMQ nodes:        3
Expected peer links per node:   2
Persistent deviation allowed:   0

A short for interval prevents transient restarts from producing unnecessary alerts, while still detecting a real cluster split long before it reaches application traffic.

The practical lesson

When recovering a distributed system, the node that is alive is not necessarily the node whose state should survive.

For RabbitMQ Streams, compare the distributed state itself.

In this incident:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

rabbitmq-slough-0
  term:     23
  commit:   3,794
  state:    pre_vote

rabbitmq-slough-1
  term:     24
  commit:   1,537,397
  state:    leader
  snapshot: 1,531,904

rabbitmq-slough-2
  term:     24
  commit:   1,537,397
  state:    follower
  snapshot: 1,531,904

The agreement between nodes 1 and 2 gave us the evidence needed to preserve the correct state instead of blindly rebuilding the cluster around whichever broker happened to be available.

But recovery was only half of the solution.

The deeper operational problem was that the RabbitMQ cluster had already degraded without generating an alert.

Recovery fixed the incident. Monitoring cluster health fixed the root cause.

For production incidents where Kubernetes resources appear healthy but applications still experience intermittent infrastructure failures, see our DevOps troubleshooting service.