Cloudflare adopts leaderless QuePaxa algorithm for global consensus
Cloudflare selected the QuePaxa consensus algorithm for its Meerkat service to handle high latency and network volatility across global data centers without relying on a single leader.
Cloudflare has introduced Meerkat, an experimental distributed consensus service designed to manage its internal global control plane. The system relies on QuePaxa, a consensus algorithm published in 2023 by researchers Tennage and Băsescu, rather than traditional leader-based approaches like Raft or Paxos. This architectural choice addresses the specific challenges of maintaining consistency across continents where network latency is high and unpredictable.
What happened
Meerkat functions as a distributed system where multiple replicas agree on an ordered log of events, such as key-value store operations. While similar systems like etcd, Consul, and Apache ZooKeeper provide these guarantees, they are typically deployed in local networks with strict latency bounds. Meerkat differs by operating at a global scale, connecting clusters across different continents via links that frequently exhibit variable latency and occasional connectivity issues.
The decision to move away from standard algorithms stems from past operational incidents. In 2020, a partial failure of a core network switch caused a consensus-powered system to degrade significantly. Although no data was lost, the dashboard operated eighty times slower than usual for six hours because the system became unavailable for writes until the network issue resolved. For a global control plane, this level of unavailability during transient network faults is unacceptable.
Traditional consensus algorithms often struggle in wide-area networks because they depend on a single leader process. If that leader experiences high latency or disconnects, the entire system stalls until a new leader is elected. Cloudflare found that these recovery mechanisms often introduce more instability than the original fault, leading to prolonged outages even after the network recovers. QuePaxa offers a path forward by removing the reliance on a single leader for making progress.
How it works
Most established consensus protocols, including Paxos and Raft, use a leader to serialize proposals and avoid conflict. In Paxos, multiple proposers can interfere with each other by submitting competing ballot numbers, a phenomenon known as destructive interference. To prevent this, systems like Raft designate one replica as the leader, allowing only that node to propose values. This works well when the network is stable, but creates a single point of failure for liveness.
When a leader fails in a Raft-like system, replicas must detect the failure and hold an election. However, fault detection in asynchronous networks is imperfect. Replicas cannot directly observe if a remote machine has crashed; they can only note that expected messages, such as heartbeats, have not arrived within a timeout window. In global networks, sudden latency spikes can cause healthy leaders to appear failed, triggering unnecessary elections.
These elections are prone to their own form of destructive interference. Multiple replicas may simultaneously candidate for leadership, competing for majority votes. If no candidate secures a majority, the election fails and restarts. This cycle, known as metastable failure, can persist long after the initial network glitch has passed. QuePaxa avoids these pitfalls by not relying on a leader to make progress, thereby sidestepping the complex and fragile recovery processes associated with leader elections.
Key details
- Meerkat is an experimental distributed consensus service for Cloudflare’s internal global control plane.
- The system uses QuePaxa, a consensus algorithm published by Tennage and Băsescu in 2023.
- Traditional algorithms like Raft and Paxos rely on a single leader, which creates vulnerabilities in high-latency wide-area networks.
- Leader-based systems suffer from destructive interference during elections, where multiple candidates block each other from gaining a majority vote.
- Fault detection in asynchronous networks is unreliable, often causing healthy leaders to be mistakenly marked as failed due to latency spikes.
- QuePaxa aims to maintain stability and liveness without requiring a leader election process when network conditions degrade.
Why it matters
For engineers building distributed systems, this shift highlights the limitations of standard tools when scaling globally. Most developers default to Raft or similar algorithms because they are well-understood and have mature implementations. However, these tools assume a level of network synchrony that does not exist on the public internet. When deploying across regions or continents, the assumption that messages will arrive within a bounded time frame breaks down, leading to the metastable failures described by Cloudflare.
Understanding the mechanics of destructive interference and metastable failure is crucial for designing resilient infrastructure. It explains why a brief two-second network outage can result in minutes or hours of degraded performance. The recovery mechanism itself becomes the bottleneck, as the system spends resources trying to re-establish leadership rather than processing requests. Recognizing these patterns helps teams choose the right consistency model for their specific network environment.
What you can do
- Evaluate whether your consensus system requires strong leader-based ordering or can tolerate alternative consistency models.
- Monitor for metastable failures where election loops persist after network connectivity is restored.
- Adjust fault detection timeouts carefully, balancing sensitivity to real failures against false positives from latency spikes.
- Consider the network topology of your deployment; global scales may require algorithms that do not depend on low-latency leader communication.
- Review existing documentation on QuePaxa and other leaderless consensus protocols for potential application in high-latency environments.
- Test system behavior under simulated network partitions to identify how recovery mechanisms perform under stress.



