We Open-Sourced a Complete PostgreSQL 17 HA Cluster: Patroni, etcd, HAProxy and a Witness

Everyone agrees PostgreSQL needs high availability. Far fewer teams have it, because the gap between "we have a replica" and "the application keeps writing when a machine dies" is filled with decisions nobody wants to make at 3 a.m.: who is the primary now, how does the application find it, and how do you avoid ending up with two of them. We built that layer, ran it, broke it on purpose, and today we are releasing it.

The problem it solves

The database must keep accepting writes if one machine dies, without waking anyone up. With two PostgreSQL servers, native streaming replication and a manual promote, three things go wrong at exactly the worst moment:

• Someone has to decide who the primary is. A human decision, under pressure, with incomplete information.
• The application has to discover the new address. Connection strings pinned to an IP do not fail over; they fail.
• A network split can leave you with two primaries. Both accept writes, both are convinced they are right, and reconciling the divergence afterwards is a manual, lossy job.

The repository ties those pieces together with components that already exist and are well understood, wired in the specific way that makes them work as one system:

Architecture at a glance

One HA model: Patroni. There is no manual promote path, no hand-rolled pg_basebackup rebuild, no dual-mode "sometimes automatic, sometimes not". Patroni elects the primary, promotes the standby, and rebuilds a broken member.

The witness is the part people try to skip, and it is the part that makes the difference between a cluster and a coin flip. With two voters, a network partition gives you 1 vs 1: neither side can prove it holds the majority, so neither can safely be primary. The third etcd — on a tiny VM, with no database on it — breaks the tie deterministically.

One replication layer, on purpose

• db1 and db2 form a single Patroni scope (pg-ha).
• The leader accepts writes; the standby applies WAL through a physical slot named after the member.
• synchronous_mode: true and synchronous_mode_strict: true: COMMIT waits for the synchronous standby.
• The standby serves read-only queries on HAProxy :5001 (hot_standby=on).
• Ubuntu watchdog: if Patroni loses the lock and cannot demote PostgreSQL in time, the node reboots itself.

Why we removed the logical replication layer

The first release carried a second concern: a read-only central server fed by PostgreSQL 17 failover slots, with publication, subscription and role-change callbacks. It worked. We removed it anyway, and the reasoning is worth more than the code was.

High availability and logical replication are different problems that happened to share a repository. Carrying both had a cost beyond file count: it forced wal_level=logical in production, it added apply workers, and — the part that actually mattered — the lab tests measured logical continuity instead of measuring failover. The project claimed to be an HA cluster while not testing the one thing an HA cluster has to prove.

Removing it took eight scripts and about 1,400 lines with it, and let the configuration fall back to what physical streaming actually needs:

The standby still answers read-only queries on :5001. If you need a separate analytics copy, add logical replication yourself — deliberately, with its own parameters and its own tests — rather than inheriting it from a cluster whose job is to survive a dead machine.

Try it in five minutes: the Docker lab

The whole cluster — three etcd nodes, two Patroni/PostgreSQL nodes and HAProxy — starts on one machine with Docker Compose. It exists so you can break things on purpose before you own them in production.

git clone https://github.com/johnvithera01/postgresql-patroni-ha.git
cd postgresql-patroni-ha

./scripts/lab/reset-lab.sh    # build and start the whole cluster
./scripts/lab/status.sh       # who is the leader, who is sync standby
./scripts/lab/setup-app.sh    # a database and a table to write into

Requirements are only Docker Compose v2 and Bash. What comes up:

psql "host=127.0.0.1 port=15000 user=postgres dbname=appdb"   # writer
psql "host=127.0.0.1 port=15001 user=postgres dbname=appdb"   # reader

Now break it on purpose

Reading about failover teaches you nothing. Watching a leader die while a writer session is open teaches you a lot:

./scripts/lab/test-switchover.sh   # planned, controlled promotion
./scripts/lab/test-failover.sh     # kill the leader, watch the election
./scripts/lab/test-ha.sh           # the full run: reset + switchover + kill

test-failover.sh kills the leader's container and then asserts three things: that Patroni promoted the standby on its own, that the already-committed row survived the promotion, and that HAProxy :5000 accepts writes again. Every write in the tests goes through HAProxy — no test pins which node is primary, because in a real failover you do not know either.

== Killing leader db2; db1 must take over with no operator action ==
 Container postgresql-patroni-ha-db2-1  Killed
automatic promotion succeeded: leader=db1
committed row before-kill-1786624502 survived the promotion (RPO zero)
== Restarting db2 to restore RPO-zero writes ==
INSERT 0 1
Automatic failover completed. The cluster writes again through HAProxy :5000.

The bug that only showed up when we ran it

Worth telling, because it is the kind of failure no static check catches. The rewritten tests passed bash -n and ShellCheck. On the first real lab run they broke like this:

== Creating appdb through the HAProxy writer (leader=db2) ==
CREATE DATABASE
ERROR:  relation "public.ha_events" does not exist
LINE 1: INSERT INTO public.ha_events (marker) VALUES ('bootstrap')

The CREATE TABLE never reached psql. The cause: docker compose exec -T inherits the caller's stdin. The helper that discovers which container is alive runs inside a command substitution — and that substitution inherited the same stdin, eating the heredoc meant for psql before psql could read it.

The fix is one redirection, applied to every probe that might be called from inside a $(…):

# scripts/lab/lib.sh
- if dc exec -T "$service" true             >/dev/null 2>&1; then
+ if dc exec -T "$service" true </dev/null  >/dev/null 2>&1; then

No linter finds that. Only running it does — which is the same reason the lab exists at all.

Production on Ubuntu: three machines

Production is three VMs, and each script is idempotent enough to be run from a checklist rather than from memory:

Decisions already baked into the scripts, so you do not re-litigate them under pressure: PostgreSQL 17 with Patroni 4.1+, synchronous_mode_strict (RPO zero), TLS on etcd and PostgreSQL, watchdog required, and a clean install plus restore rather than an in-place conversion of an existing Debian cluster.

# on db1, db2 and witness
sudo ./ubuntu/install-env.sh ubuntu/env.example
sudo nano /etc/pg-patroni-ha/env
sudo bash ubuntu/patroni/setup-etcd.sh

# witness
sudo bash ubuntu/patroni/setup-witness.sh

# db1 and db2
sudo bash ubuntu/patroni/setup-patroni-node.sh
sudo bash ubuntu/patroni/bootstrap-cluster.sh   # first node only
sudo bash ubuntu/patroni/join-replica.sh        # second node
sudo bash ubuntu/patroni/apply-dcs-tuning.sh    # current leader only
sudo bash ubuntu/patroni/setup-haproxy.sh
sudo bash ubuntu/patroni/setup-keepalived.sh
sudo bash ubuntu/monitor/install-cron.sh

apply-dcs-tuning.sh writes configuration into etcd for the whole cluster, so it belongs to the current leader — running it on both nodes is not "safer", it is redundant.

The full runbook is docs/production.md; the reasoning behind the replication model, the quorum and the explicit non-goals is in docs/architecture.md.

Day-to-day operations

sudo bash ubuntu/patroni/status.sh             # cluster state, lag, who leads
sudo bash ubuntu/patroni/switchover.sh db2    # planned, in a maintenance window
sudo bash ubuntu/patroni/failover.sh db2      # forced, when the leader is gone
sudo bash ubuntu/patroni/reinit-member.sh db1 # rebuild a member from the leader
sudo bash ubuntu/monitor/check-all.sh         # the cron checks

Use switchover.sh when the current leader is healthy and you are choosing the moment — Patroni only switches if the candidate is already the synchronous standby in the DCS, so it refuses the moves that would lose data. Use failover.sh only when the leader is already gone and Patroni has not moved on its own; if Patroni is working, you will never need it.

After a crash the old node usually rejoins through pg_rewind. When rewind is not possible, reinit-member.sh clones that member from the current leader — slower, but always correct.

The limits, stated up front

An HA design that does not tell you where it breaks is marketing. These are deliberate trade-offs, documented in the repository itself:

• RPO zero stalls writes when there is no synchronous standby. Availability of writes is traded for never losing a confirmed commit.
• Losing the witness and one database node at once loses quorum. The survivor holds no lock and will not promote itself. This is intentional — a lone node that cannot prove it is alone is exactly how split brain starts.
• Replication is not a backup. A DELETE replicates faithfully and instantly. Use pgBackRest, or an equivalent, for PITR.
• Keepalived/VRRP needs a real L2 network. The Docker lab does not exercise the VIP; validate it on the actual VMs.
• HAProxy terminates TCP, so PostgreSQL sees the proxy's IP, not the client's. Restrict the application CIDR at the firewall and in HAProxy, not only in pg_hba.conf.
• This is local HA on one L2 network. It survives a dead machine, not a dead site. Disaster recovery across regions is a different design.
• It does not migrate an existing production cluster in place. It bootstraps a new cluster, restores a validated backup on the leader, then joins the standby.

Automatic failover is not monitoring

A Patroni cluster survives the failure. It does not tell you why it happened, and it will happily survive the same failure every week without anyone noticing that a disk is filling, that a member has been reinitialized three times this month, or that the sync standby falls behind under afternoon load.

Worse, the parts of this design that protect you are also the parts that hurt quietly. A stalled write under synchronous_mode_strict looks like "the application is slow", not like "the standby is gone". A replication slot left behind by a member that never came back retains WAL until pg_wal fills the filesystem and the whole cluster stops — the most common way a perfectly healthy PostgreSQL server goes down. Replication lag that spikes at 3 a.m. and recovers by 7 is invisible to anyone who only looks after a complaint.

That is the layer PG Monitoring covers: continuous tracking of replication lag and slot retention, WAL growth, autovacuum and wraparound risk, query regressions, connection saturation and lock waits — with history, so after a failover you can prove what actually happened instead of guessing. Use the open-source cluster to stay up; use monitoring so you are not the last to know why you almost did not.

Use it, break it, ask us

The repository is MIT licensed. Clone it, fork it, run the lab, tear it apart, adapt the scripts to your environment. Issues and pull requests are welcome — CI validates Bash and ShellCheck, checks the Compose file and runs the full test-ha.sh, so a change that breaks failover does not get merged quietly.

We answer questions about the repository whether or not you are a PG Monitoring customer. Designing HA badly is expensive enough that we would rather you got it right.

References: the PostgreSQL manual documents synchronous replication, hot standby, pg_rewind and replication slots. Patroni's own replication modes documentation explains synchronous_mode_strict in detail.