> ## Documentation Index > Fetch the complete documentation index at: https://controlplanecorporation-majid-docs-content-expansion.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Postgres Multi-Location > Deploy a single PostgreSQL 17 Patroni cluster stretched across Control Plane locations, with one primary, asynchronous streaming replicas elsewhere, automatic cross-location promotion, leader-routing HAProxy, optional PgBouncer pooling, and optional backups to object storage. **Version 2.0.0 no longer creates a GVC — it deploys into one you already have, and there is no in-place upgrade path from 1.x.** A `helm upgrade` across that boundary deletes the GVC the old release created, and with it the volume sets holding your database. See [Migrating From Version 1](#migrating-from-version-1). ## Overview Postgres Multi-Location deploys **one** PostgreSQL 17 Patroni cluster whose members span several Control Plane locations: a single primary that takes all traffic, asynchronous streaming replicas in the other locations, and automatic promotion of a replica in a surviving location when the primary's location is lost. An HAProxy tier in every location routes connections to whichever member currently holds the leader lock, so applications connect to a stable name in their own region and never need to know where the primary is. Consensus is provided by a bundled [`etcd-multi-location`](/template-catalog/templates/etcd-multi-location) cluster with one member per location — that quorum, not PostgreSQL, decides what the cluster survives. This template deploys into a GVC **you already have** and requires at least **two** locations. It does not create, provision or manage a GVC, and it cannot run in a single location: the consensus store is a stretched etcd cluster with exactly one member per location. For a single-location cluster, use [PostgreSQL Highly Available](/template-catalog/templates/postgres-highly-available) instead. ### How Many Locations You Need The consensus store commits a write only when a **majority** of its members agree, and it runs exactly one member per location. | Locations | Majority | Location losses survived | What happens when one location is lost | | --------- | -------- | ------------------------ | ------------------------------------------------------------------------------------------------------- | | **2** | 2 | **0** | The surviving replica holds current data but stays **read-only**. Promotion is **manual**. | | **3** | 2 | **1** | **Automatic failover** — a replica in a surviving location is promoted and every proxy re-routes to it. | | **5** | 3 | **2** | Survives losing **two** locations. | With N locations you survive `floor((N-1)/2)` losses, so an even count buys nothing over the odd count below it. Two locations cannot form a symmetric quorum, which is why that topology is a warm standby rather than an automatic-failover cluster. ### What Gets Created * **Stateful Patroni PostgreSQL Workload** — `{release}-postgres`, running PostgreSQL 17 with Patroni. Each configured location runs `replicas` members, every member gets its own volume, and every member is individually addressable. PostgreSQL listens on `5432` and the Patroni REST API on `8008`. * **Stateful etcd Workload** — `{release}-etcd`, the bundled `etcd-multi-location` chart providing consensus, one member per location. * **HAProxy Leader-Routing Workload** *(optional, enabled by default)* — `{release}-postgres-proxy`, one tier per location, each routing to the single current primary. * **PgBouncer Workload** *(optional)* — `{release}-postgres-pgbouncer`, a connection pooler, one tier per location, pooling into that location's HAProxy. * **Cron Backup Workload** *(optional)* — `{release}-postgres-backup`, a scheduled `pg_dumpall` to object storage, running in exactly one location. * **WAL-G Sidecar** *(optional)* — A sidecar on every Patroni member; only the member holding the leader lock archives WAL and pushes base backups. * **Volume Sets** — `{release}-postgres-vs` for the PostgreSQL data directory (`ext4`, final snapshot with 7-day retention), plus the etcd cluster's own volume set. * **Secrets** — Opaque secrets holding the Patroni startup script, the HAProxy startup script, and the WAL-G backup script. The **database credentials secret is not created by this template** — see [Prerequisites](#prerequisites). * **Identities & Policies** — An identity per workload group with `reveal` on exactly the secrets in play, `view` on **exactly the one GVC this release installs into** (for the boot-time location check described in [Matching the Location List to the GVC](#matching-the-location-list-to-the-gvc)), plus a bucket-scoped cloud binding when backups are enabled. No GVC resource is created. Every resource above lands in the GVC you install into, and both startup scripts read that GVC's name from the platform at runtime rather than from Helm, so a hostname can never drift from where the workload actually runs. ## Prerequisites **An existing GVC with at least 2 locations, and `global.locations` set to match it.** This template deploys into the GVC you install it into and creates none of its own. Read the GVC's list with `cpln gvc get GVC_NAME -o yaml` and compare `spec.staticPlacement.locationLinks` before you install — see [Matching the Location List to the GVC](#matching-the-location-list-to-the-gvc) for what happens if the two disagree. PostgreSQL credentials are supplied through a [dictionary secret](/guides/create-secret/dictionary) that you create **before** installing. They are never passed through Helm values. The secret must hold exactly three keys — `username`, `password` and `database`: ```bash theme={null} cpln secret create-dictionary --name my-postgres-credentials \ --entry username=postgres \ --entry password="$(openssl rand -hex 24)" \ --entry database=mydb ``` Use plain identifiers for `username` and `database` — they are used unquoted when the database is created. Secrets are org-level, so no GVC flag is involved. Set `postgres.credentialsSecretName` to the secret's name. ```bash theme={null} cpln secret reveal my-postgres-credentials ``` Create the secret **before** installing. Without it `helm install` still reports success, but the workload sits at zero replicas in every location with `The secret ... no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` in `cpln workload get-deployments`. Creating the secret afterwards clears the wedge on its own within a few minutes — no reinstall or upgrade is needed. Backups need additional setup before they can be enabled — see [Backing Up](#backing-up). To install, follow the instructions for your preferred method: Browse, install, and manage templates visually Manage templates from your terminal }> Declare templates in your Terraform configurations Pulumi Icon Streamline Icon: https://streamlinehq.com } > Declare templates in your Pulumi programs ## Configuration The default `values.yaml` for this template: ```yaml theme={null} # ─── Locations ──────────────────────────────────────────────────────────────── # This chart deploys into the GVC you install into — it does NOT create one. # Every location listed here MUST already exist in that GVC. The platform does # not validate that: a location the GVC lacks is stored verbatim and is simply # inert, so the declared Patroni member and its etcd peer never start while every # status surface reads healthy. The members read the GVC at boot and refuse to # bootstrap in that state. # # Lives under `global` so the bundled etcd-multi-location subchart gets the same # list automatically — the two lists can then never be edited apart. # # Minimum 2 locations. 3 gives automatic failover, 5 survives losing two; # 2 gives a warm standby with MANUAL promotion. See the quorum table in the # README. `replicas` is Patroni members per location; etcd always runs 1 per # location and ignores this field. global: locations: - name: aws-us-east-1 replicas: 1 - name: aws-eu-central-1 replicas: 1 - name: aws-us-west-2 replicas: 1 # ─── PostgreSQL / Patroni ───────────────────────────────────────────────────── image: controlplanecorporation/patroni-postgres:0.7 resources: minCpu: 500m minMemory: 1Gi maxCpu: 1 maxMemory: 2Gi postgres: # REQUIRED PREREQUISITE SECRET — CREATE IT BEFORE YOU INSTALL. # A `dictionary` secret holding exactly three keys: `username`, `password` and # `database`. If it does not exist at install time the deployment WEDGES # waiting on it and looks broken. See Prerequisites in the README for the # exact `cpln secret create-dictionary` command. credentialsSecretName: my-postgres-credentials # Preferred location for the primary. It does three things: # 1. On a FRESH install it decides where the primary starts — members in other # locations wait up to 90s for this one to initialise the cluster first. # If it is down or slow they bootstrap anyway (logged as a WARNING) and the # primary starts elsewhere; move it later with patronictl switchover. # 2. It biases FAILOVER elections toward this location (failover_priority). # 3. CHANGING IT ON A LIVE CLUSTER MOVES THE LEADER: the value is baked into # the startup script, so editing it restarts every member (~2 min of # interrupted writes) and the election that follows picks this location. # To move a primary WITHOUT a restart, use patronictl switchover. # Empty = no preference, and the primary starts wherever a member gets there first. primaryLocation: "" volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) autoscaling: enabled: false maxCapacity: 100 # GiB, when autoscaling is enabled minFreePercentage: 10 # free-space trigger scalingFactor: 1.2 # growth multiplier internalAccess: type: same-gvc # options: same-gvc, same-org, workload-list # Only used when type is workload-list. This chart's OWN workloads (Patroni, # the proxy, PgBouncer, the backup cron) are added automatically — the list # also governs Patroni-to-Patroni replication and the proxy's health checks, # so a list naming only clients would cut the cluster off from itself. List # your clients here; do not list this release's workloads. workloads: [] #- //gvc/GVC_NAME/workload/WORKLOAD_NAME # ─── Leader-routing proxy ───────────────────────────────────────────────────── # One HAProxy tier per location; every one routes to the single current primary. proxy: enabled: true # automatically enabled when pgbouncer.enabled is true image: haproxy:2.9 resources: cpu: 100m memory: 128Mi minReplicas: 2 maxReplicas: 2 # ─── PgBouncer connection pooler ────────────────────────────────────────────── # One tier per location, each pooling into its location's HAProxy. pgbouncer: enabled: false image: edoburu/pgbouncer:v1.25.1-p0 poolMode: transaction # options: session, transaction, statement defaultPoolSize: 25 # real Postgres connections PgBouncer keeps per replica maxClientConn: 1000 # client connections PgBouncer accepts per replica maxDbConnections: 100 # cap on Postgres connections PER PgBouncer pod — multiply by maxReplicas minReplicas: 2 maxReplicas: 4 resources: cpu: 200m memory: 128Mi # ─── Backups ────────────────────────────────────────────────────────────────── backup: enabled: false mode: logical # logical or wal-g # `logical` mode ONLY. A cron workload runs in EVERY location of its GVC, so # without this the job would fire once per location every night and write N # copies into one bucket. Pick the ONE location it runs in — nearest your # bucket. `wal-g` mode ignores it: that archives from whichever member is # currently the primary, wherever that is. location: aws-us-east-1 resources: # applies to whichever mode is enabled cpu: 100m # 512Mi, not 128Mi: the GCP path OOMs at 128Mi with NO log output — # logical jobs merely report `failed`, and the wal-g sidecar loops on # OOMKilled while WAL archives with no base backup. AWS and MinIO are # fine at 128Mi; a default has to work for every provider. memory: 512Mi logical: image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0 # 17.1.0 = Postgres 17, 18.1.0 = Postgres 18 schedule: "0 2 * * *" # cron schedule, default is daily at 02:00 UTC walg: intervalSeconds: 21600 # seconds between base backups, default is every 6 hours # storage settings are applied to whichever mode is enabled provider: aws # options: aws, gcp, minio aws: bucket: my-postgres-bucket region: us-east-1 cloudAccountName: my-s3-cloud-account policyName: my-postgres-backup-policy # bucket-scoped IAM policy, see README prefix: postgres/backups # folder within the bucket gcp: bucket: my-postgres-bucket cloudAccountName: my-gcs-cloud-account prefix: postgres/backups # folder within the bucket minio: # a self-hosted MinIO workload, or any S3-compatible endpoint # MUST be reachable from EVERY location: wal-g's restore_command runs on # every member, and a member that cannot reach the archive never finishes # starting — its container is recycled every 160s indefinitely. A MinIO # workload in ONE location answers 503 from the others, so for wal-g use # a multi-location endpoint or an external S3-compatible service. endpoint: http://my-minio-workload:9000 # e.g. http://WORKLOAD.GVC.cpln.local:9000 in the same GVC bucket: my-postgres-bucket # REQUIRED PREREQUISITE SECRET when provider is `minio` — a `dictionary` # secret holding `accessKey` and `secretKey`. See Storage setup in the README. credentialsSecretName: my-postgres-minio-credentials prefix: postgres/backups # folder within the bucket # ─── etcd (subchart: etcd-multi-location) ───────────────────────────────────── etcd: image: controlplanecorporation/etcd:0.1 resources: cpu: 500m memory: 512Mi tuning: heartbeatIntervalMs: 250 electionTimeoutMs: 5000 # History compaction. Without it etcd's backend grows with time alone — # Patroni renews its lease every ~10s and every renewal is a revision — # until it hits the quota and etcd goes READ-ONLY. autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count) autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h) quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB volumeset: capacity: 10 internalAccess: type: same-gvc # etcd adds its OWN workload automatically, but not this chart's Patroni # workload. If you set type to workload-list here you MUST add # //gvc/YOUR_GVC/workload/RELEASE-postgres yourself, or Patroni loses its DCS. workloads: [] recovery: # EMERGENCY ONLY — see "Recovering from a lost location" in the README. forceNewClusterInLocation: "" ``` ### Locations The location list lives under `global` so that Helm passes the same list to the bundled etcd cluster. Never maintain the two lists separately. * `global.locations[].name` — A Control Plane location (e.g. `aws-us-east-1`) that **must already be one of the locations of the GVC you install into**. At least 2 are required; see [How Many Locations You Need](#how-many-locations-you-need). * `global.locations[].replicas` — Number of Patroni members in that location. etcd always runs exactly one member per location regardless of this value. Members are named `{workload}-{location}-{index}`, so they stay unique when a location runs more than one. Removing a location from the list is the supported way to shrink the cluster. Do not suspend a location instead — see [Important Notes](#important-notes). ### Matching the Location List to the GVC The platform validates the pairing in neither direction, so the template closes both. Each mismatch fails differently: | Mismatch | What happens | | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The GVC has a location `global.locations` does not list | Nothing runs there. Every tier's default `minScale`/`maxScale` are `0` and only the declared locations get a real count, so that location's deployment reads `This workload location is deactivated because maxScale is set to 0`. Harmless, and it is how you deploy into a subset of a larger GVC. | | `global.locations` lists a location the GVC does not have | The platform accepts and stores it, and it is simply inert — the declared Patroni member and its etcd peer never start. Left unchecked that quietly costs the cluster the fault tolerance it was sized for. | The members catch the second case themselves by reading their own GVC at boot, using the scoped `view` grant described in [What Gets Created](#what-gets-created). The check is deliberately asymmetric: * On a **fresh** data directory it is a **hard failure**: etcd and Patroni both log a `FATAL:` line naming the missing location and refuse to bootstrap, and the containers exit with code `1`. * On an **already-initialized** member it is a **warning only**, and the member keeps serving. A location removed from a GVC is indistinguishable from a location that is down, and surviving that is exactly what this template is for. * If the GVC read itself fails — a control-plane blip, or a missing policy — the check is **skipped with a warning**. It never stops a database on its own unavailability. The same boot check also warns when `primaryLocation` or `backup.location` names a location the GVC lacks. Look for `[patroni]` and `[etcd]` lines in `cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-postgres"}'`. ### Credentials `postgres.credentialsSecretName` names the dictionary secret described in [Prerequisites](#prerequisites). The template creates no credentials of its own, and PgBouncer reads the same secret. The credentials are written into the data directory when the cluster first bootstraps. Rotating the secret afterwards does not change the database — change the password with `ALTER ROLE` first, then update the secret to match. ### Preferred Primary Location `primaryLocation` names the location you want the primary in. Patroni treats "which member initializes an empty cluster" and "which member wins a failover" as separate questions, and the knob addresses both: * **On a fresh install it biases where the primary starts.** Since template version 1.0.2, a member outside the preferred location whose data directory is empty holds back before bootstrapping: it polls the preferred location's members and proceeds as soon as one answers as the leader — or as a replica, which means the cluster is already initialized elsewhere and waiting cannot move it. * **On a running cluster it biases failover elections** toward that location through Patroni's `failover_priority` tag, a tiebreaker between candidates that have replayed the same amount of WAL. Leaving it empty means no preference: the primary starts wherever a member gets there first. **The head start is a bounded preference, not a guarantee.** The wait is capped at 90 seconds, so if the preferred location is slow to start, down or misconfigured, another member bootstraps the cluster where it is and logs a `WARNING` naming the consequence. Move the primary afterwards with `patronictl switchover --candidate`. The wait cannot split the cluster: the consensus store is the arbiter, so a member that gives up early finds the cluster already initialized and clones as a replica instead. Before version 1.0.2 the knob rendered only the failover tag, so a fresh install raced and the primary could land in any location. **Check where the leader actually landed before judging a slow first install.** Every write goes to the primary, so a primary outside `primaryLocation` puts a cross-region hop in front of your entire write path, and schema migrations are where that shows up first. On [Grafana Multi-Location](/template-catalog/templates/grafana-multi-location), which bundles this template, the same 713 migrations took about **5 seconds** with the primary local and stretched to **minutes** when it bootstrapped in another region. In the pair of test installs measured that way, the cold install took **11 m 41 s** with the primary remote against **4 m 19 s** with it in place, and the Grafana instances waiting on the migrations restarted **15** times against **2** — every figure in that comparison comes from those two runs. `primaryLocation` is baked into the Patroni startup script, so changing it on a live cluster rewrites that script and restarts every member. That costs the full [upgrade write outage](#availability-and-planned-outages) and the election that follows moves the leader. To move a primary without a restart, use `patronictl switchover` instead. The preference biases elections; it does not fail back automatically after an outage. A cluster whose preferred location is restored keeps its current leader until the next election. ### Resources and Storage * `image` — The Patroni + PostgreSQL 17 image. * `resources.minCpu` / `resources.minMemory` — Reserved CPU and memory per member. * `resources.maxCpu` / `resources.maxMemory` — Limits per member. * `volumeset.capacity` — Initial volume size in GiB (minimum 10). Every member gets its own volume. * `volumeset.autoscaling.enabled` — Expand the volume automatically as it fills. When enabled: * `maxCapacity` — Maximum volume size in GiB. * `minFreePercentage` — Trigger a scale-up when free space drops below this percentage. * `scalingFactor` — Multiply the current capacity by this factor when scaling up. ### Internal Access `internalAccess` applies to the PostgreSQL, HAProxy and PgBouncer tiers alike. There is no public access in this version — every endpoint is internal to Control Plane. | Type | Description | | --------------- | ----------------------------------------------------------------------- | | `same-gvc` | Allow access from all workloads in the same GVC (default) | | `same-org` | Allow access from all workloads in the same organization | | `workload-list` | Allow access only from the workload links in `internalAccess.workloads` | **With `workload-list`, list only your clients.** The same list governs Patroni-to-Patroni streaming replication, the proxy's health checks of every member, PgBouncer pooling into HAProxy and the nightly dump, so a list naming only clients would cut the cluster off from itself. This release's own workloads — Patroni, the proxy, PgBouncer and the backup cron — are appended for you, and a client that is not listed is still refused. **The bundled etcd tier is a separate knob and is not covered by that.** `etcd.internalAccess` has its own list, and the etcd chart appends only its *own* workload. If you set `etcd.internalAccess.type: workload-list` you must add `//gvc/GVC_NAME/workload/RELEASE_NAME-postgres` to `etcd.internalAccess.workloads` by hand, or Patroni loses its consensus store. A firewall change is not instant. Blocking was measured at 47 seconds and unblocking at about 165 seconds on this template. Allow several minutes and re-test before concluding the setting did not apply. ### Leader-Routing Proxy Only the member holding the leader lock accepts writes; the rest are read-only. HAProxy polls each member's Patroni REST API and forwards connections to whichever one currently answers as the primary, so applications get a stable endpoint in their own location. `proxy.minReplicas` / `proxy.maxReplicas` are **per location** — the default of 2 means 2 replicas in each configured location. | Endpoint | Description | | --------------- | ---------------------------------------------------------------- | | `:8404/healthz` | Healthy when a primary backend is reachable, unhealthy otherwise | | `:8405/stats` | Live HAProxy stats, including the health of each member | The proxy is required for `logical` backups, which must dump from the current primary, and the chart refuses to render without it. Enabling PgBouncer enables the proxy implicitly, even if `proxy.enabled` is set to `false`. ### PgBouncer Connection Pooling PgBouncer multiplexes application connections into a smaller pool of real database connections. It pools into its own location's HAProxy rather than into a member, so leader routing and failover stay transparent. When enabled it becomes the connection endpoint applications use. | Mode | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `transaction` | Connection held for one transaction, then returned to the pool. Best for most web and API workloads. Not compatible with `SET` variables, temporary tables or advisory locks. | | `session` | Connection held for the whole client session. Compatible with everything, less reuse — raise `defaultPoolSize` to your expected concurrency. | | `statement` | Connection returned after every statement. Transactions are not supported. | * `pgbouncer.defaultPoolSize` — Real PostgreSQL connections kept per PgBouncer replica (default `25`). * `pgbouncer.maxClientConn` — Client connections accepted per PgBouncer replica (default `1000`). * `pgbouncer.maxDbConnections` — Cap on PostgreSQL connections **per PgBouncer replica**, not across the tier. PgBouncer instances do not coordinate, so the real ceiling is `maxReplicas × maxDbConnections`; at the shipped values that is 4 × 100 = 400 against the cluster's `max_connections` of 100, three of which are reserved for superusers. Size it so `maxReplicas × maxDbConnections` stays comfortably under 97, or clients get `remaining connection slots are reserved` under load rather than being queued. Treat `defaultPoolSize` the same way. * `pgbouncer.minReplicas` / `pgbouncer.maxReplicas` — Autoscaling bounds, **per location**. ### etcd Consensus Store The `etcd` block is passed straight through to the bundled [`etcd-multi-location`](/template-catalog/templates/etcd-multi-location) chart — pinned to **2.0.0**, the version that no longer creates a GVC — which is where its image, resources, storage, access and emergency recovery settings are documented. It reads its location list from the same `global.locations`, so the two can never be edited apart. `etcd.tuning.autoCompactionMode`, `etcd.tuning.autoCompactionRetention` and `etcd.tuning.quotaBackendBytes` control how much revision history etcd keeps and how large its backend may grow. Compaction is enabled in every version of the bundled chart and cannot be switched off, only retuned. The defaults — `periodic`, `1h` and `0` (etcd's own 2 GiB limit) — are the right settings for a Patroni consensus store and should be left alone: Patroni renews its leader lease every ten seconds or so, and every renewal is a revision, so an uncompacted backend grows with time alone until etcd goes read-only and takes the database's failover with it. See [Compaction and Backend Growth](/template-catalog/templates/etcd-multi-location#compaction-and-backend-growth) for the mechanism and the accepted value formats. `etcd.tuning.heartbeatIntervalMs` and `etcd.tuning.electionTimeoutMs` are the raft timers, tuned for cross-region round trips; raise both in proportion if your locations are more than about 250 ms apart. ## Connecting Everything is internal to Control Plane. Substitute your release name and the name of the GVC you installed the release into. | What | Where | | --------------------------------------------------- | ---------------------------------------------------------------------------- | | PostgreSQL, pooled (when PgBouncer is enabled) | `RELEASE_NAME-postgres-pgbouncer.GVC_NAME.cpln.local:5432` | | PostgreSQL, current primary (recommended otherwise) | `RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local:5432` | | PostgreSQL, one specific member | `replica-INDEX.RELEASE_NAME-postgres.LOCATION.GVC_NAME.cpln.local:5432` | | Patroni REST API | Port `8008` on the same per-member names (`/primary`, `/replica`, `/health`) | | HAProxy health and stats | `:8404/healthz` and `:8405/stats` on the proxy workload | | Credentials | The dictionary secret named by `postgres.credentialsSecretName` | The proxy endpoint in every location reaches the same primary, wherever it is. Per-member names are useful for read-only queries against a nearby replica, or as a fallback when the proxy is disabled — but a client using them has to poll the members to discover which one is currently the primary. ## Migrating From Version 1 Every release before 2.0.0 created its own GVC. **Do not `helm upgrade` a 1.x release onto 2.0.0.** Once the chart stops declaring a GVC, Helm prunes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it, including the volumes holding your database. Measured on a sibling template: everything was gone in about **six seconds**, while the command printed `upgraded successfully`. The chart refuses to render if your values still carry the 1.x `global.gvc` key, so a values-carrying upgrade fails safely before any resource is touched. That guard **cannot** fire on an upgrade run with no values file at all, because it then sees only 2.0.0's own defaults. The procedure below is the safety; the render guard is only a backstop. Install 2.0.0 as a **new release against an existing GVC**, move the data across, then remove the old release. Use `backup.mode: logical`, or run a `pg_dumpall` through the old release's proxy endpoint. On 1.0.0 and 1.0.1 that endpoint is `{release}-postgres-ml-proxy`; from 1.0.2 onwards it is `{release}-postgres-proxy`. This dump is the only copy that crosses the boundary, so confirm it is a real dump and not a zero-length object before going further. Delete `global.gvc.name` and rename `global.gvc.locations` to a top-level `global.locations`. Every location you list must already exist in the GVC you are installing into. Not the GVC the 1.x release created — that GVC is still owned by the old Helm release and goes away when you uninstall it. Load the dump through the new release's proxy — see [Restoring a Backup](#restoring-a-backup) — then repoint every application at `{new-release}-postgres-proxy.GVC_NAME.cpln.local:5432`. Uninstalling it removes the release **and the GVC it created**, taking the old volume sets with it. Do this only once the restore is verified. Values that changed: | 1.x | 2.0.0 | | --------------------------------- | ---------------------------------------------- | | `global.gvc.name` | Removed — the GVC is wherever you install | | `global.gvc.locations` | `global.locations` | | Bundled `etcd-multi-location` 1.x | Pinned to **2.0.0**, which also creates no GVC | ## Availability and Planned Outages Measured on a three-location cluster (`aws-us-east-1`, `aws-eu-central-1`, `aws-us-west-2`) with one member per location and a one-second write probe running in each location. | Event | Measured impact | | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Clean install to all three members ready | **169 s** — leader elected, both replicas streaming with no lag. A later run of 2.0.0 with the proxy and pooler on measured **175 s** for all four tiers | | Graceful loss of the member holding the leader lock | Switchover in **3.19 s**; writes unavailable **4.7–5.8 s** per location; the old member rejoined as a streaming replica after **2 m 16 s** | | **Abrupt loss of the primary** (process hung, lease left to expire) | A replica in a **surviving location** was promoted within **43 s**, bounded by the 45-second leader lease. Writes resumed through both the proxy and the pooler with no client reconfiguration, and the old primary rejoined as a streaming replica with **no split brain** | | Any `helm upgrade`, including one that changes nothing | Writes unavailable in **every** location for about **117 s** | | Restart of the etcd tier | About **19 s** of write unavailability; recovers automatically | **Treat every `helm upgrade` as a planned write outage of roughly two minutes.** Members do not restart one at a time: the field that would serialize the rollout is not retained by the platform, so all members go down together and no location can write during the window. This was measured on an upgrade that changed nothing at all, so it applies to any values change, including enabling a feature. During an etcd outage Patroni's failsafe mode keeps the primary serving writes — it held for about 60 seconds with zero failed writes. Once the replicas' own health checks start failing, the failsafe loses its precondition and the primary demotes itself, which is where the \~19 s of unavailability comes from. The cluster re-promotes and resumes on its own; no operator action is needed. ### Upgrading From 1.0.0 or 1.0.1 This section describes an upgrade **within the 1.x line**, where the chart still created its own GVC. Reaching 2.0.0 from any 1.x release is a migration rather than an upgrade — see [Migrating From Version 1](#migrating-from-version-1). **Version 1.0.2 renamed every PostgreSQL-side resource, there is no in-place upgrade path, and the upgrade DELETES the volume set holding your data. Back up before you go anywhere near it.** The `-ml` infix is gone: the workloads, secrets, identity, policy and — critically — the **volume set** are now named `{release}-postgres…` instead of `{release}-postgres-ml…`. Running `helm upgrade` from 1.0.0 or 1.0.1 creates a new, empty volume set and then names the old one in its cleanup phase; it is gone seconds later, with no reachable final snapshot despite `snapshots.createFinalSnapshot`. There is no leftover copy to recover from. | Resource | 1.0.0 / 1.0.1 | 1.0.2 | | -------------------------- | --------------------------------- | ------------------------------ | | Patroni workload | `{release}-postgres-ml` | `{release}-postgres` | | **Volume set (your data)** | `{release}-postgres-ml-vs` | `{release}-postgres-vs` | | HAProxy workload | `{release}-postgres-ml-proxy` | `{release}-postgres-proxy` | | PgBouncer workload | `{release}-postgres-ml-pgbouncer` | `{release}-postgres-pgbouncer` | | Backup workload | `{release}-postgres-ml-backup` | `{release}-postgres-backup` | | Identity, policy, secrets | `{release}-postgres-ml-…` | `{release}-postgres-…` | The bundled etcd cluster's resources were never renamed and are unaffected — which is the only reason a measured upgrade kept its data at all. **Why the data survived a measured upgrade, and why that is not a plan.** Helm creates the new workload before it deletes the old one, and because etcd's volume set kept its name the cluster still had its DCS. All three new members therefore cloned the database from the still-running old primary (`bootstrapped from leader …`). That old primary kept serving for **86 seconds** after Helm reported it deleted, and a 7.5 MB database cloned in **12 seconds**. A database that cannot finish a base backup inside that window has nothing to fall back on, because the volume holding it has already been destroyed — that failure case is *inferred, not observed*, and it is not a race worth running. Two further consequences of the same upgrade, both measured: * **No writable primary for about 2 minutes 22 seconds.** The upgrade bounces the etcd tier too, so Patroni cannot immediately expire the stale leader key. * **`{release}-postgres-ml-proxy` is deleted and never returns.** Every application connection string has to move to `{release}-postgres-proxy`; nothing recovers on its own. Migrate with a backup and restore rather than an upgrade: Use `backup.mode: logical`, or run a `pg_dumpall` against the proxy endpoint of the old release. This dump is the only copy of your data that survives the rename, so verify it before you touch the release. `cpln helm uninstall RELEASE_NAME --gvc GVC_NAME`. Install the new version, then load the dump back through the proxy — see [Restoring a Backup](#restoring-a-backup). Update each application's connection string from `{release}-postgres-ml-proxy` to `{release}-postgres-proxy`. The old hostname no longer exists. Fresh installs of 1.0.2 are unaffected — this applies only to clusters created with an earlier version. ### Application Retry Logic Applications must retry on transient database errors. Two are worth handling explicitly during a failover: * Connection-level failures — `server closed the connection unexpectedly`, `terminating connection due to administrator command`. * `ERROR: cannot execute INSERT in a read-only transaction`, seen briefly when the proxy forwards to a member that has just been demoted and its health check has not yet flipped. Exponential backoff over a window of a few seconds covers a failover; an upgrade needs a window of a couple of minutes or a maintenance pause. ### Data Loss on Failover Replication is **asynchronous**. A promoted replica applies everything it has received, so a failover can lose the transactions that had not yet reached it — bounded by the replication lag at the instant of failure. Check lag with `pg_stat_replication` on the primary; in the service mesh every replication client reports the same `client_addr`, so use `application_name` (the member name) to tell members apart. ## Operating the Cluster Members are named `{workload}-{location}-{index}`, for example `my-db-postgres-aws-us-east-1-0`. `patronictl` reads the config the startup script writes to `/tmp/patroni_config.yml`: ```bash theme={null} # Every member, its location, role and replication lag cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --container patroni-postgres \ -- patronictl -c /tmp/patroni_config.yml list # Move a live primary to another location — a planned, near-zero-downtime handover cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --container patroni-postgres \ -- patronictl -c /tmp/patroni_config.yml switchover --candidate RELEASE_NAME-postgres-LOCATION-0 --force ``` ### Failover Timing The cluster ships a fixed Patroni consensus configuration, and it is what bounds how long an abrupt loss of the primary takes to resolve: | Setting | Value | Meaning | | --------------- | ------ | --------------------------------------------------------------------------------- | | `ttl` | `45` s | How long a dead primary's leader lock survives before another member may claim it | | `retry_timeout` | `15` s | How long the primary tolerates losing the consensus store before demoting itself | | `loop_wait` | `10` s | How often the HA loop runs | Patroni enforces `loop_wait + 2 × retry_timeout <= ttl` and, when that is violated, silently substitutes values of its own rather than reporting an error — which is why the three always move together. Consensus-level settings (`ttl`, `loop_wait`, `retry_timeout`, `maximum_lag_on_failover`, failsafe mode) are written once, when the cluster is first initialized, and are not values knobs — a knob would look adjustable while only ever applying to a brand-new cluster. Read and change them on a live cluster with `patronictl`: ```bash theme={null} # What this cluster is actually running cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --container patroni-postgres \ -- patronictl -c /tmp/patroni_config.yml show-config # Change them — all three together cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --container patroni-postgres \ -- patronictl -c /tmp/patroni_config.yml edit-config --force \ -s ttl=45 -s loop_wait=10 -s retry_timeout=15 ``` ### Recovering From a Lost Location With **3 or more locations** this section does not apply — losing one location is an automatic failover. With **2 locations**, losing one loses consensus quorum permanently: the survivor holds current data but cannot be granted the leader lock, and consensus writes time out. To rebuild from the surviving member, set `etcd.recovery.forceNewClusterInLocation` to that member's location and run a `helm upgrade`. Once writes are accepted again, return the value to `""` and reprovision the failed location's members — their volumes must be reset before they rejoin. ## Backing Up Backups are disabled by default. Set `backup.enabled: true`, choose a mode, and configure a storage provider. | Mode | Shape | Runs where | Good for | | --------- | --------------------------------------------------- | ---------------------------------------- | ----------------------------------------------------------------------- | | `logical` | Scheduled `pg_dumpall` cron workload | Exactly one location — `backup.location` | Portable SQL dumps, cross-version migration, smaller databases | | `wal-g` | Sidecar on every member, active only on the primary | Follows the primary | Continuous WAL archiving and point-in-time recovery on larger databases | `backup.location` exists because **a cron workload runs in every location of its GVC**. Without it a three-location cluster would dump the same database three times a night into one bucket. Pick the location nearest the bucket — that is where the dump is read and uploaded from. It applies to `logical` mode only: the WAL-G sidecar runs on every member but only the one holding the leader lock pushes, so the archive follows the primary across a failover with no selector needed. * `backup.logical.schedule` — Cron expression for the dump (default: daily at 02:00 UTC). The dump runs through the proxy, so it always hits the current primary, and includes roles and all databases. * `backup.walg.intervalSeconds` — Seconds between base backups (default `21600`, every 6 hours). WAL segments are archived continuously in between. * `backup.resources.cpu` / `backup.resources.memory` — Resources for whichever mode is enabled. Switching `backup.mode` to or from `wal-g` restarts PostgreSQL, because it changes `archive_mode`. Enabling `logical` backups does not. After enabling `wal-g`, confirm `SHOW archive_mode` reports `on` in every location before relying on the archive — one location can take up to about ten minutes to pick up the new configuration. Every provider has been exercised in both modes. **AWS S3** and **MinIO / S3-compatible** passed in `logical` and `wal-g` mode alike, at the shipped settings. **Google Cloud Storage** works but is memory-sensitive: at `backup.resources.memory: 128Mi` it failed in both modes — a `logical` job simply reports `failed` with no error line anywhere, and the `wal-g` sidecar is OOM-killed in a loop, which takes its member out of service and triggers leader elections while WAL keeps accumulating with no base backup to restore it against. It passed at 256Mi (`logical`) and 512Mi (`wal-g`), and this template ships **512Mi**, at or above both proven values — which is why the values comment warns against lowering it. Both restores have been verified end to end: a **wal-g restore** (base backup plus WAL replay into an empty data directory, checksum-identical to source) and a **logical restore** (a `pg_dumpall` artifact replayed into an `initdb`-fresh cluster with zero errors, roles, ownership and sequence positions preserved). The **volume-set swap** in the WAL-G procedure below has **not** been exercised — that step is documented, not tested. Rehearse your restore procedure in a scratch environment before you need it. ### AWS S3 Before enabling backup with `provider: aws`, complete the following in your AWS account: 1. Create an S3 bucket. Set `backup.aws.bucket` to its name and `backup.aws.region` to its region. 2. If you do not have a Cloud Account set up, refer to the docs to [Create a Cloud Account](/guides/create-cloud-account). Set `backup.aws.cloudAccountName` to its name. 3. Create an IAM policy with the following JSON, replacing `YOUR_BUCKET_NAME`: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetObjectVersion", "s3:DeleteObjectVersion" ], "Resource": [ "arn:aws:s3:::YOUR_BUCKET_NAME", "arn:aws:s3:::YOUR_BUCKET_NAME/*" ] } ] } ``` 4. Set `backup.aws.policyName` to the name of the policy created in step 3. This bucket-scoped policy is all the workload identity needs — no broad managed policy is required. 5. Set `backup.aws.prefix` to the folder path where backups will be stored. ### GCS Before enabling backup with `provider: gcp`, complete the following in your GCP account: 1. Create a GCS bucket. Set `backup.gcp.bucket` to its name. 2. If you do not have a Cloud Account set up, refer to the docs to [Create a Cloud Account](/guides/create-cloud-account). Set `backup.gcp.cloudAccountName` to its name. 3. Add the **Storage Admin** role to the GCP service account associated with the Cloud Account. The template additionally binds the identity to `roles/storage.objectAdmin` on exactly the bucket named in `backup.gcp.bucket`. 4. Set `backup.gcp.prefix` to the folder path where backups will be stored. ### MinIO **In `wal-g` mode the endpoint must be reachable from every location.** WAL-G's `restore_command` runs on every member, and a member that cannot reach the archive never finishes starting — its container is recycled roughly every 160 seconds, indefinitely. A MinIO workload that runs in only one location answers `503` immediately from the others, which is exactly the shape a same-GVC MinIO invites: measured with two of three members looping for over 20 minutes. Nothing looks wrong from a client, because the leader stays healthy and writes keep succeeding while the cluster quietly loses all of its redundancy. For `wal-g`, point the endpoint at a MinIO deployment present in every location or at an external S3-compatible service. `logical` mode is unaffected — its cron runs only in `backup.location` — and AWS S3 and GCS are global endpoints, so they never hit this. No Cloud Account is needed — credentials are supplied as a secret. 1. Create a bucket in MinIO. Set `backup.minio.bucket` to its name. 2. Set `backup.minio.endpoint` to the MinIO S3 API address including the port. For the [minio](/template-catalog/templates/minio) template deployed in the same GVC, use `http://WORKLOAD_NAME:9000`. 3. Create a dictionary secret holding the MinIO credentials and set `backup.minio.credentialsSecretName` to its name. For the `minio` template these are its `admin.username` and `admin.password`: ```bash theme={null} cpln secret create-dictionary --name my-postgres-minio-credentials \ --entry accessKey=MINIO_ACCESS_KEY \ --entry secretKey=MINIO_SECRET_KEY ``` 4. Set `backup.minio.prefix` to the folder path where backups will be stored. ## Restoring a Backup Both restore paths have been verified end to end, though not by running these exact commands: the `pg_dumpall` artifact was replayed with `psql` into an `initdb`-fresh cluster rather than through the proxy into a live one, and the WAL-G restore was fetched and replayed into an empty directory rather than onto a swapped volume set. The steps below are the operational form of those procedures — rehearse them in a scratch environment before you need them. ### Logical Stream the dump back through the proxy, which writes to the current primary. Run this from a client with access to the bucket: ```sh theme={null} export PGPASSWORD="PASSWORD" aws s3 cp "s3://BUCKET_NAME/PREFIX/BACKUP_FILE.sql.gz" - \ | gunzip \ | psql \ --host=RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local \ --port=5432 \ --username=USERNAME \ --dbname=postgres unset PGPASSWORD ``` For GCS, replace the `aws s3 cp` with `gsutil cp "gs://BUCKET_NAME/PREFIX/BACKUP_FILE.sql.gz" -`. For MinIO, add `--endpoint-url "http://MINIO_ENDPOINT:9000"` and run `aws configure set default.s3.addressing_style path` first. ### WAL-G **The last two steps of this procedure cannot be carried out through the template as it ships.** The Patroni workload's volume set name is derived from the release name and hard-rendered into the workload, and `values.yaml` exposes only `volumeset.capacity` and `volumeset.autoscaling.*` — nothing that names or substitutes a volume set. Re-pointing the workload by hand would be reverted by the next `helm upgrade`. This is a known gap, reported rather than papered over with an untested replacement: mounting a restored volume is a manual, out-of-band operation today. The archive itself is genuinely restorable — a fetched base backup was verified as a valid PostgreSQL data directory whose system identifier matched the backup's own metadata. A point-in-time restore needs an empty data directory, so it restores into a new volume set: 1. Run `wal-g backup-list` to identify the desired backup. **Verified working from the WAL-G sidecar.** 2. Stop the Patroni workload. 3. Create a new volume set and mount it at `/var/lib/postgresql/data` on a one-off restore workload. 4. Run `wal-g backup-fetch /var/lib/postgresql/data/pgdata BACKUP_NAME`. **Verified working — it produces a complete, valid data directory.** 5. Re-point the Patroni workload at the restored volume set and start it. **Not achievable through the chart — see the warning above.** 6. Change the WAL-G prefix before re-enabling backups, or the new cluster's WAL collides with the old system identifier. ## Important Notes * **There is no upgrade path from 1.x.** Every 1.x release created its own GVC; 2.0.0 deploys into an existing one, and a `helm upgrade` across that boundary deletes the old GVC and every volume set in it. The chart refuses to render on the 1.x `global.gvc` key, but that guard cannot see an upgrade run with no values at all. Follow [Migrating From Version 1](#migrating-from-version-1): new release, restore, then uninstall the old one. * **`global.locations` must match the GVC you install into.** A GVC location this release does not list runs nothing, which is harmless; a listed location the GVC lacks makes a fresh cluster refuse to bootstrap, naming the location. See [Matching the Location List to the GVC](#matching-the-location-list-to-the-gvc). * **Back up before upgrading a 1.0.0 or 1.0.1 cluster to 1.0.2.** Resource names dropped the `-ml` infix in 1.0.2, and the upgrade **deletes** the old volume set — there is no orphaned copy and no reachable final snapshot. Data survived a measured upgrade only because the new members re-cloned from the old primary during an 86-second window before it was torn down. Back up, uninstall, reinstall and restore instead, and move every connection string to `{release}-postgres-proxy`: see [Upgrading From 1.0.0 or 1.0.1](#upgrading-from-1-0-0-or-1-0-1). * **Create the credentials secret before installing.** `postgres.credentialsSecretName` names a secret this template does not create. Without it the deployment waits on a secret that does not exist; creating it afterwards clears the wedge within a few minutes. * **Every `helm upgrade` interrupts writes in all locations for about two minutes.** Plan changes as maintenance windows. * **Replication is asynchronous**, so a failover can lose the most recent transactions — bounded by the replication lag at the moment of failure. * **Set `primaryLocation` before the first install if you care where the primary is.** Since version 1.0.2 it also biases where the primary bootstraps: members elsewhere wait up to 90 seconds for the preferred location, then bootstrap anyway and log a `WARNING`. Changing it later moves a live primary — that restarts every member, costs the full upgrade outage and triggers an election — and it does not fail back automatically after an outage. * **Never suspend a location.** Suspending and resuming one permanently withdraws its endpoints from the other locations' service discovery while every status surface still reports healthy. To remove a location, take it out of `global.locations` **and** out of the GVC. * **With `internalAccess.type: workload-list`, list only your clients** — this release's own tiers are added for you. The bundled etcd tier is separate: setting `etcd.internalAccess.type: workload-list` requires adding the Patroni workload link by hand, or Patroni loses its consensus store. * **A restore from WAL-G cannot be completed through the chart today.** The archive is restorable and `backup-fetch` produces a valid data directory, but nothing in `values.yaml` names the volume set the final step tells you to swap in. See [WAL-G](#wal-g). * **Allow about two minutes after a cold install** before believing a member is unreachable — cross-location service discovery takes that long to converge. Firewall changes can take up to about four minutes. * **Consensus-level settings are not values knobs.** They are written once, at first initialization; change them with `patronictl edit-config`. * **Cost scales with write volume multiplied by the members outside the primary's location.** Each of them receives a full copy of the WAL stream, and cross-region traffic is billed. Read-mostly workloads are inexpensive to stretch; write-heavy ones are not. ## External References Patroni clustering and automatic failover documentation Command reference for inspecting and switching over a cluster The consensus-level settings changed with `patronictl edit-config` Official PostgreSQL documentation Continuous archiving and point-in-time recovery Official etcd documentation PgBouncer configuration reference View the source files, default values, and chart definition