> ## Documentation Index > Fetch the complete documentation index at: https://controlplanecorporation-majid-docs-content-expansion.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Grafana Multi-Location > Deploy Grafana OSS across Control Plane locations behind one endpoint, with dashboards, users, sessions and datasource credentials shared through a stretched Patroni PostgreSQL cluster, and exactly one instance sending each firing alert's notification — a pinned evaluator or Redis-coordinated HA. **Version 2.0.0 no longer creates a GVC — it deploys into one you already have, and there is no in-place upgrade path from 1.x.** A `helm upgrade` across that boundary deletes the GVC the old release created and everything in it, including the database holding every dashboard, user and alert rule. See [Migrating From Version 1](#migrating-from-version-1). ## Overview Grafana Multi-Location deploys **one logical Grafana** whose UI/API instances run in every location you configure, behind a single georouted `*.cpln.app` endpoint. Every dashboard, user, org, session, alert rule and saved datasource lives in a [`postgres-multi-location`](/template-catalog/templates/postgres-multi-location) cluster stretched across the same locations, so Grafana itself holds no state: there is no volume, no session affinity and nothing to hand over when an instance is replaced. **Exactly one instance sends each firing alert's notification**, in one of two shapes you choose with `alerting.highAvailability.enabled`: * **Off (the default)** — a **separate single-replica workload** runs in one location with rule execution enabled, and it is disabled on every UI instance. Exactly-once evaluation is a property of the topology: nothing is elected at runtime, and no value of `replicas` can produce a second evaluator. Losing that location stops evaluation. * **On** — that workload is not created at all. Every UI instance in every location evaluates every rule, and a stretched Redis tier coordinates the peers so exactly one **notification** is sent. Alert evaluation then survives the loss of a location, at the cost of multiplying data-source query load. See [Alert Evaluation](#alert-evaluation) before choosing. This template deploys into a GVC **you already have**, and that GVC must have at least 2 locations — 3 with alerting HA enabled. It does not create, provision or manage a GVC. For a Grafana inside a single location, use the [Grafana](/template-catalog/templates/grafana) template instead. ### How Many Locations You Need The bundled database's consensus store ([`etcd-multi-location`](/template-catalog/templates/etcd-multi-location)) commits a write only when a **majority** of its members agree, and it runs one member per location. That arithmetic decides what survives — the Grafana tier itself has no quorum of its own. | Locations | Location losses survived | What a user sees when one location is lost | | --------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **2** | **0** | Dashboards still render at times, but logins, saves and alert-state writes fail. The survivor holds current data and stays read-only until it is promoted by hand. | | **3** | **1** | Automatic database failover; the endpoint keeps serving from the two surviving locations (derived from the quorum arithmetic — a true region outage cannot be simulated here). | | **5** | **2** | Survives losing **two** locations. | With N locations you survive `floor((N-1)/2)` losses, so an even count buys nothing over the odd count below it. Losing the location named in `alerting.location` is a separate matter — see [Alert Evaluation](#alert-evaluation). **`alerting.highAvailability.enabled: true` requires at least 3 locations**, and the chart refuses to render below that. Its Redis tier elects a master by a majority of locations (one Sentinel each), so at 2 locations losing either one leaves no quorum — the exact event the knob exists to survive. ### What Gets Created * **Standard Grafana UI Workload** — `{release}-grafana`, `replicas` instances **per location**, serving the UI and HTTP API on port `3000`. Public by default. Alert rule execution is disabled here by default, and enabled on every instance when alerting HA is on. * **Standard Alert Evaluator Workload** *(optional)* — `{release}-grafana-alerting`, the same image with **exactly one replica**, running only in `alerting.location` and never reachable from the internet. Not created when `alerting.highAvailability.enabled` is `true`, or when `alerting.enabled` is `false`. * **App Database Workloads** — The `postgres-multi-location` subchart: a stateful Patroni PostgreSQL workload with one primary and asynchronous replicas, a stateful etcd workload for consensus, and an HAProxy leader-routing tier in every location. * **Alerting Coordination Workloads** *(optional)* — The `redis-multi-location` subchart: `{release}-redis` and `{release}-sentinel`, one of each **per location**, coordinating exactly-once notification delivery. Created **only** when `alerting.highAvailability.enabled` is `true`. * **Volume Sets** — `{release}-postgres-vs` for the PostgreSQL data directory and the etcd cluster's own volume set, plus `{release}-redis-vs` and `{release}-sentinel-vs` with alerting HA on. Grafana itself has none. * **Secrets** — The subchart startup scripts and Redis configuration, and — only when `datasources.definitions` is set — `{release}-grafana-datasources`, the rendered datasource provisioning file mounted by every Grafana workload. The admin password, encryption key and database credentials are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. * **Identities & Policies** — One identity shared by every Grafana workload, with `reveal` on exactly the secrets they mount and nothing else, plus a second policy granting `view` on **exactly the one GVC this release installs into** so the boot check can read that GVC's location list. Each subchart tier gets its own identity and policy. No GVC resource is created. Every resource above lands in the GVC you install into, and nothing runs in that GVC's other locations. ## Prerequisites **An existing GVC with at least 2 locations** — 3 if you enable alerting HA, and 3 for automatic database failover. This template deploys into the GVC you install it into and creates none of its own; every entry in `global.locations` must already be one of that GVC's locations. Read them back with `cpln gvc get GVC_NAME -o yaml` and compare `spec.staticPlacement.locationLinks`. See [Matching the Location List to the GVC](#matching-the-location-list-to-the-gvc). **Three secrets must exist before you install.** The admin account is a human-facing login and `publicAccess.enabled` defaults to `true`, so its password never passes through Helm values; the encryption key and the database credentials are shared by every instance in every location. An [opaque secret](/guides/create-secret/opaque) with encoding `plain`: ```bash theme={null} printf '%s' "$(openssl rand -hex 24)" \ | cpln secret create-opaque --name my-grafana-admin-password --encoding plain -f - ``` Set `admin.passwordSecretName` to the name you used. An [opaque secret](/guides/create-secret/opaque) with encoding `plain`. Every instance reads it on every boot to decrypt datasource credentials stored in the shared database — back it up outside Control Plane and never rotate it: ```bash theme={null} printf '%s' "$(openssl rand -hex 32)" \ | cpln secret create-opaque --name my-grafana-secret-key --encoding plain -f - ``` Set `admin.secretKeySecretName` to the name you used. A [dictionary secret](/guides/create-secret/dictionary) holding exactly `username`, `password` and `database`: ```bash theme={null} cpln secret create-dictionary --name my-grafana-db-credentials \ --entry username=grafana \ --entry password="$(openssl rand -hex 24)" \ --entry database=grafana ``` Use plain identifiers for `username` and `database` — they are used unquoted when the database is created. Set `postgresML.postgres.credentialsSecretName` to the secret's name. ```bash theme={null} cpln secret reveal my-grafana-db-credentials -o json ``` Without `-o json` the command prints a table containing no secret data. **A missing prerequisite secret wedges the install rather than failing it.** `cpln helm install` still reports success while the affected workload sits at zero replicas waiting on a secret reference that never resolves, which looks like a broken install. Create all three first, and confirm with `cpln workload get-deployments {release}-grafana --gvc {gvc}` rather than trusting the Helm output. Optional features each need something created **before** you install: | Feature | What you must create first | | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Credentialed provisioned datasources | One [dictionary secret](/guides/create-secret/dictionary) per entry in `datasources.credentialSecrets`, holding the credential keys | | Authenticated SMTP | An [opaque secret](/guides/create-secret/opaque) with encoding `plain` holding the SMTP password, named in `smtp.passwordSecretName` | | Database backups | A bucket and access setup on AWS S3, Google Cloud Storage, or an S3-compatible server — see [Backing Up](#backing-up) | | Authenticating the alerting-HA Redis tier | One [opaque secret](/guides/create-secret/opaque) with encoding `plain` per password — see [Authenticating the Redis Tier](#authenticating-the-redis-tier). Optional hardening: alerting HA itself needs **no** extra secrets | Once the secrets exist, install the template using your preferred method: Browse, install, and manage templates visually Manage templates from your terminal }> Declare templates in your Terraform configurations Pulumi Icon Streamline Icon: https://streamlinehq.com } > Declare templates in your Pulumi programs ## Configuration The default `values.yaml` for this template: ```yaml theme={null} # ─── Locations ──────────────────────────────────────────────────────────────── # This chart deploys into the GVC you install into — it does NOT create one. # Every location listed here MUST already exist in that GVC. The platform does # not validate that: a location the GVC lacks is stored verbatim and is simply # inert, so nothing runs there and no deployment reports a failure. Both Grafana # tiers read the GVC at boot and log a warning naming the mismatch. Extra # locations in the GVC are fine — nothing here runs in them. # # Lives under `global` so BOTH subcharts get the same list automatically — the # postgres-multi-location database (and its own etcd, two levels down) and the # optional redis-multi-location alerting coordinator. Never maintain two lists. # # Minimum 2 locations, and minimum 3 with alerting HA enabled below. The # database tier needs 3 for automatic failover and 5 to survive losing two — # see the survival table in the README. # `replicas` here sets the POSTGRES members in that location, and nothing else. # Every other tier ignores it and has its own count: # etcd — always exactly 1 per location, not configurable. Quorum is a # majority of LOCATIONS, so a second member in one buys nothing. # Grafana — the top-level `replicas` below, applied to every location. # Redis — redisML.redis.replicasPerLocation (alerting HA only). global: locations: - name: aws-us-east-1 replicas: 1 - name: aws-eu-central-1 replicas: 1 - name: aws-us-west-2 replicas: 1 # ─── Grafana UI tier ────────────────────────────────────────────────────────── image: grafana/grafana:13.1.3 replicas: 1 # Grafana UI instances PER LOCATION resources: maxCpu: 1000m maxMemory: 1Gi minCpu: 500m minMemory: 512Mi database: maxOpenConn: 10 # per instance; see the README budget # ─── Alerting ───────────────────────────────────────────────────────────────── # enabled: false turns rule evaluation off entirely — rules can still be created # and viewed, nothing ever evaluates them. # # highAvailability.enabled: false (default) — a SEPARATE single-replica workload # in `location` evaluates every rule, and exactly-once follows from there being # only one of it. Restarting it re-notifies whatever is firing at the time, and # losing that location stops evaluation with no sign of it in the UI. # # highAvailability.enabled: true — no separate workload; every UI instance # evaluates and a stretched Redis (redisML below) picks exactly one to SEND, so # alerting survives losing a location. Needs 3+ locations. The costs: datasource # query load multiplies by (locations × replicas) because Redis dedupes the # notification and not the query; a large rule set may need resources.maxCpu # raised; and if Redis is unhealthy alerting DUPLICATES rather than going silent. alerting: enabled: true highAvailability: enabled: false # Dedicated-evaluator mode ONLY — where the one evaluator runs. Must be one of # global.gvc.locations. IGNORED when highAvailability.enabled is true. location: aws-us-east-1 # Dedicated-evaluator mode ONLY — in HA mode the UI tier's `resources` apply. resources: maxCpu: 1000m maxMemory: 1Gi minCpu: 500m minMemory: 512Mi # ─── Admin Bootstrap & Encryption (prerequisite secrets) ────────────────────── # BOTH secrets must EXIST BEFORE INSTALL — opaque, encoding: plain (see README). # The admin login is on the public internet when publicAccess.enabled is true, # so its password never transits values or the Helm release. admin: user: admin # initial admin login name (not sensitive) # The password applies only when the admin account is FIRST created; on later # boots Grafana ignores it (change it in the UI instead). applyPassword: true # set false after your first login to stop referencing the password secret passwordSecretName: my-grafana-admin-password # opaque secret holding the first-boot admin password # DIFFERENT LIFECYCLE — permanent. Encrypts datasource passwords AT REST in the # shared database; it protects nothing in transit (that is TLS, via the # datasource URL). Every instance in every location must use the SAME key, or a # datasource saved in one region is undecryptable in another and rules querying # it fail silently. Rotating it invalidates every credential already stored, so # never delete it and never rotate it. secretKeySecretName: my-grafana-secret-key # opaque secret holding the at-rest encryption key # ─── Datasources as Code (optional) ─────────────────────────────────────────── # Grafana datasource provisioning entries, passed through verbatim. Every # instance of every workload applies the same file on boot. datasources: definitions: [] # - name: Prometheus # type: prometheus # access: proxy # # Substitute BOTH parts: the workload name AND the GVC. A leftover # # placeholder resolves to nothing and the panel shows only # # "upstream connect error ... connection timeout". # url: http://YOUR_WORKLOAD.YOUR_GVC.cpln.local:9095 # isDefault: true # - name: AppDB # type: postgres # url: my-db-host:5432 # user: grafana_reader # jsonData: { database: appdb, sslmode: disable } # secureJsonData: # password: $PG_PASSWORD # interpolated from credentialSecrets below # The credentials Grafana authenticates TO each datasource with. `definitions` # renders into a plaintext provisioning file, so put the password in a # pre-created dictionary secret and write $KEY above. Every $KEY needs an entry # here, and the secrets MUST EXIST BEFORE INSTALL. credentialSecrets: [] # - name: my-grafana-ds-credentials # keys: [PG_PASSWORD] # ─── SMTP for Alert Emails (optional) ───────────────────────────────────────── # Whichever instance evaluates is what actually sends: the dedicated evaluator # workload by default, or the coordinating UI instance when alerting HA is on. smtp: enabled: false host: smtp.example.com:587 # host:port user: "" # empty = unauthenticated SMTP. IF YOU SET THIS, the relay MUST offer # STARTTLS or TLS: Grafana refuses to send credentials over an unencrypted # connection ("failed to send email: unencrypted connection") and every # notification is then lost, with the only signal a log line in the # sending workload. Hosted relays (SES, SendGrid, Mailgun, M365, Gmail) # are fine; a plain in-GVC relay is not — leave this empty for those. passwordSecretName: "" # opaque secret (encoding: plain) with the SMTP password; create BEFORE install fromAddress: grafana@example.com fromName: Grafana # ─── Access ─────────────────────────────────────────────────────────────────── # Applies to the UI tier. The dedicated evaluator workload is NEVER public; it # honours internalAccess only, so you can reach it in-GVC to manage silences. publicAccess: enabled: true # UI on the canonical *.cpln.app HTTPS endpoint internalAccess: type: same-gvc # options: none, same-gvc, same-org, workload-list # Only used when type is workload-list. This chart's OWN workloads (the UI # tier and, when it renders, the alert evaluator) are added automatically — # the list also governs the in-GVC path the README's silence procedure uses, # so a list naming only clients would deny it. List your clients here; do not # list this release's workloads. workloads: [] # workloads: # - //gvc/GVC_NAME/workload/WORKLOAD_NAME # ─── App Database (subchart: postgres-multi-location) ───────────────────────── # One Patroni cluster stretched across the same locations: a single primary, # async replicas elsewhere, automatic promotion in a surviving location. # Grafana has no read/write splitting, so EVERY query goes to the primary — # put it where most of your users are. postgresML: # The stretched etcd cluster behind Patroni. Compaction is on by default and # should stay on: without it etcd's backend grows with time alone — Patroni # renews its lease every ~10s and every renewal is a revision — until it hits # the quota and etcd goes READ-ONLY, taking the database's failover with it. etcd: tuning: autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count) autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h) quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB postgres: # REQUIRED PREREQUISITE SECRET — CREATE IT BEFORE YOU INSTALL. # A `dictionary` secret holding exactly `username`, `password` and # `database`. If it does not exist the deployment wedges waiting on it. credentialsSecretName: my-grafana-db-credentials # The database tier's own internal firewall. Leave it at same-gvc unless you # have a reason not to: this chart's Grafana workloads are NOT added to the # subchart's list automatically (a parent cannot inject a rendered name into a # subchart's values), so switching to workload-list without listing them cuts # Grafana off from its own database. The chart refuses to render in that state # and prints the exact links to add. internalAccess: type: same-gvc # options: same-gvc, same-org, workload-list workloads: [] # Preferred location for the database primary. Keep it aligned with # alerting.location so the hot path has no cross-region hop. With alerting HA # on, this also decides which Grafana location runs the schema migrations. primaryLocation: aws-us-east-1 resources: minCpu: 500m minMemory: 1Gi maxCpu: 1 maxMemory: 2Gi volumeset: capacity: 10 # initial capacity in GiB per member (minimum is 10) proxy: minReplicas: 2 # HAProxy leader-routing tier, per location maxReplicas: 2 backup: # optional database backups — see Storage setup in the README enabled: false mode: logical # logical or wal-g location: aws-us-east-1 # logical mode only: the ONE location the nightly job runs in resources: cpu: 100m # 512Mi, matching postgres-multi-location's own default. At 128Mi the GCP # path OOMs with NO log output: logical jobs merely report `failed` and the # wal-g sidecar loops on OOMKilled while WAL archives with no base backup. # Do not lower this without re-testing the GCS path. memory: 512Mi logical: image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0 schedule: "0 2 * * *" walg: intervalSeconds: 21600 provider: aws # options: aws, gcp, minio aws: bucket: my-grafana-bucket region: us-east-1 cloudAccountName: my-s3-cloud-account policyName: my-grafana-backup-policy prefix: grafana/backups gcp: bucket: my-grafana-bucket cloudAccountName: my-gcs-cloud-account prefix: grafana/backups minio: endpoint: http://my-minio-workload:9000 bucket: my-grafana-bucket credentialsSecretName: my-grafana-minio-credentials prefix: grafana/backups # ─── Alerting-HA coordination (subchart: redis-multi-location) ──────────────── # Rendered ONLY when alerting.highAvailability.enabled is true. One Redis and one # Sentinel per location; failover needs a majority of them, which is why alerting # HA requires 3+ locations. redisML: # Which server the coordination tier runs. `redis` changes nothing. `valkey` # runs BOTH the Redis and Sentinel tiers on valkeyImage — the BSD-licensed # fork of Redis 7.2, which ships redis-server/redis-cli/redis-sentinel # compatibility symlinks, so Grafana's Sentinel client is unchanged. Chosen at # INSTALL time: a data directory cannot be moved between engines. engine: redis # redis | valkey valkeyImage: valkey/valkey:8.1.9 # used for both tiers when engine is valkey # Same rule as postgresML.internalAccess above: Grafana is not added to this # list automatically, and blocked from Sentinel it boots fine and every # instance sends its own copy of every alert. The chart refuses to render if # you set workload-list without listing the Grafana workloads. firewall: internalAllowType: same-gvc # options: same-gvc, same-org, workload-list workloads: [] redis: # OPTIONAL hardening, off by default. Opaque secret (encoding: plain) whose # payload is the password; it MUST EXIST BEFORE INSTALL, and Grafana reads the # same one, so a misspelt name stops the UI tier too. "" leans on the same-GVC # firewall instead — reasonable when nothing untrusted shares the GVC, which # you now choose rather than the chart choosing for you. passwordSecretName: "" image: redis:7.4 replicasPerLocation: 1 # Redis members per location; 1 is plenty for alert coordination resources: cpu: 200m memory: 256Mi volumeset: initialCapacity: 10 # GiB per member (platform minimum); coordination data is tiny sentinel: # Same shape and same failure mode as redis.passwordSecretName, independent of # it — Sentinel is what Grafana asks for the current master. passwordSecretName: "" image: redis:7.4 resources: cpu: 200m memory: 256Mi ``` ### Locations The location list lives under `global` so that Helm passes the same list to the bundled database, through it to etcd, and to the optional Redis tier. Never maintain two location lists. * `global.locations[].name` — A Control Plane location (e.g. `aws-us-east-1`) that **must already be one of the locations of the GVC you install into**. At least 2 are required, at least 3 with `alerting.highAvailability.enabled: true`, and the chart refuses to render with fewer. * `global.locations[].replicas` — **Database** members in that location, not Grafana instances. It must be at least 1; removing a location from the list is the supported way to shrink the deployment. Grafana's own count per location is the top-level `replicas`, etcd always runs exactly one member per location, and the Redis tier has its own count (`redisML.redis.replicasPerLocation`) — all three ignore this number. ### Matching the Location List to the GVC The platform validates the pairing in neither direction. A GVC location this release does not list is harmless — nothing runs there, and its deployments read `This workload location is deactivated because maxScale is set to 0`. A location the GVC does not have is the dangerous direction: the platform accepts it, stores it, and it is simply inert, with no failed deployment to see. Both Grafana tiers read the GVC at boot, using the scoped `view` grant described in [What Gets Created](#what-gets-created), and log a warning naming any mismatch: ```bash theme={null} cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-grafana"}' | grep '\[grafana\]' ``` **A fresh install with a location the GVC lacks fails loudly, not quietly.** `alerting.location` must also appear in `global.locations` — a render-time check enforces that — so a mismatch is necessarily a `global.locations` mismatch, and the bundled etcd and Patroni tiers **refuse to bootstrap on a fresh data directory** for any location the GVC does not have. They crash-loop with a `FATAL:` line naming the location, Grafana never gets past its database gate, and nothing comes up half-working. Fixing the values and upgrading recovers the whole stack with no manual intervention. The quiet failure applies to a location **removed from the GVC after the cluster was initialized**: the database tiers then only warn and keep serving, so dashboards stay healthy while whatever ran in that location silently does not. See [Alert Evaluation](#alert-evaluation) for what that costs when the location is `alerting.location`. ### Grafana UI Tier * `image` — The Grafana OSS container image. It must provide `/bin/bash`: the chart overrides the entrypoint to run Grafana's `/run.sh` through a boot wrapper. The default Alpine-based official image does. * `replicas` — Grafana UI instances **per location**. It carries no alerting-related restriction in either mode — the only thing to watch is the connection budget below. * `resources` — `minCpu`/`minMemory` are the reservation, `maxCpu`/`maxMemory` the limit, applied per instance. * `database.maxOpenConn` — Maximum database connections **per instance**. `(replicas × locations + 1) × database.maxOpenConn` must stay at **80 or less** — the bundled cluster's `max_connections` is 100, and the remainder is headroom for Patroni and administration. The `+ 1` is the dedicated alert evaluator, and it is dropped from the arithmetic when there isn't one, i.e. with alerting HA **on** or `alerting.enabled: false`. The chart enforces the budget at render time and refuses to install with the arithmetic spelled out, so raising `replicas` past that point means lowering `database.maxOpenConn` (or lowering `replicas`). With alerting HA on, rule evaluation runs on this tier rather than on a workload of its own, so a large rule set may need `resources.maxCpu` raised above the default — `alerting.resources` is unused in that mode. Signup, anonymous access and upstream analytics are disabled by the template and are not configurable. ### Admin Credentials and Encryption Key Both credentials live in [opaque secrets](/guides/create-secret/opaque) you create before installing (see [Prerequisites](#prerequisites)) — neither ever passes through Helm values. They sit next to each other in `values.yaml` but their lifecycles are opposites: | | `admin.passwordSecretName` | `admin.secretKeySecretName` | | ---------------------------- | -------------------------------------------------------------------- | ------------------------------------------------ | | What it holds | The `admin` login password | The key encrypting stored datasource credentials | | When Grafana reads it | Only when the admin account is **first created** | On **every boot**, on every instance | | Can you stop referencing it? | Yes — set `admin.applyPassword: false` | **No.** There is no toggle | | Can you rotate or delete it? | Yes, once `applyPassword` is `false` — change the password in the UI | **Never** | Every Grafana workload carries the admin bootstrap environment deliberately. Grafana's built-in default password is the literal string `admin`, so an instance without it that won the race for the empty database would create an `admin`/`admin` account on a publicly exposed UI. Testing confirms that does not happen: `admin`/`admin` is refused with a `401` on a fresh install. The encryption key is permanent. Every instance in every location decrypts saved datasource credentials with it, so changing its payload makes all of them unreadable everywhere — and alert rules that query those datasources then fail. Back the key up outside Control Plane instead of rotating it. ### Datasources as Code `datasources.definitions` entries are standard [Grafana datasource provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources) entries, rendered verbatim into a provisioning file that every instance of every Grafana workload mounts at boot. Applying the same file concurrently from every instance is safe — testing at ten instances produced exactly one row per definition. Credentials never go into the provisioning file. Put them in a [dictionary secret](/guides/create-secret/dictionary) created before installing, list it under `datasources.credentialSecrets`, and reference each key as `$KEY`: ```yaml theme={null} datasources: definitions: - name: AppDB type: postgres url: my-db-host:5432 user: grafana_reader jsonData: { database: appdb, sslmode: disable } secureJsonData: password: $PG_PASSWORD credentialSecrets: - name: my-grafana-ds-credentials keys: [PG_PASSWORD] ``` Each listed key is exposed to every Grafana workload as an environment variable and interpolated when Grafana loads the file; the chart grants the identity `reveal` on exactly that secret. Provisioned datasources are read-only in the UI — change the value and upgrade to change one. **Substitute both parts of an in-GVC datasource URL — the workload name *and* the GVC name.** A leftover placeholder resolves to nothing, and the panel shows only `upstream connect error ... connection timeout` with no hint at the cause. A name that does not resolve times out rather than failing fast, so this looks like a network or firewall problem and is not one. ### SMTP **Authenticated SMTP requires a relay that offers STARTTLS or TLS.** Grafana refuses to send credentials over an unencrypted connection — it fails with `unencrypted connection` and **every notification is lost**, with the only signal a log line in whichever workload does the sending. Hosted relays (SES, SendGrid, Mailgun, Microsoft 365, Gmail) are unaffected; a plain in-GVC relay is not. Leave `smtp.user` empty to send unauthenticated against such a relay. Grafana sends alert notification emails through the SMTP server you configure, and **whichever instance evaluates is what actually sends them** — the dedicated evaluator by default, or the coordinating UI instance when alerting HA is on. `smtp.passwordSecretName` is required whenever `smtp.user` is set, and the password stays in a pre-created [opaque secret](/guides/create-secret/opaque) with encoding `plain`. Leave `smtp.enabled: false` if you use webhook or chat contact points instead. ### Access `publicAccess.enabled` applies to the **UI tier only**. When it is on, the tier is served on the canonical `*.cpln.app` HTTPS endpoint and Grafana's `root_url` is derived from it automatically, including for the evaluator, so links in delivered notifications point at the UI tier and open in a browser. The dedicated evaluator itself is never reachable from the internet — it gets a canonical endpoint but requests to it are refused with a `403`. With alerting HA on there is no evaluator workload, and the Redis tier is reachable from inside the GVC only. | `internalAccess.type` | Description | | --------------------- | ------------------------------------------------------------------------ | | `none` | No internal access. | | `same-gvc` | Allow access from all workloads in the same GVC (default). | | `same-org` | Allow access from all workloads in the same organization. | | `workload-list` | Allow access only from the workload links in `internalAccess.workloads`. | **With `workload-list`, list only your clients.** This chart's own workloads — the UI tier and, when it renders, the alert evaluator — are appended for you, because the same list also governs the in-GVC path the silence procedure below uses. A client that is not listed is still refused. **The two subcharts keep their own firewall lists, and Grafana is not added to them automatically** — a parent chart cannot inject a rendered workload name into a subchart's values. Switching `postgresML.internalAccess.type` or `redisML.firewall.internalAllowType` to `workload-list` without listing the Grafana workloads would cut Grafana off from its own database or its alerting coordinator. Rather than let that happen, **the chart refuses to render and prints the exact links to add**, naming the evaluator as well as the UI tier when both exist. Paste them into the subchart's own `workloads` list and the render succeeds, with no duplicate entries. A firewall change is not instant. Turning public access off was measured at about 107 seconds after the new workload version was serving, and enforcement of a subchart `workload-list` change took about four minutes. Allow several minutes and re-test before concluding a setting did not apply. ### App Database The `postgresML` block configures the bundled [`postgres-multi-location`](/template-catalog/templates/postgres-multi-location) cluster: one primary, asynchronous replicas in the other locations, an HAProxy leader-routing tier in each, and automatic promotion in a surviving location. Everything the database tier can do — pooling, restores, emergency quorum recovery, per-member addressing — is documented on that template's page. `postgresML.internalAccess` is the database tier's **own** internal firewall, separate from this chart's. Leave it at `same-gvc` unless you have a reason not to — see the warning under [Access](#access) for what `workload-list` requires there. `postgresML.primaryLocation` is a **preferred** location for the primary: the members elsewhere wait up to 90 seconds for it to initialize the cluster before bootstrapping themselves, and it also biases later elections. Keep it aligned with `alerting.location` so the evaluator's queries have no cross-region hop. With alerting HA on it also decides which Grafana location runs the schema migrations first. Grafana has **no read/write splitting** — every query, including every dashboard load, goes to the single primary. Testing confirms the shape directly: with seven Grafana instances running, all application connections were on the primary and both standbys carried none. Every location except the primary's therefore pays one cross-region round trip per query, so set `primaryLocation` where most of your users are. #### etcd History Compaction `postgresML.etcd.tuning.autoCompactionMode`, `postgresML.etcd.tuning.autoCompactionRetention` and `postgresML.etcd.tuning.quotaBackendBytes` control how much revision history the database's consensus store keeps and how large its backend may grow. Compaction has been enabled in every version of the bundled etcd chart; since template version `1.1.1` the values are also configurable here. The defaults — `periodic`, `1h` and `0` (etcd's own 2 GiB limit) — are the right settings for a Patroni consensus store and should be left alone: an etcd cluster that fills its backend goes read-only, which takes the database's failover with it. See [Compaction and Backend Growth](/template-catalog/templates/etcd-multi-location#compaction-and-backend-growth) for the mechanism and the accepted value formats. ### Alerting-HA Coordination The `redisML` block configures the bundled [`redis-multi-location`](/template-catalog/templates/redis-multi-location) subchart — one Redis and one Sentinel per location, coordinating exactly-once notification delivery. It is rendered **only** when `alerting.highAvailability.enabled` is `true`, and ignored entirely otherwise. * `redisML.engine` — `redis` (the default, which changes nothing) or `valkey`, which runs **both** the Redis and Sentinel tiers on `redisML.valkeyImage`. Valkey is the BSD-licensed fork of Redis 7.2 and ships `redis-server` / `redis-cli` / `redis-sentinel` compatibility symlinks, so Grafana's Sentinel client is unchanged. It is an **install-time** choice: a data directory cannot be moved between engines. * `redisML.valkeyImage` — The image used for both tiers when `engine` is `valkey`, at which point `redisML.redis.image` and `redisML.sentinel.image` are inert. * `redisML.firewall.internalAllowType` / `.workloads` — The coordination tier's own internal firewall. See the warning under [Access](#access) before setting it to `workload-list`. * `redisML.redis.replicasPerLocation` — Redis members per location. One is plenty: the data is a few kilobytes of coordination keys. * `redisML.redis.volumeset.initialCapacity` — GiB per member, at the platform minimum of `10`. * `redisML.{redis,sentinel}.image` / `.resources` — Passed through to the subchart. These blocks expose only a limit, so they use bare `cpu` and `memory`. * `redisML.{redis,sentinel}.passwordSecretName` — Optional authentication, empty by default. See below. #### Authenticating the Redis Tier **Authentication is off by default: enabling alerting HA is one flag and needs no extra secrets.** The tier is reachable only from inside the GVC, and that same-GVC firewall is the boundary it relies on. **Since version 2.0.0 that GVC is yours rather than this chart's** — it is the GVC you installed into, and it very likely already holds other workloads. Weigh the default accordingly and turn authentication on unless you control everything in that GVC. Write access to an authless Redis is enough to **suppress your alert notifications**, because a hostile or simply buggy neighbor can claim another peer already sent them, and alert coordination is precisely the thing you do not want failing quietly. Create either or both [opaque secrets](/guides/create-secret/opaque) with encoding `plain` **before installing**: ```bash theme={null} printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name my-grafana-redis-password --encoding plain -f - printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name my-grafana-sentinel-password --encoding plain -f - ``` Then name them: ```yaml theme={null} redisML: redis: passwordSecretName: my-grafana-redis-password sentinel: passwordSecretName: my-grafana-sentinel-password ``` The two are independent — setting one without the other is valid. Grafana authenticates with `ha_redis_password` and `ha_redis_sentinel_password`, reading the **same secrets** the Redis tier reads, so the two sides cannot drift apart. Testing confirmed both ports reject unauthenticated clients (`NOAUTH`) and a wrong password (`WRONGPASS`) on a fresh install with the passwords set from the first boot, and again after authentication is turned on for an already-running authless install. Rotating the Sentinel password was verified the same way — the new password is accepted and the old one rejected on every Sentinel — and turning a password back off also takes effect, after which that port accepts unauthenticated clients again. In the authenticated runs all six Grafana instances registered their peer keys inside the authenticated keyspace. The fresh-install result was measured through this template; enabling, rotating and removing a password afterwards was measured on the `redis-multi-location` subchart's own test run. **If you do set these names, a name that does not resolve stops the Grafana UI, not just alerting** — Grafana mounts the same secret. `cpln helm install` still reports **success**; the Redis, Sentinel and Grafana UI tiers then sit at **0 replicas** with no containers and therefore no logs. The explanation is on each workload's `status.versions[].message` — not the top-level `status.message`, which is empty — and reads `The secret no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` Create the missing secret and it recovers on its own in about 5–6 minutes; no Helm action is needed. ## Alert Evaluation `alerting.enabled` is the parent switch, on by default. Setting it to `false` creates no evaluator workload and turns rule execution off on every instance: alert rules and contact points can still be created and viewed, and nothing ever evaluates them. Testing measured exactly that — two always-firing rules present, both reporting a null evaluation timestamp, and **zero notifications over 6 m 24 s** — while the UI, logins and dashboards were unaffected. Combining `alerting.enabled: false` with `alerting.highAvailability.enabled: true` is a contradiction and fails at render time. Everything below concerns the two shapes available while alerting is on. Grafana's memberlist alerting HA coordinates instances over a UDP gossip channel, which is not available between workloads on Control Plane; Grafana's **Redis-backed** coordination needs no peer port at all, and that is what `alerting.highAvailability.enabled` turns on. ### Choosing an Alerting Mode | | `highAvailability.enabled: false` (default) | `highAvailability.enabled: true` | | -------------------------------------------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | Who evaluates rules | The dedicated `{release}-grafana-alerting` workload — 1 replica, 1 location | **Every UI instance**, in every location | | Exactly-once delivery | A property of the topology: there is only one evaluator | Redis-coordinated: peers order themselves and share a notification log | | Extra tier | None | 1 Redis + 1 Sentinel **per location** | | Losing a location | Alert evaluation **stops** until you repoint `alerting.location` and upgrade | Evaluation **continues** in the surviving locations (derived — see the note below; a true region outage cannot be simulated here) | | Minimum locations | 2 | **3** — the chart refuses to render below that | | **Data-source query load** | **1×** | **(locations × replicas)×** | | Silences | Expected not to propagate reliably — see the workaround below (not exercised in our testing) | Expected to propagate through the shared peer (upstream behaviour, not exercised in our testing) | | Where `alerting.location` and `alerting.resources` apply | Both | Neither — they are ignored | ### Read This Before Enabling HA **Turning HA on multiplies your data-source query load by the number of Grafana instances.** Every instance evaluates every rule against your Prometheus, Mimir or SQL server; Redis dedupes the **notification**, never the **query**. At the default 3 locations × 1 replica that is **3× the queries, permanently**. If your data source is already the bottleneck, that is a worse trade than losing alert evaluation when a region dies — leave the knob off. The other costs, at three locations and defaults: **+6 containers and +6 × 10 GiB volumes** for the Redis tier (minus the one evaluator container), constant cross-region heartbeat traffic to whichever location holds the Redis master, and rule evaluation landing on the UI tier's `resources` rather than on `alerting.resources`. ### Dedicated Evaluator Mode (the Default) Rule execution is disabled on the UI tier and enabled on a separate workload pinned to one replica in `alerting.location`, with zero replicas in every other location. Nothing is elected at runtime, and no value of `replicas` can produce a second evaluator. `alerting.location` is **required** in this mode and must name one of `global.locations`; a render-time check enforces that. Verified in testing at `replicas: 3` across three locations: all **9 UI instances** report rule execution disabled, the **single evaluator** reports it enabled, and the other two locations run no evaluator replica at all. In a later run at the default `replicas: 1`, an always-firing rule produced **41 notifications with 41 distinct request IDs over 26 minutes**, every one of them from the evaluator's own address. * **Losing the evaluator's replica self-heals, and re-notifies.** The platform reschedules it and evaluation resumes with no operator action — a gap of **21 seconds** between the last notification from the old replica and the first from its replacement (measured on 1.0.0, before the database gate was extended, so expect this to be a floor rather than a ceiling), with the UI tier unaffected throughout. Each replacement boots with an empty notification log and immediately re-sends whatever is currently firing, so a restart costs **duplicate** notifications rather than missed ones. * **Losing that whole location does not self-heal.** Alert evaluation stops until you run `helm upgrade` with `alerting.location` set to a surviving location, and the UI gives no sign of it — dashboards look perfectly healthy while nothing is being evaluated. This is the failure alerting HA exists to remove. * **That silent shape only happens to a location removed from the GVC after the cluster was initialized.** A **fresh** install naming a location the GVC lacks fails loudly instead: `alerting.location` must also be in `global.locations`, and the bundled etcd and Patroni tiers refuse to bootstrap on a fresh data directory for any location the GVC does not have, so the whole stack crash-loops with a named error rather than coming up half-working. Either way the boot warning names the consequence explicitly — the evaluator workload holding zero replicas everywhere, and no alert rule ever being evaluated. * **Moving `alerting.location` does not open a two-evaluator window.** Only the evaluator workload is updated; the UI tier is untouched and does not restart. In the measured relocation the old evaluator's last notification preceded the new evaluator's readiness by **57 seconds** (measured on 1.0.0, same caveat), so the two never overlapped. Expect a gap of a few minutes with no evaluation while the new one boots. * **Silences are not expected to propagate between instances** (not exercised in our testing). Without gossip, a silence created against the UI tier is not guaranteed to be honored by the evaluator, so create silences against the evaluator directly: ```bash theme={null} curl -u admin:PASSWORD http://RELEASE_NAME-grafana-alerting.GVC_NAME.cpln.local:3000/api/alertmanager/grafana/api/v2/silences ``` The API requires credentials, so the `-u` flag is not optional. **The evaluator only answers from inside `alerting.location`.** Its internal name resolves to the GVC address from every location, but the other locations have no local upstream for it and return `503`. Run the silence command from a workload replica that is running in `alerting.location`. ### Redis-Coordinated Alerting HA With `alerting.highAvailability.enabled: true` no evaluator workload is created. Every UI instance evaluates every rule, registers itself as a peer in the stretched Redis tier, and the peers order themselves so that exactly one **sends** the notification. Each instance selects its own local Sentinel and all of them resolve the same Redis master, across regions. Configure the tier — including optional authentication — under [`redisML`](#alerting-ha-coordination). Measured across three locations at `replicas: 2` (six evaluating instances): | What was measured | Result | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Exactly-once delivery | On the shipped authless default, **19 deliveries / 19 firing cycles / 19 distinct request IDs over 9 m 30 s**, against **114** if the six instances were uncoordinated. A run with the Redis tier authenticated measured **21 / 21 / 21 over 9 m 59 s** against **126** | | Notification links | `externalURL`, `generatorURL` and `silenceURL` are all the UI tier's canonical endpoint and return `200` when authenticated | | Redis unhealthy | Degrades to **duplicate** notifications — one per instance — and **never goes silent**; the longest gap during a five-minute Redis outage was 15 s, and it re-converged about **40 s** after the fault stopped, with no operator action | | A `helm upgrade`, including a no-op | **80–95 s** of duplicate notifications while the Redis tier is patched and peers churn (4 extra notifications over a 93 s window), then exactly-once resumes | **"We're getting every alert twice" almost always means the Redis tier is unhealthy, not that your alert rules are wrong.** Check the `{release}-redis` and `{release}-sentinel` workloads first. The failure direction is always duplicates, never silence. Two more properties to design around: * **Which location sends is not pinnable.** The sender is chosen by sorted peer name, and a standard workload's name carries a ReplicaSet hash that changes on every rollout — the elected sender moved from `aws-us-east-1` to `aws-us-west-2` between two installs of the same chart. Do not build routing, filtering or egress-IP allowlisting on the assumption that a particular location sends. * **After a location is lost, hand-off is not instant — and this one is derived, not timed.** Two things were measured directly: the Redis peer keys carry a **5-minute TTL**, so a dead location's peers keep their positions until it expires, and a location other than the previous sender was observed taking alerting over on its own after a rollout reordered the peers. The loss *event* itself has never been observed — no platform primitive simulates a true region outage — so "alerting continues from the survivors and only the sending instance changes" follows from those two measurements rather than from a timed failover. On the same basis, expect roughly **30 s** of extra delay before the first notification from a survivor, decaying to zero as the dead peer keys expire (derived from the 5-minute TTL and Grafana's peer-timeout default, not timed). ## Cross-Location Behavior Every instance shares one database, so there is no replication step between a write in one location and a read in another. Verified across three locations at `replicas: 2` — six UI instances plus the evaluator, each result attributed to a named individual instance: | What was tested | Result | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | Dashboard created in one location | Read back **byte-identical** from all six UI instances and the evaluator, including the internal id | | Edit made in a second location | Authoritative everywhere, including the location it was created in — writes flow in both directions | | User and org created in one location | Log in successfully against every other instance (a wrong password returns `401` as the control) | | Session cookie issued in one location | Accepted as the only credential by every other instance, returning the correct identity — **no session affinity is needed** | | Two replicas in the same location | Each sees the other's writes | | Datasource credentials | Decrypt and connect successfully on **every** instance, because all of them share `GF_SECURITY_SECRET_KEY` | Cross-region visibility was bounded at roughly 1–3 seconds by the measurement resolution rather than by the system. **The public endpoint is proximity-routed, not round-robin.** In a 100-request tabulation, all 100 requests from one client were served by a single location — the one nearest the client. Several locations give you regional redundancy and locally served traffic, not a request spread across regions. ## Connecting Substitute your release name and the name of the GVC you installed the release into. | What | Where | | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Grafana UI / API (public) | `status.canonicalEndpoint` of `{release}-grafana` — `cpln workload get RELEASE_NAME-grafana --gvc GVC_NAME -o yaml` | | Grafana UI / API (internal) | `RELEASE_NAME-grafana.GVC_NAME.cpln.local:3000` | | Alert evaluator (internal, own location only) | `RELEASE_NAME-grafana-alerting.GVC_NAME.cpln.local:3000` — dedicated-evaluator mode only | | App database (always the current primary) | `RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local:5432` | | Alerting Redis (internal only) | `replica-0.RELEASE_NAME-redis.LOCATION.GVC_NAME.cpln.local:6379` — alerting HA only | | Alerting Sentinel (internal only) | `replica-0.RELEASE_NAME-sentinel.LOCATION.GVC_NAME.cpln.local:26379`, master name `mymaster` — alerting HA only | | Admin login | `admin.user`, with the password in the secret named by `admin.passwordSecretName` — `cpln secret reveal SECRET_NAME -o json` | Health and readiness are served at `/api/health`, which reports the Grafana version and the app-database status. The [Grafana HTTP API](https://grafana.com/docs/grafana/latest/developers/http_api/) is available on the same endpoint for scripted dashboard, datasource and alert-rule management. An in-GVC request to `RELEASE_NAME-grafana.GVC_NAME.cpln.local:3000` is always served by an instance in the **caller's own** location. That is why a single write-then-read from one client proves nothing about cross-location state. ## Migrating From Version 1 Every release before 2.0.0 created its own GVC. **Do not `helm upgrade` a 1.x release onto 2.0.0.** Once the chart stops declaring a GVC, Helm prunes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it: both Grafana tiers, the whole Patroni cluster and its data volumes, the etcd store behind it, and the Redis coordination tier. Measured on a sibling template: everything was gone in about **six seconds**, while the command printed `upgraded successfully`. The chart refuses to render if your values still carry the 1.x `global.gvc` key, so a values-carrying upgrade fails safely before any resource is touched. That guard **cannot** fire on an upgrade run with no values file at all, because it then sees only 2.0.0's own defaults. The procedure below is the safety; the render guard is only a backstop. The refusal message usually names **`etcd-multi-location`** or **`postgres-multi-location`** rather than this chart. Helm renders the deepest subchart first and every chart in the stack carries the same guard on the same key, so a subchart's copy aborts the render first. The remedy is identical. Install 2.0.0 as a **new release against an existing GVC**, restore the database into it, then remove the old release. Use `postgresML.backup.mode: logical`, or run a `pg_dumpall` through the old release's `{release}-postgres-proxy` endpoint. Everything that makes up your Grafana — dashboards, users, orgs, datasources, alert rules and alert state — lives in that database, so this one dump carries all of it. Delete `global.gvc.name` and rename `global.gvc.locations` to a top-level `global.locations`. Every location you list must already exist in the GVC you are installing into. Not the GVC the 1.x release created — that GVC is still owned by the old Helm release and goes away when you uninstall it. Point `admin.secretKeySecretName` at the **same** secret the old release used. It decrypts the datasource credentials stored in the dump, and a different key makes every saved datasource credential unreadable. The [`postgres-multi-location`](/template-catalog/templates/postgres-multi-location#restoring-a-backup) page has the exact restore command; run it from a client workload in the same GVC. Uninstalling it takes the GVC it created, and the old volume sets, with it. Values that changed: | 1.x | 2.0.0 | | -------------------------- | ------------------------------------------------------------------------------ | | `global.gvc.name` | Removed — the GVC is wherever you install | | `global.gvc.locations` | `global.locations` | | `postgresML` pinned at 1.x | Pinned at **2.0.0**, which creates no GVC of its own | | `redisML` pinned at 2.x | Pinned at **3.0.0**, which creates no GVC of its own and adds `redisML.engine` | ## Availability and Planned Outages Measured on three locations (`aws-us-east-1`, `aws-eu-central-1`, `aws-us-west-2`). | Event | Measured impact | | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Cold install to all tiers ready, `replicas: 1` | **5 m 07 s** — measured at chart defaults with the primary in place. HAProxy and etcd first, then the Patroni cluster, then Grafana's 713 schema migrations (about 5 s with the primary local) | | Cold install at `replicas: 2` | About **5 m 30 s** for every replica to report ready — a shipped-code run at that setting measured 5 m 26 s with the primary in place | | Grafana-only change (for example `replicas: 1` → `3`) | **Zero downtime** — 1160 probe samples across four endpoints, no failures (measured on 1.0.0; the rollout is surge-based and unchanged since). **Does not apply to the first upgrade after an install**, which re-applies every tier once regardless of what changed | | Cold install with alerting HA on, `replicas: 2` | **4 m 22 s**, **4 m 44 s** and **6 m 11 s** in three separate runs, with the Redis and Sentinel tiers ready about a minute in | | Any `helm upgrade` that touches the database tier | The database reported unavailable for about **2 m 17 s**, and the Grafana tier took **5 m 1 s** to finish rolling every replica — measured on 1.1.1 with alerting HA on at `replicas: 2`. An earlier build whose primary bootstrapped outside `primaryLocation` was worse: about **4 minutes** to recover, one location out for **5–6 minutes**. Treat that as the worst case rather than the expected one | | Any `helm upgrade` with alerting HA on, including a no-op | **80–95 s** of duplicate alert notifications while the Redis tier is patched and the peers churn; never silence | | Loss of the evaluator's replica (HA off) | **21 s** without alert evaluation, then the replacement re-notifies whatever is firing; the UI tier is unaffected — measured on 1.0.0 before the database gate was extended, so treat it as a floor | **Treat every `helm upgrade` as a planned outage rather than a rolling one.** The bundled database members do not restart one at a time — the field that would serialize the rollout is not retained by the platform, so they go down together. Grafana's readiness probe is `/api/health`, which reports the database, so while the database is down the whole Grafana tier drops out of the load balancer and returns `503`. Changes confined to the Grafana workloads do not have this cost. **An upgrade that adds a new secret reference can pause the rollout for about 9–10 minutes while `helm upgrade` reports success.** Affected locations show `The identity ... is not allowed to reveal the secret ...` even though the grant is already in place and visible in `cpln secret access-report` — there is nothing to fix in the policy. It **clears itself with no action** (measured 9 m 0 s – 9 m 30 s), and no Helm action is required. Any change that introduces a secret reference can trigger it — for example setting `admin.applyPassword` back to `true`, enabling `smtp` with a password, adding `datasources.credentialSecrets`, or turning on `postgresML.backup` in `wal-g` mode, which wedged one location of the database tier for about 10 minutes in that subchart's own testing; because the wedged location held the primary on the pre-backup spec, **nothing was archived** for that window. It also means Helm reporting success is not evidence the new version is running — check `cpln workload get-deployments`. **A `Failed to lock database` line during a cold start is expected, and it clears itself.** Grafana runs its schema migrations under a non-blocking lock with no retry, so an instance that arrives while another holds the lock exits; the platform restarts it and the next attempt succeeds against the already-migrated schema. The chart staggers the UI instances so they do not arrive together, which removes these restarts at the default one replica per location. Two replicas in the **same** location still start together, so a few remain at `replicas: 2` — two three-location runs at that setting measured **2 and 3** container restarts, most of them this lock race. They are far more numerous when the database primary does not land in `postgresML.primaryLocation`, because the migrations then run cross-region and instances keep restarting while they finish — **15** at `replicas: 1`, against **0** at that same setting with the primary in place. Nothing needs doing either way. ## Backing Up Backups are disabled by default and cover the app database — the dashboards, users, alert rules and saved datasources that make up your Grafana. Enable them with `postgresML.backup.enabled: true`, choose `logical` (a scheduled `pg_dumpall` cron workload in the one location named by `postgresML.backup.location`) or `wal-g` (continuous archiving from whichever member is currently the primary), and complete the storage setup for your provider **before** installing. Backups come from the `postgres-multi-location` subchart and were not exercised in this template's own testing; the results below are from that template's test runs. **AWS S3** and **MinIO / S3-compatible** were exercised in both `logical` and `wal-g` modes at the shipped settings. **Google Cloud Storage** works but is memory-sensitive: it failed at `postgresML.backup.resources.memory: 128Mi` and passed at 256Mi (logical) and 512Mi (wal-g) — this template ships **512Mi**, at or above the values proven for both modes, which is why the values comment warns against lowering it. Both restores have been verified end to end there: a **wal-g restore** (base backup plus WAL replay into an empty data directory, checksum-identical to source) and a **logical restore** (`pg_dumpall` into a clean cluster, 0 errors, roles and sequences preserved). Rehearse your restore procedure before you rely on it. Create an S3 bucket. Set `postgresML.backup.aws.bucket` and `postgresML.backup.aws.region` to match. If you do not have one, [create a Cloud Account](/guides/create-cloud-account) for the AWS account holding the bucket. Set `postgresML.backup.aws.cloudAccountName` to its name. Create an IAM policy with the JSON below (replace `YOUR_BUCKET_NAME`), then set `postgresML.backup.aws.policyName` to the policy's name. This bucket-scoped policy is all the identity needs — no broad managed policy is required. ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetObjectVersion", "s3:DeleteObjectVersion" ], "Resource": [ "arn:aws:s3:::YOUR_BUCKET_NAME", "arn:aws:s3:::YOUR_BUCKET_NAME/*" ] } ] } ``` Set `postgresML.backup.aws.prefix` to the folder path within the bucket. Create a GCS bucket. Set `postgresML.backup.gcp.bucket` to its name. If you do not have one, [create a Cloud Account](/guides/create-cloud-account) for the GCP project. Set `postgresML.backup.gcp.cloudAccountName` to its name. Add the **Storage Admin** (`roles/storage.admin`) role to the service account associated with the Cloud Account. The chart additionally binds `roles/storage.objectAdmin` on exactly the bucket named in `postgresML.backup.gcp.bucket`. Set `postgresML.backup.gcp.prefix` to the folder path. Create the bucket on the server. Set `postgresML.backup.minio.bucket` to its name. No Cloud Account is needed. Set `postgresML.backup.minio.endpoint` to the S3 API address including the port. For the [MinIO](/template-catalog/templates/minio) template in the same GVC, that is `http://WORKLOAD_NAME:9000`. **The endpoint must be reachable from every location.** In `wal-g` mode every member runs `restore_command`, so a MinIO workload that runs in only one location leaves the members in the other locations in a permanent restart loop — silently, because the leader stays healthy and writes keep succeeding. Run your S3-compatible endpoint in every location of the GVC, or use S3/GCS, which are global. Create a [dictionary secret](/guides/create-secret/dictionary) and set `postgresML.backup.minio.credentialsSecretName` to its name, then set `postgresML.backup.minio.prefix` to the folder path: ```bash theme={null} cpln secret create-dictionary --name my-grafana-minio-credentials \ --entry accessKey=MINIO_ACCESS_KEY \ --entry secretKey=MINIO_SECRET_KEY ``` ## Important Notes * **Create the admin password, encryption key and database credentials secrets before installing.** The chart creates none of them; without them the deployment waits on secrets that do not exist while `helm install` reports success. * **There is no upgrade path from 1.x.** Every 1.x release created its own GVC; 2.0.0 deploys into an existing one, and a `helm upgrade` across that boundary deletes the old GVC and everything in it, database volumes included. The chart refuses to render on the 1.x `global.gvc` key, but that guard cannot see an upgrade run with no values at all. Follow [Migrating From Version 1](#migrating-from-version-1). * **Every location in `global.locations` must already exist in the GVC you install into.** A GVC location this release does not list runs nothing, which is harmless. A listed location the GVC lacks makes a fresh install crash-loop in the database tier with a named error, and — for a location removed from a GVC after initialization — silently runs nothing there. See [Matching the Location List to the GVC](#matching-the-location-list-to-the-gvc). * **Setting either subchart's firewall to `workload-list` requires listing the Grafana workloads in it.** The chart refuses to render otherwise and prints the exact links to add; this chart's own list adds its own workloads for you. * **Never rotate or delete the encryption key.** Every instance in every location decrypts stored datasource credentials with it; changing it makes them all unreadable and alert rules that query them fail. * **Every `helm upgrade` that touches the database tier is a planned outage.** The database went unavailable for about 2 m 17 s and the Grafana tier took about 5 minutes to finish rolling every replica. A build whose primary bootstrapped outside `primaryLocation` was worse — about 4 minutes to recover, one location 5–6 minutes — so treat that as the worst case. The first upgrade after any install costs this even for a Grafana-only change. Changes confined to the Grafana workloads roll with zero downtime. * **An upgrade that adds a secret reference can pause the rollout for about 9–10 minutes** while Helm reports success. It clears itself with no action, and the grant is already in place — there is nothing to fix in the policy. Enabling `smtp`, `datasources.credentialSecrets` or `postgresML.backup` are examples that trigger it. * **Alert evaluation stops if you lose `alerting.location`, and the UI will not show it.** Repoint the knob and upgrade; that restarts only the evaluator. Set `alerting.highAvailability.enabled: true` to remove that failure — read the query-load cost in [Alert Evaluation](#alert-evaluation) first. * **Silences must be created against the evaluator, from a workload in `alerting.location`** — its internal address returns `503` from every other location. With alerting HA on there is no evaluator and this does not apply. * **`alerting.location` is required while alerting is on and HA is off**, and it is ignored with HA on. It is no longer a disable switch: use `alerting.enabled: false` to stop rule evaluation entirely, which also fails at render if combined with `alerting.highAvailability.enabled: true`. * **Turning alerting HA on or off changes which workloads exist.** Enabling it deletes the `{release}-grafana-alerting` workload and adds a Redis and a Sentinel workload per location; disabling it does the reverse. It is a normal `helm upgrade`, but treat it as a planned change, not a toggle to flip during an incident. * **Alerting HA multiplies data-source query load by (locations × replicas)** — 3× at the defaults — because every instance evaluates every rule and Redis dedupes the notification, not the query. * **Alerting HA is blocked below 3 locations, on purpose.** Sentinel elects a master by a majority of locations, so at 2 locations losing either one leaves no quorum. * **With alerting HA on, an unhealthy Redis means duplicate notifications, never silence.** Check the `{release}-redis` and `{release}-sentinel` workloads before suspecting your alert rules. * **With alerting HA on, a `helm upgrade` costs 80–95 s of duplicate notifications**, even an upgrade that changes nothing, and post-outage sender hand-off is not instant — the Redis peer keys' 5-minute TTL was measured, but the hand-off itself is derived from it rather than timed against a real outage. * **Which location sends notifications is not pinnable with alerting HA on.** The sender is chosen by sorted peer name and moved between regions across two installs of the same chart, so do not route, filter or egress-allowlist on it. * **The alerting-HA Redis tier is unauthenticated by default**, reachable only from inside the GVC you install into — which since 2.0.0 is a GVC you own and may already share with other workloads. Authenticate it unless you control everything in that GVC: write access to it is enough to suppress alert notifications. See [Authenticating the Redis Tier](#authenticating-the-redis-tier). * **If you do set `redisML.redis.passwordSecretName` or `redisML.sentinel.passwordSecretName`, a wrong name stops the Grafana UI too**, because Grafana reads the same secret — and `helm install` still reports success. If a tier sits at 0 replicas after install, read `status.versions[].message` on the workload. * **Use the canonical `*.cpln.app` endpoint, not a per-location hostname.** Grafana is configured with a single absolute `root_url` (the canonical endpoint). On a per-location hostname the UI and dashboard layout load, but the POST that fetches panel data is rejected — you get a dashboard with empty panels and no error shown. The canonical endpoint is georouted and already serves from the nearest location. * **A provisioned datasource reporting `upstream connect error ... connection timeout` is almost always an unsubstituted placeholder in its URL**, not a network or firewall problem. Check the GVC segment of `datasources.definitions[].url` first — a name that does not resolve times out rather than failing fast, and nothing in the Grafana UI names the cause. * **The public endpoint is proximity-routed.** More locations means regional redundancy and locally served traffic, not requests spread across regions. * **Every location except the database primary's pays a cross-region round trip per query**, because Grafana has no read/write splitting. Set `postgresML.primaryLocation` where most of your users are. * **Scaling `replicas` has no alerting-related restriction** — it applies to the UI tier only, in every location including `alerting.location`. Watch the connection budget instead. * **`replicas: 2` or higher lengthens a cold install** to roughly five and a half minutes. Nothing is wrong; there is simply more to schedule. * **Grafana Live has no HA engine here**, so a live-streamed message reaches only the browsers connected to the same instance — carried over from the single-location [Grafana](/template-catalog/templates/grafana) template's multi-replica behaviour, and not exercised in this template's own testing. Dashboard auto-refresh, queries, alerting, provisioning, login and the API are unaffected. * **With `publicAccess.enabled: false`, links in alert notifications point at the internal GVC address** and will not open from a browser outside the GVC. * **Never suspend a location.** Suspending and resuming one permanently withdraws its endpoints from the other locations' service discovery while every status surface still reports healthy. To remove a location, take it out of `global.locations` **and** out of the GVC. * **Uninstall deletes every volume set the chart created** — the database, etcd and, with alerting HA on, Redis and Sentinel. The GVC is yours and is left alone. Enable backups if the data matters. Prerequisite secrets you created are not owned by the release and survive it. * **This template ships Grafana OSS only** — Enterprise features such as fine-grained RBAC, reporting and query caching are not available. ## External References Official Grafana documentation Alert rules, contact points, notification policies, and silences How Grafana coordinates alert evaluation across instances Reference for the datasource provisioning entries used by this template Every Grafana setting and its environment-variable name Manage dashboards, datasources, and alert rules programmatically View the source files, default values, and chart definition