Airflow

​

Overview

​

What Gets Created

• Airflow Webserver — (RELEASE_NAME-airflow-webserver): a stateful, single-replica workload running the API server, scheduler, dag-processor and triggerer in one container, serving the UI and REST API on port 8080. It is the only tier with external inbound access, and the only one that runs the boot-time GVC checks.
• Celery Workers — (RELEASE_NAME-airflow-celery-worker): a stateful workload executing DAG tasks, at a fixed airflow.celeryWorker.replicas or scaled by KEDA on the Redis queue length.
• Redis — (RELEASE_NAME-airflow-redis): the Celery broker, on its own persistent volume set.
• PostgreSQL — (RELEASE_NAME-airflow-postgres): the Airflow metadata database, on its own persistent volume set.
• Airflow Volume Set — (RELEASE_NAME-airflow-vs): a shared-filesystem volume mounted as the Airflow home by the webserver and every worker, so they all see the same DAGs and logs.
• Config Secret — (RELEASE_NAME-airflow-config): a dictionary secret holding the metadata database username and password. It carries no key material.
• Webserver Startup Secret — (RELEASE_NAME-airflow-webserver-startup): an opaque secret holding the webserver’s startup script, which runs the topology checks before Airflow’s own boot sequence.
• Identity & Two Policies — An identity bound to the workloads, with reveal on exactly the secrets they mount (the database config secret, the startup script, your auth secret, and your git token secret when git-sync uses one), plus view on the one GVC you install into so the webserver can check its own topology at boot.
• git-sync Sidecar (optional) — Polls a Git repository and syncs DAGs onto the shared volume.

​

Migrating from 1.x

Error: execution error at (airflow/templates/workload-webserver.yaml:1:4): airflow 2.0.0: the `gvc`
values key was REMOVED. This chart no longer creates a GVC — it deploys into the GVC you install
into, and `gvc.locations` became the single top-level `location`. DO NOT `helm upgrade` a 1.x
release onto 2.0.0: the upgrade drops `kind: gvc` from the manifest and Helm deletes what a chart no
longer declares, which DESTROYS that GVC and every workload, volumeset and identity inside it —
including your Airflow metadata database and your DAGs.

cpln workload exec OLD_RELEASE-airflow-postgres --gvc OLD_GVC --container postgresql -- \
  pg_dump -U username -d airflow -Fc > airflow.dump

cpln workload exec NEW_RELEASE-airflow-postgres --gvc NEW_GVC --container postgresql --stdin -- \
  pg_restore -U username -d airflow --clean --if-exists < airflow.dump

cpln workload exec OLD_RELEASE-airflow-webserver --gvc OLD_GVC --container airflow -- \
  tar -cf - -C /opt/airflow/dags . > dags.tar

cpln workload exec NEW_RELEASE-airflow-webserver --gvc NEW_GVC --container airflow --stdin -- \
  tar -xf - -C /opt/airflow/dags < dags.tar

​

Earlier Breaking Changes

• The metadata database password cannot be changed in place either. POSTGRES_PASSWORD is read only when the data directory is initialized, so pointing postgres.config.password at a new value on an existing volume leaves the database on the old one and Airflow can no longer connect. Set it at first install.
• The web UI now ships closed. If you were relying on the old 0.0.0.0/0 default, put your own CIDRs in firewallConfig.inboundAllowCIDR.
• Two JWT knobs were removed because they did nothing. jwtExpirationDelta and jwtRefreshThreshold rendered AIRFLOW__API_AUTH__JWT_EXPIRATION_DELTA and AIRFLOW__API_AUTH__JWT_REFRESH_THRESHOLD, neither of which is an option in Airflow 3.x — tokens were issued with Airflow’s own 86400-second lifetime regardless of what those values said. The live knob is airflow.auth.jwtExpirationTime; there is no refresh-threshold equivalent.

​

Prerequisites

​

A GVC Containing the Location You Configure

cpln gvc get GVC_NAME -o json

cpln gvc add-location GVC_NAME --location aws-us-east-1

cpln workload get-deployments RELEASE_NAME-airflow-webserver --gvc GVC_NAME -o yaml

This workload location is deactivated because maxScale is set to 0.

​

One Secret Must Exist Before You Install

cpln secret create-dictionary --name my-airflow-auth \
  --entry jwtSecret="$(openssl rand -base64 48)" \
  --entry fernetKey="$(openssl rand -base64 32 | tr '+/' '-_')" \
  --entry adminPassword="$(openssl rand -base64 18)"

cpln secret reveal my-airflow-auth -o yaml

printf '%s' "MY_GIT_TOKEN" | cpln secret create-opaque --name my-airflow-git-token --encoding plain -f -

cpln workload get-deployments RELEASE_NAME-airflow-webserver --gvc GVC_NAME -o yaml

​

KEDA on the GVC

cpln gvc get MY_GVC -o yaml-slim > gvc.yaml

cpln apply -f gvc.yaml

cpln gvc get MY_GVC -o yaml | grep -A1 'keda:'

​

Installation

​

Configuration

# ─── Location ─────────────────────────────────────────────────────────────────
# This chart deploys into the GVC you install into — it does NOT create one.
# Airflow runs in exactly ONE location, and this names it. It MUST already be a
# location of that GVC; the webserver reads the GVC at boot and refuses a fresh
# install if it is not (see Important Notes in the README).
# Extra locations in the GVC are fine: nothing Airflow-related runs in them.
#
# One location is not a simplification. A `shared` volumeset provisions ONE
# VOLUME PER LOCATION, so the Airflow home — DAGs and task logs — written in one
# location is invisible in another, and the metadata database and Celery broker
# are each a single volume bound to a single replica.
location: aws-us-east-1

# ─── Postgres (Airflow metadata database) ─────────────────────────────────────
# Bundled plumbing: it serves Airflow only, is unreachable from outside the GVC,
# and nobody ever types this password into a client — so it stays a value.
postgres:
  image: postgres:18
  resources:
    minCpu: 250m
    maxCpu: 500m
    minMemory: 512Mi
    maxMemory: 1024Mi
  config:
    username: username
    password: change-me-airflow-db # set at FIRST install; see Migrating in the README
    database: airflow
  volumeset:
    capacity: 10 # initial capacity in GiB (minimum is 10)

# ─── Redis (Celery broker) ────────────────────────────────────────────────────
redis:
  image: redis:7.4
  resources:
    cpu: 250m
    memory: 512Mi
  volumeset:
    capacity: 10 # initial capacity in GiB (minimum is 10)

# ─── Apache Airflow ───────────────────────────────────────────────────────────
airflow:
  webserver:
    image: apache/airflow:3.0.3
    resources:
      cpu: 2000m
      memory: 3Gi
  celeryWorker:
    image: controlplanecorporation/celery:v1
    replicas: 1 # fixed worker count; ignored when keda.enabled is true
    resources:
      cpu: 256m
      memory: 512Mi
  webPort: 8080 # port serving the Airflow UI and REST API

  auth:
    # REQUIRED PREREQUISITE SECRET — CREATE IT BEFORE YOU INSTALL. If it does not
    # exist at install time the deployment WEDGES silently and `cpln logs` returns
    # nothing at all. A `dictionary` secret holding exactly three keys:
    #   jwtSecret     — signs API access tokens
    #   fernetKey     — encrypts the Connections and Variables your DAGs use
    #                   (database passwords, cloud keys, API tokens). It CANNOT be
    #                   rotated without making every stored one unreadable.
    #   adminPassword — the Airflow UI login for `admin.username` below
    # See Prerequisites in the README for the exact command.
    secretName: my-airflow-auth
    jwtExpirationTime: 86400 # API token lifetime in seconds (Airflow's own default)

  admin:
    username: admin # its password is `adminPassword` in the secret above

  scheduler:
    dagDirListInterval: 10 # how often to rescan the DAG folder (seconds)
    minFileProcessInterval: 10 # minimum interval between processing a DAG file (seconds)

  celery:
    workerConcurrency: 1 # tasks each worker runs concurrently

volumeset:
  airflow:
    capacity: 10 # shared Airflow home (DAGs + logs), GiB (minimum is 10)

# ─── Access ───────────────────────────────────────────────────────────────────
# The Airflow UI can trigger arbitrary code and decrypt every stored Connection,
# so it ships CLOSED to the internet. Reach it with `cpln port-forward`, or add
# your own CIDRs here (e.g. - 203.0.113.0/24) to expose it. A firewall change
# takes up to a couple of minutes to propagate.
firewallConfig:
  inboundAllowCIDR: []

# Who may reach Airflow's workloads from inside the org. Every workload this
# release creates is ALWAYS allowed, whatever is set here — the webserver and the
# workers have to reach the metadata database, the broker and each other.
internalAccess:
  type: same-gvc # options: same-gvc, same-org, workload-list
  workloads: [] # only with workload-list, e.g. - //gvc/GVC_NAME/workload/WORKLOAD_NAME

# ─── DAG delivery (git-sync) ──────────────────────────────────────────────────
gitSync:
  enabled: false
  repo: "" # git repository URL (e.g. https://github.com/YOUR_ORG/dags)
  branch: main
  period: 60s # how often to sync
  subPath: "" # optional subfolder within the repo containing DAGs
  auth:
    # OPTIONAL PREREQUISITE SECRET, private repos only. An `opaque` secret
    # (encoding `plain`) whose payload is a personal access token, e.g.
    # `my-airflow-git-token`. Empty = public repo, no credentials sent.
    secretName: ""

# ─── KEDA autoscaling (Celery workers) ────────────────────────────────────────
# OFF by default, because KEDA is a GVC-LEVEL setting this chart can no longer
# turn on for you: enabling it means editing the GVC you install into (see
# Prerequisites in the README). Leave it off and workers run at a fixed
# `airflow.celeryWorker.replicas`.
#
# Turning it on WITHOUT enabling KEDA on the GVC fails SILENTLY: the workload is
# accepted, never autoscales, and simply sits at `minScale` forever — at
# minScale 0 that is zero workers and every task queues with no error anywhere
# (measured 2026-08-28). The webserver checks the GVC at boot and says so.
keda:
  enabled: false
  minScale: 1 # minimum number of Celery workers; 0 = scale to zero when idle
  maxScale: 3 # maximum number of Celery workers
  scaleToZeroDelay: 300 # idle time before scaling to zero (seconds)
  listLength: 3 # Redis queue length that triggers a scale-up
  cooldownPeriod: 1 # cooldown between scaling events (seconds)
  initialCooldownPeriod: 1 # cooldown after startup before scaling begins (seconds)
  pollingInterval: 4 # how often KEDA queries Redis (seconds)

​

Location

• location — the single location name every workload in the release is pinned to. It must already be a location of the GVC you install into.

​

PostgreSQL Metadata Database

• postgres.image — The PostgreSQL image.
• postgres.resources — Reservation and limit for the database container (minCpu / maxCpu / minMemory / maxMemory).
• postgres.config.username / postgres.config.password / postgres.config.database — The metadata database credentials. This is bundled plumbing: it serves Airflow only, is unreachable from outside the GVC, and no human ever types it, so it stays a value. Set the password at first install — the shipped change-me-airflow-db is a published placeholder, and PostgreSQL only reads it when the data directory is initialized.
• postgres.volumeset.capacity — Database volume size in GiB (minimum 10).

​

Redis Broker

• redis.image — The Redis image.
• redis.resources — CPU and memory limit for the broker (this block exposes only a limit).
• redis.volumeset.capacity — Broker volume size in GiB (minimum 10).

​

Airflow

• airflow.webserver.image / airflow.celeryWorker.image — Images for the webserver and the Celery workers.
• airflow.webserver.resources / airflow.celeryWorker.resources — CPU and memory limits for each tier.
• airflow.celeryWorker.replicas — The fixed number of Celery workers. Ignored when keda.enabled is true, which hands the replica count to KEDA instead.
• airflow.webPort — The port serving the UI and the REST API.
• airflow.auth.secretName — Name of the prerequisite dictionary secret holding jwtSecret, fernetKey and adminPassword. It must exist before you install.
• airflow.auth.jwtExpirationTime — API access-token lifetime in seconds, applied as AIRFLOW__API_AUTH__JWT_EXPIRATION_TIME. The shipped 86400 is Airflow’s own default.
• airflow.admin.username — The admin login paired with the adminPassword key of your auth secret.
• airflow.scheduler.dagDirListInterval — How often the DAG folder is rescanned, in seconds.
• airflow.scheduler.minFileProcessInterval — Minimum interval between processing the same DAG file, in seconds.
• airflow.celery.workerConcurrency — Tasks each Celery worker runs concurrently.

​

Storage

• volumeset.airflow.capacity — Size in GiB of the shared Airflow home holding DAGs and logs (minimum 10).

​

Access

• firewallConfig.inboundAllowCIDR — CIDR ranges allowed to reach the webserver from the internet. The shipped [] means closed: the canonical endpoint returns 403 and the UI is reachable only over a port-forward. Add your own ranges (for example 203.0.113.0/24) to expose it.

​

DAG Delivery With git-sync

​

KEDA Autoscaling

[airflow] FATAL: keda.enabled is true in your values, but GVC 'my-gvc' has spec.keda.enabled = false.
KEDA is a GVC-LEVEL setting this chart cannot turn on for you.

​

Wide Fan Outs Can Silently Lose Tasks

​

Connecting to Airflow

​

First Run Sequence

cpln port-forward RELEASE_NAME-airflow-webserver 8080:8080 --gvc GVC_NAME

​

API Access

curl -X POST http://localhost:8080/auth/token \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "ADMIN_PASSWORD"}'

curl http://localhost:8080/api/v2/dags \
  -H "Authorization: Bearer ACCESS_TOKEN"

​

Important Notes

• The auth secret must exist before you install. Without it the deployment wedges with no log output at all; see Prerequisites for the one diagnostic that names it, and expect recovery to take up to about ten and a half minutes after you create it.
• The Fernet key can never be rotated — every stored Connection and Variable is encrypted under it. Generate it once, keep it, and back it up somewhere you will still have it later.
• A wide simultaneous fan-out silently loses roughly a third of its tasks — see Wide Fan Outs Can Silently Lose Tasks.
• Never upgrade a 1.x release onto 2.0.0 in place — it deletes the GVC the 1.x chart created and everything in it, including your metadata database and DAGs. Migrate to a new release: Migrating from 1.x.
• location must be a location of your GVC. If it is not, the install still succeeds and then nothing starts anywhere, with no container to produce a log line. Check cpln workload get-deployments for This workload location is deactivated because maxScale is set to 0. on every location.
• Airflow runs in a single location by construction — the shared Airflow home provisions one volume per location, so a second location would be a second, disconnected set of DAGs and logs.
• Enable KEDA on the GVC before setting keda.enabled: true. Getting the order wrong is accepted with no error and leaves the workers stuck at keda.minScale forever — zero workers, and silently queuing tasks, at minScale: 0.
• Set postgres.config.password at first install. It is bundled plumbing, but the shipped placeholder is published and the value cannot be changed once the data directory exists.
• The UI is closed by default and a firewall change takes up to a few minutes to apply, so an endpoint still returning 403 right after an upgrade is not necessarily broken.
• The first Helm upgrade after an install re-applies resources even with identical values, restarting the webserver and the database — the UI was unavailable for about three minutes in testing. Later upgrades are clean.
• Changing the admin password means editing the auth secret and restarting the webserver; the password file is rewritten from the secret on every container start.
• Uninstalling deletes all three volume sets, including the DAGs and logs on the shared Airflow home. Export anything you need first.

​

External References