Rolling upgrades — zero-downtime procedure

This is the supported path for upgrading ProxmoxVEx behind a load balancer without dropping the UI. It composes three primitives the codebase provides:

• SIGTERM drain (spec 092/US084): docker stop sends SIGTERM. The app marks itself draining — /api/healthz returns 503 immediately, so the orchestrator/LB pulls the node — waits for in-flight requests inside a 25s deadline, checkpoints running bulk jobs to interrupted, then exits.
• Container HEALTHCHECK + readiness probe (US083/US026): /api/healthz is the deep check (db, clusters, license, disk). Docker’s HEALTHCHECK uses the shallow /api/health liveness; the LB and this procedure should wait on /api/healthz.
• Boot-time schema convergence + pre-migration backup (US073/US074): every boot runs idempotent _init_db convergence — the migrate phase — and takes a pg_dump to config/backups/pre-migrate-*.dump first whenever the live schema differs from the stored fingerprint.

The sequence: drain → migrate → swap

Per node, one at a time (never in parallel — that would take the whole fleet down):

1. Drain — docker stop -t 45 proxmoxvex. SIGTERM starts the drain: healthz 503s, the LB marks the node down and routes around it, in-flight requests finish inside the 25s window. The stop timeout must exceed the drain window (45s covers it with slack).
2. Swap — docker compose pull the new image tag and up -d.
3. Migrate + verify — boot runs schema convergence (with a pre-migration backup taken automatically), then serves. Poll GET /api/healthz until 200 (up to ~4 min covers cold migrations). Only then touch the next node.

scripts/deploy-rolling.sh automates exactly this:

VEX_NODES="vex1 vex2" \
VEX_IMAGE=proxmoxvex:1.2.477 \
VEX_ROLLBACK_IMAGE=proxmoxvex:1.2.476 \
scripts/deploy-rolling.sh

Env knobs: VEX_CONTAINER (default proxmoxvex), VEX_COMPOSE_DIR, VEX_HEALTH_PORT (5000), VEX_HEALTH_TIMEOUT (240s), VEX_STOP_TIMEOUT (45s).

Rollback

If a node fails to return healthy within VEX_HEALTH_TIMEOUT, the script repins VEX_ROLLBACK_IMAGE on that node and aborts — remaining nodes are left untouched, so the fleet keeps serving on the old version while you investigate. Because convergence is additive (CREATE IF NOT EXISTS / guarded ADD COLUMN), a rollback to the previous image is safe on the already-migrated schema; the pre-migration pg_dump in config/backups/ covers the destructive-change case (see scripts/backup.sh / python -m ProxmoxVEx.cli restore).

Optional: two-instance HA

With PROXMOXVEX_HA=1 (spec 092/US088) both instances share a DB leader lock — only the leader dispatches scheduled jobs. During a staggered restart the standby holds the lock while the old leader drains, so even mid-upgrade there is never a window with two active writers.