Engineering design review: smtp-proxy scaling and durability hardening, proposing a two-tier Postfix architecture to fix single-replica throughput ceilings and a durable-mail-loss risk within Hetzner's per-node storage-volume limits.

Engineering design review — 2026-07-16

smtp-proxy: scaling & durability under inbound load

The shared inbound relay for every customer domain runs as one pod with no durable queue and no rate limits. It already throttled once. This proposes a two-tier Postfix split that fixes both — without adding a new stateful dependency, and without exceeding Hetzner's per-node volume ceiling.

#546 tracking issue #505 volume ceiling (closed, cited) #502 Stalwart (closed, declined) docs/superpowers/specs/2026-07-16-smtp-proxy-scaling-design.md

Four root causes, one incident already

Proactive hardening, not incident response — but CPUThrottlingHigh already fired once, on 2026-07-14, and the fix applied was a one-time vertical bump, not a structural one.

A

No burst headroom

replicas: 1, no HPA anywhere in infra/mail-proxy. Every connection for every customer domain funnels through one pod.

CPU 500m → 1000m, 2026-07-14 — vertical fix only
B

No connection or rate limiting

mynetworks = 0.0.0.0/0, smtpd_relay_restrictions = permit. A burst has no degrade-gracefully path — just CPU starvation. Same root cause as the open-relay posture.

no smtpd_client_connection_rate_limit set
C

No durable queue

Spool lives on the pod's writable layer. Any message between accept and durable commit is lost on crash — and that window scales with throughput, not just backend-outage duration.

no PVC mounted for the postfix spool
D

Coupled scaling

dovecot-imap (client-pull) shares a replica count with the SMTP relay (sender-push) for no technical reason.

4 containers, 1 Deployment, 1 replica count

One pod terminates everything

Traefik matches HostSNI(*) on ports 25 / 587 / 143 and routes all of it to a single ClusterIP-backed pod.

Today
Internet
TraefikHostSNI(*), :25/587/143
smtp-proxy pod — replicas: 1
postfixrelay-only, no PVC
transport-lookupdomain→namespace
postfix-exporter
dovecot-imapcoupled, apk-install-at-boot
mailserver×10 customer namespaces

Every customer's inbound mail and IMAP traffic depends on this one pod being up. A crash between accept and delivery loses whatever was in flight.

Two-tier Postfix: stateless front door, fixed-size durable backstop

Tier 1 scales freely and holds nothing. Tier 2 is small, fixed, and durable — reusing Postfix's own native retry/bounce logic instead of building new state machinery.

Proposed — P1 + P2
Traefik
Tier 1 — HPA, zero PVCs
postfix (front door)one delivery attempt
transport-lookup + postfix-exporterco-located, unchanged
mailserverbackend reachable → done
↑ scales N replicas — PVC count stays at zero
Tier 1
Tier 2 — queue/retryPVC-backed, fixed 1–2 pods
mailserverbackend unreachable → held & retried

250 OK is only sent after durable commit — to the backend, or to Tier 2. A Tier 1 crash before that point means the original sender's own MTA never got 250 and retries on its own.

Proposed — dovecot split
Traefik :143
imap-proxyown Deployment, independent scaling

No longer shares a replica count or a pod with the SMTP relay.

Hetzner CSI is RWO-only, 16 volumes per node

10of 16 used
k3s-applications-krd

Verified live, 2026-07-16. This node is already close to the ceiling from ordinary customer PVC growth (established by #505). A design that scales PVC count 1:1 with relay replicas competes directly with that headroom — which is why Tier 2's PVC count is fixed, not proportional to Tier 1's replica count.

Grilled and ruled out, on the record

OptionVerdictWhy
Two-tier Postfix chosen Reuses Postfix's own proven retry/backoff/bounce logic. Fixed PVC count regardless of Tier 1 replicas. No new stateful dependency.
N PVCs, one per replica rejected Scales PVC consumption 1:1 with replica count — directly collides with the 16/node ceiling and ordinary customer growth.
Redis / DB-backed queue rejected Achieves the same fixed-storage outcome, but means hand-rebuilding retry backoff and RFC 3464 bounce generation against a new dependency that doesn't exist in this cluster yet.
Stalwart central mail tier (#502) closed Bigger idea — retires per-customer volumes entirely. Explicitly declined: newer, not proven at scale, at the exact layer where failure means silently lost mail.
Dedicated ingress per dominant tenant deferred No evidence of a noisy-neighbor problem today. Named as an escape hatch, not built.

Each phase ships and verifies independently

P2 — scale-out

Go horizontal

P3 — observability

Trust the signal