TenantFlow

Overview

A Go control plane that treats "the worker died at 3 AM" as a normal input, not an edge case. The API only starts workflows; Temporal workers do the work as sagas of idempotent activities across PostgreSQL, Keycloak, quotas, and backups, so half-finished operations are compensated instead of left behind. Every tenant gets its own database and dedicated role with PUBLIC CONNECT revoked. Failed runs drain to a DLQ and resume with a single call, a reconcile workflow re-drives drifted tenants to their declared state, and runtime chaos injection proves the recovery paths end-to-end. A Next.js admin dashboard, Prometheus metrics with a per-tenant cost collector, and Grafana dashboards make it all observable.

Highlights

• Orchestrated provision, upgrade, migrate, backup, restore, delete, and reconcile as durable Temporal workflows that survive worker crashes mid-run
• Implemented saga compensations and a failed-runs DLQ whose one-call retry resumes a stuck deletion instead of blindly replaying it
• Isolated tenants with database-per-tenant, dedicated owner roles, and Keycloak OIDC with role-based access on every mutation
• Load-tested 100 tenants per scenario with 0 errors and 0 leftover databases, roles, or identities after teardown (p95 provision 1.26s shared, 2.75s dedicated)
• Proved recovery paths with runtime chaos injection, a documented failure matrix, and 8 ADRs recording the design decisions