Engineering Principles

A single reference mapping general distributed-systems, security, and delivery best practices onto what Codevertex actually does. Each principle below either points at one of the detailed pages in this section, or states the platform's actual current position in one or two lines. This page doesn't repeat the detail — read the linked page for the real code paths, defaults, and gotchas. If you're starting new infra work or reviewing someone else's, start here, then follow the links.

Reliability & resilience

• Circuit breakers, retries, timeouts — every outbound S2S call goes through shared/service-client (breaker + exponential backoff + tracing). See Resilience — Circuit Breakers & Retries.
• Idempotency, outbox, dead-letter-adjacent handling — domain writes and their events commit atomically via the transactional outbox; consumers claim-before-acting for irreversible side effects. See Idempotency & the Outbox Pattern.
• Health checks / liveness & readiness probes — every service's Helm chart (charts/app/templates/deployment.yaml) defines both; the control-plane's own kube-apiserver probe thresholds have separately been tuned for this cluster's single-node disk-contention profile (see the platform's incident history if you're touching probe timing on a control-plane component — it's not the same tuning problem as an app pod).
• PodDisruptionBudgets — every app service ships with one (minAvailable: 1 or equivalent), so a voluntary disruption (rollout, node drain) can't take every replica down at once. When adding a new stateful component (a database, a queue), give it a PDB too — app services get one by default via the shared chart, but hand-rolled StatefulSets outside that chart don't automatically inherit it.
• Backups & disaster recovery — each service owns and schedules its own tenant-scoped backup CronJob (devops-k8s/manifests/backup/); there is deliberately no platform-wide, all-tenants dump exposed through any UI. Recovery beyond a single-node failure (multi-region, cross-datacenter DR) is out of scope for the current architecture.
• Rollbacks — GitOps is the rollback mechanism: ArgoCD's self-heal reverts any manual cluster edit back to the committed manifest, so a bad change is reverted by reverting the commit (or the release-bot's tag-bump), never by hand-editing the live resource. A manual kubectl fix during an active incident is a valid stopgap, but treat it as temporary until the corresponding commit lands — self-heal will otherwise silently undo it.
• Race conditions / concurrent-replica hazards — a recurring class of bug on this platform is logic that's correct for one replica but wrong for N (an in-process WebSocket hub with no cross-pod relay, a scheduled job with no locking). The fix is either a Redis-backed relay (for hubs/caches that must be visible cluster-wide) or a Postgres advisory lock (for singleton jobs like inventory-api's end-of-life purge scheduler) — never "just don't scale that service past 1 replica" as a way to dodge the problem.

Scalability

• Connection pooling — every service connects through PgBouncer, never directly at Postgres. See Connection Pooling & PgBouncer.
• Horizontal autoscaling — every service ships with an HPA (CPU/memory targets, 2-replica HA floor as the platform standard even where the max is also 2). A VPA object also exists per service for right-sizing observation; KEDA (event-driven autoscaling, e.g. scaling on queue depth) is installed platform-wide as a capability but isn't yet driving any service's scaling decisions — reach for it if you're adding a queue-depth-sensitive workload rather than tuning CPU-based HPA thresholds to approximate the same thing.
• Read replicas — a handful of specific heavy, staleness-tolerant read endpoints route to a Postgres streaming replica via a dedicated PgBouncer alias, opt-in per endpoint with a fallback-to-primary default. See Read Replicas — this is the pattern to extend, not reinvent, if you find a report/list endpoint that's genuinely primary-load-heavy.
• Caching — see Caching.
• N+1 avoidance — see N+1 Queries & Preloading.
• Sharding, partitioning, leader election, CAP-theorem tradeoffs — not part of the current architecture: each domain runs a single-primary Postgres database (per-service, not sharded), appropriate at the platform's current scale. Where the platform does need one-active-worker semantics (schedulers, purge jobs), it uses a Postgres advisory lock rather than a dedicated leader-election system — see the race-conditions note above. Revisit sharding/partitioning only if a specific table's write volume, not just its row count, becomes the bottleneck — right-sizing via VPA and read-replica offload are the first tools to reach for.
• Backpressure / queueing — NATS JetStream (see Event Architecture) is the platform's durable queue; consumers use QueueSubscribe for multi-replica-safe, at-least-once delivery rather than a bespoke retry loop.

Data & consistency

• Database migrations — Ent + Atlas versioned migrations, never a hand-edited schema. See Go Backend — Ent + Atlas Migrations.
• Distributed transactions / sagas — the platform avoids classic two-phase-commit-style distributed transactions; cross-service consistency is achieved via the outbox-plus-event-consumption pattern (each service commits its own state atomically, and downstream effects happen through idempotent event consumers, not a coordinated multi-service transaction). See Idempotency & the Outbox Pattern and Event Architecture.
• Additive-first schema changes — when a fix or feature can be expressed as a value inside an entity's existing metadata (JSON) field rather than a new column, prefer that — it avoids a migration entirely for genuinely optional, non-relational data. Reach for a real migration once the field needs to be queried/indexed/validated at the DB level, not just stored.
• Service data ownership — each service owns its own domain's data outright; cross-service reads happen via S2S calls or event-driven projections, never a second service reading another's tables directly. See Cross-Service Data Ownership.

Security & IAM

• Secrets — platform secrets are provisioned as Kubernetes Secrets via a generate-or-preserve script; a subset are migrated to Sealed Secrets (in-cluster asymmetric encryption, so the plaintext never needs to touch a CI system or an engineer's shell after the initial seal) with the rest queued for migration in ascending blast-radius order. See Secrets Management.
• AuthN/AuthZ — SSO is centralized in auth-api (OIDC/OAuth against one shared identity provider config, never duplicated per-service), with JWKS kid-based key rotation, TOTP/WebAuthn MFA, and login-endpoint rate limiting. Authorization follows the Trinity Authorization Pattern (JWT claims ∪ service-level RBAC sync). Never build a second OAuth/OIDC integration in a downstream service — it's a pure SSO client.
• S2S auth — one shared INTERNAL_SERVICE_KEY, always with an explicit tenant header, never inferred from the key alone. See Service-to-Service (S2S) Conventions.
• Transport security — TLS everywhere via cert-manager (letsencrypt-prod/staging issuers); no service terminates its own certs by hand.
• Injection defense — every service's ORM (Ent) parameterizes query values by construction; the rare raw-SQL path (e.g. a dynamic table name in a cross-table cleanup job) should only ever interpolate a value drawn from a fixed, internal enumeration (never user input) and should still parameterize every actual data value ($1, $2, ...) even when the identifier itself is interpolated.
• Rate limiting as an abuse/security control, not just a scalability one — see Rate Limiting; this is also the platform's brute-force defense on auth-sensitive endpoints (e.g. login), not only a capacity-protection measure.
• Edge protection — public hostnames sit behind Cloudflare (TLS + IP preservation + proxying); a managed WAF ruleset is the platform's chosen first layer of request-level filtering ahead of any self-hosted WAF investment, kept in monitor-then-block rollout order to avoid false-positive blocking real traffic.
• Network-level isolation — services are separated by Kubernetes namespace; pod-to-pod network policy enforcement is being introduced incrementally, piloted on a single low-traffic service before wider rollout, rather than deployed fleet-wide in one pass given the risk of a missed allow-rule causing a live outage.

Delivery & operations

• CI/CD & GitOps — GitHub Actions builds and pushes each service's image on merge; ArgoCD (with self-heal) is the only path into the live cluster. image.tag in devops-k8s/apps/*/values.yaml is CI-managed — never hand-edited, and never built/pushed via a local docker build.
• Infrastructure as code — the shared charts/app Helm chart is the single deployment template every service's values.yaml configures; a hand-rolled manifest outside that chart is the exception, not the default.
• API versioning — every service exposes /api/v1/...; there's no tooling yet for introducing a breaking /v2 on a given service, so a breaking change today means coordinating client updates rather than dual-serving both versions.
• Structured logging & tracing — zap via shared/httpware fleet-wide, with request-ID propagation and OpenTelemetry spans on every S2S call. See Observability.
• API documentation — Swagger/OpenAPI generation follows a specific gating convention (which environments serve docs, and how). See Swagger/OpenAPI Docs Gating.

Explicitly not in scope today

Some items on any general best-practices checklist genuinely aren't part of Codevertex's current architecture, by choice or by scale: chaos engineering, formal SLO/SLI/error-budget tracking, multi-region/cross-datacenter disaster recovery, a general-purpose feature-flag system, and a metered-serverless cost model (the platform runs on fixed cluster capacity, not per-invocation billing, so classic "cold start" and "serverless limits" concerns don't apply). These aren't oversights to work around — they're simply not the shape of problem this platform has today. If one of these becomes relevant (e.g. the platform expands to a second region), it deserves its own dedicated design pass rather than a checklist-driven retrofit.