Cluster Management

Pneuma provisions one GKE cluster per zone for every team whose Logos entry lists platform_managed_project.kubernetes_engine.locations, requested through the Nomos Agent. Each cluster has a regional control plane (HA across three zones) with nodes pinned to a single zone — keeping Istio's locality-aware routing within the zone and preventing cross-zone scheduling hot spots.

Clusters are named {team}-{zone} (e.g., pt-pneuma-us-east1-b) and filtered per environment: sandbox and non-production deploy two zones (one per region), production deploys all six.

Namespace Provisioning

Teams request namespaces through the Nomos Agent, which records them as platform_managed_project.kubernetes_engine.namespaces in the team's Logos entry. Pneuma's onboarding workspace provisions those namespaces and for each one:

• creates a dedicated Workload Identity service account, and
• binds the team's GitHub Actions service account (from Corpus) as a namespace-scoped namespace-admin via RBAC.

This gives each team self-serve deployment into its own namespaces without access to other teams' namespaces or the cluster itself.

Architecture Decision Records

This page includes Architecture Decision Records documenting the key design decisions.

Components

Component	Description
gke-cluster	Regional GKE cluster with nodes pinned to a single zone. KMS-encrypted etcd, Workload Identity, CIS GKE Benchmark hardened, Fleet-enrolled.
node-pool	Managed node pool with auto-provisioning, auto-repair, and auto-upgrade
fleet	GKE Fleet registration for multi-cluster service discovery and ingress
workload-identity	Kubernetes-to-GCP service account mapping — pods authenticate to GCP without keys

The zonal Authentik and agentgateway workspaces look up only their selected Pneuma cluster and use a single provider connection per zone. Other teams' clusters are not prerequisites for those workspaces; shared add-ons retain multi-cluster deployment.

Sandbox Teardown

The manually dispatched Sandbox Destroy workflow tears down all team clusters in both sandbox zones, including pt-pneuma-us-east1-b; it is not a selective cluster pause. It refreshes main state without applying infrastructure changes, then removes agentgateway parameters before its runtime and Authentik configuration before its routes and server releases. Diagnostics are removed before Istio manifests, manifests before their controllers, and all workloads before onboarding namespaces and GKE clusters.

The workflow retains the main workspace and Corpus-managed infrastructure, including the Authentik Cloud SQL instance, so it does not eliminate all sandbox costs. Authentik configuration teardown removes managed identity-provider configuration from that retained database. Every apply retains its existing environment approval; never run sandbox deployment concurrently with teardown.

Core Invariants

• etcd is KMS-encrypted at rest — database_encryption with state = "ENCRYPTED" is hardcoded in the GKE module.
• Workload Identity is enabled on every node pool — no static credentials for pod-level GCP access.
• Shielded nodes with Secure Boot and integrity monitoring are enforced on every node — no unverified boot path.
• Client certificate authentication is permanently disabled — issue_client_certificate = false is hardcoded.
• Dataplane V2 (eBPF) is hardcoded as the network datapath — no legacy kube-proxy on any cluster.

Architecture Decision Records

Per-Zone Clusters with Regional Control Planes

Status	Date	Deciders
Accepted ✅	April 2026	Pneuma

Context and Problem Statement

Teams need dedicated clusters with independent lifecycles. A shared cluster couples every tenant to the slowest consumer's upgrade cadence and makes one team's misconfiguration everyone's problem.

The cluster topology must also work with Istio's locality-aware routing — spreading nodes across all zones causes cross-zone traffic by default.

Decision

Teams request cluster locations through the Nomos Agent, recorded under platform_managed_project.kubernetes_engine.locations in Logos. Corpus provisions the GCP project; Pneuma provisions clusters for every declared location. Adding a cluster requires only a Nomos request.

Regional control plane with single-zone node placement. Each cluster has a regional control plane (HA across three zones) but nodes pinned to one zone via node_locations. This keeps Istio routing zone-local by default and avoids cross-zone scheduling hot spots. Clusters scale by adding zones; each zone is independently upgradeable and recoverable.

Five add-on layers. The workload runtime is decomposed into independently deployed layers:

Layer	Tool	Concern
Cluster Management	GKE	Compute, networking, Workload Identity, Fleet enrollment
Service Mesh	Istio	mTLS, traffic management, ingress
Certificate Management	cert-manager	Istio CA, mTLS PKI via istio-csr
Policy Enforcement	OPA Gatekeeper	Admission control and audit
Observability	Datadog Operator	Metrics, logs, and traces

Each layer maps to a subdirectory workspace in pt-pneuma, deployed in order via GitHub Actions needs dependencies.

Alternatives Considered

• One shared cluster per environment — Rejected. Single point of failure; blast radius is the entire environment; cannot scale horizontally; creates cross-team upgrade coordination overhead.
• Bundle all add-ons into one deployment step — Rejected. CRD ordering constraints (e.g., cert-manager CRDs before Istio certificate resources) require explicit sequencing. Separate workspaces make the order visible and enforceable.

Consequences

• Zone failure is contained — other clusters continue serving
• Add-on upgrades apply per cluster without touching cluster infrastructure
• Teams add or change clusters via Logos only — no pt-pneuma changes required
• Adding a zone requires claiming a CIDR slot from the Corpus IPAM plan

Centralized Namespace Provisioning with Per-Team RBAC

Status	Date	Deciders
Accepted ✅	July 2026	Pneuma

Context and Problem Statement

Namespace provisioning needs a single source of truth. Letting each team manage namespaces from its own repo would fragment cluster credentials across many pipelines and make cluster-wide invariants (naming, Workload Identity, ambient mesh enrollment) impossible to enforce.

At the same time, each team needs to deploy workloads from its own CI/CD pipeline without accessing other teams' namespaces or the cluster itself.

Decision

Pneuma centralizes all namespace provisioning in a single pipeline via per-cluster provider for_each. Teams request namespaces through Nomos; Pneuma creates them and grants each team scoped access through RBAC:

• A dedicated Workload Identity service account per namespace
• The team's GitHub Actions service account (from Corpus) bound as a namespace-scoped namespace-admin

Teams deploy into their own namespaces from their own pipelines. No cross-team access is possible.

Alternatives Considered

• Each team provisions namespaces from its own repo — Rejected. Fragments cluster credentials across many pipelines and prevents centrally enforced invariants.
• Grant each team cluster-admin — Rejected. Violates inter-team trust boundary; any team could access every other team's namespaces.

Consequences

• Cluster-wide invariants (naming, Workload Identity, ambient mesh policy) are enforced in one place
• Each team can self-serve deployments into its own namespaces only
• Every namespace gets a Workload Identity service account; runtime GCP bindings are requested separately via the Nomos Agent
• The cluster set must reach zone workspaces as TF_VAR_clusters because provider for_each cannot read module outputs