Service Mesh

Istio runs on every GKE cluster as a single multi-cluster ambient mesh via GKE Fleet. It provides workload identity and mTLS via ztunnel, L4 policy enforcement in the data plane, centralized gateway authentication, and an ingress gateway backed by Cloud Armor WAF and Datadog AAP. Ingress uses the vendor-neutral Kubernetes Gateway API.

• Ambient data plane: ztunnel runs on each node and establishes identity-based mTLS and L4 policy enforcement without per-pod sidecars.
• Ingress gateway: External traffic to mesh workloads enters exclusively through pneuma's dedicated Gateway API Envoy data plane, backed by MCI global load balancer, Cloud Armor WAF, Datadog AAP, and Authentik external authorization. OpenBao is outside the mesh and has separate ingress endpoints.
• Routing and auth: Teams request routes and auth policy through the Nomos Agent, which records them in Logos; pneuma renders HTTPRoutes and gateway policies for namespaces enrolled in the ambient mesh.
• Waypoint-on-demand: Advanced L7 routing, traffic shaping, and richer observability require a waypoint; workloads stay ambient by default without always-on proxies.
• cert-manager: cert-manager-istio-csr signs workload identities requested by ztunnel for mTLS across the mesh.

Architecture Decision Records

This page includes Architecture Decision Records documenting the key design decisions.

Components

Component	Description
istio-control-plane	istiod deployed via Helm on every cluster — manages traffic policy, service discovery, and certificate distribution for the mesh
ztunnel	Per-node ambient dataplane component that provides identity, mTLS, L4 policy enforcement, and telemetry for workloads in mesh-enabled namespaces
waypoint	Optional L7 proxy provisioned on demand when a namespace or service needs advanced routing, policy, or observability beyond ambient L4 functionality
gateway	Gateway API Gateway (gatewayClassName istio) on pneuma clusters only. The dedicated Envoy data plane terminates external traffic and is separate from ambient workload mode
gateway-auth	Istio RequestAuthentication and AuthorizationPolicy resources on the gateway data plane. See Gateway Authentication
shared-ingress policies	Shared Cloud Armor security policy (OWASP rules, rate limiting, adaptive DDoS) and SSL policy, both managed by the Pneuma main workspace
http-route	Gateway API HTTPRoute per team host, co-located with the backend Service in the team's namespace
destination-rule	Ambient traffic policy configuration for cross-cluster routing and service selection when a waypoint or explicit policy is needed
peer-authentication	Mesh-wide mTLS enforcement at the ambient dataplane boundary

Multi-Cluster Mesh

All GKE clusters join a GKE Fleet and form a single ambient Istio mesh. Fleet membership enables cross-cluster endpoint discovery: a route on a pneuma cluster can reach a pod on a member team cluster with no additional configuration.

Each cluster runs its own istiod control plane and has ztunnel deployed on the node pool. A control-plane failure on one cluster does not affect workloads on another, and workloads remain ambient-only unless a waypoint is explicitly provisioned.

Gateway and Member Cluster Roles

Role	Clusters	Responsibilities
Gateway	pt-pneuma-*	Shared ingress gateway, MCI global load balancer, Cloud Armor WAF, Datadog AAP, and HTTPRoute rendering for all teams
Member	Future team clusters	ztunnel ambient dataplane and optional waypoint proxies; no public gateway — receives traffic from the mesh

DNS and Ingress Routing

External DNS for mesh routes points to pneuma's gateway IPs:

Record	Target
{team}.{env}.osinfra.io	MCI global IP (anycast, lowest-latency zone)
{zone}.{team}.{env}.osinfra.io	Zonal load balancer in that zone
authentik.{env}.osinfra.io, agentgateway.{env}.osinfra.io	MCI global IP
{zone}.authentik.{env}.osinfra.io, {zone}.agentgateway.{env}.osinfra.io	Zonal load balancer in that zone

Traffic to a member team's host (for example {team}.sb.osinfra.io) enters pneuma's gateway, matches the HTTPRoute for that host, and forwards across the mesh to the team's cluster. Member clusters have no public IP.

The shared Authentik and agentgateway hosts are also available directly on each regional ingress. For example, non-production exposes authentik.nonprod.osinfra.io through MCI and us-east1-b.authentik.nonprod.osinfra.io through the us-east1-b regional ingress. Production omits the environment label (us-east1-b.authentik.osinfra.io). The same pattern applies to agentgateway.

Gateway Certificate

The MCI load balancer terminates TLS with Google-managed certificates named istio-gateway-mci-<hash>, one per set of gateway domains. Managed certificate domains are immutable and GCP refuses to delete a certificate the load balancer still references, so adding a team creates a new certificate and the old one must stay attached until the swap completes. Adding a domain therefore takes three pt-pneuma deployments, each one finishing in every environment before the next starts:

1. Retain and attach. Register the domain (for a team, its logos workspace in the helpers PR) and list it in istio_gateway_pending_teams (or istio_gateway_pending_hosts for hosts such as agentgateway) in locals.tofu, and reset istio_gateway_attach_retained_certificates to true (a previous rotation leaves it false). The main workspace creates the new certificate and keeps the previous one, and the regional workspaces attach both.
2. Detach. Once the new certificate is ACTIVE, stop attaching the retained certificate by setting istio_gateway_attach_retained_certificates = false in locals.tofu. The regional workspaces detach the old certificate while its resource stays in state.
3. Delete. After ingress has reconciled and the load balancer no longer references the old certificate, remove the pending entry. The next deployment deletes the old certificate. Doing this before step 2 fails with a GCP resource-in-use error.

The update-gateway-certificate skill in pt-pneuma walks through these steps, and the Nomos Agent reminds you to run it when onboarding a team with GKE clusters. New managed certificates can take up to an hour to provision; existing hosts keep serving from the old certificate meanwhile.

Shared Ingress Policies

The Pneuma main workspace owns the Cloud Armor security policy and the SSL policy, both named shared-ingress. These are shared ingress infrastructure, not resources owned by the Arche Istio module.

The Istio gateway and OpenBao use separate, service-specific BackendConfig resources that reference the shared Cloud Armor security policy. Their ingress FrontendConfig resources reference the shared SSL policy. Sharing these policies does not share backend configuration or route OpenBao traffic through Istio.

Cross-Cluster Routing

Gateway API HTTPRoute backendRefs must resolve to a local Service on the gateway cluster. For member teams, pneuma creates a selectorless stub Service (and its namespace) on the gateway cluster. Fleet endpoint discovery fills that stub's endpoints from the owning team's clusters only — no pods with that service name exist elsewhere — so traffic routes exclusively to the correct team without explicit DestinationRule subsets.

Each HTTPRoute lives in the same namespace as its backend Service, so no ReferenceGrant is required. The shared Gateway authorizes attachment from all namespaces (allowedRoutes.namespaces.from: All).

Logos-Declared Routes

Teams request routes (service, port, optional path) on a mesh-enabled namespace through the Nomos Agent, which records them in the Logos team spec; the consumer flow is documented in Expose and Protect a Route. Pneuma renders each declaration into an HTTPRoute with:

• hostnames derived from the team's authoritative DNS zone (dns_subdomain) — never team-supplied text
• backendRef pointing to the Service in the team's prefixed namespace on the gateway cluster
• no requirement for per-pod sidecar injection because the workload traffic is handled by the ambient ztunnel data plane

The shared Gateway carries a single catch-all HTTPS listener (no hostname filter, wildcard TLS cert). Route changes take effect on the next pneuma pipeline run.

For the route declared in the consumer example, Pneuma renders an HTTPRoute in the st-ethos-api namespace on the gateway cluster, serving ethos.osinfra.io/api:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-ethos
  namespace: st-ethos-api
spec:
  parentRefs:
    - name: gateway
      namespace: istio-ingress
  hostnames:
    - ethos.osinfra.io
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      backendRefs:
        - name: api-service
          port: 8080

What's served vs not:

Scenario	Result
ethos.osinfra.io/api from a declared route	✅ Served — pneuma binds the team's subdomain
us-east1-b.ethos.osinfra.io/api (zonal probe)	✅ Served — derived from the same subdomain
A team claiming other-team.osinfra.io	❌ Impossible — teams never supply a hostname
An HTTPRoute applied directly to a gateway cluster	❌ Teams have no RBAC there

Consumer troubleshooting steps are in Expose and Protect a Route.

Gateway Authentication

Auth enforcement happens at the gateway data plane, independent of workload ambient mode. See Gateway Authentication for rendering, request flow, and ownership, and Expose and Protect a Route for the declaration contract.

AgentGateway

AgentGateway is an internal application gateway for ordinary HTTP connectivity. It uses the shared Istio ingress gateway and Authentik browser authorization; it does not add a separate public ingress path.

client -> shared Istio Gateway -> Authentik browser authorization -> internal AgentGateway -> workload

Pneuma owns the AgentGateway integration. Its environment hostname (agentgateway.<env>.osinfra.io, with the environment label omitted in production) resolves to MCI; each gateway zone also has a direct regional hostname (<zone>.agentgateway.<env>.osinfra.io). Both hosts use the same route and Authentik browser authorization. Traffic protection and external exposure remain the responsibility of the shared Istio gateway and its authentication policies.

End-to-End Validation

The istio-test workspace deploys a lightweight metadata service into each team's prefixed istio-test namespace (pt-pneuma-istio-test, etc.). A validation script checks every global and zonal endpoint, confirming the returned cluster name matches the expected team and zone.

Core Invariants

• Ambient mTLS is enforced on every cluster via ztunnel and PeerAuthentication in strict mode — no plaintext traffic between workloads enrolled in the ambient mesh.
• The ingress gateway runs only on pneuma clusters — member clusters have no public endpoint.
• HTTPRoute hostnames are derived from the team's dns_subdomain — teams cannot serve traffic on another team's subdomain.
• Member namespace names carry the team-key prefix ({team_key}-{namespace}) — no cross-team endpoint aggregation in the mesh.
• Ambient-only mode is the default target state — a permanent mixed sidecar/ambient deployment and always-on waypoints are both rejected.

Architecture Decision Records

Ambient-only workload mesh with waypoint-on-demand

Status	Date	Deciders
Accepted ✅	September 2026	Pneuma

Context and Problem Statement

The platform needs a single workload mesh that is predictable, secure, and easy to operate across many GKE clusters. A permanent mix of sidecars and ambient data planes would create two lifecycle models for the same workloads, duplicate policy and telemetry semantics, and increase operational complexity during upgrades. Always-on waypoints would add unnecessary L7 proxy overhead for namespaces that do not need advanced routing or observability.

Decision

Pneuma standardizes on an ambient-only workload mesh. All mesh-enabled namespaces enroll their workloads into the ambient data plane via ztunnel; the gateway remains a dedicated Envoy data plane for ingress and auth, and waypoint proxies are created only when a service or namespace genuinely needs L7 policy, routing, or observability. The target state rejects both permanently mixed sidecar/ambient workloads and always-on waypoint usage.

Alternatives Considered

• Permanent mixed mode — Rejected. It keeps two parallel workload data planes, creates conflicting telemetry and policy expectations, and complicates upgrades.
• Always-on waypoints for every namespace — Rejected. It adds unnecessary proxy overhead and turns a lightweight ambient mesh into a sidecar-like deployment model for every workload.
• A separate team-owned ingress mesh — Rejected. It fragments platform ownership, duplicates policy, and makes cross-cluster routing more complex.

Consequences

• Workloads remain lightweight and consistent across all clusters.

• Advanced L7 policy and telemetry are introduced only when needed through explicit waypoint deployment.

• Gateway auth remains independent of ambient workload mode, keeping ingress enforcement centralized and predictable.

• Platform operators maintain a simpler upgrade path because the mesh has one primary data plane.

• TLS termination, WAF, and threat detection happen at a single controlled point for every team

• Adding a team requires only a Nomos request — no gateway or load balancer changes

• Pneuma gateway clusters are critical infrastructure — their availability determines reachability of all teams

Team-Prefixed Namespace Isolation in the Mesh

Status	Date	Deciders
Accepted ✅	July 2026	Pneuma

Context and Problem Statement

In a single Fleet mesh, cross-cluster endpoint discovery aggregates endpoints by service name and namespace. If two teams deploy a Service with the same name in the same namespace, fleet discovery merges their endpoints — traffic becomes non-deterministic.

Decision

All team namespaces are provisioned with a team-key prefix: {name} in the team spec becomes {team_key}-{name} in the cluster (e.g., pt-pneuma-istio-test). Pneuma is prefixed the same way — no platform-team exception.

For member teams, pneuma creates a selectorless stub Service on the gateway cluster in the prefixed namespace. Fleet discovery returns endpoints only from the owning team's clusters, achieving isolation through naming alone.

Alternatives Considered

• Explicit DestinationRule subsets with cluster labels — Rejected. Requires a DestinationRule for each team/service, each of which must stay in sync with topology changes.
• Locality failover rules — Rejected. Routes by proximity, not ownership — cannot guarantee traffic stays within a team's cluster.
• Separate gateways per team — Rejected. Contradicts single-gateway-owner decision.

Consequences

• Each team's namespace DNS is globally unique — no cross-team endpoint aggregation
• Namespace names in Logos specs remain unprefixed; the prefix is applied at provision time
• The istio-test validation provably reaches only the correct team's cluster

Kubernetes Gateway API over Native Istio Gateway

Status	Date	Deciders
Accepted ✅	July 2026	Pneuma

Context and Problem Statement

Istio provides two models for ingress: its native Gateway/VirtualService CRDs, and the vendor-neutral Kubernetes Gateway API (Gateway/HTTPRoute). The native model couples routing to Istio-specific resources, requires a separately managed Helm-deployed data plane, and gates all route changes behind pneuma PRs since teams cannot own route intent through VirtualService.

Decision

Use the Kubernetes Gateway API. Pneuma owns a shared Gateway (gatewayClassName istio); istiod reconciles it and auto-provisions the gateway-istio data plane. Routing uses HTTPRoute instead of VirtualService: each route is co-located with its backend Service and attaches to the shared Gateway via parentRefs. Cross-cluster backends resolve through selectorless stub Services with Fleet-filled endpoints.

Alternatives Considered

• Native Istio Gateway/VirtualService — Rejected. Vendor-specific API; all route changes gated behind pneuma PRs; requires a separately managed Helm data plane.
• Manual gateway data plane with Gateway API routing — Rejected. The Helm chart is not Gateway-API-aware; auto-provisioning by istiod is less code and idiomatic.

Consequences

• Routing is vendor-neutral; teams own route intent in Logos
• Data plane is Istio-owned, removing the Helm release
• Cross-cluster backends require selectorless stub Services on the gateway cluster

Links

• Kubernetes Gateway API
• Istio Kubernetes Gateway API support
• GKE Gateway API

IaC-Layer Subdomain Isolation (Single Catch-All Listener)

Status	Date	Deciders
Accepted ✅	July 2026	Pneuma

Context and Problem Statement

Each team must serve traffic only on its own <subdomain>.osinfra.io host. The Gateway-native approach — per-team HTTPS listeners with concrete hostnames — fails on GCP: the fronting global external ALB (GFE) does not forward client SNI to backends. Per-team listeners build only SNI-matched Envoy filter chains with no default chain, so every GFE connection hits NR filter_chain_not_found → universal 502.

Decision

Keep a single catch-all HTTPS listener (no hostname filter, wildcard cert, allowedRoutes.namespaces.from: All) so the GFE→Envoy hop always matches. Move subdomain isolation to the IaC layer:

• Teams request routes through Nomos (PR-reviewed); they never supply hostnames
• Pneuma derives HTTPRoute hostnames from the team's dns_subdomain
• Only the pneuma pipeline applies manifests to gateway clusters; teams have no RBAC there

Alternatives Considered

• Per-team SNI listeners — Rejected. GCP L7 ALB does not forward client SNI; returns universal 502.
• L4 TLS-passthrough load balancer — Rejected. Preserves SNI but drops L7 features (Cloud Armor WAF, URL maps) and requires public ACME certs. Revisit only if teams gain direct HTTPRoute apply rights.
• OPA Gatekeeper admission policy — Rejected. Adds Rego + webhook + second source of truth to replicate a guarantee the pipeline already provides.

Consequences

• Subdomain isolation is enforced in reviewed IaC and the deploy pipeline, not the data plane
• End-to-end TLS, L7 ALB features, and Gateway API are all retained
• Wildcard cert covers all team hosts; adding a team requires no per-team platform action
• Residual risk: a rendering bug or compromised pipeline could mis-bind a hostname — same trust surface as all pneuma-managed resources