Kubernetes Production Best Practices: Multi-AZ Reliability & Security

1. The High-Availability Baseline: Why Single-Zone & Default Configurations Fail

Deploying Kubernetes workloads using default configuration manifests is one of the most hazardous mistakes in cloud infrastructure. Out-of-the-box, Kubernetes is designed for flexibility, not enterprise resilience. By default, pods run with root privileges, have unlimited access to host memory and CPU, can initiate arbitrary outbound network calls, and are scheduled onto whatever node happens to have capacity—often clustering all replicas into a single availability zone.

When a cloud provider experiences an isolated availability zone power outage, network partition, or fiber cut, organizations with uncalibrated clusters suffer catastrophic total service downtime. Even routine maintenance events—such as Kubernetes control plane upgrades, node AMI security patches, or Karpenter spot node reclamations—can cause cascading 502 Bad Gateway outages if pods lack disruption budgets and graceful termination hooks.

To achieve true 99.99% availability on AWS EKS or GCP GKE, engineering teams must configure multi-AZ topology spread constraints, guaranteed Quality of Service (QoS) resource sizing, strict network segmentation, and automated canary rollouts.

2. Cluster Topology & Blast Radius: Multi-AZ Node Pools, Control Plane Isolation & Ingress

Enterprise Kubernetes clusters distribute computing nodes and stateful workloads across three separate physical Availability Zones within an AWS region (e.g. us-east-1a, us-east-1b, and us-east-1c). The following architectural blueprint illustrates the production multi-AZ topology:

`` [ External Internet Traffic / Route 53 DNS ] │ ▼ [ AWS Network Load Balancer (Cross-Zone Load Balancing Enabled) ] │ ├── Zone A: Private Subnet 1 ──► Ingress Controller Pod 1 ├── Zone B: Private Subnet 2 ──► Ingress Controller Pod 2 └── Zone C: Private Subnet 3 ──► Ingress Controller Pod 3 │ ┌─────────────────────────────┼─────────────────────────────┐ ▼ ▼ ▼ [ Worker Node Zone A ] [ Worker Node Zone B ] [ Worker Node Zone C ] ├── Pod Replica 1 ├── Pod Replica 2 ├── Pod Replica 3 ├── Cilium eBPF Agent ├── Cilium eBPF Agent ├── Cilium eBPF Agent └── Node Exporter └── Node Exporter └── Node Exporter │ │ │ └─────────────────────────────┴─────────────────────────────┘ │ (TopologySpreadConstraint: maxSkew 1) ▼ [ High-Availability Workload Quorum Maintained At All Times ] ``

Key topological pillars:

3. Production Workload Manifest: TopologySpreadConstraints, PDB & Resource Guarantees

The following production deployment manifest illustrates the battle-tested configuration required for high-concurrency microservices, featuring guaranteed QoS, TopologySpreadConstraints, readiness probes, and rootless container security:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: core-api-service
  namespace: production
  labels:
    app.kubernetes.io/name: core-api-service
    app.kubernetes.io/part-of: technofreaks-platform
spec:
  replicas: 6
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0 # Zero downtime during rolling updates
  selector:
    matchLabels:
      app: core-api-service
  template:
    metadata:
      labels:
        app: core-api-service
    spec:
      # Spread replicas evenly across AWS Availability Zones
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: core-api-service
      # Enforce strict container security (PSS Restricted)
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: application
          image: 111122223333.dkr.ecr.us-east-1.amazonaws.com/core-api:v2.4.1
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          ports:
            - containerPort: 8080
              name: http
          # Guaranteed QoS: Memory requests EQUAL limits to prevent OOM eviction
          resources:
            requests:
              cpu: "1000m"
              memory: "2Gi"
            limits:
              memory: "2Gi"
              # CPU limit omitted to prevent CFS kernel throttling
          # Graceful termination hook to drain in-flight connections
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]
          startupProbe:
            httpGet:
              path: /healthz/startup
              port: 8080
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz/live
              port: 8080
            periodSeconds: 10
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: core-api-service-pdb
  namespace: production
spec:
  minAvailable: 75%
  selector:
    matchLabels:
      app: core-api-service

4. Zero-Trust Network Policies & Pod Security Standards (PSS Restricted)

In a standard Kubernetes cluster, any pod can communicate with any other pod across namespaces. If an attacker compromises a public-facing frontend pod via an unpatched vulnerability, they can initiate internal lateral movement to query internal Redis caches, sensitive metadata endpoints (169.254.169.254), or relational databases.

Production clusters enforce Default-Deny Kubernetes NetworkPolicies powered by Cilium eBPF or Calico:

In addition, namespace labels enforce Pod Security Standards (PSS) Restricted mode, automatically rejecting any manifest that attempts to mount host paths, run as root, or escalate Linux privileges.

5. Graceful Termination, PreStop Hooks & Zero-Downtime Rolling Deploys

During a rolling deployment, Kubernetes terminates old pods while starting new ones. However, there is an inherent race condition in how Kubernetes distributes termination events:

When a pod is scheduled for termination, two asynchronous events happen simultaneously: the container runtime sends a SIGTERM signal to the application process, and the kube-proxy / ingress controller updates iptables/eBPF endpoints to remove the pod from service load balancing. Because endpoint propagation across worker nodes takes 1 to 5 seconds, incoming client requests are still routed to the old pod after it has received SIGTERM, resulting in abrupt connection resets and HTTP 502 errors.

The solution is mandatory inclusion of a `preStop` sleep hook: sleep 10. The hook forces the application container to pause for 10 seconds before passing the SIGTERM signal to the application code. This gives ingress controllers and load balancers ample time to drain the endpoint from routing tables, ensuring zero dropped in-flight customer transactions.

6. Common Kubernetes Anti-Patterns & Troubleshooting Runbook

Platform teams frequently encounter several severe operational anti-patterns:

7. Enterprise Kubernetes Production Readiness Checklist

Before promoting any application to production Kubernetes clusters, verify these five requirements:

1. Topology Spread Constraints: Confirm topologySpreadConstraints with maxSkew: 1 is configured across all Availability Zones.

2. PodDisruptionBudget: Verify a PDB enforcing at least minAvailable: 66% exists for every Deployment.

3. Guaranteed QoS Memory: Confirm memory requests equal memory limits to prevent sudden kernel OOM kills.

4. PreStop Grace Period: Ensure container lifecycle includes a sleep 10 preStop hook and appropriate termination grace period.

5. Non-Root User Enforcement: Verify securityContext.runAsNonRoot: true and read-only root filesystems are enforced.