Troubleshooting Resource Shortages in EKS Clusters

Contents

Introduction

Amazon EKS handles container orchestration in cloud-native environments and is core infrastructure for application deployment. As clusters grow and workloads increase, however, various resource shortages can arise.

"What if a service that was working yesterday suddenly cannot create Pods?"

This post walks through the three resource shortages most frequently encountered in production environments and how to resolve them.

Each section covers the root cause, specific solutions, and preventive measures, in that order.

1. Resolving Pod IP Exhaustion

Situation and Symptoms

Actual error message:

Code
Warning  FailedCreatePodSandBox  pod/my-app-xxx  Failed to create pod sandbox: 
rpc error: code = Unknown desc = failed to set up sandbox container: 
networkPlugin cni failed to set up pod "my-app-xxx_default" network: 
add cmd: failed to assign an IP address to container

How to diagnose the problem:

bash
# 1. 노드별 IP 사용량 확인
kubectl get nodes -o custom-columns=NAME:.metadata.name,PODS:.status.capacity.pods,ALLOCATABLE:.status.allocatable.pods

# 2. VPC CNI 로그 확인
kubectl logs -n kube-system -l k8s-app=aws-node --tail=100

# 3. ENI 사용 현황 확인
aws ec2 describe-network-interfaces --filters "Name=description,Values=aws-K8S-*"

As the number of Pods in an EKS cluster increases, IP addresses run out because the default VPC CNI configuration limits the number of IPs each node can receive.

Solutions

1.1 Adjust WARM_IP_TARGET

First, let's look at what the key VPC CNI settings mean.

Key settings:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  # Pod 생성 시 IP 부족으로 인한 지연을 방지하기 위해 미리 확보할 IP 개수
  warm_ip_target: "10"  # 기본값: 1
  # 노드당 최소한 보장할 IP 개수 (갑작스런 트래픽 증가 대비)
  minimum_ip_target: "5"
  # 노드당 최대 ENI 개수 제한 (인스턴스 타입별로 상한 존재)
  max_eni: "4"
  # 새로운 ENI 할당 시 미리 준비해둘 개수
  warm_eni_target: "1"

Effects of the configuration:

1.2 Enable Prefix Delegation

What is Prefix Delegation?
The traditional approach assigns individual IP addresses to each ENI. Prefix Delegation assigns a /28 subnet block (16 IPs) at once, using IPs more efficiently.

Comparison:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  # Prefix Delegation 기능 활성화
  enable_prefix_delegation: "true"
  # 미리 할당받을 prefix 개수 (각 prefix = 16개 IP)
  warm_prefix_target: "1"
  # prefix 내에서 미리 확보할 IP 개수  
  warm_ip_target: "5"
  # prefix가 고갈되기 전 새 prefix를 요청할 임계점
  minimum_ip_target: "3"

Precautions:

1.3 Configure Custom Networking

Use this when security requirements or IP range separation call for it.

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  aws_vpc_k8s_cni_custom_network_cfg: "true"
yaml
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: pod-subnet-config
spec:
  subnet: subnet-xxxxxxxxx
  securityGroups:
    - sg-xxxxxxxxx

1.4 Initial Design Considerations

2. Insufficient Resource Limits/Requests

Situation and Symptoms

Actual error message:

Code
Warning  FailedScheduling  pod/my-app-xxx  0/3 nodes are available: 
3 Insufficient cpu, 2 Insufficient memory.

How to diagnose the problem:

bash
# 1. 클러스터 전체 리소스 현황
kubectl top nodes

# 2. 특정 노드의 상세 리소스 정보
kubectl describe node <node-name>

# 3. Pending Pod 상세 정보 확인
kubectl describe pod <pod-name>

# 4. 리소스 사용률이 높은 Pod 찾기
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top pods --all-namespaces --sort-by=memory

When growing Pod counts exhaust node CPU/memory resources, Pods remain Pending.

Kubernetes 1.33 In-Place Pod Resize (Beta)

Starting with Kubernetes 1.33, resources can be adjusted without restarting the Pod.

yaml
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: app
    image: nginx
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        cpu: "200m"
        memory: "256Mi"
    # 1.33에서 추가된 필드
    resizePolicy:
    - resourceName: cpu
      restartPolicy: NotRequired
    - resourceName: memory
      restartPolicy: NotRequired

Resource adjustment command:

bash
kubectl patch pod example-pod --patch='
{
  "spec": {
    "containers": [{
      "name": "app",
      "resources": {
        "requests": {"cpu": "200m", "memory": "256Mi"},
        "limits": {"cpu": "400m", "memory": "512Mi"}
      }
    }]
  }
}'

Solutions for Versions Before 1.33

2.1 Right-sizing with Kyverno

What is Kyverno?
Kyverno is a Kubernetes-native policy engine that manages cluster resources through policies defined in YAML. You can write policies using familiar Kubernetes syntax instead of a complex scripting language.

Key Kyverno features:

Why right-sizing is needed:
Many development teams set resource requests too high or omit them entirely, wasting cluster resources or causing performance issues. Kyverno lets you manage this through policies.

Resource requirement validation policy:

yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-pod-resources
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: validate-resources
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Resource requests and limits are required"
      pattern:
        spec:
          containers:
          - name: "*"
            resources:
              requests:
                memory: "?*"
                cpu: "?*"
              limits:
                memory: "?*"
                cpu: "?*"

Automatic resource allocation policy:

yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-resources
spec:
  background: true
  rules:
  - name: add-default-resources
    match:
      any:
      - resources:
          kinds:
          - Pod
    mutate:
      patchStrategicMerge:
        spec:
          containers:
          - (name): "*"
            resources:
              requests:
                +(memory): "128Mi"
                +(cpu): "100m"
              limits:
                +(memory): "256Mi"
                +(cpu): "200m"

Limiting excessive resource requests:

yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: limit-resources
spec:
  validationFailureAction: enforce
  rules:
  - name: check-cpu-limit
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "CPU limit cannot exceed 2 cores"
      pattern:
        spec:
          containers:
          - name: "*"
            resources:
              limits:
                cpu: "<=2000m"

2.2 Using Vertical Pod Autoscaler (VPA)

Install and configure VPA:

bash
# VPA 설치
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler/
./hack/vpa-install.sh

Configure a VPA policy:

yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Auto"  # Auto: 자동 적용, Off: 권장값만 제공, Initial: 최초 생성시만
  resourcePolicy:
    containerPolicies:
    - containerName: app
      maxAllowed:
        cpu: 1
        memory: 500Mi
      minAllowed:
        cpu: 100m
        memory: 50Mi
      # 리소스 조정 시 적용할 배율
      controlledResources: ["cpu", "memory"]

Scenario comparing before and after VPA:

Before:

yaml
# 개발자가 임의로 설정한 리소스 (과도하게 할당)
resources:
  requests:
    cpu: "1000m"      # 실제 사용량: 200m
    memory: "2Gi"     # 실제 사용량: 512Mi
  limits:
    cpu: "2000m"
    memory: "4Gi"

Check the VPA analysis:

bash
kubectl describe vpa my-app-vpa

Example VPA recommendations:

Code
Recommendation:
  Container Recommendations:
    Container Name:  app
    Lower Bound:
      Cpu:     150m
      Memory:  256Mi
    Target:
      Cpu:     250m    # 실제 사용량 기반 권장값
      Memory:  512Mi   # 실제 사용량 기반 권장값
    Upper Bound:
      Cpu:     400m
      Memory:  1Gi

After:

yaml
# VPA가 자동 조정한 리소스 (효율적 할당)
resources:
  requests:
    cpu: "250m"       # 75% 절약
    memory: "512Mi"   # 75% 절약
  limits:
    cpu: "400m"       # 80% 절약
    memory: "1Gi"     # 75% 절약

Effects:

3. Insufficient Spot Instance Capacity

Situation and Symptoms

Actual error message:

Code
# CloudWatch 이벤트 또는 EC2 콘솔에서 확인 가능
EC2 Auto Scaling group activity:
Status: Failed
StatusReason: We currently do not have sufficient m5.large capacity 
in the Availability Zone you requested (us-west-2a). 
Our system will be working on provisioning additional capacity.

# Karpenter 사용 시
Warning  FailedScheduling  pod/my-app-xxx  
no nodes available to schedule pod, 
insufficient capacity for instance type m5.large in zone us-west-2a

How to diagnose the problem:

bash
# 1. Auto Scaling Group 활동 히스토리 확인
aws autoscaling describe-scaling-activities --auto-scaling-group-name <asg-name>

# 2. Spot Instance 가격 및 가용성 확인
aws ec2 describe-spot-price-history --instance-types m5.large --availability-zone us-west-2a

# 3. Karpenter 노드 프로비저닝 로그 확인 (Karpenter 사용 시)
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter

# 4. 클러스터 오토스케일러 로그 확인 (CA 사용 시)
kubectl logs -n kube-system -l app=cluster-autoscaler

Main problem scenarios:

Solutions

3.1 Use Diverse Instance Types

Strategy:
To improve Spot Instance availability, combine different instance families and sizes. Relying on a single type affects the entire system when that type's capacity is insufficient.

Karpenter configuration (recommended):

yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: diversified-spot-nodepool
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        # 다양한 인스턴스 패밀리 포함
        - key: node.kubernetes.io/instance-type
          operator: In
          values: 
            # 범용 인스턴스 (M 시리즈)
            - "m5.large"
            - "m5.xlarge"
            - "m5a.large"
            - "m5a.xlarge"
            - "m5n.large"
            - "m5n.xlarge"
            # 컴퓨팅 최적화 (C 시리즈)
            - "c5.large"
            - "c5.xlarge"
            - "c5a.large"
            - "c5a.xlarge"
            - "c5n.large"
            - "c5n.xlarge"
            # 메모리 최적화 (R 시리즈)
            - "r5.large"
            - "r5.xlarge"
        # 다중 AZ 지원
        - key: topology.kubernetes.io/zone
          operator: In
          values: 
            - "us-west-2a"
            - "us-west-2b"
            - "us-west-2c"
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: default
  # 중단 정책 설정
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s
    expireAfter: 2160h # 90일

Auto Scaling Group configuration:

yaml
# ASG 사용 시 Launch Template 설정
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-status
  namespace: kube-system
data:
  nodes.max: "100"
  # 다양한 인스턴스 타입 지원을 위한 MixedInstancesPolicy 활용

3.2 Strategies for Spot Instance Interruptions

Handle Spot Instance interruption signals:

yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: spot-interrupt-handler
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: spot-interrupt-handler
  template:
    metadata:
      labels:
        app: spot-interrupt-handler
    spec:
      serviceAccount: spot-interrupt-handler
      hostNetwork: true
      containers:
      - name: spot-interrupt-handler
        image: amazon/aws-node-termination-handler:v1.19.0
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: SPOT_INTERRUPT_THRESHOLD
          value: "120" # 2분 전 미리 알림
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      tolerations:
      - operator: Exists

Configure a PodDisruptionBudget:

yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2  # 최소 2개 Pod는 항상 유지
  selector:
    matchLabels:
      app: my-app

3.3 Mixed Instance Policy (When Using an ASG)

json
{
  "InstancesDistribution": {
    "OnDemandAllocationStrategy": "prioritized",
    "OnDemandBaseCapacity": 2,
    "OnDemandPercentageAboveBaseCapacity": 20,
    "SpotAllocationStrategy": "diversified",
    "SpotInstancePools": 4,
    "SpotMaxPrice": "0.50"
  },
  "LaunchTemplate": {
    "LaunchTemplateSpecification": {
      "LaunchTemplateId": "lt-xxxxxxxxx",
      "Version": "$Latest"
    },
    "Overrides": [
      {
        "InstanceType": "m5.large",
        "WeightedCapacity": 1
      },
      {
        "InstanceType": "m5.xlarge",
        "WeightedCapacity": 2
      },
      {
        "InstanceType": "c5.large",
        "WeightedCapacity": 1
      }
    ]
  }
}

3.4 Real-time Monitoring and Alerts

CloudWatch-based Spot Instance monitoring:

yaml
# CloudWatch 알람 설정 (AWS CLI)
aws cloudwatch put-metric-alarm \
  --alarm-name "SpotInstanceTermination" \
  --alarm-description "Spot instance termination detected" \
  --metric-name "SpotInstanceTerminating" \
  --namespace "AWS/EC2Spot" \
  --statistic "Sum" \
  --period 300 \
  --threshold 1 \
  --comparison-operator "GreaterThanOrEqualToThreshold" \
  --evaluation-periods 1 \
  --alarm-actions "arn:aws:sns:us-west-2:123456789012:spot-alerts"

Prometheus + Grafana monitoring:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: spot-monitoring-rules
  namespace: monitoring
data:
  spot-rules.yml: |
    groups:
    - name: spot-instances
      rules:
      # Spot Instance 중단 예측
      - alert: SpotInstanceInterruptionWarning
        expr: aws_spot_instance_interruption_warning > 0
        for: 30s
        labels:
          severity: warning
        annotations:
          summary: "Spot instance interruption warning"
          description: "Spot instance {{ $labels.instance_id }} will be terminated in 2 minutes"
      
      # 노드 용량 부족
      - alert: InsufficientSpotCapacity
        expr: increase(kube_node_status_condition{condition="Ready",status="false"}[10m]) > 2
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Multiple spot instances unavailable"
          description: "{{ $value }} nodes became unavailable in the last 10 minutes"
      
      # 클러스터 오토스케일러 실패
      - alert: ClusterAutoscalerFailure
        expr: increase(cluster_autoscaler_failed_scale_ups_total[15m]) > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Cluster autoscaler failed to scale up"
          description: "Autoscaler failed to provision nodes: {{ $labels.reason }}"

Slack alert integration:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
data:
  alertmanager.yml: |
    global:
      slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
    
    route:
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 1h
      receiver: 'spot-alerts'
    
    receivers:
    - name: 'spot-alerts'
      slack_configs:
      - channel: '#eks-alerts'
        title: 'EKS Spot Instance Alert'
        text: |
          {{ range .Alerts }}
          *Alert:* {{ .Annotations.summary }}
          *Description:* {{ .Annotations.description }}
          *Severity:* {{ .Labels.severity }}
          {{ end }}

Production Operations Guide and Best Practices

1. Proactive Monitoring

Monitor key metrics:

bash
# 일일 점검 스크립트 예시
#!/bin/bash

echo "=== EKS 클러스터 리소스 현황 체크 ==="

# 1. 노드별 Pod 밀도 확인
echo "1. 노드별 Pod 사용률:"
kubectl get nodes -o custom-columns=NAME:.metadata.name,PODS:.status.capacity.pods,ALLOCATABLE:.status.allocatable.pods | awk 'NR==1 || $2/$3 > 0.8'

# 2. IP 주소 사용률 확인
echo "2. VPC CNI IP 사용 현황:"
kubectl get pods -n kube-system -l k8s-app=aws-node -o jsonpath='{range .items[*]}{.spec.nodeName}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}'

# 3. 리소스 사용률 Top 10
echo "3. CPU 사용률 상위 Pod:"
kubectl top pods --all-namespaces --sort-by=cpu | head -10

echo "4. 메모리 사용률 상위 Pod:"
kubectl top pods --all-namespaces --sort-by=memory | head -10

# 5. Pending Pod 확인
echo "5. Pending 상태 Pod:"
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

Predictive capacity planning:

2. Cost Optimization Strategies

Optimize the Spot/On-Demand ratio:

yaml
# 권장 비율: Spot 70%, On-Demand 30%
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: cost-optimized-nodepool
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
      # Spot Instance 우선, 부족 시 On-Demand로 대체
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: default
  # 비용 최적화를 위한 빠른 스케일 다운
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 10s

Automate resource right-sizing:

bash
# VPA 권장값 적용 스크립트
#!/bin/bash
for deployment in $(kubectl get deployments --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}'); do
  namespace=$(echo $deployment | cut -d' ' -f1)
  name=$(echo $deployment | cut -d' ' -f2)
  
  # VPA 권장값 확인
  recommendation=$(kubectl get vpa ${name}-vpa -n $namespace -o jsonpath='{.status.recommendation.containerRecommendations[0].target}' 2>/dev/null)
  
  if [ ! -z "$recommendation" ]; then
    echo "Deployment $namespace/$name 리소스 조정 권장: $recommendation"
  fi
done

3. High Availability and Failure Recovery

Multi-AZ deployment strategy:

yaml
# Pod Anti-Affinity를 활용한 분산 배치
apiVersion: apps/v1
kind: Deployment
metadata:
  name: high-availability-app
spec:
  replicas: 6
  template:
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - high-availability-app
              topologyKey: topology.kubernetes.io/zone
          - weight: 50
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - high-availability-app
              topologyKey: kubernetes.io/hostname

Automatic failure recovery mechanism:

yaml
# Health Check 및 자동 재시작
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: app
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 2

Closing Notes

Resource shortages are hard to avoid when operating EKS clusters. As this post shows, though, understanding each problem's root cause and applying an appropriate solution makes them manageable.

Key points:

  1. Resolve Pod IP exhaustion by optimizing VPC CNI settings and using Prefix Delegation
  2. Address resource shortages with Kubernetes 1.33 features and management through Kyverno/VPA
  3. Improve reliability for Spot Instance issues through diversification and active monitoring

Operational principles:

In particular, Kubernetes 1.33's in-place Pod resize feature will substantially change operations. Adjusting resources without restarting Pods brings operations closer to zero downtime.

Checklist: A Roadmap for Resolving Resource Shortages

Actions You Can Take Now

Step 1: Diagnose the current state (30 minutes)

Step 2: Emergency response (1 hour)

Step 3: Medium- to long-term improvements (1 week)

Step 4: Establish operational processes (1 month)

Success Metrics

Technical metrics:

Business metrics:

I hope this guide helps you operate reliable, cost-efficient EKS clusters through continuous monitoring and capacity planning.

References:

Diagram

100%

Drag to pan; pinch or use the wheel to zoom. Keyboard: arrow keys pan, + − zoom, 0 fits, Esc closes. The same content is in “Read the diagram as text” below the figure.