Comparing 3 Cilium Installation Modes on EKS: ENI vs CNI Chaining vs BYOCNI (Overlay)

Overview

Choosing how to install Cilium on EKS effectively means deciding who assigns Pod IPs and how (VPC ENI vs overlay), whether the VPC can route Pod IPs directly (Direct routing vs Encapsulation), and to what extent to retain the default AWS add-ons (AWS VPC CNI, kube-proxy).

This post reviews the following 3 sources: what each covers, how each setup works, and the differences, advantages, and disadvantages of the configurations.

Writing/validation criteria: Compare the documents and blogs from the perspectives of "Can the instructions be reproduced as written?" and "What operational constraints arise?" (Versions/defaults can change over time, so always check the latest documented Helm values and EKS add-on status before applying a configuration.)

TL;DR (Conclusion)

• ENI mode (recommended/direct approach): Remove AWS VPC CNI and let Cilium handle ENI/IPAM. It is a good path to VPC-native routing + kube-proxy-free operation in one step, but requires substantial changes to AWS permissions and operations.
• CNI chaining mode (practical compromise): Retain AWS VPC CNI and chain Cilium as a "second CNI" to add eBPF policies/LB/encryption and other features. It is relatively easy to apply to existing clusters, but introduces the operational complexity of two coexisting CNIs.
• BYOCNI(Overlay) mode (avoiding ENI IP limits): Without AWS VPC CNI, Cilium builds the Pod network using an overlay (VXLAN/Geneve). It removes ENI-based IP constraints, but increases encapsulation overhead, MTU concerns, and troubleshooting difficulty.

1) What Does Each Source Cover?

(1) eks-eni-install: A Tutorial on Installing Cilium in ENI Mode on EKS

This post walks through installing Cilium in ENI mode on EKS.

• Main approach
  • Create the cluster with addonsConfig.disableDefaultAddons: true, disabling the default networking add-ons (AWS VPC CNI, kube-proxy)
  • When installing Cilium with Helm, enable eni.enabled=true, ipam.mode=eni, routingMode=native, kubeProxyReplacement=true, and other options.
• Overall workflow
  1. Create the EKS cluster (default add-ons disabled) → 2) Install Cilium (ENI/IPAM + kube-proxy replacement) → 3) Create the node group → 4) Validate with cilium status and cilium connectivity test

The post notes a limitation: the cluster and node group cannot be created together when default add-ons are disabled. Since this depends on the evolving support in eksctl/EKS, checking the eksctl documentation at the time of implementation is prudent.

(2) byonci-overlay-install: A Comparison/Installation Tutorial for Chaining vs Overlay(BYOCNI)

Part 2 of the series covers 2 additional installation modes.

1. CNI chaining mode

• Disable default add-ons when creating the cluster, but in addons: explicitly include vpc-cni to install AWS VPC CNI.
• Install Cilium with cni.chainingMode=aws-cni, cni.exclusive=false, routingMode=native, and other options.

2. BYOCNI(Overlay) mode

• Without including AWS VPC CNI (= effectively BYOCNI), Cilium builds an overlay network.
• The key point is escaping AWS ENI-based Pod IP limits, such as the number of IPs assignable per node.

(3) cni-chaining-aws-cni: The Official Reference for Chaining Mode

The official documentation defines clear responsibilities for the AWS VPC CNI + Cilium combination.

• AWS VPC CNI: ENI-based device configuration + IPAM
• Cilium: Attach eBPF to those devices (policies/LB/encryption, etc.)

It also highlights operational details that are easy to miss.

• AWS VPC CNI plugin version requirements (for example, v1.11.2 or later)
• Existing Pods must be restarted (chaining is not applied automatically to running Pods)
• (Advanced) How to integrate EKS Security Groups for Pods (Pod ENI)

2) How Do the Components and Workflow Differ?

The key is which CNI runs first when a Pod is created, and which component is responsible for its IP.

(A) ENI Mode: Cilium Handles ENI/IPAM Too

• AWS VPC CNI (aws-node) is removed/disabled.
• Pod IPs come from ENIs (= directly routable in the VPC), so overlay encapsulation is unnecessary.
• The blog example also disables kube-proxy and enables kubeProxyReplacement=true.

(B) CNI Chaining Mode: AWS VPC CNI Handles IPAM, Cilium Handles eBPF

• Pod IPs are still ENI-based and use VPC-native routing.
• The key operational difference is that there are 2 CNIs (aws-node + cilium).
• This also explains why the official documentation requires restarting existing Pods: the chaining step has not been applied to already-created network devices.

(C) BYOCNI(Overlay) Mode: Cilium Builds the Pod Network with an Overlay

• Rather than making Pod IPs directly routable in the VPC, traffic between nodes is encapsulated.
• The aim is to escape VPC-native constraints such as ENI IP limits and address planning.
• In return, operations must handle MTU, overhead, and network visibility issues.

3) Configuration Differences (Decision Points)

Comparison at a Glance

4) Advantages and Disadvantages (Operational Perspective)

ENI Mode

Advantages

• VPC-native Pod IPs (no overlay/encapsulation needed) → Benefits for performance/simplicity
• A natural path to kube-proxy-free operation with kubeProxyReplacement=true
• Works well with EKS's ability to create clusters without default add-ons (the workflow used in the blog)

Disadvantages/precautions

• Replacing AWS VPC CNI means substantial operational changes (existing standards, runbooks, and troubleshooting processes change)
• Correct Helm values are important:
  • When setting k8sServiceHost, the blog explicitly warns to remove https:// (an incorrect value crashes the agent)
  • Before copying the blog's egressMasqueradeInterfaces=eth+ example, check the node interface names (for example, eth0 or ens5)

CNI Chaining Mode

Advantages

• Makes it easier to introduce Cilium policies/observability/features gradually while retaining AWS VPC CNI
• Official documentation gives relatively clear prerequisites, validation steps, and upgrade considerations
• (Advanced) A guide is available for integrating EKS Security Groups for Pods

Disadvantages/precautions

• Having 2 CNIs means 2 times as many failure/debugging paths (it becomes harder to trace which component changed packets and where)
• As the official documentation states, restarting existing Pods is effectively required to apply chaining
• aws-node has version requirements (the documentation recommends v1.11.2+)
• The official documentation mentions the label vps.amazonaws.com/has-trunk-attached, but its command example uses vpc.amazonaws.com/has-trunk-attached. In practice, vpc.amazonaws.com/has-trunk-attached is common, so check the actual label key in your cluster when applying this.

BYOCNI(Overlay) Mode

Advantages

• Relatively free from ENI-based Pod IP limits, including assignable IPs per node and address planning
• Network design may become simpler because the VPC does not need to route Pod IPs directly (but tunnels are introduced instead)

Disadvantages/precautions

• Encapsulation overhead + MTU issues increase performance concerns and potential failure points
• Security group, network policy, and observability (Flow) issues shift from VPC routing problems to tunnel/MTU/path problems, broadening the problem space

5) Which Should I Choose? (A Practical Guide)

• A new EKS cluster, a desire for standardization, and consideration of kube-proxy-free operation → ENI mode is the 1st candidate
• An existing EKS cluster where you want to retain AWS VPC CNI and first introduce Cilium policies/observability → CNI chaining mode is practical
• Many Pods per node (ENI IPs are the bottleneck), with VPC address planning/ENI limits repeatedly causing problems → Consider BYOCNI(Overlay), while preparing to handle MTU and overlay operating costs

5.1) If Pod IP Exhaustion Is the Starting Point (Avoiding Misunderstandings)

Many teams consider Cilium because available Pod IPs run out quickly. First, clarify the following.

• What is causing the IP shortage?
  • (1) Are Pods per node limited by the node instance type's ENI/IP limits?
  • (2) Is the subnet CIDR itself too small, exhausting addresses across the VPC?

The conclusion is fairly simple.

• Even if you replace the CNI with Cilium in ENI mode or chain it with AWS VPC CNI, VPC IP exhaustion may not be fundamentally resolved if Pods still use VPC-routable IPs (ENI-based IPs).
  • If the issue is the per-node ENI/IP limit, AWS-native tuning such as changing instance types, adding nodes, or Prefix Delegation is often a more direct solution.
• Options that truly move away from VPC IP exhaustion generally use Pod IPs outside the VPC (an overlay IP pool), such as Overlay(BYOCNI), but introduce trade-offs with AWS-native integration.

5.2) Trade-offs for AWS-Native Integration

When changing networking to resolve Pod IP exhaustion, whether the VPC can route/observe Pod IPs directly has a major impact on the usability of AWS-native features.

CNI Chaining (Retain AWS VPC CNI)

• Advantage: AWS VPC CNI continues to handle ENI/IPAM, offering the strongest continuity with existing EKS operations/integrations.
• Disadvantage: Pods continue consuming VPC IPs, so the IP exhaustion problem is likely to remain.

Cilium ENI Mode (Replace AWS VPC CNI)

• Advantage: Uses a model where Pod IPs are directly routable in the VPC (ENI-based), making the network model relatively simple compared with an overlay.
• Disadvantage: Operational/tuning practices that depend on aws-node (AWS VPC CNI), such as aws-node parameters and runbooks, may need to be redefined. Before adoption, list and review the features your organization expects from aws-node.

Overlay(BYOCNI)

• Advantage: The most direct option for escaping VPC IP exhaustion, depending on the design.
• Disadvantage: Because Pod IPs are not directly routable in the VPC, configurations assuming VPC-reachable/routable Pod IPs may be constrained, such as:
  • AWS Load Balancer Controller designs based on target-type: ip (requiring direct VPC access to Pod IPs)
  • Approaches that provided straightforward Pod-level network visibility/troubleshooting through VPC Flow Logs/security groups/NACLs
  • More failure points caused by MTU/encapsulation issues

5.3) AWS-Native Solutions to Check First for IP Shortages (Checklist)

Before moving to an overlay, these are among the most commonly used solutions on EKS.

(1) AWS VPC CNI Prefix Delegation

If the Pod count per node is restricted by ENI/IP limits, Prefix Delegation often makes a noticeable difference. However, if the subnet CIDR is too small, the overall address space will still run out first.

Conceptual example: Set Prefix Delegation environment variables on aws-node

kubectl -n kube-system set env ds/aws-node \
  ENABLE_PREFIX_DELEGATION=true \
  WARM_PREFIX_TARGET=1

(2) Expand/Add Subnet CIDRs (Address Planning)

If the entire VPC is running short of addresses, changing the CNI eventually reaches the same limit, so expanding/adding subnet CIDRs is the more direct approach.

(3) Node Instance Type/Node Count Strategy

Since ENI/IP limits vary by instance type, changing instance types or adding nodes is often the simplest solution when the IP bottleneck is per node.

(4) Consider IPv6/Dual Stack (If Feasible)

If organizational/service constraints allow it, IPv6 can structurally alleviate address exhaustion. However, ecosystem and integration requirements need review.

6) Reproducible Checklist (Common Steps)

(1) Check Cluster Add-on Status

kubectl get ds -n kube-system
kubectl get pods -n kube-system

(2) Check API Server Host/Port (Common Blog Workflow)

kubectl cluster-info

When setting k8sServiceHost, the ENI mode blog explicitly says to remove https://.

(3) Validate the Installation

cilium status --wait
cilium connectivity test

(4) For Chaining Mode, Plan to Restart Existing Pods

According to the official documentation, chaining is not applied automatically to running Pods. Include a maintenance window (rolling restart plan).

Closing Notes

The three sources cover essentially the same topic at different depths.

• The 2 blogs focus on the steps needed to complete an EKS installation (a series)
• The official documentation focuses on operational constraints and validation points for chaining mode (a reference)

Choose ENI for performance/simplicity/a conventional approach, chaining for gradual adoption, and overlay(BYOCNI) for avoiding ENI IP limits. Whatever you choose, planning for the cluster's add-on state at creation (aws-node/kube-proxy), Pod restarts, and validation routines greatly improves the chances of operational success.