Managing Secrets in Kubernetes: Sealed Secrets vs SOPS

While recently working on a project in a GitOps environment, I ran into a question. Deployment automation with Argo CD was working well, but how could I safely manage the secrets in the Helm chart's values.yaml file?

yaml

# 이런 식으로 평문 시크릿이 values.yaml에...
database:
  host: "db.example.com"
  username: "admin"
  password: "super-secret-password"  # 😱 Git에 그대로 노출

Looking for Kubernetes secret management tools led me to Sealed Secrets and SOPS. After testing both, I found that they take different approaches, each with its own strengths and weaknesses.

Why Does Kubernetes Secret Management Matter?

Practical Problems

The secret management problems you encounter during development are more complicated than you might expect.

bash

# 1. Git 저장소에 시크릿 노출 위험
git add values.yaml  # 평문 패스워드가 포함된 파일

# 2. 환경별 시크릿 관리의 어려움
dev-values.yaml      # 개발 환경 시크릿
staging-values.yaml  # 스테이징 환경 시크릿  
prod-values.yaml     # 운영 환경 시크릿

# 3. 팀원 간 시크릿 공유 문제
# Slack으로 패스워드 공유? 😰

The GitOps Secret Dilemma

This is especially challenging when using GitOps tools such as Argo CD.

Code

Git Repository (모든 설정 저장) ← GitOps 철학
       ↕️
   시크릿도 Git에?
   보안은 어떻게?

Tools for managing encrypted secrets help address these concerns.

Sealed Secret과 SOPS의 시크릿 흐름

Line guide

워크플로의 명령과 결과
SOPS가 쓰는 암호화 백엔드

Sealed Secret과 SOPS의 시크릿 흐름

Sealed Secret은 kubeseal로 봉인한 매니페스트를 Git에 두고 클러스터 컨트롤러가 Secret으로 푼다. SOPS는 KMS 같은 백엔드 키로 values 값만 암호화해 Git에 두고 helm-secrets로 설치한다.

Read the diagram as text

Sealed Secret: kubectl create --dry-run으로 만든 Secret 매니페스트를 kubeseal이 SealedSecret으로 바꾼다.
SealedSecret(encryptedData)을 Git에 저장한다.
클러스터의 Sealed Secret Controller가 SealedSecret을 자동 복호화한다.
컨트롤러가 Secret 리소스를 만든다.
SOPS: 평문 values.yaml(.gitignore로 제외)을 sops -e로 암호화해 values.enc.yaml을 만든다. 파일 구조는 두고 값만 암호화한다.
암호화 키는 AWS KMS, Azure Key Vault, GPG 같은 백엔드를 쓴다.
values.enc.yaml을 Git에 저장한다.
helm-secrets 플러그인으로 암호화된 파일을 써서 바로 설치한다.

Sealed Secrets: A Kubernetes-Native Approach

Concept and How It Works

Sealed Secrets is a Kubernetes controller developed by Bitnami. The core idea is simple.

yaml

# 일반 Secret (평문)
apiVersion: v1
kind: Secret
data:
  password: cGFzc3dvcmQ=  # base64 인코딩 (암호화 아님)

# SealedSecret (암호화됨)  
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: database-secret
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEQAx...  # 실제 암호화!

Workflow

Here is the workflow I used in practice.

bash

# 1. Sealed Secret Controller 설치
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

# 2. kubeseal CLI 설치  
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz

# 3. 시크릿 암호화
echo -n mypassword | kubectl create secret generic mysecret --dry-run=client --from-file=password=/dev/stdin -o yaml | kubeseal -o yaml > mysealedsecret.yaml

# 4. Git에 안전하게 저장
git add mysealedsecret.yaml
git commit -m "Add encrypted secret"

Advantages

Kubernetes-native: CRD-based, integrating naturally with k8s
Ready to use: Available immediately after installing the controller
Automatic decryption: Automatically creates Secret resources in the cluster

Disadvantages

In practice, I ran into a few limitations.

bash

# 헬름 차트와의 궁합 문제
# values.yaml의 각 시크릿을 일일이 찾아서 SealedSecret으로 변환해야 함
database:
  password: "secret123"  # 이걸 찾아서
  
# 이렇게 바꿔야 함
database:
  password:
    valueFrom:
      secretKeyRef:
        name: database-secret
        key: password

Helm chart changes required: The existing values.yaml structure must change
Individual secret management: A separate SealedSecret resource is needed for each secret
Cluster-dependent: Each cluster uses a different key pair

SOPS: File-Level Encryption

Concept and Philosophy

SOPS (Secret OPerationS) was developed by Mozilla and is now a CNCF Sandbox project. Its approach is completely different from Sealed Secrets.

yaml

# 원본 values.yaml
database:
  host: db.example.com
  username: admin
  password: secret123

# SOPS로 암호화한 후
database:
  host: db.example.com
  username: admin  
  password: ENC[AES256_GCM,data:Tr7o1T8w5...,iv:1234...,tag:abcd...]

The key idea: Selectively encrypt only the values while preserving the file structure.

Practical Workflow

The workflow with SOPS was very simple.

bash

# 1. SOPS 설치
curl -LO https://github.com/mozilla/sops/releases/download/v3.8.1/sops-v3.8.1.linux.amd64
sudo mv sops-v3.8.1.linux.amd64 /usr/local/bin/sops
chmod +x /usr/local/bin/sops

# 2. 헬름 차트 다운로드
helm pull grafana/grafana
tar -xzf grafana-*.tgz

# 3. values.yaml 수정 (평문으로 편하게)
vi grafana/values.yaml
adminPassword: "my-super-secret-admin-password"

# 4. 암호화
sops -e grafana/values.yaml > values.enc.yaml

# 5. Git에 안전하게 저장  
echo "values.yaml" >> .gitignore  # 평문 파일 제외
git add values.enc.yaml
git commit -m "Add encrypted grafana config"

Integration with helm-secrets

SOPS really shows its strengths when used with the helm-secrets plugin.

bash

# helm-secrets 플러그인 설치
helm plugin install https://github.com/jkroepke/helm-secrets

# 암호화된 파일로 바로 설치!
helm secrets install grafana grafana/grafana -f values.enc.yaml

# ArgoCD와 연동
# ArgoCD가 자동으로 복호화해서 배포

Comparing the Two Tools

Here is a comparison based on my experience using both.

Category	Sealed Secrets	SOPS
Installation complexity	🟢 Easy (controller only)	🟡 Moderate (CLI + plugin)
Helm chart compatibility	🔴 Difficult (structural changes required)	🟢 Excellent (use as is)
Learning curve	🟢 Easy	🟡 Moderate
GitOps friendliness	🟢 Good	🟢 Very good
Multi-environment support	🔴 Complex (per-cluster keys)	🟢 Simple (using KMS)
Backup/migration	🔴 Difficult	🟢 Easy
Ecosystem	🟡 Kubernetes only	🟢 Supports various tools

Recommendations by Use Case

When to choose Sealed Secrets:

bash

# 1. 순수 쿠버네티스 매니페스트만 사용
# 2. 헬름 차트 커스터마이징 가능
# 3. 클러스터별 독립적 시크릿 관리 선호

When to choose SOPS:

bash

# 1. 헬름 차트 기반 배포 (대부분의 경우)
# 2. 기존 values.yaml 구조 유지 필요  
# 3. 다양한 환경에서 동일한 워크플로우 원함
# 4. GitOps와 CI/CD 파이프라인 통합

My Conclusion and Recommendations

After using both tools myself, I concluded that SOPS is more practical.

Main Reasons for Choosing SOPS

1. Natural Integration with the Helm Ecosystem

bash

# 기존 헬름 차트를 그대로 사용 가능
helm pull bitnami/postgresql
# values.yaml 수정
# sops로 암호화
# helm secrets로 배포

2. A Simple Development Workflow

bash

# 로컬에서는 평문으로 편집
vi values.yaml

# Git에는 암호화해서 저장
sops -e values.yaml > values.enc.yaml
git add values.enc.yaml

3. Support for Various Encryption Backends

bash

# AWS KMS
sops --kms "arn:aws:kms:us-east-1:123456789:key/12345678" -e secrets.yaml

# Azure Key Vault  
sops --azure-kv "https://myvault.vault.azure.net/keys/mykey/version" -e secrets.yaml

# GPG (온프레미스)
sops --pgp "user@example.com" -e secrets.yaml

4. Stability as a CNCF Project

Acceptance as a CNCF Sandbox project in 2023 also secured its long-term sustainability.

Recommendations for Practical Adoption

Initial Adoption

bash

# 1. 작은 프로젝트부터 시작
# 2. GPG 키 기반으로 시작 (클라우드 KMS 학습 전)
# 3. helm-secrets 플러그인 활용

Production Adoption

bash

# 1. 클라우드 KMS 활용 (AWS KMS, Azure Key Vault 등)
# 2. 환경별 암호화 키 분리
# 3. ArgoCD와 연동하여 GitOps 구현

Next Steps

If you are reading this, I recommend testing both tools yourself.

Set up a local cluster with k3s or minikube
Test both tools with a simple Helm chart
Choose the tool that fits your environment and workflow

Personally, I think SOPS + helm-secrets is currently the most practical choice for most Kubernetes environments. But Sealed Secrets may be more suitable depending on your environment and requirements, so I encourage you to try them yourself.

Secret management is fundamental to security, but it should not be so complex that it hurts developer productivity. Choose an appropriate tool to balance security and convenience.

References: