I run applications and a few add-ons in a Kubernetes cluster, and DNS query failures were intermittently causing Pods to die.
I had experienced something similar on AWS before. I knew that running a DNS server ourselves or using a DNS cache could help avoid overloading Route 53 Resolver with queries.
If you would like more details about the AWS case, the links below should be helpful.
- https://repost.aws/ko/knowledge-center/vpc-find-cause-of-failed-dns-queries
- https://repost.aws/ko/knowledge-center/dns-resolution-failures-ec2-linux
Drawing on that experience, I looked for a similar approach and learned that NodeLocal DNSCache can reduce DNS query load in Kubernetes as well.
Deploying NodeLocal DNSCache is straightforward using the official Kubernetes documentation. It also seems that managed Kubernetes services from cloud providers may offer NodeLocal DNSCache as an add-on.
- https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns
- (Aliyun) https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/configure-nodelocal-dnscache
First, I changed the CoreDNS logging level to Debug, then compared the logs in the CoreDNS Pods before and after deploying NodeLocal DNSCache.
The number of log entries in CoreDNS clearly decreased after deploying NodeLocal DNSCache. During about a month of monitoring, no query failure issues occurred.
If you have a similar problem, give NodeLocal DNSCache a try. If you know a better approach, please share it! 🫡