Skip to main content
inspsol

2 years ago

CKA - Configure & Use Cluster DNS at 01.5 ...

Hi,

In this video you are using busybox for the nslookup ...

when i use it I get an error: exit code 1

Any idea why this could be ?

I am wondering if it highlights a problem, since I don't seem to get full access to the pods ) see the other question )


This is the command list I entered and the responses.
...

ubuntu@ip-172-31-4-184:~/pod-job$ kubectl expose deployment bit-of-nothing --type=ClusterIP --port 80

service "bit-of-nothing" exposed

ubuntu@ip-172-31-4-184:~/pod-job$ kubectl get services

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

bit-of-nothing ClusterIP 10.101.78.245 <none> 80/TCP 36s

dp-nginx-1-8 NodePort 10.96.33.37 <none> 80:31552/TCP 1h

kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d

...

ubuntu@ip-172-31-4-184:~/pod-job$ kubectl exec -it busybox -- nslookup bit-of-nothing

Server: 10.96.0.10

Address 1: 10.96.0.10

nslookup: can't resolve 'bit-of-nothing'

command terminated with exit code 1

...

This is the /etc/resolv.conf


ubuntu@ip-172-31-4-184:~/pod-job$ kubectl exec -it busybox -- cat /etc/resolv.conf

nameserver 10.96.0.10

search default.svc.cluster.local svc.cluster.local cluster.local eu-west-2.compute.internal

options ndots:5

...

KUBE DNS is running:

kube-dns-6f4fd4bdf-d9m4r 3/3 Running 0 1d

...

The Endpoints:


ubuntu@ip-172-31-4-184:~/pod-job$ kubectl get endpoints kube-dns -n kube-system

NAME ENDPOINTS AGE

kube-dns 10.244.0.2:53,10.244.0.2:53 1d

...


AND ..

ubuntu@ip-172-31-4-184:~/pod-job$ kubectl get svc -n kube-system

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 1d

...

And Finally ... I have looked through the logs, but I don't see anything out of the ordinary ... but maybe someone can so I paste them all here !!!

...

This is the log data.

...


ubuntu@ip-172-31-4-184:~/pod-job$ kubectl logs -n kube-system $(kubectl get pods -n kube-system -l k8s-app=kube-dns -o name) -c kubedns

I0228 19:55:02.438787 1 dns.go:48] version: 1.14.6-3-gc36cb11

I0228 19:55:02.441980 1 server.go:69] Using configuration read from directory: /kube-dns-config with period 10s

I0228 19:55:02.442100 1 server.go:112] FLAG: --alsologtostderr="false"

I0228 19:55:02.442135 1 server.go:112] FLAG: --config-dir="/kube-dns-config"

I0228 19:55:02.442157 1 server.go:112] FLAG: --config-map=""

I0228 19:55:02.442216 1 server.go:112] FLAG: --config-map-namespace="kube-system"

I0228 19:55:02.442239 1 server.go:112] FLAG: --config-period="10s"

I0228 19:55:02.442278 1 server.go:112] FLAG: --dns-bind-address="0.0.0.0"

I0228 19:55:02.442312 1 server.go:112] FLAG: --dns-port="10053"

I0228 19:55:02.442331 1 server.go:112] FLAG: --domain="cluster.local."

I0228 19:55:02.442372 1 server.go:112] FLAG: --federations=""

I0228 19:55:02.442393 1 server.go:112] FLAG: --healthz-port="8081"

I0228 19:55:02.442411 1 server.go:112] FLAG: --initial-sync-timeout="1m0s"

I0228 19:55:02.442428 1 server.go:112] FLAG: --kube-master-url=""

I0228 19:55:02.442455 1 server.go:112] FLAG: --kubecfg-file=""

I0228 19:55:02.442474 1 server.go:112] FLAG: --log-backtrace-at=":0"

I0228 19:55:02.442494 1 server.go:112] FLAG: --log-dir=""

I0228 19:55:02.442543 1 server.go:112] FLAG: --log-flush-frequency="5s"

I0228 19:55:02.442565 1 server.go:112] FLAG: --logtostderr="true"

I0228 19:55:02.442583 1 server.go:112] FLAG: --nameservers=""

I0228 19:55:02.442611 1 server.go:112] FLAG: --stderrthreshold="2"

I0228 19:55:02.442627 1 server.go:112] FLAG: --v="2"

I0228 19:55:02.442643 1 server.go:112] FLAG: --version="false"

I0228 19:55:02.442663 1 server.go:112] FLAG: --vmodule=""

I0228 19:55:02.442721 1 server.go:194] Starting SkyDNS server (0.0.0.0:10053)

I0228 19:55:02.443501 1 server.go:213] Skydns metrics enabled (/metrics:10055)

I0228 19:55:02.443554 1 dns.go:146] Starting endpointsController

I0228 19:55:02.443575 1 dns.go:149] Starting serviceController

I0228 19:55:02.444850 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]

I0228 19:55:02.444913 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]

I0228 19:55:02.943830 1 dns.go:170] Initialized services and endpoints from apiserver

I0228 19:55:02.943848 1 server.go:128] Setting up Healthz Handler (/readiness)

I0228 19:55:02.943858 1 server.go:133] Setting up cache handler (/cache)

I0228 19:55:02.943864 1 server.go:119] Status HTTP port 8081

...

The DNSMASQ

...


ubuntu@ip-172-31-4-184:~/pod-job$ kubectl logs -n kube-system $(kubectl get pods -n kube-system -l k8s-app=kube-dns -o name) -c dnsmasq

I0228 19:55:04.316812 1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}

I0228 19:55:04.317170 1 nanny.go:94] Starting dnsmasq [-k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]

I0228 19:55:04.660555 1 nanny.go:119]

W0228 19:55:04.660594 1 nanny.go:120] Got EOF from stdout

I0228 19:55:04.660649 1 nanny.go:116] dnsmasq[8]: started, version 2.78 cachesize 1000

I0228 19:55:04.660670 1 nanny.go:116] dnsmasq[8]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify

I0228 19:55:04.660690 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain ip6.arpa

I0228 19:55:04.660717 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa

I0228 19:55:04.660761 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain cluster.local

I0228 19:55:04.660831 1 nanny.go:116] dnsmasq[8]: reading /etc/resolv.conf

I0228 19:55:04.660850 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain ip6.arpa

I0228 19:55:04.660865 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa

I0228 19:55:04.660880 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain cluster.local

I0228 19:55:04.660902 1 nanny.go:116] dnsmasq[8]: using nameserver 172.31.0.2#53

I0228 19:55:04.660975 1 nanny.go:116] dnsmasq[8]: read /etc/hosts - 7 addresses

...

The Sidecar log.

...


ubuntu@ip-172-31-4-184:~/pod-job$ kubectl logs -n kube-system $(kubectl get pods -n kube-system -l k8s-app=kube-dns -o name) -c sidecar

I0228 19:55:06.122985 1 main.go:51] Version v1.14.6-3-gc36cb11

I0228 19:55:06.123183 1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})

I0228 19:55:06.123237 1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}

I0228 19:55:06.123297 1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}



Many thanks



Image of chadrm
chadrm
2 years ago
Thanks for the thorough question!  It really helps!  I'm not sure specifically what's going on.  Like you said, it all seems normal and ordinary.  It's like kube-dns didn't register the service somehow.  What if you try an nslookup on a different service, like the nodeport one?  Does it still return an error?
Image of hareshadvani
hareshadvani
2 weeks ago
I had a similar problem today. I went thru the lab to deploy nginx and then it asks to use busybox to see the dns record but it did not find a dns record even though I had the deployment, 2 nginx pods, and the service was running as a NodePort. Curl worked but busybox could not get a dns record.

Is it always automatic for a dns record to be created? and exactly what name does it use for the DNS record? is it the "app" label declared in the deployment.yaml?  Thank you in advance
Image of
1 week ago
Hello! Yes, every service defined in the cluster is assigned a DNS name. The name is the service name, followed by the namespace, followed by the base domain name. For example, kubernetes.default.svc.cluster.local

A pod will follow the dns name with pod ip, namespace, followed by the base domain name. for example 10-24-3-20.default.pod.cluster.local.

Here is a lesson that will further explain cluster DNS:

https://linuxacademy.com/cp/courses/lesson/course/4018/lesson/5/module/327

Image of
1 week ago
  @hareshtemp1  @hareshadvani
Image of hareshadvani
hareshadvani
1 week ago
Thank you. That was great info. I especially appreciate you provided a link to the appropriate module rather than just a simple comment. Thank you so much - you are THE BEST!

I found a few issues - hopefully this can help others with this and other DNS issues. (1) busybox is not the best - it caused me a lot of trouble here - nslookups worked and then failed - it was not consistent at all. (2) I installed alpine instead (kubectl run -i -tt alpine --image=alpine) then kubectl attach -i full-name-of-alpine-pod. (3) I also did a few other things but I really believe my whole issue was related to a bad busybox. (4) it also helps to know what specific syntax of dns names will get resolved and what won't. Your module notes show very clearly the syntax of the dns records for pods and services and knowing the search suffixes in the /etc/resolv.conf. Very helpful.

Thanks again for your great course and continued help.