Kubernetes metrics | Error from server (ServiceUnavailable) | Quick 2 fixes

Introduction

Kubernetes metrics are required to run top command. To know the resource utilization of pods and nodes this will help us. While trying to set up the metric we could come across a few issues. For future reference this has been documented, may be helpful for someone those who face similar issues.

I have used an old version of metric till now, It’s time for upgrade. Tried with latest version of metric server deployment.

If you are looking to Setup the Kubernetes Cluster using Ansible, have a look at this.

Installing Latest Metrics

Latest Kubernetes metrics manifests can be downloaded from here, as of when I’m started the upgrade the latest version os v0.4.1.

Thought, the issues would have fixed on the latest version and still they exist. I have used the latest and still face few of issue.

$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml

Output for reference

ansible@k8mas1:~$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
ansible@k8mas1:~$ 

Fixing Kubernetes Metrics for Pods

This guide will walk through small fix.

ansible@k8mas1:~$ kubectl top pods
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
ansible@k8mas1:~$

Even installed with latest metric and getting unformatted output or metric not available error.

ansible@k8mas1:~$ kubectl top pods
W1203 23:45:07.204217   20547 top_pod.go:265] Metrics not available for pod default/apacheweb-5c8fcdd556-hq8mz, age: 16m9.204210978s
error: Metrics not available for pod default/apacheweb-5c8fcdd556-hq8mz, age: 16m9.204210978s
ansible@k8mas1:~$ 

Edit the metric server deployment in kube-system namespace.

$ kubectl edit deployments.apps -n kube-system metrics-server 

Append with two options, from

    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443


to this

    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-insecure-tls=true
        - --kubelet-preferred-address-types=InternalIP

Save and exit from deployment using wq!

ansible@k8mas1:~$ kubectl edit deployments.apps -n kube-system metrics-server 
deployment.apps/metrics-server edited
ansible@k8mas1:~$

Let’s describe the deployment now, we should get the appended arguments.

ansible@k8mas1:~$ kubectl describe deployments.apps -n kube-system metrics-server 
Name:                   metrics-server
Namespace:              kube-system
CreationTimestamp:      Thu, 03 Dec 2020 23:26:08 +0400
Labels:                 k8s-app=metrics-server
Annotations:            deployment.kubernetes.io/revision: 5
Selector:               k8s-app=metrics-server
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=metrics-server
  Service Account:  metrics-server
  Containers:
   metrics-server:
    Image:      k8s.gcr.io/metrics-server/metrics-server:v0.4.1
    Port:       4443/TCP
    Host Port:  4443/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=4443
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --kubelet-insecure-tls=true
      --kubelet-use-node-status-port
    Liveness:     http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:https/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /tmp from tmp-dir (rw)
  Volumes:
   tmp-dir:
    Type:               EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:             
    SizeLimit:          <unset>
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   metrics-server-798bd55c4d (1/1 replicas created)
Events:          <none>
ansible@k8mas1:~$

Now metric for pods should work fine.

ansible@k8mas1:~$ kubectl top pods
NAME                         CPU(cores)   MEMORY(bytes)   
apacheweb-5c8fcdd556-hq8mz   1m           6Mi             
apacheweb-5c8fcdd556-mxmc2   1m           6Mi             
apacheweb-5c8fcdd556-rlvhn   1m           6Mi             
apacheweb-5c8fcdd556-x7vr7   1m           6Mi             
apacheweb-5c8fcdd556-zsvkp   1m           6Mi             
somepod                      0m           2Mi             
ansible@k8mas1:~$ 

Fixing Kubernetes Metrics for Worker Nodes

While trying to list the top command, getting unknown for worker nodes.

ansible@k8mas1:~$ kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%     
k8nod1   163m         8%     742Mi           9%          
k8mas1   <unknown>                           <unknown>               <unknown>               <unknown>               
k8nod2   <unknown>                           <unknown>               <unknown>               <unknown>               
ansible@k8mas1:~$ 

Once again edit the deployment of Kubernetes metrics under the kube-system namespace.

$ kubectl edit deployments.apps -n kube-system metrics-server

Add this below dns policy or at the end of the container section above the restart Policy.

      hostNetwork: true

Run the top command, it should work now.

ansible@k8mas1:~$ kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8mas1   113m         5%     1270Mi          16%       
k8nod1   56m          2%     741Mi           9%        
k8nod2   55m          2%     698Mi           8%        
ansible@k8mas1:~$ 

That’s it, we have successfully fixed the metric issues for pods and nodes.

Conclusion

To conclude on fixing the Kubernetes Metrics issue in newly deployed Kubernetes cluster or an existing cluster is easier by editing the deployment yaml files. Will come up with a new fix on other guide, Subscribe to our newsletter and stay close for more upcoming how-to guides.