Kubernetes Troubleshooting Scenarios

Scenario: "Bilbao": Basic Kubernetes Problems

Level: Easy

Type: Fix

Access: Email

Description: There's a Kubernetes Deployment with an Nginx pod and a Load Balancer declared in the manifest.yml file. The pod is not coming up. Fix it so that you can access the Nginx container through the Load Balancer.

There's no "sudo" (root) access.

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: Running curl 10.43.216.196 returns the default Nginx Welcome page.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: K8s Playground - Free

Level: Easy

Type: Playground

Access: Email

Description: This is a Kubernetes sandbox for you to play with and experiment.

It comes with an nginx.yaml playbook. You can try for example k apply -f nginx.yaml (you can use "k" as an alias for "kubectl".)

The Helm binary is also installed.

Free account:
The free account sandbox runs on a 1 GB of RAM VM. As usual, there is no Internet access.

Paid accounts (Pro/Pro+/Business:
The playground in this case runs on a 2 GB of RAM VM. It has Internet access (to pull your own images for example) and twice the time.

Test: (No test and no "Check Solution" button. No Clues.)

Time to Solve: 20 minutes.

Scenario: K8s Playground - Pro

Level: Easy

Type: Playground

Access: Paid

Description: This is a Kubernetes sandbox for you to play with and experiment.

It comes with an nginx.yaml playbook. You can try for example k apply -f nginx.yaml (you can use "k" as an alias for "kubectl".)

The Helm binary is also installed.

Free account:
The free account sandbox runs on a 1 GB of RAM VM. As usual, there is no Internet access.

Paid accounts (Pro/Pro+/Business:
The playground in this case runs on a 2 GB of RAM VM. It has Internet access (to pull your own images for example) and twice the time.

Test: (No test and no "Check Solution" button. No Clues.)

Time to Solve: 60 minutes.

Scenario: "Buenos Aires": Kubernetes Pod Crashing

Level: Medium

Type: Fix

Access: Email

Description: There are two pods: "logger" and "logshipper" living in the default namespace. Unfortunately, logshipper has an issue (crashlooping) and is forbidden to see what logger is trying to say. Could you help fix Logshipper?

Do not change the K8S definition of the logshipper pod.

NOTE: Wait about 2 minutes after VM start for k3s to settle, then about 30 seconds after the pod is stably Ready before using "Check My Solution".

Test: kubectl get pods -l app=logshipper --no-headers -o json | jq -r '.items[] | "\(.status.containerStatuses[0].ready)"' returns true and stays that way.

The logshipper pod specification has not been changed.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Poznań": Helm Chart Issue in Kubernetes

Level: Medium

Type: Fix

Access: Paid

Description: NOTE: Prompt may take a few extra seconds to be responsive while the k3s environment gets ready. Root access is not needed for this challenge ("admin" user cannot sudo).

A DevOps engineer created a Helm Chart web_chart with a custom nginx site, however he still gets the default nginx index.html.

You can check for example with POD_IP=$(kubectl get pods -n default -o jsonpath='{.items[0].status.podIP}') and curl -s "${POD_IP}">.

In addition he should set replicas to 3.

The chart is not working as expected. Fix the configurations so you get the custom HTML page from any nginx pod.

Credit Kamil Błaż

Test: Doing curl on the default port (:80) of any nginx pod returns a Welcome SadServers page. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Bengaluru": Kubernetes StatefulSet least known characteristic

Level: Medium

Type: Fix

Access: Email

Description: There's a Kubernetes cluster (created with "k3d") with two worker nodes and two pods on the node k3d-cluster-agent-0: a Deployment demo-deployment-... and a StatefulSet demo-statefulset-0. Their manifests are identical except for the different kind of K8s resource.

Make the node hosting the pods unavailable (it "goes down" or "crashes" without being deleted from k8s), for example with: docker stop k3d-mycluster-agent-0.

After waiting for about a minute (tolerationSeconds in the manifest is 30s, we shorten the K8S 5 minutes default so you don't have to wait so much, plus a grace period), both pods are marked as Terminating. While the Deployment pod is evicted and deployed onto the remaining available node k3d-cluster-agent-1, the StatefulSet demo-statefulset-0 is not (Why?).

Make the StatefulSet pod demo-statefulset-0 run on the available node.

Note: you can use k as a shortcut for kubectl.

Test: Node k3d-cluster-agent-0 is NotReady. Both the Deployment and the StatefulSet are running on the node k3d-cluster-agent-1
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Ruaka": Kubernetes pod in distress

Level: Medium

Type: Fix

Access: Paid

Description: A developer wants to deploy an open-source tool on Kubernetes. The tool unfortunately has limited documentation.

They built a helm chart and a container image. When the application is deployed, for some reason the server in Kubernetes doesn't seem to work but when the binary is started on their laptop/machine it works perfectly.

The application server is deployed by Helm. The command they used is: helm upgrade --install ruaka charts/ruaka.

Debug and help the developer find the issue. NOTE: Do not change or delete any current Helm field value in the chart, only add if needed.

Remember to give enough time to k8S after you apply a change before checking the solution.

Test: kubectl get pod shows the ruaka application pod up and running, while no Helm fields have been taken out from the applicaiton chart.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Kilifi": Speculative Misallocation!

Level: Medium

Type: Fix

Access: Email

Description: A developer is having trouble deploying an application on a preconfigured cluster.

The application kilifi is to be deployed on kubernetes in the default namespace.
The application server is deployed by helm. The command they used is helm install kilifi charts/kilifi.
The application operates correctly with ~210 MB of memory, but 256 MB is recommended.

Swap should remain disabled in the cluster.

Debug and help the developer fix any issue with deployment.

Test: The kilifi application runs properly; it's Service on :3333/healthz returns "kilifi ready to serve".

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Solanea": ClickHouse mad house

Level: Medium

Type: Do

Access: Paid

Description: You have a ClickHouse installation CHI running on a Kubernetes cluster and a set of requests (located at ~/data/requests.csv) that you must populate into the http_requests table in the monitoring database (table may not exist in all pod instances).
Do this insert in all pod instances of the database.
The user and password to connect to the database are default.
The keeper pods provide clickhouse replication services.

Test: You are able to query the database and see the data:

clickhouse-client -h --password default -q 'SELECT COUNT(*) FROM monitoring.http_requests'

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.

Scenario: "Lyon": Migrate Ingress-NGINX to Traefik

Level: Medium

Type: Do

Access: Paid

Description: Ingress-NGINX is being retired. As the DevOps Engineer, you will replace it with Traefik on the production Kubernetes cluster in a private VPC. This scenario is a local proof-of-concept for that migration.

The current K8s cluster has a "Hello World" pod running, i.e.: curl hello.lyon.local returns "Hello world" (see note 1). You should be able to see the same content delivered via Traefik once the ingress-nginx is down.

Notes: 1: Wait at the start until k8s is fully up before doing curl, otherwise you get 503, you can check for ex with k get pod -n ingress-nginx
2: The k8s manifests are under the ~/app dir.
3: ingress-nginx was deployed with a Helm chart.
4: The Helm chart for traefik is available under /home/admin/traefik (The Traefik image is already loaded in k3s).
5: Traefik dashboard and probes/metrics port by default is :8080 but that's used by the system; use a different port or disable.
6: The domain hello.lyon.local is actually pointing to the localhost.
7: The ingress must be listening on port 80 for any IP so it can respond to localhost:80 or actually to *:80

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: When the command curl -i hello.lyon.local is executed, it returns the message Hello World, while only the traefik pod must be present (instead of ingress-nginx).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.

Scenario: "Aswan": K8s Service Connectivity Failure

Level: Medium

Type: Fix

Access: Paid

Description: Our microservices application on Kubernetes cannot reach the backend-db service from the frontend deployment; curl from the frontend pod fails.

Your task is to diagnose and fix the connectivity problems so the frontend can reach the backend database. There may be more than one issue.

NOTE: wait for all the pods to be running at the beginning of the exercise.

Test: Curl from inside the frontend pod to the backend-db service returns "Hello from Backend DB!": FRONTEND_POD=$(kubectl get pods -l app=frontend -o jsonpath='{.items[0].metadata.name}')
kubectl exec $FRONTEND_POD -- curl -s backend-db returns "Hello from Backend DB!"

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Kyoto": The Gion ticket API will not start

Level: Medium

Type: Fix

Access: Email

Description: The Gion ticket office API should answer on port :80. After last night's change the office never came back.

Manifests are in /home/admin/app. There is a short handover note in /home/admin/HANDOVER.txt. Wait until the Kubernetes node is Ready after boot.

GET /health must return {"status":"ok","office":"gion-tickets"}. Do not move the API off port 80.

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: curl -s http://127.0.0.1/health returns {"status":"ok","office":"gion-tickets"} and the gion-api pod in the gion namespace is Ready (1/1).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Singara": Docker and Kubernetes web app not working.

Level: Hard

Type: Fix

Access: Email

Description: There's a k3s Kubernetes install you can access with kubectl. The Kubernetes YAML manifests under /home/admin have been applied. Get the webapp Deployment in the web namespace to a Running pod that serves its HTTP response (a town/city name). Running the image with host Docker alone is not enough to pass the check.

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: curl localhost:8888 returns a value from the webapp deployed Kubernetes pod.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Amygdala": Do you have enough insight to see the secrets?

Level: Hard

Type: Fix

Access: Paid

Description: Troubleshoot and fix a Kubernetes web application running in the app namespace. Make the deployment run successfully so that it returns Hello handsome! when you curl it.

Fix first your admin user access to the local Kubernetes cluster; the KUBECONFIG environment variable must be set to $HOME/.kube/config.

You have full admin access to a Vault server (containing the secrets you need) from the admin user. All the used manifests for the application are placed on the /home/admin/manifests directory.

Test: Running: POD_IP=$(kubectl get po -n app -l app=app -o jsonpath='{.items[0].status.podIP}') curl http://$POD_IP returns Hello handsome!.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.

Scenario: "Tigoni": Patch and Pray!

Level: Hard

Type: Fix

Access: Email

Description: A developer wants to upgrade their stateful application. This application handles their archives/backups.

The application tigoni is deployed on kubernetes in the default namespace.
The application server is deployed by Helm. The command they used is helm install tigoni charts/tigoni.

Upgrade the tigoni application to v2.0.0. The image already exists in the local repository.
Debug and help the developer fix any issue with the upgrade.
Everytime v1.0.0 is launched the archiving code starts on a clean slate.

Test: The tigoni pod with version v2 runs correctly (its endpoint :3000/healthz displays serverVersion:v2)

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.