SadServers: Linux, DevOps & SRE Labs | Job Interview Prep

Troubleshooting Scenarios: Kubernetes

Linux & Bash

- Linux commands, Bash scripting
- Systemd
- Networking, DNS
- Storage
- SSH, Firewall
- Libraries
- Cron and more...

Web Servers

- Nginx
- Apache
- HAProxy
- Caddy
- Gunicorn
- uWSGI
- HTTPS/TLS

Databases

- PostgreSQL
- MySQL
- SQLite
- Redis
- ClickHouse
- MongoDB
- etcd

Data Processing

- CSV
- JSON
- SQL queries

Docker

- Building images
- Multi-stage builds
- Volumes
- Networks
- Docker Compose
- Podman

Kubernetes

- kubectl
- Helm
- K8S Roles & Permissions
- Services
- Namespaces
- Deployments, StatefulSets
- ConfigMaps, Secrets

Tooling / Applications

- Git
- Rabbitmq
- Envoy
- Vault
- Harbor
- Prometheus
- Jenkins

Hacking

- Capture the Flag (CTF) Challenges
- Code Vulnerabilities
- Privilege Escalation

Languages

- Python
- Golang
- PHP
- Java
- Node.js
- C

helm kubernetes

realistic / interviews new pro

Easy

#	Name	Time	Type
1	"Bilbao": Basic Kubernetes Problems	10 m	Fix	"Bilbao": Basic Kubernetes Problems Scenario: "Bilbao": Basic Kubernetes Problems Level: Easy Type: Fix Access: Email Description: There's a Kubernetes Deployment with an Nginx pod and a Load Balancer declared in the manifest.yml file. The pod is not coming up. Fix it so that you can access the Nginx container through the Load Balancer. There's no "sudo" (root) access. Test: Running `curl 10.43.216.196` returns the default Nginx Welcome page. See /home/admin/agent/check.sh for the test that "Check My Solution" runs. Time to Solve: 10 minutes.

Medium

#	Name	Time	Type
1	"Buenos Aires": Kubernetes Pod Crashing	20 m	Fix	"Buenos Aires": Kubernetes Pod Crashing Scenario: "Buenos Aires": Kubernetes Pod Crashing Level: Medium Type: Fix Access: Email Description: There are two pods: "logger" and "logshipper" living in the default namespace. Unfortunately, logshipper has an issue (crashlooping) and is forbidden to see what logger is trying to say. Could you help fix Logshipper? Do not change the K8S definition of the logshipper pod. Use "sudo". Because k8s takes a minute or two to change the pod state initially, the check for the scenario is made to fail in the first two minutes. Credit Srivatsav Kondragunta Test: `kubectl get pods -l app=logshipper --no-headers -o json \| jq -r '.items[] \| "\(.status.containerStatuses[0].ready)"'` returns `true` Time to Solve: 20 minutes.
2	"Poznań": Helm Chart Issue in Kubernetes	15 m	Fix Pro	"Poznań": Helm Chart Issue in Kubernetes Scenario: "Poznań": Helm Chart Issue in Kubernetes Level: Medium Type: Fix Access: Paid Description: NOTE: Prompt may take a few extra seconds to be responsive while the k3s environment gets ready. Root access is not needed for this challenge ("admin" user cannot sudo). A DevOps engineer created a Helm Chart web_chart with a custom nginx site, however he still gets the default nginx index.html. You can check for example with `POD_IP=$(kubectl get pods -n default -o jsonpath='{.items[0].status.podIP}')` and `curl -s "${POD_IP}">`. In addition he should set replicas to 3. The chart is not working as expected. Fix the configurations so you get the custom HTML page from any nginx pod. Credit Kamil Błaż Test: Doing `curl` on the default port (:80) of any nginx pod returns a `Welcome SadServers` page. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.
3	"Bengaluru": Kubernetes StatefulSet least known characteristic	15 m	Fix	"Bengaluru": Kubernetes StatefulSet least known characteristic Scenario: "Bengaluru": Kubernetes StatefulSet least known characteristic Level: Medium Type: Fix Access: Email Description: There's a Kubernetes cluster (created with "k3d") with two worker nodes and two pods on the node k3d-cluster-agent-0: a Deployment demo-deployment-... and a StatefulSet demo-statefulset-0. Their manifests are identical except for the different kind of K8s resource. Make the node hosting the pods unavailable (it "goes down" or "crashes" without being deleted from k8s), for example with: `docker stop k3d-mycluster-agent-0`. After waiting for about a minute (tolerationSeconds in the manifest is 30s, we shorten the K8S 5 minutes default so you don't have to wait so much, plus a grace period), both pods are marked as Terminating. While the Deployment pod is evicted and deployed onto the remaining available node k3d-cluster-agent-1, the StatefulSet demo-statefulset-0 is not (Why?). Make the StatefulSet pod demo-statefulset-0 run on the available node. Note: you can use `k` as a shortcut for `kubectl`. Test: Node k3d-cluster-agent-0 is NotReady. Both the Deployment and the StatefulSet are running on the node k3d-cluster-agent-1 The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.
4	"Ruaka": Kubernetes pod in distress	15 m	Fix Pro	"Ruaka": Kubernetes pod in distress Scenario: "Ruaka": Kubernetes pod in distress Level: Medium Type: Fix Access: Paid Description: A developer wants to deploy an open-source tool on Kubernetes. The tool unfortunately has limited documentation. They built a helm chart and a container image. When the application is deployed, for some reason the server in Kubernetes doesn't seem to work but when the binary is started on their laptop/machine it works perfectly. The application server is deployed by Helm. The command they used is: helm upgrade --install ruaka charts/ruaka. Debug and help the developer find the issue. NOTE: Do not change or delete any current Helm field value in the chart, only add if needed. Remember to give enough time to k8S after you apply a change before checking the solution. Test: `kubectl get pod` shows the ruaka application pod up and running, while no Helm fields have been taken out from the applicaiton chart. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.
5	"Kilifi": Speculative Misallocation!	20 m	Fix	"Kilifi": Speculative Misallocation! Scenario: "Kilifi": Speculative Misallocation! Level: Medium Type: Fix Access: Email Description: A developer is having trouble deploying an application on a preconfigured cluster. The application kilifi is to be deployed on kubernetes in the default namespace. The application server is deployed by helm. The command they used is helm install kilifi charts/kilifi. The application operates correctly with ~210 MB of memory, but 256 MB is recommended. Swap should remain disabled in the cluster. Debug and help the developer fix any issue with deployment. Test: The kilifi application runs properly; it's Service on :3333/healthz returns "kilifi ready to serve". The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.
6	"Solanea": ClickHouse mad house	20 m	Do Pro	"Solanea": ClickHouse mad house Scenario: "Solanea": ClickHouse mad house Level: Medium Type: Do Access: Paid Description: You have a ClickHouse installation CHI running on a Kubernetes cluster and a set of requests (located at ~/data/requests.csv) that you must populate into the http_requests table in the monitoring database (table may not exist in all pod instances). Do this insert in all pod instances of the database. The user and password to connect to the database are default. The keeper pods provide clickhouse replication services. Test: You are able to query the database and see the data: `clickhouse-client -h --password default -q 'SELECT COUNT() FROM monitoring.http_requests'` The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve:* 20 minutes.

Hard

#	Name	Time	Type
1	"Singara": Docker and Kubernetes web app not working.	20 m	Fix	"Singara": Docker and Kubernetes web app not working. Scenario: "Singara": Docker and Kubernetes web app not working. Level: Hard Type: Fix Access: Email Description: There's a k3s Kubernetes install you can access with kubectl. The Kubernetes YAML manifests under `/home/admin` have been applied. The objective is to access from the host the "webapp" web server deployed and find what message it serves (it's a name of a town or city btw). In order to pass the check, the webapp Docker container should not be run separately outside Kubernetes as a shortcut. Test: `curl localhost:8888` returns a value from the webapp deployed Kubernetes pod. Time to Solve: 20 minutes.
2	"Amygdala": Do you have enough insight to see the secrets?	20 m	Fix Pro	"Amygdala": Do you have enough insight to see the secrets? Scenario: "Amygdala": Do you have enough insight to see the secrets? Level: Hard Type: Fix Access: Paid Description: Troubleshoot and fix a Kubernetes web application running in the app namespace. Make the deployment run successfully so that it returns Hello handsome! when you curl it. Fix first your admin user access to the local Kubernetes cluster; the KUBECONFIG environment variable must be set to $HOME/.kube/config. You have full admin access to a Vault server (containing the secrets you need) from the admin user. All the used manifests for the application are placed on the /home/admin/manifests directory. Test: Running: `POD_IP=$(kubectl get po -n app -l app=app -o jsonpath='{.items[0].status.podIP}') curl http://$POD_IP` returns `Hello handsome!`. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.
3	"Tigoni": Patch and Pray!	20 m	Fix	"Tigoni": Patch and Pray! Scenario: "Tigoni": Patch and Pray! Level: Hard Type: Fix Access: Email Description: A developer wants to upgrade their stateful application. This application handles their archives/backups. The application tigoni is deployed on kubernetes in the default namespace. The application server is deployed by Helm. The command they used is helm install tigoni charts/tigoni. Upgrade the tigoni application to v2.0.0. The image already exists in the local repository. Debug and help the developer fix any issue with the upgrade. Everytime v1.0.0 is launched the archiving code starts on a clean slate. Test: The tigoni pod with version v2 runs correctly (its endpoint :3000/healthz displays serverVersion:v2) The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.