Realistic-interviews Troubleshooting Scenarios

Scenario: "Hamburg": Find the AWS EC2 volume

Level: Easy

Type: Do

Access: Paid

Description: We have a lot of AWS EBS volumes, the description of which we have save to a file with: aws ec2 describe-volumes > aws-volumes.json.
One of the volumes contains important data and we need to identify which volume (its ID), but we only remember these characteristics: gp3, created before 30/09/2025 , Size < 64 , Iops < 1500, Throughput > 300.

Find the correct volume and put its InstanceId into the ~/mysolution file, e.g.: echo "i-00000000000000000" > ~/mysolution

Test: Running md5sum /home/admin/mysolution returns e7e34463823bf7e39358bf6bb24336d8 (we also accept the file without a new line at the end).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 60 minutes.

Scenario: "Bologna": counting ELB 5xx errors

Level: Easy

Type: Do

Access: Paid

Description: Operations handed you a classic AWS Elastic Load Balancer access log at /home/admin/elb.log. Each line is one request. Fields are space-separated; the quoted HTTP request starts at field 12, so the numeric fields before it are fixed-width columns.

Field 8 is the ELB status code and field 9 is the backend status code returned by the target instance. Count how many log lines have a backend status code in the 5xx range (500 through 599). Write that integer — digits only — to /home/admin/solution.txt. For example: echo 42 > ~/solution.txt

The log mixes successful responses, redirects, client errors, and server errors; only backend 5xx responses count toward your answer.

Test: The MD5 checksum of your answer file md5sum /home/admin/solution.txt is b73ce398c39f506af761d2277d853a92 (we also accept the correct count with a trailing newline in the file).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Genova": cgroups problem

Level: Easy

Type: Fix

Access: Email

Description: This small VM runs sad-api (a lightweight health endpoint on port 9090) and sad-batch (a nightly ETL-style job that allocates a lot of RAM).

After a recent deploy, starting sad-batch caused memory use to spike and sad-api was killed by the OOM killer. On-call stopped the batch service before handing you the host.

A legacy cgroup v2 launcher under /opt/sad/ is supposed to enforce a 128M hard limit on cgroup sad-batch, but the cap never applies.

sad-batch is intentionally stopped and disabled when you log in. Read /home/admin/incident-notes.txt for context. Fix the cgroup configuration so /sys/fs/cgroup/sad-batch/memory.max is 134217728 before you start the batch job again.

Do not change sad-api; it should keep running on 127.0.0.1:9090.

Test: sad-api is active and curl http://127.0.0.1:9090/ returns SadServers - API OK.

The cgroup v2 hard limit is in place: cat /sys/fs/cgroup/sad-batch/memory.max prints 134217728 (128 MiB).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Bergen": Port already in use

Level: Easy

Type: Fix

Access: Email

Description: There's an application at /home/admin/standalone that needs to run successfully but currently it fails.

Fix the environment so the binary can run without errors, without changing the binary itself, and without breaking the web app served on port :80.

Test: Running /home/admin/standalone prints OK and curl http://localhost:80 returns hello SadServers.


The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Woluwe": Too many images

Level: Medium

Type: Fix

Access: Paid

Description: A pipeline created a lot of Docker images locally for a web app. All these images except for one contain a typo introduced by a developer: there's an incorrect image instruction to pipe "HelloWorld" to "index.htmlz" instead of using the correct "index.html"
Find which image doesn't have the typo (and uses the correct "index.html"), tag this correct image as "prod" (rather than fixing the current prod image) and then deploy it with docker run -d --name prod -p 3000:3000 prod so it responds correctly to HTTP requests on port :3000 instead of "404 Not Found".

Test: curl http://localhost:3000 should respond with HelloWorld;529

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Torino": Optimize grande Docker image

Level: Medium

Type: Do

Access: Paid

Description: A Torino Node.js application is located in the ~/torino-app directory.
You can run it directly with: nohup node app.js > app.log 2>&1 &. You can also verify that it works by running: curl localhost:3000

There is already a torino Docker image built with the Dockerfile in ~/torino-app, but the resulting image size is 916 MB.

Your task is to optimize the Docker image size:
1. Build a new Docker image for the Torino application, also called torino:latest but with a total size under 122 MB
2. Create and run a container using this optimized image.

NOTE: You can only use the existing Docker images in the server.
To build a Node application you need to COPY in your Dockerfile, besides the app.js , the package*.json files and without Internet access, the node_modules directory, since you cannot RUN npm install.

Test: The torino Docker image is less than 122 MB and curl http://localhost:3000 returns Hello from Torino!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "San Juan": mucho Traefik

Level: Medium

Type: Fix

Access: Email

Description: There is a Traefik load balancer that must be up and running. The server and the backend services are managed by Docker Compose. Running curl -s app.sadserver | head -n1 must return the host ID of one of the backend servers, running the command again must return a new host ID. The server seems to be working some times, some others fails or just times out.

The round-robin configuration should make the webserver iterate through the back-end servers.

Test: curl -s app.sadserver | head -n1 returns something like Hostname:

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Suzhou": MongoDB replicas!

Level: Medium

Type: Fix

Access: Email

Description: A new MongoDB replica set has been setup in the development environment trough /home/admin/app/rs0.js, however, a variety or errors are showing up when trying to bring it up. You should bring up all the replica servers, get them communicating to each other and make sure the replica set is working as it should.

The status of the first replica can be checked via systemctl status mongo1 same for the replicas mongo2 and mongo3. The logs are also in a separate file for each replica under the directory /var/log/mongodb. To initilize the replica set again: mongosh --file app/rs0.js

Note: The default configuration file /etc/mongo.conf is not the problem.

Test: mongosh --eval "rs.status()" | grep health returns the status of all the replicas

       health: 1,       health: 1,       health: 1, 

Time to Solve: 20 minutes.

Scenario: "Lyon": Migrate Ingress-NGINX to Traefik

Level: Medium

Type: Do

Access: Paid

Description: Ingress-NGINX is being retired. As the DevOps Engineer, you will replace it with Traefik on the production Kubernetes cluster in a private VPC. This scenario is a local proof-of-concept for that migration.

The current K8s cluster has a "Hello World" pod running, i.e.: curl hello.lyon.local returns "Hello world" (see note 1). You should be able to see the same content delivered via Traefik once the ingress-nginx is down.

Notes: 1: Wait at the start until k8s is fully up before doing curl, otherwise you get 503, you can check for ex with k get pod -n ingress-nginx
2: The k8s manifests are under the ~/app dir.
3: ingress-nginx was deployed with a Helm chart.
4: The Helm chart for traefik is available under /home/admin/traefik (The Traefik image is already loaded in k3s).
5: Traefik dashboard and probes/metrics port by default is :8080 but that's used by the system; use a different port or disable.
6: The domain hello.lyon.local is actually pointing to the localhost.
7: The ingress must be listening on port 80 for any IP so it can respond to localhost:80 or actually to *:80

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: When the command curl -i hello.lyon.local is executed, it returns the message Hello World, while only the traefik pod must be present (instead of ingress-nginx).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.

Scenario: "Stockholm": DNS health check issue

Level: Medium

Type: Fix

Access: Paid

Description: The internal status portal on this host should answer on http://127.0.0.1:9167/ with a body containing OK.

It worked until operations ran a package cleanup.

The portal service (stockholm-portal) only runs after a DNS health check at /usr/local/bin/stockholm-dns-check.sh succeeds.

Make the necessary changes so the portal works again.

Do not modify /usr/local/bin/stockholm-dns-check.sh.

Test: The health script /usr/local/bin/stockholm-dns-check.sh runs successfully, stockholm-portal is active, and curl http://127.0.0.1:9167/ returns a response whose body contains OK.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Cordoba": df is lying (or is it du?)

Level: Medium

Type: Fix

Access: Email

Description: Monitoring reports that the root filesystem is under pressure, but a quick du of /var/log shows almost nothing in the logs of the running application at /var/log/cordoba-app.

Find what is holding the space and reclaim it so df and du agree again for practical purposes; currently there's a ~300 MB discrepancy on the root partition /

The service unit is cordoba-hoarder.service.

Test: df -h / and sudo du -sh / report the same used space after reclaiming the ~300 MB discrepancy.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Tallinn": BuildKit & Docker build mismatch

Level: Medium

Type: Fix

Access: Paid

Description: This VM runs a tiny container app, tallinn-service, whose only job is to print an API version string (for example tallinn-api-version=1.4.0). The image is built from /home/admin/tallinn-app with docker build.

The dev team raised the API contract to 2.0.0 in src/api_version.txt and ran a new build, but QA still rejects the image tagged tallinn-app:current: it reports 1.4.0 at runtime. A recent CI log is in /home/admin/build.log.

Fix the docker build outcome so the deploy image matches what the sources ask for.

Fix the image tagged tallinn-app:current so the on-disk contract file and the shipped binary both report API 2.0.0.

Test: Image tallinn-app:current exists, /etc/tallinn/api_version is 2.0.0, and /usr/local/bin/tallinn-service prints tallinn-api-version=2.0.0.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Verona": Apache Portal Won't Open

Level: Medium

Type: Fix

Access: Email

Description: An internal Apache portal was migrated to this host. The document root is /var/www/portal.

The site root at http://localhost/ does not serve the expected homepage, and a legacy bookmark at /reports no longer reaches the status page (direct access to /status/ works).

Find and fix what keeps the portal root and the legacy redirect from working. Adding missing content is allowed.

Test: Ready.

curl -L http://localhost/reports returns a first line of SadServers - Status OK.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Modena": Ansible Deploy Won't Publish

Level: Medium

Type: Fix

Access: Paid

Description: This host publishes an internal status page by running Ansible locally against the Docker container status-app (port 8888 on localhost maps to the container's HTTP port).

The playbook tree lives in /home/admin/deploy/. After a refactor, ansible-playbook site.yml no longer leaves a working status endpoint — curl http://localhost:8888/ does not return the expected line.

Fix the Ansible project and run the playbook successfully so the status page is served from the container.

Test: curl http://localhost:8888/ returns a first line of SadServers - Modena OK.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Parma": Debugging Terraform Issues

Level: Medium

Type: Fix

Access: Email

Description: This host publishes a machine-readable status marker using Terraform with a local backend. The project lives in /home/admin/infra/ and should write /var/local/platform-status.txt.

After a refactor, terraform plan and terraform apply no longer succeed, and the status file is missing or stale.

Fix the Terraform project and apply it so the marker is published again.

(Note: Internet access is not needed).

Test: The first line of /var/local/platform-status.txt is SadServers - Parma OK.

Running terraform plan in /home/admin/infra/ reports no changes pending (clean state).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Ravenna": Logs Missing in ELK Pipeline

Level: Medium

Type: Fix

Access: Paid

Description: You are on call for the orders-api service. Central logging uses a small ELK stack on Docker Compose: an application container, Filebeat, Logstash, and Elasticsearch.

Operations reports that no order events show up in Elasticsearch, even though the application container is healthy and keeps writing logs. SRE left notes that the service contract specifies plain-text log lines.

The stack lives under /home/admin/ravenna and is managed with Docker Compose. Elasticsearch is reachable on the VM at http://127.0.0.1:9200.

Notes: 1. Wait until all four containers are Up before debugging (docker compose -f /home/admin/ravenna/docker-compose.yml ps). Elasticsearch can take up to two minutes to become healthy.
2. Internet access is not needed; container images are preloaded in the local Docker engine.

Test: At least one document containing order_shipped is indexed in Elasticsearch under the orders-* index pattern.

Quick check:

 curl -s 'http://127.0.0.1:9200/orders-*/_search?q=order_shipped&size=1' | jq . 

Time to Solve: 30 minutes.

Scenario: "Luxor": PostgreSQL analytics queries crawl

Level: Medium

Type: Fix

Access: Email

Description: Our analytics API serves customer sales counts from a PostgreSQL database. Requests to the customer lookup endpoint take several seconds and often time out, even though the database server looks healthy — no obvious CPU, memory, or disk exhaustion.

The API runs as systemd service sad-analytics-api on port 9090. Application code lives under /opt/sad/. Database credentials for debugging: connect as saduser to database analytics_db (password: sadpassword). The main table is sales_data.

Find why customer lookups are slow and restore acceptable API response times.

Test: This API request must complete quickly (total response time under 500 ms):
curl -s -w "\nTotal time: %{time_total}s\n" http://127.0.0.1:9090/api/customers/1234/sales/count

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Banha" : Intermittent 504 Gateway Time-Out errors

Level: Medium

Type: Fix

Access: Paid

Description: Our Python Flask web application, served via Gunicorn and Nginx, is experiencing intermittent 504 Gateway Time-Out errors on concurrent requests. Users report that the application works perfectly at times, but fails unexpectedly at others. Your task is to identify the root cause of these sporadic failures and implement a permanent fix to ensure the applications consistent stability on port 80.

Test: curl http://localhost/ returns "Hello from Flask!" consistently, even on parallel load: seq 15 | xargs -n1 -P15 curl -sf --max-time 5 http://localhost/ && echo OK || echo FAIL.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Aswan": K8s Service Connectivity Failure

Level: Medium

Type: Fix

Access: Paid

Description: Our microservices application on Kubernetes cannot reach the backend-db service from the frontend deployment; curl from the frontend pod fails.

Your task is to diagnose and fix the connectivity problems so the frontend can reach the backend database. There may be more than one issue.

NOTE: wait for all the pods to be running at the beginning of the exercise.

Test: Curl from inside the frontend pod to the backend-db service returns "Hello from Backend DB!": FRONTEND_POD=$(kubectl get pods -l app=frontend -o jsonpath='{.items[0].metadata.name}')
kubectl exec $FRONTEND_POD -- curl -s backend-db returns "Hello from Backend DB!"

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Spa": The Docker Compose healthcheck that does not heal

Level: Medium

Type: Fix

Access: Paid

Description: The thermal baths booking API should answer on port :5000. After last week's outage the previous engineer added a Docker HEALTHCHECK and restart: unless-stopped so the stack would recycle itself if the probe failed. The front desk is still down.

The Compose project is in /home/admin/spa. There is a short handover note in that directory.

The booking API application itself is fine when it is running: if you restart the container, GET /health comes back. You do not need to rewrite or "fix" the API service code. The real gap is recovery — when the API process inside the container dies again, nothing brings the service back on its own. Make the stack recover automatically from that failure so /health returns {"status":"ok"} without a manual restart.

The process will not usually die on its own during the exercise. To replay the incident, stop the API worker inside the container (leave the container running): docker exec spa-api sh -c 'kill "$(cat /tmp/spa.pid)"' — before a fix, /health stays down. After you add automatic recovery, the same kill should bring /health back without a manual docker restart. The recovery must still work after a host reboot (do not leave the API dead on boot).

Test: curl -s http://127.0.0.1:5000/health returns {"status":"ok"} and the spa-api container is healthy.
A one-off docker restart is not enough: after the API worker dies again inside the container, the service must recover on its own.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Kyoto": The Gion ticket API will not start

Level: Medium

Type: Fix

Access: Email

Description: The Gion ticket office API should answer on port :80. After last night's change the office never came back.

Manifests are in /home/admin/app. There is a short handover note in /home/admin/HANDOVER.txt. Wait until the Kubernetes node is Ready after boot.

GET /health must return {"status":"ok","office":"gion-tickets"}. Do not move the API off port 80.

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: curl -s http://127.0.0.1/health returns {"status":"ok","office":"gion-tickets"} and the gion-api pod in the gion namespace is Ready (1/1).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Dakar": The half-installed package

Level: Medium

Type: Fix

Access: Email

Description: The checkout platform's local health agent stopped working after a package upgrade was interrupted. The checkout-agent binary may already be on disk, but its systemd service is not healthy.

Restore the package and service so the agent runs again and still comes back after a reboot. Do not wipe the package database under /var/lib/dpkg, and do not replace the service with a hand-written unit of your own.

Test: checkout-agent is install ok installed; checkout-agent.service is active and enabled; its unit under /lib/systemd/system/ is owned by the package.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Anatolia": compromised server

Level: Hard

Type: Fix

Access: Paid

Description: This web server has been compromised and is not serving the home page anymore, those troubleshooting skills you have as DevOps are urgently needed to solve the mystery of the missed home page and restore the integrity of the server.

Note: The default configuration files under /etc/apache2 are not the problem.

This scenario is based on a real server that was "hacked". Ideally you'd recover from infrastrucrure as code playbooks and clean data backups on a new server with the vulnerabilities fixed. Instead, in this exercise you are asked to clean manually the compromised server, restore it to a working condition and ideally, find how the server was broken into. The solution test only checks that the web service is working.

Test: curl localhost must return SadServer - Anatolia

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.

Scenario: "Sapporo": ephemeral tokens

Level: Hard

Type: Fix

Access: Paid

Description: The Sapporo gate API on this host should answer on http://127.0.0.1:9180/ with a body containing OK.

A background service writes short-lived tokens to /var/lib/sapporo/pulse (each value is visible for only a fraction of a second, then the file is cleared again). The gate compares /home/admin/sapporo/active-token against the latest emitted token.

The installed collector at /home/admin/sapporo-collector.sh (triggered by sapporo-collector.timer once per minute) never keeps up; active-token stays empty or stale and the gate keeps failing.

Fix collection so the current token is captured reliably and the gate returns OK.

Test: curl http://127.0.0.1:9180/ returns a response whose body contains OK, and /home/admin/sapporo/active-token holds a token matching the current pulse (format SAPPORO- followed by eight hex digits).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.