Realistic-interviews Troubleshooting Scenarios

Scenario: "Saskatoon": counting IPs.

Level: Easy

Type: Do

Access: Email

Description: There's a web server access log file at /home/admin/access.log. The file consists of one line per HTTP request, with the requester's IP address at the beginning of each line (first column).

Find what's the IP address that has the most requests in this file (there's no tie; the IP is unique). Write the solution into a file /home/admin/highestip.txt. For example, if your solution is "1.2.3.4", you can do echo "1.2.3.4" > /home/admin/highestip.txt

NOTE: The solution IP shows 482 times, ie grep -c -F -f highestip.txt access.log returns 482, if your solution has a different (lower) number you got the wrong most common IP.

Test: The SHA1 checksum of the IP address sha1sum /home/admin/highestip.txt is 6ef426c40652babc0d081d438b9f353709008e93 (just a way to verify the solution without giving it away, we also accept the right IP with no ending newline in the file.)

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Bilbao": Basic Kubernetes Problems

Level: Easy

Type: Fix

Access: Email

Description: There's a Kubernetes Deployment with an Nginx pod and a Load Balancer declared in the manifest.yml file. The pod is not coming up. Fix it so that you can access the Nginx container through the Load Balancer.

There's no "sudo" (root) access.

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: Running curl 10.43.216.196 returns the default Nginx Welcome page.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: Linux Server Review - Guided Learning

Level: Easy

Type: Do

Access: Email

Description: This is a guided learning scenario. Follow this Linux Server Review Scenario Guide

The purpose of this scenario is to review a Linux server and be able to answer questions like:

• What's the purpose of the server?
• What's the hardware (CPU / RAM / disk / net) utilization of the server? is there a problem there?
• What is running and what's going on in the server?

Test: (there's no test)

Time to Solve: 30 minutes.

Scenario: "Hamburg": Find the AWS EC2 volume

Level: Easy

Type: Do

Access: Paid

Description: We have a lot of AWS EBS volumes, the description of which we have save to a file with: aws ec2 describe-volumes > aws-volumes.json.
One of the volumes contains important data and we need to identify which volume (its ID), but we only remember these characteristics: gp3, created before 30/09/2025 , Size < 64 , Iops < 1500, Throughput > 300.

Find the correct volume and put its InstanceId into the ~/mysolution file, e.g.: echo "i-00000000000000000" > ~/mysolution

Test: Running md5sum /home/admin/mysolution returns e7e34463823bf7e39358bf6bb24336d8 (we also accept the file without a new line at the end).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 60 minutes.

Scenario: "Bologna": counting ELB 5xx errors

Level: Easy

Type: Do

Access: Paid

Description: Operations handed you a classic AWS Elastic Load Balancer access log at /home/admin/elb.log. Each line is one request. Fields are space-separated; the quoted HTTP request starts at field 12, so the numeric fields before it are fixed-width columns.

Field 8 is the ELB status code and field 9 is the backend status code returned by the target instance. Count how many log lines have a backend status code in the 5xx range (500 through 599). Write that integer — digits only — to /home/admin/solution.txt. For example: echo 42 > ~/solution.txt

The log mixes successful responses, redirects, client errors, and server errors; only backend 5xx responses count toward your answer.

Test: The MD5 checksum of your answer file md5sum /home/admin/solution.txt is b73ce398c39f506af761d2277d853a92 (we also accept the correct count with a trailing newline in the file).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Genova": cgroups problem

Level: Easy

Type: Fix

Access: Email

Description: This small VM runs sad-api (a lightweight health endpoint on port 9090) and sad-batch (a nightly ETL-style job that allocates a lot of RAM).

After a recent deploy, starting sad-batch caused memory use to spike and sad-api was killed by the OOM killer. On-call stopped the batch service before handing you the host.

A legacy cgroup v2 launcher under /opt/sad/ is supposed to enforce a 128M hard limit on cgroup sad-batch, but the cap never applies.

sad-batch is intentionally stopped and disabled when you log in. Read /home/admin/incident-notes.txt for context. Fix the cgroup configuration so /sys/fs/cgroup/sad-batch/memory.max is 134217728 before you start the batch job again.

Do not change sad-api; it should keep running on 127.0.0.1:9090.

Test: sad-api is active and curl http://127.0.0.1:9090/ returns SadServers - API OK.

The cgroup v2 hard limit is in place: cat /sys/fs/cgroup/sad-batch/memory.max prints 134217728 (128 MiB).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Bergen": Port already in use

Level: Easy

Type: Fix

Access: Email

Description: There's an application at /home/admin/standalone that needs to run successfully but currently it fails.

Fix the environment so the binary can run without errors, without changing the binary itself, and without breaking the web app served on port :80.

Test: Running /home/admin/standalone prints OK and curl http://localhost:80 returns hello SadServers.


The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Istanbul": Delete unattached AWS EBS volumes

Level: Easy

Type: Do

Access: Public

Description: Finance found orphaned EBS volumes racking up cost in this account. Your job is to find every volume that is not attached to an EC2 instance and delete it.

Do not delete volumes that are attached (in-use). Do not stop or terminate instances.

The AWS CLI is already configured against the local endpoint. Start with:
aws ec2 describe-volumes

NOTE: this scenario was done using Floci, a local AWS emulator; there are no real AWS resources.

Test: Delete every EBS volume that is not attached to an EC2 instance. Attached (in-use) volumes and running instances must remain.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Manhattan": can't write data into database.

Level: Medium

Type: Fix

Access: Public

Description: Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don't need to know details about it (although it may help).

Helpful Postgres information: it's a service that listens to a port (:5432) and writes to disk in a data directory, the location of which is defined in the data_directory parameter of the configuration file /etc/postgresql/14/main/postgresql.conf. In our case Postgres is managed by systemd as a unit with name postgresql.

Test: (from default admin user) sudo -u postgres psql -c "insert into persons(name) values ('jane smith');" -d dt

Should return:INSERT 0 1

Time to Solve: 20 minutes.

Scenario: "Tokyo": can't serve web file

Level: Medium

Type: Fix

Access: Email

Description: There's a web server serving a file /var/www/html/index.html with content "hello sadserver" but when we try to check it locally with an HTTP client like curl 127.0.0.1:80, nothing is returned. This scenario is not about the particular web server configuration and you only need to have general knowledge about how web servers work.

Test: curl 127.0.0.1:80 should return: hello sadserver

Time to Solve: 15 minutes.

Scenario: "Cape Town": Borked Nginx

Level: Medium

Type: Fix

Access: Email

Description: There's an Nginx web server installed and managed by systemd. Running curl -I 127.0.0.1:80 returns curl: (7) Failed to connect to localhost port 80: Connection refused , fix it so when you curl you get the default Nginx page.

Test: curl -Is 127.0.0.1:80|head -1 returns HTTP/1.1 200 OK

Time to Solve: 15 minutes.

Scenario: "Melbourne": WSGI with Gunicorn

Level: Medium

Type: Fix

Access: Paid

Description: There is a Python WSGI web application at /home/admin/wsgi.py that should serve the string Hello, world!. It is run by Gunicorn, fronted by nginx (both under systemd). Request flow: client (curl) → nginx (:80) → Gunicorn (Unix domain socket) → wsgi.py.

Gunicorn is already configured to listen on the Unix socket file /run/gunicorn/gunicorn.sock (see /etc/systemd/system/gunicorn.service). Nginx must proxy to that same filesystem path.

Make curl to localhost on port 80 return Hello, world! using this nginx → Gunicorn → WSGI setup.

Test: curl -s http://localhost returns Hello, world! (served via Gunicorn and nginx; both services must be running).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.

Scenario: "Woluwe": Too many images

Level: Medium

Type: Fix

Access: Paid

Description: A pipeline created a lot of Docker images locally for a web app. All these images except for one contain a typo introduced by a developer: there's an incorrect image instruction to pipe "HelloWorld" to "index.htmlz" instead of using the correct "index.html"
Find which image doesn't have the typo (and uses the correct "index.html"), tag this correct image as "prod" (rather than fixing the current prod image) and then deploy it with docker run -d --name prod -p 3000:3000 prod so it responds correctly to HTTP requests on port :3000 instead of "404 Not Found".

Test: curl http://localhost:3000 should respond with HelloWorld;529

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Torino": Optimize grande Docker image

Level: Medium

Type: Do

Access: Paid

Description: A Torino Node.js application is located in the ~/torino-app directory.
You can run it directly with: nohup node app.js > app.log 2>&1 &. You can also verify that it works by running: curl localhost:3000

There is already a torino Docker image built with the Dockerfile in ~/torino-app, but the resulting image size is 916 MB.

Your task is to optimize the Docker image size:
1. Build a new Docker image for the Torino application, also called torino:latest but with a total size under 122 MB
2. Create and run a container using this optimized image.

NOTE: You can only use the existing Docker images in the server.
To build a Node application you need to COPY in your Dockerfile, besides the app.js , the package*.json files and without Internet access, the node_modules directory, since you cannot RUN npm install.

Test: The torino Docker image is less than 122 MB and curl http://localhost:3000 returns Hello from Torino!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "San Juan": mucho Traefik

Level: Medium

Type: Fix

Access: Email

Description: There is a Traefik load balancer that must be up and running. The server and the backend services are managed by Docker Compose. Running curl -s app.sadserver | head -n1 must return the host ID of one of the backend servers, running the command again must return a new host ID. The server seems to be working some times, some others fails or just times out.

The round-robin configuration should make the webserver iterate through the back-end servers.

Test: curl -s app.sadserver | head -n1 returns something like Hostname:

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Suzhou": MongoDB replicas!

Level: Medium

Type: Fix

Access: Email

Description: A new MongoDB replica set has been setup in the development environment trough /home/admin/app/rs0.js, however, a variety or errors are showing up when trying to bring it up. You should bring up all the replica servers, get them communicating to each other and make sure the replica set is working as it should.

The status of the first replica can be checked via systemctl status mongo1 same for the replicas mongo2 and mongo3. The logs are also in a separate file for each replica under the directory /var/log/mongodb. To initilize the replica set again: mongosh --file app/rs0.js

Note: The default configuration file /etc/mongo.conf is not the problem.

Test: mongosh --eval "rs.status()" | grep health returns the status of all the replicas

       health: 1,       health: 1,       health: 1, 

Time to Solve: 20 minutes.

Scenario: "Lyon": Migrate Ingress-NGINX to Traefik

Level: Medium

Type: Do

Access: Paid

Description: Ingress-NGINX is being retired. As the DevOps Engineer, you will replace it with Traefik on the production Kubernetes cluster in a private VPC. This scenario is a local proof-of-concept for that migration.

The current K8s cluster has a "Hello World" pod running, i.e.: curl hello.lyon.local returns "Hello world" (see note 1). You should be able to see the same content delivered via Traefik once the ingress-nginx is down.

Notes: 1: Wait at the start until k8s is fully up before doing curl, otherwise you get 503, you can check for ex with k get pod -n ingress-nginx
2: The k8s manifests are under the ~/app dir.
3: ingress-nginx was deployed with a Helm chart.
4: The Helm chart for traefik is available under /home/admin/traefik (The Traefik image is already loaded in k3s).
5: Traefik dashboard and probes/metrics port by default is :8080 but that's used by the system; use a different port or disable.
6: The domain hello.lyon.local is actually pointing to the localhost.
7: The ingress must be listening on port 80 for any IP so it can respond to localhost:80 or actually to *:80

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: When the command curl -i hello.lyon.local is executed, it returns the message Hello World, while only the traefik pod must be present (instead of ingress-nginx).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.

Scenario: "Stockholm": DNS health check issue

Level: Medium

Type: Fix

Access: Paid

Description: The internal status portal on this host should answer on http://127.0.0.1:9167/ with a body containing OK.

It worked until operations ran a package cleanup.

The portal service (stockholm-portal) only runs after a DNS health check at /usr/local/bin/stockholm-dns-check.sh succeeds.

Make the necessary changes so the portal works again.

Do not modify /usr/local/bin/stockholm-dns-check.sh.

Test: The health script /usr/local/bin/stockholm-dns-check.sh runs successfully, stockholm-portal is active, and curl http://127.0.0.1:9167/ returns a response whose body contains OK.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Cordoba": df is lying (or is it du?)

Level: Medium

Type: Fix

Access: Email

Description: Monitoring reports that the root filesystem is under pressure, but a quick du of /var/log shows almost nothing in the logs of the running application at /var/log/cordoba-app.

Find what is holding the space and reclaim it so df and du agree again for practical purposes; currently there's a ~300 MB discrepancy on the root partition /

The service unit is cordoba-hoarder.service.

Test: df -h / and sudo du -sh / report the same used space after reclaiming the ~300 MB discrepancy.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Tallinn": BuildKit & Docker build mismatch

Level: Medium

Type: Fix

Access: Paid

Description: This VM runs a tiny container app, tallinn-service, whose only job is to print an API version string (for example tallinn-api-version=1.4.0). The image is built from /home/admin/tallinn-app with docker build.

The dev team raised the API contract to 2.0.0 in src/api_version.txt and ran a new build, but QA still rejects the image tagged tallinn-app:current: it reports 1.4.0 at runtime. A recent CI log is in /home/admin/build.log.

Fix the docker build outcome so the deploy image matches what the sources ask for.

Fix the image tagged tallinn-app:current so the on-disk contract file and the shipped binary both report API 2.0.0.

Test: Image tallinn-app:current exists, /etc/tallinn/api_version is 2.0.0, and /usr/local/bin/tallinn-service prints tallinn-api-version=2.0.0.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Verona": Apache Portal Won't Open

Level: Medium

Type: Fix

Access: Email

Description: An internal Apache portal was migrated to this host. The document root is /var/www/portal.

The site root at http://localhost/ does not serve the expected homepage, and a legacy bookmark at /reports no longer reaches the status page (direct access to /status/ works).

Find and fix what keeps the portal root and the legacy redirect from working. Adding missing content is allowed.

Test: Ready.

curl -L http://localhost/reports returns a first line of SadServers - Status OK.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Modena": Ansible Deploy Won't Publish

Level: Medium

Type: Fix

Access: Paid

Description: This host publishes an internal status page by running Ansible locally against the Docker container status-app (port 8888 on localhost maps to the container's HTTP port).

The playbook tree lives in /home/admin/deploy/. After a refactor, ansible-playbook site.yml no longer leaves a working status endpoint — curl http://localhost:8888/ does not return the expected line.

Fix the Ansible project and run the playbook successfully so the status page is served from the container.

Test: curl http://localhost:8888/ returns a first line of SadServers - Modena OK.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Parma": Debugging Terraform Issues

Level: Medium

Type: Fix

Access: Email

Description: This host publishes a machine-readable status marker using Terraform with a local backend. The project lives in /home/admin/infra/ and should write /var/local/platform-status.txt.

After a refactor, terraform plan and terraform apply no longer succeed, and the status file is missing or stale.

Fix the Terraform project and apply it so the marker is published again.

(Note: Internet access is not needed).

Test: The first line of /var/local/platform-status.txt is SadServers - Parma OK.

Running terraform plan in /home/admin/infra/ reports no changes pending (clean state).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Ravenna": Logs Missing in ELK Pipeline

Level: Medium

Type: Fix

Access: Paid

Description: You are on call for the orders-api service. Central logging uses a small ELK stack on Docker Compose: an application container, Filebeat, Logstash, and Elasticsearch.

Operations reports that no order events show up in Elasticsearch, even though the application container is healthy and keeps writing logs. SRE left notes that the service contract specifies plain-text log lines.

The stack lives under /home/admin/ravenna and is managed with Docker Compose. Elasticsearch is reachable on the VM at http://127.0.0.1:9200.

Notes: 1. Wait until all four containers are Up before debugging (docker compose -f /home/admin/ravenna/docker-compose.yml ps). Elasticsearch can take up to two minutes to become healthy.
2. Internet access is not needed; container images are preloaded in the local Docker engine.

Test: At least one document containing order_shipped is indexed in Elasticsearch under the orders-* index pattern.

Quick check:

 curl -s 'http://127.0.0.1:9200/orders-*/_search?q=order_shipped&size=1' | jq . 

Time to Solve: 30 minutes.

Scenario: "Luxor": PostgreSQL analytics queries crawl

Level: Medium

Type: Fix

Access: Email

Description: Our analytics API serves customer sales counts from a PostgreSQL database. Requests to the customer lookup endpoint take several seconds and often time out, even though the database server looks healthy — no obvious CPU, memory, or disk exhaustion.

The API runs as systemd service sad-analytics-api on port 9090. Application code lives under /opt/sad/. Database credentials for debugging: connect as saduser to database analytics_db (password: sadpassword). The main table is sales_data.

Find why customer lookups are slow and restore acceptable API response times.

Test: This API request must complete quickly (total response time under 500 ms):
curl -s -w "\nTotal time: %{time_total}s\n" http://127.0.0.1:9090/api/customers/1234/sales/count

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Banha" : Intermittent 504 Gateway Time-Out errors

Level: Medium

Type: Fix

Access: Paid

Description: Our Python Flask web application, served via Gunicorn and Nginx, is experiencing intermittent 504 Gateway Time-Out errors on concurrent requests. Users report that the application works perfectly at times, but fails unexpectedly at others. Your task is to identify the root cause of these sporadic failures and implement a permanent fix to ensure the applications consistent stability on port 80.

Test: curl http://localhost/ returns "Hello from Flask!" consistently, even on parallel load: seq 15 | xargs -n1 -P15 curl -sf --max-time 5 http://localhost/ && echo OK || echo FAIL.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Aswan": K8s Service Connectivity Failure

Level: Medium

Type: Fix

Access: Paid

Description: Our microservices application on Kubernetes cannot reach the backend-db service from the frontend deployment; curl from the frontend pod fails.

Your task is to diagnose and fix the connectivity problems so the frontend can reach the backend database. There may be more than one issue.

NOTE: wait for all the pods to be running at the beginning of the exercise.

Test: Curl from inside the frontend pod to the backend-db service returns "Hello from Backend DB!": FRONTEND_POD=$(kubectl get pods -l app=frontend -o jsonpath='{.items[0].metadata.name}')
kubectl exec $FRONTEND_POD -- curl -s backend-db returns "Hello from Backend DB!"

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Spa": The Docker Compose healthcheck that does not heal

Level: Medium

Type: Fix

Access: Paid

Description: The thermal baths booking API should answer on port :5000. After last week's outage the previous engineer added a Docker HEALTHCHECK and restart: unless-stopped so the stack would recycle itself if the probe failed. The front desk is still down.

The Compose project is in /home/admin/spa. There is a short handover note in that directory.

The booking API application itself is fine when it is running: if you restart the container, GET /health comes back. You do not need to rewrite or "fix" the API service code. The real gap is recovery — when the API process inside the container dies again, nothing brings the service back on its own. Make the stack recover automatically from that failure so /health returns {"status":"ok"} without a manual restart.

The process will not usually die on its own during the exercise. To replay the incident, stop the API worker inside the container (leave the container running): docker exec spa-api sh -c 'kill "$(cat /tmp/spa.pid)"' — before a fix, /health stays down. After you add automatic recovery, the same kill should bring /health back without a manual docker restart. The recovery must still work after a host reboot (do not leave the API dead on boot).

Test: curl -s http://127.0.0.1:5000/health returns {"status":"ok"} and the spa-api container is healthy.
A one-off docker restart is not enough: after the API worker dies again inside the container, the service must recover on its own.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Kyoto": The Gion ticket API will not start

Level: Medium

Type: Fix

Access: Email

Description: The Gion ticket office API should answer on port :80. After last night's change the office never came back.

Manifests are in /home/admin/app. There is a short handover note in /home/admin/HANDOVER.txt. Wait until the Kubernetes node is Ready after boot.

GET /health must return {"status":"ok","office":"gion-tickets"}. Do not move the API off port 80.

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: curl -s http://127.0.0.1/health returns {"status":"ok","office":"gion-tickets"} and the gion-api pod in the gion namespace is Ready (1/1).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Dakar": The half-installed package

Level: Medium

Type: Fix

Access: Email

Description: The checkout platform's local health agent stopped working after a package upgrade was interrupted. The checkout-agent binary may already be on disk, but its systemd service is not healthy.

Restore the package and service so the agent runs again and still comes back after a reboot. Do not wipe the package database under /var/lib/dpkg, and do not replace the service with a hand-written unit of your own.

Test: checkout-agent is install ok installed; checkout-agent.service is active and enabled; its unit under /lib/systemd/system/ is owned by the package.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Auckland": S3 delete & archive old backups

Level: Medium

Type: Do

Access: Email

Description: Ops left years of database dumps in s3://auckland-backups. Keys use an absolute date prefix:
backups/YYYY-MM-DD/...

Apply this retention policy (use the date in the key, ignore LastModified):

• Before 2023-01-01 — delete those objects
• From 2023-01-01 through 2024-12-31 — keep them but set storage class to GLACIER
• On or after 2025-01-01 — leave as Standard
• Keys without that date prefix — leave untouched

Test: In s3://auckland-backups: objects with key date < 2023-01-01 are gone; dates 2023-01-01.. 2024-12-31 are GLACIER (or deep archive); dates ≥ 2025-01-01 and non-dated keys remain Standard.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Cork": The catalog won't open

Level: Medium

Type: Fix

Access: Email

Description: After last week's host reboot the product catalog Postgres database will not stay up. Inserts into the catalog database fail. Ops moved application data off the root disk onto a dedicated volume some time ago; root itself still looks healthy.

You do not need deep Postgres expertise. The server listens on :5432 when healthy. The data directory is set in /etc/postgresql/*/main/ (see data_directory). systemd manages the unit named postgresql.

Test: As admin: sudo -u postgres psql -d catalog -c "INSERT INTO products(name) VALUES ('jane smith');"

Should return: INSERT 0 1

The fix must still work after a reboot.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Karakorum": WTFIT – What The Fun Is This?

Level: Hard

Type: Fix

Access: Paid

Description: (NOTE: this is not a new scenario but an existing Pro one temporarily available to all users as the last Advent of SysAdmin 2025 scenario).

There's a binary at /home/admin/wtfit that nobody knows how it works or what it does ("what the fun is this"). Someone remembers something about wtfit needing to communicate to a service in order to start.

Run this wtfit program so it doesn't exit with an error, fixing or working around things that you need but are broken in this server.

Test: Running /home/admin/wtfit returns OK.

Time to Solve: 40 minutes.

Scenario: "Anatolia": compromised server

Level: Hard

Type: Fix

Access: Paid

Description: This web server has been compromised and is not serving the home page anymore, those troubleshooting skills you have as DevOps are urgently needed to solve the mystery of the missed home page and restore the integrity of the server.

Note: The default configuration files under /etc/apache2 are not the problem.

This scenario is based on a real server that was "hacked". Ideally you'd recover from infrastrucrure as code playbooks and clean data backups on a new server with the vulnerabilities fixed. Instead, in this exercise you are asked to clean manually the compromised server, restore it to a working condition and ideally, find how the server was broken into. The solution test only checks that the web service is working.

Test: curl localhost must return SadServer - Anatolia

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.

Scenario: "Sapporo": ephemeral tokens

Level: Hard

Type: Fix

Access: Paid

Description: The Sapporo gate API on this host should answer on http://127.0.0.1:9180/ with a body containing OK.

A background service writes short-lived tokens to /var/lib/sapporo/pulse (each value is visible for only a fraction of a second, then the file is cleared again). The gate compares /home/admin/sapporo/active-token against the latest emitted token.

The installed collector at /home/admin/sapporo-collector.sh (triggered by sapporo-collector.timer once per minute) never keeps up; active-token stays empty or stale and the gate keeps failing.

Fix collection so the current token is captured reliably and the gate returns OK.

Test: curl http://127.0.0.1:9180/ returns a response whose body contains OK, and /home/admin/sapporo/active-token holds a token matching the current pulse (format SAPPORO- followed by eight hex digits).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.

Scenario: "Dublin": registration queue overload (HTTP 429)

Level: Hard

Type: Fix

Access: Paid

Description: Open enrollment is live and the student registration portal is rejecting traffic.

The app runs on this VM (no containers): Redis-backed background jobs, systemd workers, manual deploy. During the spike the portal at localhost:5000 returns HTTP 429 on POST /register e.g. curl -i -X POST http://127.0.0.1:5000/register (header and payload not needed to simplify).

Services live under systemd as registration-app, registration-worker@N, and registration-loadgen (the load generator simulates ongoing external enrollment traffic — treat it like real users arriving; do not "fix" the outage by turning that traffic off). Redis queue name: registration:jobs. Useful endpoints on port 5000:

GET  /          portal home
POST /register  enqueue a registration (429 when overloaded)
GET  /metrics   Redis queue lengths (queue_jobs, queue_failed, ...)

Test: registration-loadgen stays active, curl -s http://127.0.0.1:5000/metrics shows queue_jobs below 50, and curl -s -o /dev/null -w '%{http_code}\n' -X POST http://127.0.0.1:5000/register returns 202 (not 429) repeatedly.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute.

Time to Solve: 40 minutes.