SadServers Linux & DevOps Troubleshooting Scenarios

Scenario: "Bergen": Port already in use

Level: Easy

Type: Fix

Access: Email

Description: There's an application at /home/admin/standalone that needs to run successfully but currently it fails.

Fix the environment so the binary can run without errors, without changing the binary itself, and without breaking the web app served on port :80.

Test: Running /home/admin/standalone prints OK and curl http://localhost:80 returns hello SadServers.


The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Philadelphia": htop metrics disagree

Level: Medium

Type: Fix

Access: Email

Description: This host exports load and uptime to /var/lib/sad/sysinfo.txt for a small internal dashboard. htop and btop are installed for troubleshooting.

On-call reports the dashboard shows load stuck at 0.00 and an old uptime, while the live process viewers look correct. A leftover lab service leaves a defunct (zombie) child in the process list.

Read /home/admin/incident-notes.txt. Fix the exporter and the zombie lab so published metrics match the real kernel counters and no zombie children remain under the lab parent.

Test: The 1st line of /var/lib/sad/sysinfo.txt is SadServers - Philadelphia OK. The load_1m and uptime_seconds values in that file match /proc/loadavg and /proc/uptime (within a small tolerance).
sad-zombie-lab is active and has no zombie children.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Luxor": PostgreSQL analytics queries crawl

Level: Medium

Type: Fix

Access: Email

Description: Our analytics API serves customer sales counts from a PostgreSQL database. Requests to the customer lookup endpoint take several seconds and often time out, even though the database server looks healthy — no obvious CPU, memory, or disk exhaustion.

The API runs as systemd service sad-analytics-api on port 9090. Application code lives under /opt/sad/. Database credentials for debugging: connect as saduser to database analytics_db (password: sadpassword). The main table is sales_data.

Find why customer lookups are slow and restore acceptable API response times.

Test: This API request must complete quickly (total response time under 500 ms):
curl -s -w "\nTotal time: %{time_total}s\n" http://127.0.0.1:9090/api/customers/1234/sales/count

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Banha" : Intermittent 504 Gateway Time-Out errors

Level: Medium

Type: Fix

Access: Paid

Description: Our Python Flask web application, served via Gunicorn and Nginx, is experiencing intermittent 504 Gateway Time-Out errors on concurrent requests. Users report that the application works perfectly at times, but fails unexpectedly at others. Your task is to identify the root cause of these sporadic failures and implement a permanent fix to ensure the applications consistent stability on port 80.

Test: curl http://localhost/ returns "Hello from Flask!" consistently, even on parallel load: seq 15 | xargs -n1 -P15 curl -sf --max-time 5 http://localhost/ && echo OK || echo FAIL.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Aswan": K8s Service Connectivity Failure

Level: Medium

Type: Fix

Access: Paid

Description: Our microservices application on Kubernetes cannot reach the backend-db service from the frontend deployment; curl from the frontend pod fails.

Your task is to diagnose and fix the connectivity problems so the frontend can reach the backend database. There may be more than one issue.

NOTE: wait for all the pods to be running at the beginning of the exercise.

Test: Curl from inside the frontend pod to the backend-db service returns "Hello from Backend DB!": FRONTEND_POD=$(kubectl get pods -l app=frontend -o jsonpath='{.items[0].metadata.name}')
kubectl exec $FRONTEND_POD -- curl -s backend-db returns "Hello from Backend DB!"

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Nairn": Phantom Gitea Deploy

Level: Medium

Type: Fix

Access: Email

Description: Every deployment through the Gitea Actions pipeline reports full success: checkout, build, deploy, service reload, and health check all pass with green checkmarks. Despite this, users keep seeing the previous version of the app after every deploy.

~/webapp is a git clone of the Gitea repo. Edit a file there, commit, and git push to trigger the Actions pipeline. Then figure out why the running webapp service never actually serves the version it just deployed — and fix it so a successful pipeline run always results in that version being served. For example:

cd ~/webapp && echo "v2" > index.html && git commit -am "bump" && git push

Test: The file /srv/app/current/index.html on the host and the content served by the webapp at http://localhost:80/ must match.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute.

Time to Solve: 20 minutes.

Scenario: "Vienna": The Vanishing Process

Level: Medium

Type: Fix

Access: Email

Description: Our monitoring agent monagent keeps "crashing" — or at least that's what it looks like. Every few minutes it seems to disappear, but:

- systemctl status monagent always reports it as active (running)
- journalctl -u monagent shows no errors, panics, or OOM kills
- CPU and memory usage are completely normal
- Restarting the service "fixes" it, but only for a few minutes

Find out what is actually happening and correct /usr/local/bin/monagent-watchdog.sh so it tracks the running process correctly, without disabling the watchdog..

Test: The watchdog unit is still active and, after a monagent restart, it does not kill/restart the newly started worker due to stale PID tracking.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute.

Time to Solve: 15 minutes.