SadServers Linux & DevOps Troubleshooting Scenarios

Scenario: "Bergen": Port already in use

Level: Easy

Type: Fix

Access: Email

Description: There's an application at /home/admin/standalone that needs to run successfully but currently it fails.

Fix the environment so the binary can run without errors, without changing the binary itself, and without breaking the web app served on port :80.

Test: Running /home/admin/standalone prints OK and curl http://localhost:80 returns hello SadServers.


The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Nara": No ls on this system

Level: Easy

Type: Do

Access: Email

Description: Common directory-listing tools on this host are missing or refuse to run. A document named shosoin.tag was misplaced somewhere under /home/admin/records.

Write its full absolute path (one line) to /home/admin/solution.txt, for example: echo "/home/admin/records/some/dir/shosoin.tag" > /home/admin/solution.txt

NOTE: There are at least 9 different ways to find the solution in this server (shown in the clues).

Test: md5sum /home/admin/solution.txt returns 8d3b739ebccb41c7c39e608d7a3e0bd6 (the solution without the trailing newline is also accepted).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Nairn": Phantom Gitea Deploy

Level: Medium

Type: Fix

Access: Email

Description: Every deployment through the Gitea Actions pipeline reports full success: checkout, build, deploy, service reload, and health check all pass with green checkmarks. Despite this, users keep seeing the previous version of the app after every deploy.

~/webapp is a git clone of the Gitea repo. Edit a file there, commit, and git push to trigger the Actions pipeline. Then figure out why the running webapp service never actually serves the version it just deployed — and fix it so a successful pipeline run always results in that version being served. For example:

cd ~/webapp && echo "v2" > index.html && git commit -am "bump" && git push

Test: The file /srv/app/current/index.html on the host and the content served by the webapp at http://localhost:80/ must match.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute.

Time to Solve: 20 minutes.

Scenario: "Vienna": The Vanishing Process

Level: Medium

Type: Fix

Access: Email

Description: Our monitoring agent monagent keeps "crashing" — or at least that's what it looks like. Every few minutes it seems to disappear, but:

- systemctl status monagent always reports it as active (running)
- journalctl -u monagent shows no errors, panics, or OOM kills
- CPU and memory usage are completely normal
- Restarting the service "fixes" it, but only for a few minutes

Find out what is actually happening and correct /usr/local/bin/monagent-watchdog.sh so it tracks the running process correctly, without disabling the watchdog..

Test: The watchdog unit is still active and, after a monagent restart, it does not kill/restart the newly started worker due to stale PID tracking.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute.

Time to Solve: 15 minutes.

Scenario: "Hangzhou": The web server one-second delay that never happens

Level: Medium

Type: Fix

Access: Email

Description: West Lake Quotes is a staging market-data gateway. GET /v1/quote on port :80 should wait about one second and then return the matching engine payload. The engine itself listens on 127.0.0.1:5000 and must stay fast.

After last Thursday's nginx change the JSON is still correct, but the courtesy delay never happens — quotes come back in a few milliseconds. The mobile QA build depends on that wait (spinner and retry logic).

There is a short handover note in /home/admin/HANDOVER.txt. Do not take the gateway off port 80, and do not add latency to the engine.

Test: curl -s http://127.0.0.1/v1/quote returns the engine JSON (fields westlake and WLTEA) and the request takes roughly one second.

curl -s http://127.0.0.1:5000/v1/quote still returns the same payload immediately.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Spa": The Docker Compose healthcheck that does not heal

Level: Medium

Type: Fix

Access: Email

Description: The thermal baths booking API should answer on port :5000. After last week's outage the previous engineer added a Docker HEALTHCHECK and restart: unless-stopped so the stack would recycle itself if the probe failed. The front desk is still down.

The Compose project is in /home/admin/spa. There is a short handover note in that directory.

The booking API application itself is fine when it is running: if you restart the container, GET /health comes back. You do not need to rewrite or "fix" the API service code. The real gap is recovery — when the API process inside the container dies again, nothing brings the service back on its own. Make the stack recover automatically from that failure so /health returns {"status":"ok"} without a manual restart.

The process will not usually die on its own during the exercise. To replay the incident, stop the API worker inside the container (leave the container running): docker exec spa-api sh -c 'kill "$(cat /tmp/spa.pid)"' — before a fix, /health stays down. After you add automatic recovery, the same kill should bring /health back without a manual docker restart. The recovery must still work after a host reboot (do not leave the API dead on boot).

Test: curl -s http://127.0.0.1:5000/health returns {"status":"ok"} and the spa-api container is healthy.
A one-off docker restart is not enough: after the API worker dies again inside the container, the service must recover on its own.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.