SadServers Linux & DevOps Troubleshooting Scenarios

Scenario: "Nara": No ls on this system

Level: Easy

Type: Do

Access: Email

Description: Common directory-listing tools on this host are missing or refuse to run. A document named shosoin.tag was misplaced somewhere under /home/admin/records.

Write its full absolute path (one line) to /home/admin/solution.txt, for example: echo "/home/admin/records/some/dir/shosoin.tag" > /home/admin/solution.txt

NOTE: There are at least 9 different ways to find the solution in this server (shown in the clues).

Test: md5sum /home/admin/solution.txt returns 8d3b739ebccb41c7c39e608d7a3e0bd6 (the solution without the trailing newline is also accepted).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Hangzhou": The web server one-second delay that never happens

Level: Medium

Type: Fix

Access: Email

Description: West Lake Quotes is a staging market-data gateway. GET /v1/quote on port :80 should wait about one second and then return the matching engine payload. The engine itself listens on 127.0.0.1:5000 and must stay fast.

After last Thursday's nginx change the JSON is still correct, but the courtesy delay never happens — quotes come back in a few milliseconds. The mobile QA build depends on that wait (spinner and retry logic).

There is a short handover note in /home/admin/HANDOVER.txt. Do not take the gateway off port 80, and do not add latency to the engine.

Test: curl -s http://127.0.0.1/v1/quote returns the engine JSON (fields westlake and WLTEA) and the request takes roughly one second.

curl -s http://127.0.0.1:5000/v1/quote still returns the same payload immediately.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Spa": The Docker Compose healthcheck that does not heal

Level: Medium

Type: Fix

Access: Paid

Description: The thermal baths booking API should answer on port :5000. After last week's outage the previous engineer added a Docker HEALTHCHECK and restart: unless-stopped so the stack would recycle itself if the probe failed. The front desk is still down.

The Compose project is in /home/admin/spa. There is a short handover note in that directory.

The booking API application itself is fine when it is running: if you restart the container, GET /health comes back. You do not need to rewrite or "fix" the API service code. The real gap is recovery — when the API process inside the container dies again, nothing brings the service back on its own. Make the stack recover automatically from that failure so /health returns {"status":"ok"} without a manual restart.

The process will not usually die on its own during the exercise. To replay the incident, stop the API worker inside the container (leave the container running): docker exec spa-api sh -c 'kill "$(cat /tmp/spa.pid)"' — before a fix, /health stays down. After you add automatic recovery, the same kill should bring /health back without a manual docker restart. The recovery must still work after a host reboot (do not leave the API dead on boot).

Test: curl -s http://127.0.0.1:5000/health returns {"status":"ok"} and the spa-api container is healthy.
A one-off docker restart is not enough: after the API worker dies again inside the container, the service must recover on its own.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Kyoto": The Gion ticket API will not start

Level: Medium

Type: Fix

Access: Email

Description: The Gion ticket office API should answer on port :80. After last night's change the office never came back.

Manifests are in /home/admin/app. There is a short handover note in /home/admin/HANDOVER.txt. Wait until the Kubernetes node is Ready after boot.

GET /health must return {"status":"ok","office":"gion-tickets"}. Do not move the API off port 80.

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: curl -s http://127.0.0.1/health returns {"status":"ok","office":"gion-tickets"} and the gion-api pod in the gion namespace is Ready (1/1).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Dakar": The half-installed package

Level: Medium

Type: Fix

Access: Email

Description: The checkout platform's local health agent stopped working after a package upgrade was interrupted. The checkout-agent binary may already be on disk, but its systemd service is not healthy.

Restore the package and service so the agent runs again and still comes back after a reboot. Do not wipe the package database under /var/lib/dpkg, and do not replace the service with a hand-written unit of your own.

Test: checkout-agent is install ok installed; checkout-agent.service is active and enabled; its unit under /lib/systemd/system/ is owned by the package.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Cadiz": Cut the live wires

Level: Hard

Type: Fix

Access: Paid

Description: Your host is wired into a demolition circuit. cable processes are the wires. They talk only to a root supervisor, fuse, to keep it convinced the circuit is intact. Some cables are live; the others are decoys. deton processes (also root) are the charges. They talk only to fuse, never to the cables. That supervisor is what fires them. You cannot kill deton or fuse. You do not have general root access. You can only cut cables.

To cut a cable, kill -9 it. Cut all the live cables (and only those), and cut them together. Any other cut resets the circuit: fuse fires the deton charges and rolls a new live set.

Test: The charges are gone (deton has exited) and the root-owned stamp /var/lib/fuse/disarmed exists. That stamp is created only by the supervisor when the live wires drop together. Creating a file in your home directory does not count. The stamp remains across reboot.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 40 minutes.