SadServers Linux & DevOps Troubleshooting Scenarios

Linux & Bash

- Linux commands, Bash scripting
- Systemd
- Networking, DNS
- Storage
- SSH, Firewall
- Libraries
- Cron and more...

Web Servers

- Nginx
- Apache
- HAProxy
- Caddy
- Gunicorn
- uWSGI
- HTTPS/TLS

Databases

- PostgreSQL
- MySQL
- SQLite
- Redis
- ClickHouse
- MongoDB
- etcd

Data Processing

- CSV
- JSON
- SQL queries

Docker

- Building images
- Multi-stage builds
- Volumes
- Networks
- Docker Compose
- Podman

Kubernetes

- kubectl
- Helm
- K8S Roles & Permissions
- Services
- Namespaces
- Deployments, StatefulSets
- ConfigMaps, Secrets

Tooling / Applications

- Git
- Rabbitmq
- Envoy
- Vault
- Harbor
- Prometheus
- Jenkins

Hacking

- Capture the Flag (CTF) Challenges
- Code Vulnerabilities
- Privilege Escalation

Languages

- Python
- Golang
- PHP
- Java
- Node.js
- C

advent2025 ai apache bash c caddy clickhouse cron csv data processing disk volumes dns docker envoy etcd ftp git golang gunicorn hack haproxy harbor hashicorp vault helm java jenkins json kubernetes linux-other mongodb mysql nginx node.js php podman postgres prometheus python rabbitmq redis sql sqlite ssh ssl supervisord systemd traefik

realistic / interviews new pro business

Easy

#	Name	Time	Type
1	"Porto": Port audit without net tools	15 m	Do New	"Porto": Port audit without net tools Scenario: "Porto": Port audit without net tools Level: Easy Type: Do Access: Email Description: The security team removed common network recon utilities from this host. Your job is to determine which TCP ports on localhost (`127.0.0.1`) are accepting connections. The ports to check are listed in /home/admin/ports-to-scan.txt (one port per line). Write your results to /home/admin/port-audit.txt with one line per port, sorted by port number (ascending), using this format: PORT STATUS where STATUS is exactly `open` or `closed` (lowercase). A template file /home/admin/port-audit.txt is available with values per port "open\|closed", delete the separator and the incorrect value per port or re-create the file. The following are not available on this system (removed or restricted): `ss`, `netstat`, `nmap`, `nc`, `telnet`, `curl`, `lsof`, `tcpdump`. NOTE: you don't have root (superuser) access. Test: The file /home/admin/port-audit.txt exists and correctly reports whether each port in /home/admin/ports-to-scan.txt is `open` or `closed` on `127.0.0.1`. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.
2	"Edinburgh": FTP catalog sync failure	10 m	Fix New	"Edinburgh": FTP catalog sync failure Scenario: "Edinburgh": FTP catalog sync failure Level: Easy Type: Fix Access: Email Description: The warehouse API on this host should answer on `http://127.0.0.1:9178/` with a body containing `OK`. It depends on a catalog file pulled from the internal FTP mirror at `127.0.0.1`. The sync job at /home/admin/edinburgh-sync.sh is failing; edinburgh-sync is in a failed state and the API never comes up healthy. Fix the sync so the catalog is downloaded and the API works again. Test: /var/lib/edinburgh/catalog.txt exists with the correct inventory data, the service edinburgh-api is active, and `curl http://127.0.0.1:9178/` returns a response whose body contains `OK`. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 10 minutes.

Medium

#	Name	Time	Type
1	"Stockholm": DNS health check issue	20 m	Fix Pro New	"Stockholm": DNS health check issue Scenario: "Stockholm": DNS health check issue Level: Medium Type: Fix Access: Paid Description: The internal status portal on this host should answer on `http://127.0.0.1:9167/` with a body containing `OK`. It worked until operations ran a package cleanup. The portal service (stockholm-portal) only runs after a DNS health check at /usr/local/bin/stockholm-dns-check.sh succeeds. Make the necessary changes so the portal works again. Do not modify /usr/local/bin/stockholm-dns-check.sh. Test: The health script /usr/local/bin/stockholm-dns-check.sh runs successfully, stockholm-portal is active, and `curl http://127.0.0.1:9167/` returns a response whose body contains `OK`. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.
2	"Cordoba": df is lying (or is it du?)	10 m	Fix New	"Cordoba": df is lying (or is it du?) Scenario: "Cordoba": df is lying (or is it du?) Level: Medium Type: Fix Access: Email Description: Monitoring reports that the root filesystem is under pressure, but a quick `du` of /var/log shows almost nothing in the logs of the running application at /var/log/cordoba-app. Find what is holding the space and reclaim it so `df` and `du` agree again for practical purposes; currently there's a ~300 MB discrepancy on the root partition / The service unit is cordoba-hoarder.service. Test: `df -h /` and `sudo du -sh /` report the same used space after reclaiming the ~300 MB discrepancy. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 10 minutes.
3	"Tallinn": BuildKit & Docker build mismatch	30 m	Fix Pro New	"Tallinn": BuildKit & Docker build mismatch Scenario: "Tallinn": BuildKit & Docker build mismatch Level: Medium Type: Fix Access: Paid Description: This VM runs a tiny container app, tallinn-service, whose only job is to print an API version string (for example `tallinn-api-version=1.4.0`). The image is built from /home/admin/tallinn-app with `docker build`. The dev team raised the API contract to 2.0.0 in src/api_version.txt and ran a new build, but QA still rejects the image tagged tallinn-app:current: it reports 1.4.0 at runtime. A recent CI log is in /home/admin/build.log. Fix the `docker build` outcome so the deploy image matches what the sources ask for. Fix the image tagged tallinn-app:current so the on-disk contract file and the shipped binary both report API `2.0.0`. Test: Image tallinn-app:current exists, /etc/tallinn/api_version is `2.0.0`, and `/usr/local/bin/tallinn-service` prints `tallinn-api-version=2.0.0`. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 30 minutes.

Hard

#	Name	Time	Type
1	"Sapporo": ephemeral tokens	40 m	Fix Pro New	"Sapporo": ephemeral tokens Scenario: "Sapporo": ephemeral tokens Level: Hard Type: Fix Access: Paid Description: The Sapporo gate API on this host should answer on `http://127.0.0.1:9180/` with a body containing `OK`. A background service writes short-lived tokens to /var/lib/sapporo/pulse (each value is visible for only a fraction of a second, then the file is cleared again). The gate compares /home/admin/sapporo/active-token against the latest emitted token. The installed collector at /home/admin/sapporo-collector.sh (triggered by sapporo-collector.timer once per minute) never keeps up; active-token stays empty or stale and the gate keeps failing. Fix collection so the current token is captured reliably and the gate returns `OK`. Test: `curl http://127.0.0.1:9180/` returns a response whose body contains `OK`, and /home/admin/sapporo/active-token holds a token matching the current pulse (format `SAPPORO-` followed by eight hex digits). The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 40 minutes.

Kubernetes Playgrounds

#	Name	Time	Type
1	K8s Playground - Free	20 m	Playground New	K8s Playground - Free Playground: K8s Playground - Free Level: Easy Type: Playground Access: Email Description: This is a Kubernetes sandbox for you to play with and experiment. It comes with an nginx.yaml playbook. You can try for example `k apply -f nginx.yaml` (you can use "k" as an alias for "kubectl".) The Helm binary is also installed. Free account: The free account sandbox runs on a 1 GB of RAM VM. As usual, there is no Internet access. Paid accounts (Pro/Pro+/Business: The playground in this case runs on a 2 GB of RAM VM. It has Internet access (to pull your own images for example) and twice the time. Time to Play: 20 minutes.
2	K8s Playground - Pro	60 m	Playground Pro New	K8s Playground - Pro Playground: K8s Playground - Pro Level: Easy Type: Playground Access: Paid Description: This is a Kubernetes sandbox for you to play with and experiment. It comes with an nginx.yaml playbook. You can try for example `k apply -f nginx.yaml` (you can use "k" as an alias for "kubectl".) The Helm binary is also installed. Free account: The free account sandbox runs on a 1 GB of RAM VM. As usual, there is no Internet access. Paid accounts (Pro/Pro+/Business: The playground in this case runs on a 2 GB of RAM VM. It has Internet access (to pull your own images for example) and twice the time. Time to Play: 60 minutes.

SadServers Linux & DevOps Troubleshooting Scenarios

Linux & Bash

Web Servers

Databases

Data Processing

Docker

Kubernetes

Tooling / Applications

Hacking

Languages

Easy

"Porto": Port audit without net tools

"Edinburgh": FTP catalog sync failure

Medium

"Stockholm": DNS health check issue

"Cordoba": df is lying (or is it du?)

"Tallinn": BuildKit & Docker build mismatch

Hard

"Sapporo": ephemeral tokens

Kubernetes Playgrounds

Send Us Feedback

Get Notified