(Beta for registered users; some features still in progress)

SadServers Linux & DevOps Troubleshooting Scenarios

Linux & Bash

- Linux commands, Bash scripting
- Systemd
- Networking, DNS
- Storage
- SSH, Firewall
- Libraries
- Cron and more...

Web Servers

- Nginx
- Apache
- HAProxy
- Caddy
- Gunicorn
- uWSGI
- HTTPS/TLS

Databases

- PostgreSQL
- MySQL
- SQLite
- Redis
- ClickHouse
- MongoDB
- etcd

Data Processing

- CSV
- JSON
- SQL queries

Docker

- Building images
- Multi-stage builds
- Volumes
- Networks
- Docker Compose
- Podman

Kubernetes

- kubectl
- Helm
- K8S Roles & Permissions
- Services
- Namespaces
- Deployments, StatefulSets
- ConfigMaps, Secrets

Infrastructure As Code

- Ansible
- Terraform

Observability

- ELK
- Prometheus

Tooling / Applications

- Git
- Rabbitmq
- Envoy
- Vault
- Harbor
- Jenkins

Hacking

- Capture the Flag (CTF) Challenges
- Code Vulnerabilities
- Privilege Escalation

Languages

- Python
- Golang
- PHP
- Java
- Node.js
- C

advent2025 ai ansible apache aws cli bash c caddy clickhouse cron csv data processing disk volumes dns docker elk envoy etcd financial finops ftp git gitea golang gunicorn hack haproxy harbor hashicorp vault helm java jenkins json kubernetes linux-other mongodb mysql nginx node.js packages php podman postgres prometheus python rabbitmq redis sql sqlite ssh ssl supervisord systemd terraform traefik

realistic / interviews new pro business

Medium

#	Name	Time	Type
1	"Cork": The catalog won't open	20 m	Fix New	"Cork": The catalog won't open Scenario: "Cork": The catalog won't open Level: Medium Type: Fix Access: Email Description: After last week's host reboot the product catalog Postgres database will not stay up. Inserts into the catalog database fail. Ops moved application data off the root disk onto a dedicated volume some time ago; root itself still looks healthy. You do not need deep Postgres expertise. The server listens on `:5432` when healthy. The data directory is set in /etc/postgresql//main/* (see data_directory). systemd manages the unit named postgresql. Test: As admin: `sudo -u postgres psql -d catalog -c "INSERT INTO products(name) VALUES ('jane smith');"` Should return: `INSERT 0 1` The fix must still work after a reboot. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.
2	"Bournemouth": Same job, three shells	20 m	Do New	"Bournemouth": Same job, three shells Scenario: "Bournemouth": Same job, three shells Level: Medium Type: Do Access: Email Description: This box is a small shell lab. Debian's `/bin/sh` is dash, your login shell is bash, and zsh is installed. Cron, Docker `CMD` lines, and most `#!/bin/sh` scripts do not run bash — and `zsh script.sh` does not honour a POSIX shebang. Read /home/admin/lab/00-START-HERE.txt and run the demos with: `/home/admin/lab/compare.sh /home/admin/lab/demos/01-identity.sh` (there are ten snippets under demos/; run-all.sh runs every one). Then write a script /home/admin/fleet-report.sh that prints a host status table from /home/admin/lab/inventory/ (hosts, disk.tsv, access.log). Rules: - One header line exactly: `HOST DISK ERR STATUS` - Then one line per host from hosts, in that file's order: `NAME DISK ERR STATUS` (single spaces, no tabs) - DISK is the integer from disk.tsv - ERR is how many access.log lines have that host and code `500` - STATUS is `crit` if DISK >= 90 or ERR >= 5; else `warn` if DISK >= 80 or ERR >= 1; else `ok` The same file must print that table with no extra lines when run as: `sh /home/admin/fleet-report.sh`, `bash /home/admin/fleet-report.sh`, and `zsh /home/admin/fleet-report.sh`. demos/10-broken-report.sh is a bash sketch that is not a valid solution. Test: /home/admin/fleet-report.sh exists. `sh`, `bash`, and `zsh` each print the same report for the inventory files, matching the rules above. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.
3	"Reykjavik": Degraded software RAID	30 m	Fix Business New	"Reykjavik": Degraded software RAID Scenario: "Reykjavik": Degraded software RAID Level: Medium Type: Fix Access: Business Description: Warehouse app data lives on a Linux software RAID volume mounted at /srv/reykjavik (see /srv/reykjavik/warehouse.dat). Overnight monitoring fired a disk-failure alert: a RAID member failed and redundancy for that volume is no longer OK. The systemd health gate `reykjavik-raid-health.service` is unhappy for the same reason. Restore full RAID redundancy for the warehouse volume. Keep the existing warehouse.dat intact. Test: `/srv/reykjavik` is mounted from a healthy RAID1 array with two working members and no failed devices, and `cat /srv/reykjavik/warehouse.dat` prints `reykjavik-warehouse-ok`. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute. Time to Solve: 30 minutes.

Kubernetes Playgrounds

#	Name	Time	Type
1	K8s Playground - Free	20 m	Playground	K8s Playground - Free Playground: K8s Playground - Free Level: Easy Type: Playground Access: Email Description: This is a Kubernetes sandbox for you to play with and experiment. It comes with an nginx.yaml playbook. You can try for example `k apply -f nginx.yaml` (you can use "k" as an alias for "kubectl".) The Helm binary is also installed. Free account: The free account sandbox runs on a 1 GB of RAM VM. As usual, there is no Internet access. Paid accounts (Pro/Pro+/Business: The playground in this case runs on a 2 GB of RAM VM. It has Internet access (to pull your own images for example) and twice the time. Time to Play: 20 minutes.
2	K8s Playground - Pro	60 m	Playground Pro	K8s Playground - Pro Playground: K8s Playground - Pro Level: Easy Type: Playground Access: Paid Description: This is a Kubernetes sandbox for you to play with and experiment. It comes with an nginx.yaml playbook. You can try for example `k apply -f nginx.yaml` (you can use "k" as an alias for "kubectl".) The Helm binary is also installed. Free account: The free account sandbox runs on a 1 GB of RAM VM. As usual, there is no Internet access. Paid accounts (Pro/Pro+/Business: The playground in this case runs on a 2 GB of RAM VM. It has Internet access (to pull your own images for example) and twice the time. Time to Play: 60 minutes.