SadServers Linux & DevOps Troubleshooting Scenarios

Linux & Bash

- Linux commands, Bash scripting
- Systemd
- Networking, DNS
- Storage
- SSH, Firewall
- Libraries
- Cron and more...

Web Servers

- Nginx
- Apache
- HAProxy
- Caddy
- Gunicorn
- uWSGI
- HTTPS/TLS

Databases

- PostgreSQL
- MySQL
- SQLite
- Redis
- ClickHouse
- MongoDB
- etcd

Data Processing

- CSV
- JSON
- SQL queries

Docker

- Building images
- Multi-stage builds
- Volumes
- Networks
- Docker Compose
- Podman

Kubernetes

- kubectl
- Helm
- K8S Roles & Permissions
- Services
- Namespaces
- Deployments, StatefulSets
- ConfigMaps, Secrets

Infrastructure As Code

- Ansible
- Terraform

Observability

- ELK
- Prometheus

Tooling / Applications

- Git
- Rabbitmq
- Envoy
- Vault
- Harbor
- Jenkins

Hacking

- Capture the Flag (CTF) Challenges
- Code Vulnerabilities
- Privilege Escalation

Languages

- Python
- Golang
- PHP
- Java
- Node.js
- C

advent2025 ai ansible apache bash c caddy clickhouse cron csv data processing disk volumes dns docker elk envoy etcd ftp git golang gunicorn hack haproxy harbor hashicorp vault helm java jenkins json kubernetes linux-other mongodb mysql nginx node.js php podman postgres prometheus python rabbitmq redis sql sqlite ssh ssl supervisord systemd terraform traefik

realistic / interviews new pro business

Medium

#	Name	Time	Type
1	"Philadelphia": htop metrics disagree	20 m	Fix New	"Philadelphia": htop metrics disagree Scenario: "Philadelphia": htop metrics disagree Level: Medium Type: Fix Access: Email Description: This host exports load and uptime to /var/lib/sad/sysinfo.txt for a small internal dashboard. htop and btop are installed for troubleshooting. On-call reports the dashboard shows load stuck at 0.00 and an old uptime, while the live process viewers look correct. A leftover lab service leaves a defunct (zombie) child in the process list. Read `/home/admin/incident-notes.txt`. Fix the exporter and the zombie lab so published metrics match the real kernel counters and no zombie children remain under the lab parent. Test: The 1st line of /var/lib/sad/sysinfo.txt is `SadServers - Philadelphia OK`. The `load_1m` and `uptime_seconds` values in that file match /proc/loadavg and /proc/uptime (within a small tolerance). sad-zombie-lab is active and has no zombie children. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.
2	"Luxor": PostgreSQL analytics queries crawl	20 m	Fix New	"Luxor": PostgreSQL analytics queries crawl Scenario: "Luxor": PostgreSQL analytics queries crawl Level: Medium Type: Fix Access: Email Description: Our analytics API serves customer sales counts from a PostgreSQL database. Requests to the customer lookup endpoint take several seconds and often time out, even though the database server looks healthy — no obvious CPU, memory, or disk exhaustion. The API runs as systemd service sad-analytics-api on port `9090`. Application code lives under `/opt/sad/`. Database credentials for debugging: connect as saduser to database analytics_db (password: sadpassword). The main table is sales_data. Find why customer lookups are slow and restore acceptable API response times. Test: This API request must complete quickly (total response time under 500 ms): `curl -s -w "\nTotal time: %{time_total}s\n" http://127.0.0.1:9090/api/customers/1234/sales/count` The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.
3	"Banha" : Intermittent 504 Gateway Time-Out errors	20 m	Fix New	"Banha" : Intermittent 504 Gateway Time-Out errors Scenario: "Banha" : Intermittent 504 Gateway Time-Out errors Level: Medium Type: Fix Access: Email Description: Our Python Flask web application, served via Gunicorn and Nginx, is experiencing intermittent 504 Gateway Time-Out errors on concurrent requests. Users report that the application works perfectly at times, but fails unexpectedly at others. Your task is to identify the root cause of these sporadic failures and implement a permanent fix to ensure the applications consistent stability on port 80. Test: `curl http://localhost/` returns "Hello from Flask!" consistently, even on parallel load: `seq 15 \| xargs -n1 -P15 curl -sf --max-time 5 http://localhost/ && echo OK \|\| echo FAIL`. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.

Kubernetes Playgrounds

#	Name	Time	Type
1	K8s Playground - Free	20 m	Playground	K8s Playground - Free Playground: K8s Playground - Free Level: Easy Type: Playground Access: Email Description: This is a Kubernetes sandbox for you to play with and experiment. It comes with an nginx.yaml playbook. You can try for example `k apply -f nginx.yaml` (you can use "k" as an alias for "kubectl".) The Helm binary is also installed. Free account: The free account sandbox runs on a 1 GB of RAM VM. As usual, there is no Internet access. Paid accounts (Pro/Pro+/Business: The playground in this case runs on a 2 GB of RAM VM. It has Internet access (to pull your own images for example) and twice the time. Time to Play: 20 minutes.
2	K8s Playground - Pro	60 m	Playground Pro	K8s Playground - Pro Playground: K8s Playground - Pro Level: Easy Type: Playground Access: Paid Description: This is a Kubernetes sandbox for you to play with and experiment. It comes with an nginx.yaml playbook. You can try for example `k apply -f nginx.yaml` (you can use "k" as an alias for "kubectl".) The Helm binary is also installed. Free account: The free account sandbox runs on a 1 GB of RAM VM. As usual, there is no Internet access. Paid accounts (Pro/Pro+/Business: The playground in this case runs on a 2 GB of RAM VM. It has Internet access (to pull your own images for example) and twice the time. Time to Play: 60 minutes.