SadServers: Linux, DevOps & SRE Labs | Job Interview Prep

Troubleshooting Scenarios: Web Servers

Linux & Bash

- Linux commands, Bash scripting
- Systemd
- Networking, DNS
- Storage
- SSH, Firewall
- Libraries
- Cron and more...

Web Servers

- Nginx
- Apache
- HAProxy
- Caddy
- Gunicorn
- uWSGI
- HTTPS/TLS

Databases

- PostgreSQL
- MySQL
- SQLite
- Redis
- ClickHouse
- MongoDB
- etcd

Data Processing

- CSV
- JSON
- SQL queries

Docker

- Building images
- Multi-stage builds
- Volumes
- Networks
- Docker Compose
- Podman

Kubernetes

- kubectl
- Helm
- K8S Roles & Permissions
- Services
- Namespaces
- Deployments, StatefulSets
- ConfigMaps, Secrets

Tooling / Applications

- Git
- Rabbitmq
- Envoy
- Vault
- Harbor
- Prometheus
- Jenkins

Hacking

- Capture the Flag (CTF) Challenges
- Code Vulnerabilities
- Privilege Escalation

Languages

- Python
- Golang
- PHP
- Java
- Node.js
- C

apache caddy gunicorn haproxy nginx ssl traefik

realistic / interviews new pro

Easy

#	Name	Time	Type
1	"Geneva": Renew an SSL Certificate	10 m	Fix	"Geneva": Renew an SSL Certificate Scenario: "Geneva": Renew an SSL Certificate Level: Easy Type: Fix Access: Email Description: There's an Nginx web server running on this machine, configured to serve a simple "Hello, World!" page over HTTPS. However, the SSL certificate is expired. Create a new SSL certificate for the Nginx web server with the same Issuer and Subject (same domain and company information). Test: Certificate should not be expired: `echo \| openssl s_client -connect localhost:443 2>/dev/null \| openssl x509 -noout -dates` and the subject of the certificate should be the same as the original one: `echo \| openssl s_client -connect localhost:443 2>/dev/null \| openssl x509 -noout -subject` The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute Time to Solve: 10 minutes.
2	"Fukuoka": Forbidden Association	15 m	Fix Pro	"Fukuoka": Forbidden Association Scenario: "Fukuoka": Forbidden Association Level: Easy Type: Fix Access: Paid Description: There's a web server running on this host but `curl localhost` returns the default 404 Not Found page. Fix the issue so that a file is served correctly and the message Welcome to the Real Site! is returned. Test: Running `curl localhost` should return HTTP 200 with the message Welcome to the Real Site!. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.

Medium

#	Name	Time	Type
1	"Tokyo": can't serve web file	15 m	Fix	"Tokyo": can't serve web file Scenario: "Tokyo": can't serve web file Level: Medium Type: Fix Access: Email Description: There's a web server serving a file /var/www/html/index.html with content "hello sadserver" but when we try to check it locally with an HTTP client like `curl 127.0.0.1:80`, nothing is returned. This scenario is not about the particular web server configuration and you only need to have general knowledge about how web servers work. Test: `curl 127.0.0.1:80` should return: `hello sadserver` Time to Solve: 15 minutes.
2	"Cape Town": Borked Nginx	15 m	Fix	"Cape Town": Borked Nginx Scenario: "Cape Town": Borked Nginx Level: Medium Type: Fix Access: Email Description: There's an Nginx web server installed and managed by systemd. Running `curl -I 127.0.0.1:80` returns `curl: (7) Failed to connect to localhost port 80: Connection refused` , fix it so when you curl you get the default Nginx page. Test: `curl -Is 127.0.0.1:80\|head -1` returns `HTTP/1.1 200 OK` Time to Solve: 15 minutes.
3	"Melbourne": WSGI with Gunicorn	20 m	Fix Pro	"Melbourne": WSGI with Gunicorn Scenario: "Melbourne": WSGI with Gunicorn Level: Medium Type: Fix Access: Paid Description: There is a Python WSGI web application file at /home/admin/wsgi.py , the purpose of which is to serve the string "Hello, world!". This file is served by a Gunicorn server which is fronted by an nginx server (both servers managed by systemd). So the flow of an HTTP request is: Web Client (curl) -> Nginx -> Gunicorn -> wsgi.py . The objective is to be able to curl the localhost (on default port :80) and get back "Hello, world!", using the current setup. Test: `curl -s http://localhost` returns `Hello, world!` (serving the wsgi.py file via Gunicorn and Nginx) Time to Solve: 20 minutes.
4	"Lisbon": etcd SSL cert troubles	20 m	Fix	"Lisbon": etcd SSL cert troubles Scenario: "Lisbon": etcd SSL cert troubles Level: Medium Type: Fix Access: Email Description: There's an etcd server running on https://localhost:2379 , get the value for the key "foo", ie `etcdctl get foo` or `curl https://localhost:2379/v2/keys/foo` Test: `etcdctl get foo` returns `bar`. Time to Solve: 20 minutes.
5	"Tarifa": Between Two Seas	20 m	Fix Pro	"Tarifa": Between Two Seas Scenario: "Tarifa": Between Two Seas Level: Medium Type: Fix Access: Paid Description: There are three Docker containers defined in the docker-compose.yml file: an HAProxy accepting connetions on port :5000 of the host, and two nginx containers, not exposed to the host. The person who tried to set this up wanted to have HAProxy in front of the (backend or upstream) nginx containers load-balancing them but something is not working. Test: Running `curl localhost:5000` several times returns both `hello there from nginx_0` and `hello there from nginx_1` Check /home/admin/agent/check.sh for the test that "Check My Solution" runs. Time to Solve: 20 minutes.
6	"Moyogalpa": Security Snag. The Trials of Mary and John	30 m	Fix	"Moyogalpa": Security Snag. The Trials of Mary and John Scenario: "Moyogalpa": Security Snag. The Trials of Mary and John Level: Medium Type: Fix Access: Email Description: Mary and John are working on a Golang web application, and the security team has asked them to implement security measures. Unfortunately, they have broken the application, and it no longer functions. They need your help to fix it. The fixed application should be able to allow clients to communicate with the application over HTTPS without ignoring any checks. (eg: `curl https://webapp:7000/users.html`) and serve its static files. Test: `curl https://webapp:7000/users.html` should return the content of file. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 30 minutes.
7	"Depok": Nginx with Brotli	15 m	Fix Pro	"Depok": Nginx with Brotli Scenario: "Depok": Nginx with Brotli Level: Medium Type: Fix Access: Paid Description: You are tasked to add compression to the company website. The website is running on an Nginx server, and you decide to add Brotli compression to it. Brotli has became very popular these days because of its high compression ratio. It's a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding, and 2nd order context modeling. For this purpose, you decided to compile the brotli modules yourself and add them to the Nginx server. The location of the Brotli source code is at /home/admin/ngx_brotli. The nginx source code (needed to compile the modules) is located at /home/admin/nginx-1.18.0. From the ngx_brotli repository first you need to compile the brotli dependencies and then configure and make modules for Nginx. Afer that you need to add the modules to the Nginx configuration. After installing the modules, you need to make sure the responses from the server are being served with compression. Create a port-forward to port 80 from the server to your computer and check the header Content-Encoding, responses must return br for Brotli compression. You can also use `curl -H "Accept-Encoding: br, gzip" -I http://localhost` to check the header. Something nice about Brotli is that it fails over to gzip if the client doesn't support Brotli, so `curl -H "Accept-Encoding: gzip" -I http://localhost` should return gzip instead. Test: `curl -H "Accept-Encoding: br" -sI http://localhost` returns the header Content-Encoding: br. The "Check My Solution" button runs the script _/home/admin/agent/check.sh_, which you can see and execute. Time to Solve: 15 minutes.
8	"Batumi": Troubleshoot "A" cannot connect to "B"	20 m	Fix Pro	"Batumi": Troubleshoot "A" cannot connect to "B" Scenario: "Batumi": Troubleshoot "A" cannot connect to "B" Level: Medium Type: Fix Access: Paid Description: (To learn the skills to solve this challenge, see Can't Connect to a Service: Linux Troubleshooting Guide) There is a web server (Caddy) on HTTP port :80 but `curl http://127.0.0.1` doesn't work. Find out what's wrong and make the necessary fixes so the web server returns a URL. Note: as a limitation, the file /home/admin/db_connector.py must not be modified so that the challenge is considered solved properly. The web server has to respond on the IP address 127.0.0.1; not only on "localhost". Test: The command `curl http://127.0.0.1` returns a URL address. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.
9	"Toulon": Denied Lamp	15 m	Fix Business	"Toulon": Denied Lamp Scenario: "Toulon": Denied Lamp Level: Medium Type: Fix Access: Business Description: The security team has asked again Mary and John to implemente more security measures. Unfortunately, this time they have broken the LAMP stack (Apache with PHP) so the frontend is unable get an answer upstream, thus, they need your help again to fix it. The fixed application should be able to serve the content from the webserver, the problem is a network connectivity, although the logs have valuable informatiion, it has nothing to do with the configuration of the apache server. Test: `curl localhost \| head -n1` returns `SadServers - LAMP Stack` The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.
10	"Auderghem": Containers miscommunication	15 m	Fix Pro	"Auderghem": Containers miscommunication Scenario: "Auderghem": Containers miscommunication Level: Medium Type: Fix Access: Paid Description: There is an nginx Docker container that listens on port 80, the purpose of which is to redirect the traffic to two other containers statichtml1 and statichtml2 but this redirection is not working. Fix the problem. IMPORTANT. You can restart all containers, but don't stop or remove them. Test: The nginx container must redirect the traffic to the statichtml1 and statichtml2 containers: `curl http://localhost` returns the Welcome to nginx default page `curl http://localhost/1` returns HelloWorld;1 `curl http://localhost/2` returns HelloWorld;2 The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.
11	"Marseille": Rocky security	15 m	Fix Pro	"Marseille": Rocky security Scenario: "Marseille": Rocky security Level: Medium Type: Fix Access: Paid Description: As the Christmas shopping season approaches, the security team has asked Mary and John to implemente more security measures. Unfortunately, this time they have broken the LAMP stack; the frontend is unable get an answer from upstream, thus they need your help again to fix it. The application should be able to serve the content from the webserver. Note for Pro users: direct SSH access is not avaiable (yet) for this scenario. Test: `curl localhost \| head -n1` returns `SadServers - LAMP Stack` The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.
12	"San Juan": mucho Traefik	20 m	Fix New	"San Juan": mucho Traefik Scenario: "San Juan": mucho Traefik Level: Medium Type: Fix Access: Email Description: There is a Traefik load balancer that must be up and running. The server and the backend services are managed by Docker Compose. Running `curl -s app.sadserver \| head -n1` must return the host ID of one of the backend servers, running the command again must return a new host ID. The server seems to be working some times, some others fails or just times out. The round-robin configuration should make the webserver iterate through the back-end servers. Test: `curl -s app.sadserver \| head -n1` returns something like `Hostname:` The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.

Hard

#	Name	Time	Type
1	"Roseau": Hack a Web Server	30 m	Hack	"Roseau": Hack a Web Server Scenario: "Roseau": Hack a Web Server Level: Hard Type: Hack Access: Email Description: There is a secret stored in a file that the local Apache web server can provide. Find this secret and have it as a /home/admin/secret.txt file. Note that in this server the admin user is not a sudoer. Also note that the password crackers Hashcat and Hydra are installed from packages and John the Ripper binaries have been built from source in /home/admin/john/run Test: `sha1sum /home/admin/secret.txt \|awk '{print $1}'` returns `cc2c322fbcac56923048d083b465901aac0fe8f8` Time to Solve: 30 minutes.