SadServers Linux & DevOps Troubleshooting Scenarios

Scenario: "Saint John": what is writing to this log file?

Level: Easy

Type: Fix

Access: Public

Description: A developer created a testing program that is continuously writing to a log file /var/log/bad.log and filling up disk. You can check for example with tail -f /var/log/bad.log.
This program is no longer needed. Find it and terminate it. Do not delete the log file.

Test: The log file size doesn't change (within a time interval bigger than the rate of change of the log file).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Saskatoon": counting IPs.

Level: Easy

Type: Do

Access: Email

Description: There's a web server access log file at /home/admin/access.log. The file consists of one line per HTTP request, with the requester's IP address at the beginning of each line (first column).

Find what's the IP address that has the most requests in this file (there's no tie; the IP is unique). Write the solution into a file /home/admin/highestip.txt. For example, if your solution is "1.2.3.4", you can do echo "1.2.3.4" > /home/admin/highestip.txt

NOTE: The solution IP shows 482 times, ie grep -c -F -f highestip.txt access.log returns 482, if your solution has a different (lower) number you got the wrong most common IP.

Test: The SHA1 checksum of the IP address sha1sum /home/admin/highestip.txt is 6ef426c40652babc0d081d438b9f353709008e93 (just a way to verify the solution without giving it away, we also accept the right IP with no ending newline in the file.)

The

Time to Solve: 15 minutes.

Scenario: "The Command Line Murders"

Level: Easy

Type: Do

Access: Public

Description: This is the Command Line Murders with a small twist as in the solution is different

Enter the name of the murderer in the file /home/admin/mysolution, for example echo "John Smith" > ~/mysolution

Hints are at the base of the /home/admin/clmystery directory. Enjoy the investigation!

Test: md5sum ~/mysolution returns 9bba101c7369f49ca890ea96aa242dd5

(You can always see /home/admin/agent/check.sh to see how the solution is evaluated).

Time to Solve: 20 minutes.

Scenario: "Taipei": Come a-knocking

Level: Easy

Type: Hack

Access: Email

Description: There is a web server on port :80 protected with Port Knocking. Find the one "knock" needed (sending a SYN to a single port, not a sequence) so you can curl localhost.

Test: Executing curl localhost returns a message with md5sum fe474f8e1c29e9f412ed3b726369ab65. (Note: the resulting md5sum includes the new line terminator: echo $(curl localhost))

Time to Solve: 15 minutes.

Scenario: "Resumable Server": Linux Upskill Challenge

Level: Easy

Type: Do

Access: Email

Description: This is a Debian 11 server without a challenge; it's for you to do as you please.

It's meant in principle for guided tutorials like the Linux Upskill Challenge, with some limitations (there's still no outgoing Internet access).

Once you are done with your task, you can stop (pause) your server from your dashboard.

From the dashboard at any time you can restart the stopped server to continue your tasks or you can stop the server if it's running. You can also destroy it at any time.

After 30 mins, the server will be stopped. You can restart it from the dashboard.

This is a "beta" feature and there can be some issues.

Test: None

Time to Solve: 30 minutes.

Scenario: "Lhasa": Easy Math

Level: Easy

Type: Do

Access: Email

Description: There's a file /home/admin/scores.txt with two columns (the first number is a line number and the second one is a test score for example).

Find the average (more precisely; the arithmetic mean: sum of numbers divided by how many numbers are there) of the numbers in the second column (find the average score).

Use exactly two digits to the right of the decimal point. i. e., use exaclty two "decimal digits" without any rounding. Eg: if average = 21.349 , the solution is 21.34. If average = 33.1 , the solution is 33.10.

Save the solution in the /home/admin/solution file, for example: echo "123.45" > ~/solution

Tip: There's bc, Python3, Golang and sqlite3 installed in this VM.

Test: md5sum /home/admin/solution returns 6d4832eb963012f6d8a71a60fac77168 solution

Time to Solve: 15 minutes.

Scenario: "Bucharest": Connecting to Postgres

Level: Easy

Type: Fix

Access: Email

Description: A web application relies on the PostgreSQL 13 database present on this server. However, the connection to the database is not working. Your task is to identify and resolve the issue causing this connection failure. The application connects to a database named app1 with the user app1user and the password app1user.

Credit PykPyky

Test: Running PGPASSWORD=app1user psql -h 127.0.0.1 -d app1 -U app1user -c '\q' succeeds (does not return an error).

Time to Solve: 10 minutes.

Scenario: "Bilbao": Basic Kubernetes Problems

Level: Easy

Type: Fix

Access: Email

Description: There's a Kubernetes Deployment with an Nginx pod and a Load Balancer declared in the manifest.yml file. The pod is not coming up. Fix it so that you can access the Nginx container through the Load Balancer.

There's no "sudo" (root) access.

Test: Running curl 10.43.216.196 returns the default Nginx Welcome page.

See /home/admin/agent/check.sh for the test that "Check My Solution" runs.

Time to Solve: 10 minutes.

Scenario: "Apia": Needle in a Haystack

Level: Easy

Type: Do

Access: Paid

Description: In a directory /home/admin/data, there are multiple files, all of them with same content. One of these files has been modified, a word was added. You need to identify which word it is and put it in the solution file (both newline terminated or not are accepted).

Test: md5sum /home/admin/solution should return 55aba155290288b58e9b778c8f616560 or 2eeefea9fc4b16ea624bed5c67a49d80

Check My Solution: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can

Time to Solve: 20 minutes.

Scenario: "Gitega": Find the Bad Git Commit

Level: Easy

Type: Do

Access: Paid

Description: The directory at /home/admin/git has a Git repository with a Golang program and a test for it.

To execute the test, from this "git" directory run: go test. The last (current HEAD) commit fails the test. Suppose the first commit passed the test.

Find the (long hash) commit that first broke the test and enter it in the /home/admin/solution file. For example: echo 9e80a7eb1b09385e93ab4a76cb2c93beec48fd9f > /home/admin/solution

Test: Doing md5sum /home/admin/solution returns f7db1bb6b7bfcd66a4eb66782804b39d.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Minneapolis": Break a CSV file

Level: Easy

Type: Do

Access: Email

Description: Break the Comma Separated Valued (CSV) file data.csv in the /home/admin/ directory into exactly 10 smaller files of about the same size named data-00.csv, data-01.csv, ... , data-09.csv files in the same directory. All the files should have the same header (first line with column names) as data.csv. None of the smaller files should be bigger than 32KB.

Note: to simplify, disregard broken lines in your files (ie, you can break a file at any point, not just at a newline). The resulting files don't have to be proper CSV files.

Test: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Saint Paul": Merge Many CSVs files

Level: Easy

Type: Do

Access: Email

Description: Join (merge) all the 338 files in /home/admin/polldayregistrations_enregistjourduscrutin?????.csv into one single /home/admin/all.csv file with the contents of all the CSV files in any order. There should be only one line with the names of the columns as a header.

Test: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Bata": Find in /proc

Level: Easy

Type: Do

Access: Email

Description: A spy has left a password in a file in /proc/sys . The contents of the file start with "secret:" (without the quotes).

Find the file and save the word after "secret:" to the file /home/admin/secret.txt with a newline at the end (e.g. if the file contents were "secret:password" do: echo "password" > /home/admin/secret.txt).

(Note there's no root/sudo access in this scenario).

Test: Running md5sum /home/admin/secret.txt returns a7fcfd21da428dd7d4c5bb4c2e2207c4

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Geneva": Renew an SSL Certificate

Level: Easy

Type: Fix

Access: Email

Description: There's an Nginx web server running on this machine, configured to serve a simple "Hello, World!" page over HTTPS. However, the SSL certificate is expired.

Create a new SSL certificate for the Nginx web server with the same Issuer and Subject (same domain and company information).

Test: Certificate should not be expired: echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -dates and the subject of the certificate should be the same as the original one: echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -subject

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute

Time to Solve: 10 minutes.

Scenario: Linux Server Review - Guided Learning

Level: Easy

Type: Do

Access: Email

Description: This is a guided learning scenario. Follow this Linux Server Review Scenario Guide

The purpose of this scenario is to review a Linux server and be able to answer questions like:

• What's the purpose of the server?
• What's the hardware (CPU / RAM / disk / net) utilization of the server? is there a problem there?
• What is running and what's going on in the server?

Test: (there's no test)

Time to Solve: 30 minutes.

Scenario: "Tokamachi": Troubleshooting a Named Pipe

Level: Easy

Type: Fix

Access: Email

Description: There's a process reading from the named pipe /home/admin/namedpipe.

If you run this command that writes to that pipe:

/bin/bash -c 'while true; do echo "this is a test message being sent to the pipe" > /home/admin/namedpipe; done' &

And check the reader log with tail -f reader.log

You'll see that after a minute or so it works for a while (the reader receives some messages) and then it stops working (no more received messages are printed to the reader log or it takes a long time to process one). Troubleshoot and fix (for example changing the writer command) so that the writer keeps sending the messages and the reader is able to read all of them.

Test: There should be a process running where a message is being sent to the pipe and that while that is running, another message can be sent to the pipe and read back.
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Yokohama": Linux Users Working Together

Level: Easy

Type: Fix

Access: Paid

Description: There are four Linux users working together in a project in this server: abe, betty, carlos, debora.

First, they have asked you as the sysadmin, to make it so each of these four users can read the project files of the other users in the /home/admin/shared directory, but none of them can modify a file that belongs to another user. Users should be able modify their own files.

Secondly, they have asked you to modify the file shared/ALL so that any of these four users can write more content to it, but previous (existing) content cannot be altered.

Test: All users (abe, betty, carlos, debora) can write to their own files. None of them can write to another user's file.
All users can add more content (append)) to the shared/project_ALL file but none can change its current content.
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Fukuoka": Forbidden Association

Level: Easy

Type: Fix

Access: Paid

Description: There's a web server running on this host but curl localhost returns the default 404 Not Found page.

Fix the issue so that a file is served correctly and the message Welcome to the Real Site! is returned.

Test: Running curl localhost should return HTTP 200 with the message Welcome to the Real Site!.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Kampot": A New Port

Level: Easy

Type: Do

Access: Email

Description: A Python app serving simulated bank data runs as root and listens on port 20280. The app is managed by supervisor and cannot be stopped or reconfigured to use a different port.

An internal legacy monitoring system expects the service to be available on port 80, but the app is hardcoded to 20280 for security and legacy reasons. Your task is to make the service accessible on port 80 locally.

Test: curl localhost:80/accounts returns [{"id":1,"name":"Alice","type":"Checking"},{"id":2,"name":"Bob","type":"Savings"},{"id":3,"name":"Charlie","type":"Business"}]

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Rio de Janeiro": Do we have another option?

Level: Easy

Type: Fix

Access: Paid

Description: This scenario server is dedicated to Jenkins, a Java application managed by systemd. Jenkins is failing to start. Troubleshoot and find the problem, then apply the solution so Jenkins runs properly.

Test: The service must return the string "Sign in - Jenkins" amongst some other html code. You can check with the command curl -s localhost:8888/login | grep Jenkins | head -n1

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Nuuk": More SSH Troubles

Level: Easy

Type: Fix

Access: Paid

Description: (NOTE: if you are a Pro user, you cannot SSH directly into this VM; click the "Open the Server Terminal" button to use the web browser instead).

SSH seems broken in this server. The user admin has an id_ed25519 SSH key pair in their ~/.ssh directory with the public key in ~/.ssh/authorized_keys but ssh 127.0.0.1 won't work.

Test: You can ssh locally, i.e. ssh admin@127.0.0.1 works.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Cairo": Time for a Timer

Level: Easy

Type: Fix

Access: Email

Description: A critical health check script at /opt/scripts/health.sh is supposed to run every 10 seconds. This check is triggered by a systemd timer.
The script's job is to check the local Nginx server and write its status (e.g., "STATUS: OK") to the log file at /var/log/health.log.
The log file is not being updated, and it appears the health check is failing.

Find out why the health check system is broken and fix it. The check will pass once the /var/log/health.log file is being correctly updated by the timer with a STATUS: OK message.

Test: The /opt/scripts/health.sh script writes STATUS: OK to /var/log/health.log every 10 seconds.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Alexandria": The Vanishing Backups

Level: Easy

Type: Fix

Access: Email

Description: A critical backup cron job has silently stopped working 3 days ago. The backup script is located at /opt/backup/backup.sh and should create daily backups in /var/backups/daily/, but no new backups have been created recently.

Looking at the backup directory, you can see old backup files from a few days ago, proving the system used to work. However, there are no error emails, no obvious error logs, and the cron service appears to be running normally.

Fix ALL issues preventing the backups from running, so that backups are created successfully and reliably.

Test directory: /var/backups/daily/
Backup script: /opt/backup/backup.sh

Test: The solution will be validated by checking if a backup file has been created in the last 10 minutes.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 5 minutes.

Scenario: "Kortenberg": Can't touch this!

Level: Easy

Type: Fix

Access: Paid

Description: Is "All I want for Christmas is you" already everywhere?. A bit unrelated, someone messed up the permissions in this server, the admin user can't list new directories and can't write into new files. Fix the issue.
NOTE: Besides solving the problem in your current admin shell session, you need to fix it permanently, as in a new login shell for user "admin" (like the one initiated by the scenario checker) should have the problem fixed as well.

Test: The admin user in a separate Bash login session should be able to create a new directory in your /home/admin directory, as well as being able to create a file into this new directory and add text into the new file.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Hamburg": Find the AWS EC2 volume

Level: Easy

Type: Do

Access: Paid

Description: We have a lot of AWS EBS volumes, the description of which we have save to a file with: aws ec2 describe-volumes > aws-volumes.json.
One of the volumes contains important data and we need to identify which volume (its ID), but we only remember these characteristics: gp3, created before 30/09/2025 , Size < 64 , Iops < 1500, Throughput > 300.

Find the correct volume and put its "InstanceId" into the ~/mysolution file, e.g.: echo "vol-00000000000000000" > ~/mysolution

Test: Running md5sum /home/admin/mysolution returns e7e34463823bf7e39358bf6bb24336d8 (we also accept the file without a new line at the end).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Manhattan": can't write data into database.

Level: Medium

Type: Fix

Access: Public

Description: Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don't need to know details about it (although it may help).

Helpful Postgres information: it's a service that listens to a port (:5432) and writes to disk in a data directory, the location of which is defined in the data_directory parameter of the configuration file /etc/postgresql/14/main/postgresql.conf. In our case Postgres is managed by systemd as a unit with name postgresql.

Test: (from default admin user) sudo -u postgres psql -c "insert into persons(name) values ('jane smith');" -d dt

Should return:INSERT 0 1

Time to Solve: 20 minutes.

Scenario: "Tokyo": can't serve web file

Level: Medium

Type: Fix

Access: Email

Description: There's a web server serving a file /var/www/html/index.html with content "hello sadserver" but when we try to check it locally with an HTTP client like curl 127.0.0.1:80, nothing is returned. This scenario is not about the particular web server configuration and you only need to have general knowledge about how web servers work.

Test: curl 127.0.0.1:80 should return: hello sadserver

Time to Solve: 15 minutes.

Scenario: "Cape Town": Borked Nginx

Level: Medium

Type: Fix

Access: Email

Description: There's an Nginx web server installed and managed by systemd. Running curl -I 127.0.0.1:80 returns curl: (7) Failed to connect to localhost port 80: Connection refused , fix it so when you curl you get the default Nginx page.

Test: curl -Is 127.0.0.1:80|head -1 returns HTTP/1.1 200 OK

Time to Solve: 15 minutes.

Scenario: "Salta": Docker container won't start.

Level: Medium

Type: Fix

Access: Email

Description: There's a "dockerized" Node.js web application in the /home/admin/app directory. Create a Docker container so you get a web app on port :8888 and can curl to it. For the solution to be valid, there should be only one running Docker container.

Test: curl localhost:8888 returns Hello World! from a running container.

Time to Solve: 15 minutes.

Scenario: "Venice": Am I in a container?

Level: Medium

Type: Do

Access: Email

Description: Try and figure out if you are inside a container (like a Docker one for example) or inside a Virtual Machine (like in the other scenarios).

Test: This scenario doesn't have a test (hence also no "Check My Solution" either).

Time to Solve: 15 minutes.

Scenario: "Oaxaca": Close an Open File

Level: Medium

Type: Fix

Access: Email

Description: The file /home/admin/somefile is open for writing by some process. Close this file without killing the process.

Test: lsof /home/admin/somefile returns nothing.

Time to Solve: 15 minutes.

Scenario: "Melbourne": WSGI with Gunicorn

Level: Medium

Type: Fix

Access: Paid

Description: There is a Python WSGI web application file at /home/admin/wsgi.py , the purpose of which is to serve the string "Hello, world!". This file is served by a Gunicorn server which is fronted by an nginx server (both servers managed by systemd). So the flow of an HTTP request is: Web Client (curl) -> Nginx -> Gunicorn -> wsgi.py . The objective is to be able to curl the localhost (on default port :80) and get back "Hello, world!", using the current setup.

Test: curl -s http://localhost returns Hello, world! (serving the wsgi.py file via Gunicorn and Nginx)

Time to Solve: 20 minutes.

Scenario: "Lisbon": etcd SSL cert troubles

Level: Medium

Type: Fix

Access: Email

Description: There's an etcd server running on https://localhost:2379 , get the value for the key "foo", ie etcdctl get foo or curl https://localhost:2379/v2/keys/foo

Test: etcdctl get foo returns bar.

Time to Solve: 20 minutes.

Scenario: "Kihei": Surely Not Another Disk Space Scenario

Level: Medium

Type: Fix

Access: Email

Description: There is a /home/admin/kihei program. Make the changes necessary so it runs succesfully, without deleting the /home/admin/datafile file.

Test: Running /home/admin/kihei returns Done..

Time to Solve: 30 minutes.

Scenario: "Unimak Island": Fun with Mr Jason

Level: Medium

Type: Do

Access: Paid

Description: Using the file station_information.json , find the station_id where "has_kiosk" is false and "capacity" is greater than 30.

Save the station_id of the solution in the /home/admin/mysolution file, for example: echo "ec040a94-4de7-4fb3-aea0-ec5892034a69" > ~/mysolution

You can use the installed utilities jq, gron, jid as well as Python3 and Golang.

Test: md5sum /home/admin/mysolution returns 8d8414808b15d55dad857fd5aeb2aebc

Time to Solve: 15 minutes.

Scenario: "Ivujivik": Parlez-vous Français?

Level: Medium

Type: Do

Access: Paid

Description: Given the CSV file /home/admin/table_tableau11.csv, find the Electoral District Name/Nom de circonscription that has the largest number of Rejected Ballots/Bulletins rejetés and also has a population of less than 100,000.

The initial CSV file may be corrupted or invalid in a way that can be fixed without changing its data.

Installed in the VM are: Python3, Go, sqlite3, miller directly and PostgreSQL, MySQL in Docker images.

Save the solution in the /home/admin/mysolution , with the name as it is in the file, for example: echo "Trois-Rivières" > ~/mysolution (the solution must be terminated by newline).

Test: md5sum /home/admin/mysolution returns e399d171f21839a65f8f8ab55ed1e1a1

Time to Solve: 20 minutes.

Scenario: "Paris": Where is my webserver?

Level: Medium

Type: Hack

Access: Email

Description: A developer put an important password on his webserver localhost:5000 . However, he can't find a way to recover it. This scenario is easy to to once you realize the one "trick".

Find the password and save it in /home/admin/mysolution , for example: echo "somepassword" > ~/mysolution

Scenario credit: PuppiestDoggo

Test: md5sum ~/mysolution returns d8bee9d7f830d5fb59b89e1e120cce8e

Time to Solve: 15 minutes.

Scenario: "Buenos Aires": Kubernetes Pod Crashing

Level: Medium

Type: Fix

Access: Email

Description: There are two pods: "logger" and "logshipper" living in the default namespace. Unfortunately, logshipper has an issue (crashlooping) and is forbidden to see what logger is trying to say. Could you help fix Logshipper?

Do not change the K8S definition of the logshipper pod. Use "sudo".

Because k8s takes a minute or two to change the pod state initially, the check for the scenario is made to fail in the first two minutes.

Credit Srivatsav Kondragunta

Test: kubectl get pods -l app=logshipper --no-headers -o json | jq -r '.items[] | "\(.status.containerStatuses[0].ready)"' returns true

Time to Solve: 20 minutes.

Scenario: "Tarifa": Between Two Seas

Level: Medium

Type: Fix

Access: Paid

Description: There are three Docker containers defined in the docker-compose.yml file: an HAProxy accepting connetions on port :5000 of the host, and two nginx containers, not exposed to the host.

The person who tried to set this up wanted to have HAProxy in front of the (backend or upstream) nginx containers load-balancing them but something is not working.

Test: Running curl localhost:5000 several times returns both hello there from nginx_0 and hello there from nginx_1

Check /home/admin/agent/check.sh for the test that "Check My Solution" runs.

Time to Solve: 20 minutes.

Scenario: "Marrakech": Word Histogram

Level: Medium

Type: Do

Access: Email

Description: Find in the file frankestein.txt the second most frequent word and save in UPPER (capital) case in the /home/admin/mysolution file.

A word is a string of characters separated by space or newlines or punctuation symbols .,:; . Disregard case ('The', 'the' and 'THE' is the same word) and for simplification consider the apostrophe as another character not as punctuation ("it's" would be a word, distinct from "it" and "is"). Also disregard plurals ("car" and "cars" are different words) and other word variations (don't do "stemming").

We are providing a shorter test.txt file where the second most common word in upper case is "WORLD", so we could save this solution as: echo "WORLD" > /home/admin/mysolution

This problem can be done with a programming language (Python, Golang and sqlite3) or with common Linux utilities.

Test: echo "SOLUTION" | md5sum returns 19bf32b8725ec794d434280902d78e18

See /home/admin/agent/check.sh for the test that "Check My Solution" runs.

Time to Solve: 20 minutes.

Scenario: "Rosario": Restore a MySQL database

Level: Medium

Type: Fix

Access: Email

Description: A developer created a database named 'main' but now some data is missing in the database. You need to restore the database using the the dump "/home/admin/backup.sql".
The issue is that the developer forgot the root password for the MariaDB server.
If you encounter an issue while restoring the database, fix it.

Credit: Sebastian Segovia

Test: The database, once restored, has a table named "solution".

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Abaokoro": Restore MySQL Databases Spooked by a Ghost

Level: Medium

Type: Fix

Access: Paid

Description: There are three databases that need to be restored. You need to create three databases called "first", "second" and "third" and restore the databases using the file "/home/admin/dbs_to_restore.zip".
If you encounter an issue while restoring the database, fix it.

Credit: Sebastian Segovia

Test: All databases, once restored, have a table named "foo".

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Poznań": Helm Chart Issue in Kubernetes

Level: Medium

Type: Fix

Access: Paid

Description: NOTE: Prompt may take a few extra seconds to be responsive while the k3s environment gets ready. Root access is not needed for this challenge ("admin" user cannot sudo).

A DevOps engineer created a Helm Chart web_chart with a custom nginx site, however he still gets the default nginx index.html.

You can check for example with POD_IP=$(kubectl get pods -n default -o jsonpath='{.items[0].status.podIP}') and curl -s "${POD_IP}">.

In addition he should set replicas to 3.

The chart is not working as expected. Fix the configurations so you get the custom HTML page from any nginx pod.

Credit Kamil Błaż

Test: Doing curl on the default port (:80) of any nginx pod returns a Welcome SadServers page. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Manado": How much do you press?

Level: Medium

Type: Do

Access: Paid

Description: You have been tasked with compressing the file /home/admin/names, which is 35147 bytes, to a size smaller than 9400 bytes. You can use any compressing tool at your disposal (there are many available in the server), also you can modify the file without deleting anything in it. Put the solution (compressed file) in the /home/user/admin/solution directory with the default extension used by the compression tool (example: ~/solution/names.gzip).

Test: The size of the compressed file is smaller than 9400 bytes.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Warsaw": Prometheus can't scrape the webserver

Level: Medium

Type: Fix

Access: Paid

Description: A developer created a golang application that is exposing the /metrics endpoint. They have a problem with scraping the metrics from the application. They asked you to help find the problem.

Full source code of the application is available at the /home/admin/app directory.

Credit Kamil Błaż

Test: The endpoint http://localhost:9000/metrics should return HTTP code 200.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Moyogalpa": Security Snag. The Trials of Mary and John

Level: Medium

Type: Fix

Access: Email

Description: Mary and John are working on a Golang web application, and the security team has asked them to implement security measures. Unfortunately, they have broken the application, and it no longer functions. They need your help to fix it.

The fixed application should be able to allow clients to communicate with the application over HTTPS without ignoring any checks. (eg: curl https://webapp:7000/users.html) and serve its static files.

Test: curl https://webapp:7000/users.html should return the content of file.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Helsingør": The first walls of postgres physical replication

Level: Medium

Type: Fix

Access: Email

Description: You're setting up a PostgreSQL database with replication, you decided to use Docker along with Docker Compose to make it easier to manage and test, after a few hours of work you finished the job and the master database is up and running, but you're having trouble with the replica.

You need to figure out what's wrong with the replica and fix it.

Since you are using Docker Compose, you can check the status of the running containers using docker compose ps or docker ps will do the job too). You may also want to check the logs of the containers.

All definition for the containers are inside the docker-compose.yml file. You can stand up the environment by running docker compose up -d and set it down by running `docker compose down`.

If you make any change to the docker-compose.yml file, you can restart the containers by running docker compose up -d --force-recreate.

Test: Postgres replica container works.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Bekasi": Supervisor is still around

Level: Medium

Type: Fix

Access: Paid

Description: There is an nginx service running on port 443, it is the main web server for the company and looks like a new employee has deployed some changes to the configuration of supervisor and now it is not working as expected.

If you try to access curl -k https://bekasi it should return Hello SadServers! but for some reason it is not.

You cannot modify files from the /home/admin/bekasi folder in order to pass the check.sh

You must find out what the issue is and fix it.

Test: curl -k https://bekasi returns Hello SadServers!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Depok": Nginx with Brotli

Level: Medium

Type: Fix

Access: Paid

Description: You are tasked to add compression to the company website. The website is running on an Nginx server, and you decide to add Brotli compression to it.

Brotli has became very popular these days because of its high compression ratio. It's a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding, and 2nd order context modeling.

For this purpose, you decided to compile the brotli modules yourself and add them to the Nginx server.

The location of the Brotli source code is at /home/admin/ngx_brotli. The nginx source code (needed to compile the modules) is located at /home/admin/nginx-1.18.0. From the ngx_brotli repository first you need to compile the brotli dependencies and then configure and make modules for Nginx. Afer that you need to add the modules to the Nginx configuration.

After installing the modules, you need to make sure the responses from the server are being served with compression.

Create a port-forward to port 80 from the server to your computer and check the header Content-Encoding, responses must return br for Brotli compression. You can also use curl -H "Accept-Encoding: br, gzip" -I http://localhost to check the header.

Something nice about Brotli is that it fails over to gzip if the client doesn't support Brotli, so curl -H "Accept-Encoding: gzip" -I http://localhost should return gzip instead.

Test: curl -H "Accept-Encoding: br" -sI http://localhost returns the header Content-Encoding: br.

The "Check My Solution" button runs the script _/home/admin/agent/check.sh_, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Tukaani": XZ LZMA Library Compromised

Level: Medium

Type: Fix

Access: Paid

Description: (You can learn about Linux Libraries before starting this scenario).

The Linux shared library liblzma.so has been compromised (the real compromised XZ Utils liblzma has not been used). The liblzma.so at the path /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5 is the good one. Consider the same library liblzma.so.5.2.5 at other paths as compromised or malicious (ideally we would have used other real versions with different checksums).

Find all instances of this "malicious" liblzma library (remember, it's the same library but in different directory locations) and make it so none of the running processes use it, while the applications "webapp" and "jobapp" (both of which managed by systemd) still run properly (eg, stopping those applications is not a solution).

Test: lsof | grep liblzma.so.5 returns only the liblzma in the path: /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Atrani": Modify a SQlite3 Database

Level: Medium

Type: Fix

Access: Paid

Description: A developer created a script /home/admin/readdb.py that tests access to a database. Without modifying the readdb.py file, change the database so that running the script returns the string "John Karmack".

Test: Running /home/admin/readdb.py returns "John Karmack".

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Nerdearla Buenos Aires": Troubleshoot "A" no se conecta con "B"

Level: Medium

Type: Fix

Access: Email

Description: Hay un servidor web (Caddy) en el puerto HTTP :80 pero curl http://127.0.0.1 no funciona. Descubre lo que pasa y haz los arreglos necesarios y el servidor web te dará una URL.

Nota: como limitación, el fichero /home/admin/db_connector.py no se debe modificar para que el problema se considere bien resuelto.
El servidor web debe responder en la dirección IP 127.0.0.1; no sólamente en "localhost".

Test: El comando curl http://127.0.0.1 devuelve una dirección URL.

El botón "Check My Solution" corre el script /home/admin/agent/check.sh, que puedes ver y ejecutar.

Time to Solve: 30 minutes.

Scenario: "Minneapolis with a Vengeance": Break a CSV file

Level: Medium

Type: Do

Access: Email

Description: Break the Comma Separated Valued (CSV) file data.csv in the /home/admin/ directory into exactly 10 smaller files of about the same size named data-00.csv, data-01.csv, ... , data-09.csv files in the same directory. All the files should have the same header (first line with column names) as data.csv. None of the smaller files should be bigger than 32KB.

Note: unlike the original Minneapolis scenario, here the resulting files have to be proper CSV files.

As a helper tool, you can run the program check_csv.py to check if your data-??.cs files look like proper CSV files.

Test: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Hanoi": Find the Multitasking Users

Level: Medium

Type: Do

Access: Paid

Description: The Hanoi office has a Linux server with a large number of user accounts and groups. The system administrators need to identify which users belong to multiple groups for better access management.

Given two files, `users.txt` and `groups.txt`, create a new file `/home/admin/multi-group-users.txt` containing the usernames of users who belong to more than one group, one username per line, sorted alphabetically.

The `users.txt` file contains a list of usernames, one per line. The `groups.txt` file contains group names and their members, in the format `group_name:user1,user2,user3`.

Test: Running md5sum /home/admin/multi-group-users.txt returns dc0ae86caae7125d21df03a0ab29d8ae

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Batumi": Troubleshoot "A" cannot connect to "B"

Level: Medium

Type: Fix

Access: Paid

Description: (To learn the skills to solve this challenge, see Can't Connect to a Service: Linux Troubleshooting Guide)

There is a web server (Caddy) on HTTP port :80 but curl http://127.0.0.1 doesn't work. Find out what's wrong and make the necessary fixes so the web server returns a URL.

Note: as a limitation, the file /home/admin/db_connector.py must not be modified so that the challenge is considered solved properly.
The web server has to respond on the IP address 127.0.0.1; not only on "localhost".

Test: The command curl http://127.0.0.1 returns a URL address.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Bengaluru": Kubernetes StatefulSet least known characteristic

Level: Medium

Type: Fix

Access: Email

Description: There's a Kubernetes cluster (created with "k3d") with two worker nodes and two pods on the node k3d-cluster-agent-0: a Deployment demo-deployment-... and a StatefulSet demo-statefulset-0. Their manifests are identical except for the different kind of K8s resource.

Make the node hosting the pods unavailable (it "goes down" or "crashes" without being deleted from k8s), for example with: docker stop k3d-mycluster-agent-0.

After waiting for about a minute (tolerationSeconds in the manifest is 30s, we shorten the K8S 5 minutes default so you don't have to wait so much, plus a grace period), both pods are marked as Terminating. While the Deployment pod is evicted and deployed onto the remaining available node k3d-cluster-agent-1, the StatefulSet demo-statefulset-0 is not (Why?).

Make the StatefulSet pod demo-statefulset-0 run on the available node.

Note: you can use k as a shortcut for kubectl.

Test: Node k3d-cluster-agent-0 is NotReady. Both the Deployment and the StatefulSet are running on the node k3d-cluster-agent-1
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Constanta": Jumping Frog

Level: Medium

Type: Hack

Access: Email

Description: This is a "hacking" or Capture The Flag challenge. You need to copy the message at /home/user3/secret.txt into the /home/admin/solution.txt file.

Test: Running md5sum /home/admin/solution.txt returns the hash 7fe16554d0b326309d980314cebc2994

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Bharuch": Lost in Translation

Level: Medium

Type: Fix

Access: Paid

Description: There's a Docker container that runs a web server on port 3000, but it's not working.

Using the tooling and resources provided in the server, make the container run correctly.

Test: curl http://localhost:3000 should return "Hello from sadservers!"

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Quito": Control One Container from Another

Level: Medium

Type: Do

Access: Email

Description: You have a running container named docker-access. Another container nginx is present but in a stopped state. Your goal is to start the nginx container from inside the docker-access container.

You must not start the nginx container from the host system or any other container that is not docker-access. You can restart this docker-access container.

Test: Executing docker ps inside the docker-access container: docker exec docker-access docker ps succeeds.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Bangalore": Envoy Panics

Level: Medium

Type: Fix

Access: Email

Description: You have an Envoy proxy routing traffic to two unhealthy backend services.

When the number of healthy backends falls under a panic threshold (default 50%), Envoy enters a panic mode and it will either send traffic to all upstream hosts or to none at all.

We are simulating this condition by returning an HTTP 503 status code from the /health endpoint in both backends. In our case Envoy is sending traffic to all upstream services.

Your objective is to change the panic Envoy behaviour so that it does not route any traffic to unhealthy services and instead Envoy returns "no healthy upstream". (Do not change anything in the backend services). There can also be other Envoy configuration issues you need to fix.

Test: Running curl localhost:10000 should return no healthy upstream.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Budapest": User Creation

Level: Medium

Type: Do

Access: Email

Description: Given the file user_list.txt you must create all the users specified in the file with their corresponding passwords.

The entries in the user_list.txt file are stored as username;password

Test: All users are created with the right password.
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Tokelau": Delete from history

Level: Medium

Type: Do

Access: Email

Description: Inspired by this nixCraft article.

The objective of this exercise is to delete all the Bash history lines that contain the term foo.

Clearing out or deleting the history file /home/admin/.bash_history is not allowed. Note that in our case, new commands (including the ones to try and delete "foo" from history) are also appended to the history file.

Test: Running history |grep "foo" returns nothing.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Bizerte": The Slow Application

Level: Medium

Type: Fix

Access: Email

Description: A Python web application running on port 5000 from the /opt directory is experiencing severe performance issues; every request takes more than 5 seconds to complete.
The application is supposed to use the redis-server cache service for speed.

Your mission is to diagnose the performance bottleneck and restore the application to its normal, fast response time.

Do not change the Python application file slow_app.py.

Test: curl localhost:5000 returns Data from FAST cache!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Ruaka": Kubernetes pod in distress

Level: Medium

Type: Fix

Access: Paid

Description: A developer wants to deploy an open-source tool on Kubernetes. The tool unfortunately has limited documentation.

They built a helm chart and a container image. When the application is deployed, for some reason the server in Kubernetes doesn't seem to work but when the binary is started on their laptop/machine it works perfectly.

The application server is deployed by Helm. The command they used is: helm upgrade --install ruaka charts/ruaka.

Debug and help the developer find the issue. NOTE: Do not change or delete any current Helm field value in the chart, only add if needed.

Remember to give enough time to k8S after you apply a change before checking the solution.

Test: kubectl get pod shows the ruaka application pod up and running, while no Helm fields have been taken out from the applicaiton chart.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Campina Grande": Give me my cert, Vault

Level: Medium

Type: Fix

Access: Paid

Description: A web application running at https://nginx.example.com has an expired certificate. Issue a new certificate using the Hashicorp Vault running on the server.
The Vault instance is already unsealed and initialized, and you have full admin access with the admin user.

Test: Running curl https://nginx.example.com returns Hello!.

The certificate presented by Nginx is issued by the Vault PKI (check using openssl verify -CAfile /usr/local/share/ca-certificates/vault-pki-ca.crt /etc/nginx/ssl/cert.pem).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Kampala": Strange Script Error

Level: Medium

Type: Fix

Access: Email

Description: A developer has been working on Linux deployment scripts on their machine and then transferred the files to a Linux server. However, when they try to execute the scripts, they encounter the mysterious error:

-bash: cannot execute: required file not found

The scripts appear to be syntactically correct, but something is preventing them from executing properly. The developer needs your help to identify and fix the issue so the deployment can proceed.

There are several script files in /home/admin/deploy/ that need to be fixed before the deployment process can work correctly.

Test: All script files in /home/admin/deploy/ should execute without the cannot execute: required file not found error.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Bondo": Split my pile!

Level: Medium

Type: Fix

Access: Email

Description: A developer wants to run a program that splits their pile of their data into compressed parts for efficient transport across their network. Unfortunately when the tool runs it never completes.

The application binary in question is called bondo located in /home/admin/bondo.

Run it, then debug and help the developer find the issue.

Test: Executing /home/admin/bondo as admin returns part files generation completed!.

The file /etc/fstab has not been modified and the solution would work on reboot.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Atlantis": Not found

Level: Medium

Type: Fix

Access: Paid

Description: There is a small "C" application in the /home/admin/app directory. Create the Docker container "app" with a small footprint and minimalistic so you get a hello binary that returns a greeting in Atlantean (Docker multi-stage build). The binary application is automatically called when running docker run app

Test: docker run app returns SOO-puhk

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Kilifi": Speculative Misallocation!

Level: Medium

Type: Fix

Access: Email

Description: A developer is having trouble deploying an application on a preconfigured cluster.

The application kilifi is to be deployed on kubernetes in the default namespace.
The application server is deployed by helm. The command they used is helm install kilifi charts/kilifi.
The application operates correctly with ~210 MB of memory, but 256 MB is recommended.

Swap should remain disabled in the cluster.

Debug and help the developer fix any issue with deployment.

Test: The kilifi application runs properly; it's Service on :3333/healthz returns "kilifi ready to serve".

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Solanea": ClickHouse mad house

Level: Medium

Type: Do

Access: Paid

Description: You have a ClickHouse installation CHI running on a Kubernetes cluster and a set of requests (located at ~/data/requests.csv) that you must populate into the http_requests table in the monitoring database (table may not exist in all pod instances).
Do this insert in all pod instances of the database.
The user and password to connect to the database are default.
The keeper pods provide clickhouse replication services.

Test: You are able to query the database and see the data:

clickhouse-client -h --password default -q 'SELECT COUNT(*) FROM monitoring.http_requests'

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Tunis": Redis Replication Problem

Level: Medium

Type: Fix

Access: Paid

Description: A Redis master-replica setup is running on this server, with the master on port 6379 and the replica on port 6380. Both instances show as "connected" when you check their status, but data synchronization has silently broken.

Recent writes to the master don't appear on the replica, even though there are no obvious errors in the logs and both Redis instances appear healthy.

Fix the replication issues so that data written to the master (port 6379) immediately appears on the replica (port 6380) without data loss.

Master: localhost:6379
Replica: localhost:6380
Password: masterpass123

A helper test script is available at /home/admin/test_replication.sh

Test: The solution will be validated by writing a test key to the master and verifying it appears on the replica within 2 seconds.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Toulon": Denied Lamp

Level: Medium

Type: Fix

Access: Business

Description: The security team has asked again Mary and John to implemente more security measures. Unfortunately, this time they have broken the LAMP stack (Apache with PHP) so the frontend is unable get an answer upstream, thus, they need your help again to fix it.

The fixed application should be able to serve the content from the webserver, the problem is a network connectivity, although the logs have valuable informatiion, it has nothing to do with the configuration of the apache server.

Test: curl localhost | head -n1 returns SadServers - LAMP Stack

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Bermuda": Command not found

Level: Medium

Type: Fix

Access: Business

Description: While working with a distro with a very small footprint, we just found out that there are some basic commands not present, this was supposed to be a security feature, after all this is just a small server, however, the web content was not deployed. Your task is to decompress the file /home/admin/web.zip and move the file home.html in it to /var/www/html/index.html

Test: The service must return the string "Homepage". You can check with the command curl -s localhost

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Auderghem": Containers miscommunication

Level: Medium

Type: Fix

Access: Paid

Description: There is an nginx Docker container that listens on port 80, the purpose of which is to redirect the traffic to two other containers statichtml1 and statichtml2 but this redirection is not working.
Fix the problem.

IMPORTANT. You can restart all containers, but don't stop or remove them.

Test: The nginx container must redirect the traffic to the statichtml1 and statichtml2 containers:

curl http://localhost returns the Welcome to nginx default page
curl http://localhost/1 returns HelloWorld;1
curl http://localhost/2 returns HelloWorld;2

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Marseille": Rocky security

Level: Medium

Type: Fix

Access: Paid

Description: As the Christmas shopping season approaches, the security team has asked Mary and John to implemente more security measures. Unfortunately, this time they have broken the LAMP stack; the frontend is unable get an answer from upstream, thus they need your help again to fix it.

The application should be able to serve the content from the webserver.

Note for Pro users: direct SSH access is not avaiable (yet) for this scenario.

Test: curl localhost | head -n1 returns SadServers - LAMP Stack

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Woluwe": Too many images

Level: Medium

Type: Fix

Access: Paid

Description: A pipeline created a lot of Docker images locally for a web app. All these images except for one contain a typo introduced by a developer: there's an incorrect image instruction to pipe "HelloWorld" to "index.htmlz" instead of using the correct "index.html"
Find which image doesn't have the typo (and uses the correct "index.html"), tag this correct image as "prod" (rather than fixing the current prod image) and then deploy it with docker run -d --name prod -p 3000:3000 prod so it responds correctly to HTTP requests on port :3000 instead of "404 Not Found".

Test: curl http://localhost:3000 should respond with HelloWorld;529

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "La Rinconada": Elevating privileges

Level: Medium

Type: Hack

Access: Email

Description: You are logged in as the user "admin" without general "sudo" privileges.
The system administrator has granted you limited "sudo" access; this was intended to allow you to read log files.

Your mission is to find a way to exploit this limited sudo permission to gain a full root shell and read the secret file at /root/secret.txt
Copy the content of /root/secret.txt into the /home/admin/solution.txt file, for example: cat /root/secret.txt > /home/admin/solution.txt (the "admin" user must be able to read the file).

Test: As the user "admin", md5sum /home/admin/solution.txt returns 52a55258e4d530489ffe0cc4cf02030c (we also accept the hash of the same secret string without an ending newline).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Annapurna": High privileges

Level: Medium

Type: Hack

Access: Email

Description: You are logged in as the user admin.

You have been tasked with auditing the admin user privileges in this server; "admin" should not have sudo (root) access.

Exploit this server so you as the admin user can read the file /root/mysecret.txt
Save the content of /root/mysecret.txt to the file /home/admin/mysolution.txt , for example: echo "secret" > ~/mysolution.txt

Test: Running md5sum /home/admin/mysolution.txt returns 47ee165a2262476f6866902a93f2a41d. (We also accept the md5sum of the same file without a newline at the end).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Podgorica": Docker to Podman migration

Level: Medium

Type: Do

Access: Paid

Description: You have been tasked with migrating this future web server from using Docker (which uses a daemon) to rootless Podman.
There is already an Nginx Podman image on the server, and your objective is to manage the container created from it using systemd, so the it starts automatically on reboot and continues running unless explicity stopped (the same behaviour expected from a Docker-managed container).
Create a systemd service named container-nginx.service that manages the Podman Nginx container. Enable and start this service.

NOTES: Although a quadlet file solution should be valid, the check script is still not accounting for it.

There is no need to reboot the VM, although if you want you could reboot it from the command line with /sbin/shutdown -r now and refresh or reopen the web console.

Test: The checker script will test if the container-nginx.service is active and enabled, and if it can stop and start the service. It will also verify that curl localhost:8888 returns the default "Welcome to nginx" web page.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Torino": Optimize grande Docker image

Level: Medium

Type: Do

Access: Paid

Description: A Torino Node.js application is located in the ~/torino-app directory.
You can run it directly with: nohup node app.js > app.log 2>&1 &. You can also verify that it works by running: curl localhost:3000

There is already a torino Docker image built with the Dockerfile in ~/torino-app, but the resulting image size is 916 MB.

Your task is to optimize the Docker image size:
1. Build a new Docker image for the Torino application, also called torino:latest but with a total size under 122 MB
2. Create and run a container using this optimized image.

NOTE: You can only use the existing Docker images in the server.
To build a Node application you need to COPY in your Dockerfile, besides the app.js , the package*.json files and without Internet access, the node_modules directory, since you cannot RUN npm install.

Test: The torino Docker image is less than 122 MB and curl http://localhost:3000 returns Hello from Torino!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Socorro, NM": Optimize Podman image

Level: Medium

Type: Do

Access: Paid

Description: The podman image localhost/prod:latest contains a static website.
Initially the image size is 261 MB and contains 100 layers.

Your task:
1. Optimize the image localhost/prod:latest so that its size is less than 200 MB, using the same tag.
2. Run a container named "check" from the optimized image: podman run -d --name check -p 8888:80 localhost/prod:latest so that curl localhost:8888 returns 100 lines.

Test: The podman image localhost/prod:latest size is less than 200 MB and running curl localhost:8888 from a container named "check" created from the image retuns 100 lines.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Sumé": Tied in a Knot

Level: Medium

Type: Fix

Access: Email

Description: A DNS server running Knot DNS is serving the zone sadservers.internal (see ls /var/lib/knot/zones/), but users are reporting that they cannot access blog.sadservers.internal neither api.sadservers.internal. Your task is to diagnose and fix the DNS issues so the services become accessible.
You can manage Knot DNS with sudo knotc commands.

Note: the 203.0.113.0/24 range is part of TEST-NET-3, a block reserved by RFC 5737 for documentation and examples, making it a Bogon IP range.

IMPORTANT. Do not change the Nginx configurations under /opt/services/ for the solution to work.

Test: You are able to access the blog and the API services: curl blog.sadservers.internal returns Welcome to blog.sadservers.internal
curl api.sadservers.internal returns {"status": "ok", "service": "api.sadservers.internal"}

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "San Juan": mucho Traefik

Level: Medium

Type: Fix

Access: Email

Description: There is a Traefik load balancer that must be up and running. The server and the backend services are managed by Docker Compose. Running curl -s app.sadserver | head -n1 must return the host ID of one of the backend servers, running the command again must return a new host ID. The server seems to be working some times, some others fails or just times out.

The round-robin configuration should make the webserver iterate through the back-end servers.

Test: curl -s app.sadserver | head -n1 returns something like Hostname:

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Suzhou": MongoDB replicas!

Level: Medium

Type: Fix

Access: Email

Description: A new MongoDB replica set has been setup in the development environment trough /home/admin/app/rs0.js, however, a variety or errors are showing up when trying to bring it up. You should bring up all the replica servers, get them communicating to each other and make sure the replica set is working as it should.

The status of the first replica can be checked via systemctl status mongo1 same for the replicas mongo2 and mongo3. The logs are also in a separate file for each replica under the directory /var/log/mongodb. To initilize the replica set again: mongosh --file app/rs0.js

Note: The default configuration file /etc/mongo.conf is not the problem.

Test: mongosh --eval "rs.status()" | grep health returns the status of all the replicas

       health: 1,       health: 1,       health: 1, 

Time to Solve: 20 minutes.

Scenario: "Jakarta": it's always DNS.

Level: Hard

Type: Fix

Access: Public

Description: Can't ping google.com. It returns ping: google.com: Name or service not known. Expected is being able to resolve the hostname. (Note: currently the VMs can't ping outside so there's no automated check for the solution).

Test: ping google.com should return something like PING google.com (172.217.2.46) 56(84) bytes of data.

Time to Solve: 20 minutes.

Scenario: "Bern": Docker web container can't connect to db container.

Level: Hard

Type: Fix

Access: Email

Description: There are two Docker containers running, a web application (Wordpress or WP) and a database (MariaDB) as back-end, but if we look at the web page, we see that it cannot connect to the database. curl -s localhost:80 |tail -4 returns:

<body id="error-page"> <div class="wp-die-message"><h1>Error establishing a database connection</h1></div></body> </html>

This is not a Wordpress code issue (the image is :latest with some network utilities added). What you need to know is that WP uses "WORDPRESS_DB_" environment variables to create the MySQL connection string. See the ./html/wp-config.php WP config file for example (from /home/admin).

Test: sudo docker exec wordpress mysqladmin -h mysql -u root -ppassword ping . The wordpress container is able to connect to the database in the mariadb container and returns mysqld is alive.

Time to Solve: 20 minutes.

Scenario: "Singara": Docker and Kubernetes web app not working.

Level: Hard

Type: Fix

Access: Email

Description: There's a k3s Kubernetes install you can access with kubectl. The Kubernetes YAML manifests under /home/admin have been applied. The objective is to access from the host the "webapp" web server deployed and find what message it serves (it's a name of a town or city btw). In order to pass the check, the webapp Docker container should not be run separately outside Kubernetes as a shortcut.

Test: curl localhost:8888 returns a value from the webapp deployed Kubernetes pod.

Time to Solve: 20 minutes.

Scenario: "Hong-Kong": can't write data into database.

Level: Hard

Type: Fix

Access: Paid

Description: (Similar to "Manhattan" scenario but harder). Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don't need to know details about it (although it may help).

Postgres information: it's a service that listens to a port (:5432) and writes to disk in a data directory, the location of which is defined in the data_directory parameter of the configuration file /etc/postgresql/14/main/postgresql.conf. In our case Postgres is managed by systemd as a unit with name postgresql.

Test: sudo -u postgres psql -c "insert into persons(name) values ('jane smith');" -d dt

Should return:INSERT 0 1

Time to Solve: 20 minutes.

Scenario: "Pokhara": SSH and other sshenanigans

Level: Hard

Type: Fix

Access: Paid

Description: A user client was added to the server, as well as their SSH public key.
The objective is to be able to SSH locally (there's only one server) as this user client using their ssh keys. This is, if as root you change to this user sudo su; su client, you should be able to login with ssh: ssh localhost.

Test: As user admin: sudo -u client ssh client@localhost 'pwd' returns /home/client

Time to Solve: 30 minutes.

Scenario: "Roseau": Hack a Web Server

Level: Hard

Type: Hack

Access: Email

Description: There is a secret stored in a file that the local Apache web server can provide. Find this secret and have it as a /home/admin/secret.txt file.

Note that in this server the admin user is not a sudoer.

Also note that the password crackers Hashcat and Hydra are installed from packages and John the Ripper binaries have been built from source in /home/admin/john/run

Test: sha1sum /home/admin/secret.txt |awk '{print $1}' returns cc2c322fbcac56923048d083b465901aac0fe8f8

Time to Solve: 30 minutes.

Scenario: "Belo-Horizonte": A Java Enigma

Level: Hard

Type: Fix

Access: Paid

Description: (Credit for the idea: fuero)

There is a one-class Java application in your /home/admin directory. Running the program will print out a secret code, or you may be able to extract the secret from the class file without executing it but I'm not providing any special tools for that.

Put the secret code in a /home/admin/solution file, eg echo "code" > /home/admin/solution.

Test: md5sum /home/admin/solution |awk '{print $1}' returns 9d2bd7aabb26681eacd9444da6b6643c

Time to Solve: 20 minutes.

Scenario: "Chennai": Pull a Rabbit from a Hat

Level: Hard

Type: Fix

Access: Paid

Description: There is a RabbitMQ (RMQ) cluster defined in a docker-compose.yml file.

Bring this system up and then run the producer.py script in such a way that is able to send messages to RMQ. In particular you have to send the message "hello-lwc".

- RMQ is a queuing system: messages are put in the queue with a "producer" and they are taken out from the other side by a "consumer". The queue name has to be the same for both.

- To send the message "hello-lwc": python3 ~/producer.py hello-lwc. Should return Message sent to RabbitMQ. "IncompatibleProtocolError" means RMQ is not working properly.

- To test consuming it: python3 ~/consumer.py, this will retrieve the next message from the queue and print it. Once everything is working send more than one message so there's at least one in the queue when the validation runs.

- Do not change the consumer.py and producer.py files; if you do the Check My Solution will fail.

Test: python3 ~/consumer.py returns hello-lwc

See /home/admin/agent/check.sh for the exact test.

Time to Solve: 30 minutes.

Scenario: "Monaco": Disappearing Trick

Level: Hard

Type: Hack

Access: Email

Description: There is a web server on :5000 with a form. POSTing the correct form password into this web service will return a secret.

Save this secret provided by the web page (not the password you sent to it) to /home/admin/mysolution, for example: echo "SecretFromWebSite" > ~/mysolution

TIP: a developer worked on the web server code in this VM, using the same 'admin' account.

Scenario credit: PuppiestDoggo

Test: md5sum /home/admin/mysolution returns a250aa19f16dda6f9fcef286f035ec4b

Time to Solve: 30 minutes.

Scenario: "Florence": Database Migration Hell

Level: Hard

Type: Fix

Access: Paid

Description: You are working as a DevOps Engineer in a company and another team member left the company and left the docker-compose.yml of a database-backed web application unfinished.

Generally, the problem revolves around the database migration and docker compose.

Additionally on front of the application there is an Nginx server and you need to fix the proper access to it as well.

The source of code is in /home/admin/app

Credit Kamil Błaż

Test: curl --cacert /etc/nginx/certs/sadserver.crt https://sadserver.local returns a message containing "ready to serve requests"

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute

Time to Solve: 30 minutes.

Scenario: "Zaragoza": Test changing critical files

Level: Hard

Type: Do

Access: Paid

Description: The goal is to make the script /home/admin/agent/check.sh return OK, without editing the original /etc/hosts file.

Think of testing changes in the critical directory /etc in a safe way. In this case, adding "127.0.0.1 my.local.test" to /etc/hosts .

There would be many ways of trying to do this with "sudo" access, like the usual procedure of making a copy of the config file, editing there and copying or renaming back to the original file. In our case, to avoid all those simple solutions, there is no general "sudo" privileges in this scenario (but there may be for some commands).

Test: The string my.local.test is in /etc/hosts

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Amygdala": Do you have enough insight to see the secrets?

Level: Hard

Type: Fix

Access: Paid

Description: Troubleshoot and fix a Kubernetes web application running in the app namespace. Make the deployment run successfully so that it returns Hello handsome! when you curl it.

Fix first your admin user access to the local Kubernetes cluster; the KUBECONFIG environment variable must be set to $HOME/.kube/config.

You have full admin access to a Vault server (containing the secrets you need) from the admin user. All the used manifests for the application are placed on the /home/admin/manifests directory.

Test: Running: POD_IP=$(kubectl get po -n app -l app=app -o jsonpath='{.items[0].status.podIP}') curl http://$POD_IP returns Hello handsome!.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Tigoni": Patch and Pray!

Level: Hard

Type: Fix

Access: Email

Description: A developer wants to upgrade their stateful application. This application handles their archives/backups.

The application tigoni is deployed on kubernetes in the default namespace.
The application server is deployed by Helm. The command they used is helm install tigoni charts/tigoni.

Upgrade the tigoni application to v2.0.0. The image already exists in the local repository.
Debug and help the developer fix any issue with the upgrade.
Everytime v1.0.0 is launched the archiving code starts on a clean slate.

Test: The tigoni pod with version v2 runs correctly (its endpoint :3000/healthz displays serverVersion:v2)

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Cabedelo": Harbor full of issues

Level: Hard

Type: Fix

Access: Paid

Description: You need to build and push a docker image without changing the Dockerfile to your company's Harbor registry, which is running at harbor.sadservers.local, with its home directory at /opt/harbor. You have full admin access with admin:Harbor12345 credential. The source code and the Dockerfile are in the ~/app directory. The image name must be harbor.sadservers.local/images/app:1.0.0. It is also expected that the application will be up and running at localhost:5000 in a container named app.

IMPORTANT. Do not:
1. Generate new internal certificates
2. Change the Dockerfile
3. Change the /opt/harbor.yml file

Test: You are able to pull the application image from Harbor:
docker rmi harbor.sadservers.local/images/app:1.0.0
docker pull harbor.sadservers.local/images/app:1.0.0


You can access the application; curl localhost:5000 returns Hello world!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Karakorum": WTFIT – What The Fun Is This?

Level: Hard

Type: Fix

Access: Paid

Description: (NOTE: this is not a new scenario but an existing Pro one temporarily available to all users as the last Advent of SysAdmin 2025 scenario).

There's a binary at /home/admin/wtfit that nobody knows how it works or what it does ("what the fun is this"). Someone remembers something about wtfit needing to communicate to a service in order to start.

Run this wtfit program so it doesn't exit with an error, fixing or working around things that you need but are broken in this server.

Test: Running /home/admin/wtfit returns OK.

Time to Solve: 20 minutes.