Pro Troubleshooting Scenarios

Scenario: "Apia": Needle in a Haystack

Level: Easy

Type: Do

Access: Paid

Description: In a directory /home/admin/data, there are multiple files, all of them with same content. One of these files has been modified, a word was added. You need to identify which word it is and put it in the solution file (both newline terminated or not are accepted).

Test: md5sum /home/admin/solution should return 55aba155290288b58e9b778c8f616560 or 2eeefea9fc4b16ea624bed5c67a49d80

Check My Solution: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can

Time to Solve: 20 minutes.

Scenario: "Gitega": Find the Bad Git Commit

Level: Easy

Type: Do

Access: Paid

Description: The directory at /home/admin/git has a Git repository with a Golang program and a test for it.

To execute the test, from this "git" directory run: go test. The last (current HEAD) commit fails the test. Suppose the first commit passed the test.

Find the (long hash) commit that first broke the test and enter it in the /home/admin/solution file. For example: echo 9e80a7eb1b09385e93ab4a76cb2c93beec48fd9f > /home/admin/solution

Test: Doing md5sum /home/admin/solution returns f7db1bb6b7bfcd66a4eb66782804b39d.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Yokohama": Linux Users Working Together

Level: Easy

Type: Fix

Access: Paid

Description: There are four Linux users working together in a project in this server: abe, betty, carlos, debora.

First, they have asked you as the sysadmin, to make it so each of these four users can read the project files of the other users in the /home/admin/shared directory, but none of them can modify a file that belongs to another user. Users should be able modify their own files.

Secondly, they have asked you to modify the file shared/ALL so that any of these four users can write more content to it, but previous (existing) content cannot be altered.

Test: All users (abe, betty, carlos, debora) can write to their own files. None of them can write to another user's file.
All users can add more content (append)) to the shared/project_ALL file but none can change its current content.
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Fukuoka": Forbidden Association

Level: Easy

Type: Fix

Access: Paid

Description: There's a web server running on this host but curl localhost returns the default 404 Not Found page.

Fix the issue so that a file is served correctly and the message Welcome to the Real Site! is returned.

Test: Running curl localhost should return HTTP 200 with the message Welcome to the Real Site!.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Melbourne": WSGI with Gunicorn

Level: Medium

Type: Fix

Access: Paid

Description: There is a Python WSGI web application file at /home/admin/wsgi.py , the purpose of which is to serve the string "Hello, world!". This file is served by a Gunicorn server which is fronted by an nginx server (both servers managed by systemd). So the flow of an HTTP request is: Web Client (curl) -> Nginx -> Gunicorn -> wsgi.py . The objective is to be able to curl the localhost (on default port :80) and get back "Hello, world!", using the current setup.

Test: curl -s http://localhost returns Hello, world! (serving the wsgi.py file via Gunicorn and Nginx)

Time to Solve: 20 minutes.

Scenario: "Unimak Island": Fun with Mr Jason

Level: Medium

Type: Do

Access: Paid

Description: Using the file station_information.json , find the station_id where "has_kiosk" is false and "capacity" is greater than 30.

Save the station_id of the solution in the /home/admin/mysolution file, for example: echo "ec040a94-4de7-4fb3-aea0-ec5892034a69" > ~/mysolution

You can use the installed utilities jq, gron, jid as well as Python3 and Golang.

Test: md5sum /home/admin/mysolution returns 8d8414808b15d55dad857fd5aeb2aebc

Time to Solve: 15 minutes.

Scenario: "Ivujivik": Parlez-vous Français?

Level: Medium

Type: Do

Access: Paid

Description: Given the CSV file /home/admin/table_tableau11.csv, find the Electoral District Name/Nom de circonscription that has the largest number of Rejected Ballots/Bulletins rejetés and also has a population of less than 100,000.

The initial CSV file may be corrupted or invalid in a way that can be fixed without changing its data.

Installed in the VM are: Python3, Go, sqlite3, miller directly and PostgreSQL, MySQL in Docker images.

Save the solution in the /home/admin/mysolution , with the name as it is in the file, for example: echo "Trois-Rivières" > ~/mysolution (the solution must be terminated by newline).

Test: md5sum /home/admin/mysolution returns e399d171f21839a65f8f8ab55ed1e1a1

Time to Solve: 20 minutes.

Scenario: "Tarifa": Between Two Seas

Level: Medium

Type: Fix

Access: Paid

Description: There are three Docker containers defined in the docker-compose.yml file: an HAProxy accepting connetions on port :5000 of the host, and two nginx containers, not exposed to the host.

The person who tried to set this up wanted to have HAProxy in front of the (backend or upstream) nginx containers load-balancing them but something is not working.

Test: Running curl localhost:5000 several times returns both hello there from nginx_0 and hello there from nginx_1

Check /home/admin/agent/check.sh for the test that "Check My Solution" runs.

Time to Solve: 20 minutes.

Scenario: "Abaokoro": Restore MySQL Databases Spooked by a Ghost

Level: Medium

Type: Fix

Access: Paid

Description: There are three databases that need to be restored. You need to create three databases called "first", "second" and "third" and restore the databases using the file "/home/admin/dbs_to_restore.zip".
If you encounter an issue while restoring the database, fix it.

Credit: Sebastian Segovia

Test: All databases, once restored, have a table named "foo".

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Poznań": Helm Chart Issue in Kubernetes

Level: Medium

Type: Fix

Access: Paid

Description: NOTE: Prompt may take a few extra seconds to be responsive while the k3s environment gets ready. Root access is not needed for this challenge ("admin" user cannot sudo).

A DevOps engineer created a Helm Chart web_chart with a custom nginx site, however he still gets the default nginx index.html.

You can check for example with POD_IP=$(kubectl get pods -n default -o jsonpath='{.items[0].status.podIP}') and curl -s "${POD_IP}">.

In addition he should set replicas to 3.

The chart is not working as expected. Fix the configurations so you get the custom HTML page from any nginx pod.

Credit Kamil Błaż

Test: Doing curl on the default port (:80) of any nginx pod returns a Welcome SadServers page. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Manado": How much do you press?

Level: Medium

Type: Do

Access: Paid

Description: You have been tasked with compressing the file /home/admin/names, which is 35147 bytes, to a size smaller than 9400 bytes. You can use any compressing tool at your disposal (there are many available in the server), also you can modify the file without deleting anything in it. Put the solution (compressed file) in the /home/user/admin/solution directory with the default extension used by the compression tool (example: ~/solution/names.gzip).

Test: The size of the compressed file is smaller than 9400 bytes.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Warsaw": Prometheus can't scrape the webserver

Level: Medium

Type: Fix

Access: Paid

Description: A developer created a golang application that is exposing the /metrics endpoint. They have a problem with scraping the metrics from the application. They asked you to help find the problem.

Full source code of the application is available at the /home/admin/app directory.

Credit Kamil Błaż

Test: The endpoint http://localhost:9000/metrics should return HTTP code 200.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Bekasi": Supervisor is still around

Level: Medium

Type: Fix

Access: Paid

Description: There is an nginx service running on port 443, it is the main web server for the company and looks like a new employee has deployed some changes to the configuration of supervisor and now it is not working as expected.

If you try to access curl -k https://bekasi it should return Hello SadServers! but for some reason it is not.

You cannot modify files from the /home/admin/bekasi folder in order to pass the check.sh

You must find out what the issue is and fix it.

Test: curl -k https://bekasi returns Hello SadServers!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Depok": Nginx with Brotli

Level: Medium

Type: Fix

Access: Paid

Description: You are tasked to add compression to the company website. The website is running on an Nginx server, and you decide to add Brotli compression to it.

Brotli has became very popular these days because of its high compression ratio. It's a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding, and 2nd order context modeling.

For this purpose, you decided to compile the brotli modules yourself and add them to the Nginx server.

The location of the Brotli source code is at /home/admin/ngx_brotli. The nginx source code (needed to compile the modules) is located at /home/admin/nginx-1.18.0. From the ngx_brotli repository first you need to compile the brotli dependencies and then configure and make modules for Nginx. Afer that you need to add the modules to the Nginx configuration.

After installing the modules, you need to make sure the responses from the server are being served with compression.

Create a port-forward to port 80 from the server to your computer and check the header Content-Encoding, responses must return br for Brotli compression. You can also use curl -H "Accept-Encoding: br, gzip" -I http://localhost to check the header.

Something nice about Brotli is that it fails over to gzip if the client doesn't support Brotli, so curl -H "Accept-Encoding: gzip" -I http://localhost should return gzip instead.

Test: curl -H "Accept-Encoding: br" -sI http://localhost returns the header Content-Encoding: br.

The "Check My Solution" button runs the script _/home/admin/agent/check.sh_, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Tukaani": XZ LZMA Library Compromised

Level: Medium

Type: Fix

Access: Paid

Description: (You can learn about Linux Libraries before starting this scenario).

The Linux shared library liblzma.so has been compromised (the real compromised XZ Utils liblzma has not been used). The liblzma.so at the path /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5 is the good one. Consider the same library liblzma.so.5.2.5 at other paths as compromised or malicious (ideally we would have used other real versions with different checksums).

Find all instances of this "malicious" liblzma library (remember, it's the same library but in different directory locations) and make it so none of the running processes use it, while the applications "webapp" and "jobapp" (both of which managed by systemd) still run properly (eg, stopping those applications is not a solution).

Test: lsof | grep liblzma.so.5 returns only the liblzma in the path: /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Atrani": Modify a SQlite3 Database

Level: Medium

Type: Fix

Access: Paid

Description: A developer created a script /home/admin/readdb.py that tests access to a database. Without modifying the readdb.py file, change the database so that running the script returns the string "John Karmack".

Test: Running /home/admin/readdb.py returns "John Karmack".

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Hanoi": Find the Multitasking Users

Level: Medium

Type: Do

Access: Paid

Description: The Hanoi office has a Linux server with a large number of user accounts and groups. The system administrators need to identify which users belong to multiple groups for better access management.

Given two files, `users.txt` and `groups.txt`, create a new file `/home/admin/multi-group-users.txt` containing the usernames of users who belong to more than one group, one username per line, sorted alphabetically.

The `users.txt` file contains a list of usernames, one per line. The `groups.txt` file contains group names and their members, in the format `group_name:user1,user2,user3`.

Test: Running md5sum /home/admin/multi-group-users.txt returns dc0ae86caae7125d21df03a0ab29d8ae

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Batumi": Troubleshoot "A" cannot connect to "B"

Level: Medium

Type: Fix

Access: Paid

Description: (To learn the skills to solve this challenge, see Can't Connect to a Service: Linux Troubleshooting Guide)

There is a web server (Caddy) on HTTP port :80 but curl http://127.0.0.1 doesn't work. Find out what's wrong and make the necessary fixes so the web server returns a URL.

Note: as a limitation, the file /home/admin/db_connector.py must not be modified so that the challenge is considered solved properly.
The web server has to respond on the IP address 127.0.0.1; not only on "localhost".

Test: The command curl http://127.0.0.1 returns a URL address.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Bharuch": Lost in Translation

Level: Medium

Type: Fix

Access: Paid

Description: There's a Docker container that runs a web server on port 3000, but it's not working.

Using the tooling and resources provided in the server, make the container run correctly.

Test: curl http://localhost:3000 should return "Hello from sadservers!"

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Ruaka": Kubernetes pod in distress

Level: Medium

Type: Fix

Access: Paid

Description: A developer wants to deploy an open-source tool on Kubernetes. The tool unfortunately has limited documentation.

They built a helm chart and a container image. When the application is deployed, for some reason the server in Kubernetes doesn't seem to work but when the binary is started on their laptop/machine it works perfectly.

The application server is deployed by Helm. The command they used is: helm upgrade --install ruaka charts/ruaka.

Debug and help the developer find the issue. NOTE: Do not change or delete any current Helm field value in the chart, only add if needed.

Remember to give enough time to k8S after you apply a change before checking the solution.

Test: kubectl get pod shows the ruaka application pod up and running, while no Helm fields have been taken out from the applicaiton chart.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Campina Grande": Give me my cert, Vault

Level: Medium

Type: Fix

Access: Paid

Description: A web application running at https://nginx.example.com has an expired certificate. Issue a new certificate using the Hashicorp Vault running on the server.
The Vault instance is already unsealed and initialized, and you have full admin access with the admin user.

Test: Running curl https://nginx.example.com returns Hello!.

The certificate presented by Nginx is issued by the Vault PKI (check using openssl verify -CAfile /usr/local/share/ca-certificates/vault-pki-ca.crt /etc/nginx/ssl/cert.pem).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Atlantis": Not found

Level: Medium

Type: Fix

Access: Paid

Description: There is a small "C" application in the /home/admin/app directory. Create the Docker container "app" with a small footprint and minimalistic so you get a hello binary that returns a greeting in Atlantean (Docker multi-stage build). The binary application is automatically called when running docker run app

Test: docker run app returns SOO-puhk

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Hong-Kong": can't write data into database.

Level: Hard

Type: Fix

Access: Paid

Description: (Similar to "Manhattan" scenario but harder). Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don't need to know details about it (although it may help).

Postgres information: it's a service that listens to a port (:5432) and writes to disk in a data directory, the location of which is defined in the data_directory parameter of the configuration file /etc/postgresql/14/main/postgresql.conf. In our case Postgres is managed by systemd as a unit with name postgresql.

Test: sudo -u postgres psql -c "insert into persons(name) values ('jane smith');" -d dt

Should return:INSERT 0 1

Time to Solve: 20 minutes.

Scenario: "Pokhara": SSH and other sshenanigans

Level: Hard

Type: Fix

Access: Paid

Description: A user client was added to the server, as well as their SSH public key.
The objective is to be able to SSH locally (there's only one server) as this user client using their ssh keys. This is, if as root you change to this user sudo su; su client, you should be able to login with ssh: ssh localhost.

Test: As user admin: sudo -u client ssh client@localhost 'pwd' returns /home/client

Time to Solve: 30 minutes.

Scenario: "Belo-Horizonte": A Java Enigma

Level: Hard

Type: Fix

Access: Paid

Description: (Credit for the idea: fuero)

There is a one-class Java application in your /home/admin directory. Running the program will print out a secret code, or you may be able to extract the secret from the class file without executing it but I'm not providing any special tools for that.

Put the secret code in a /home/admin/solution file, eg echo "code" > /home/admin/solution.

Test: md5sum /home/admin/solution |awk '{print $1}' returns 9d2bd7aabb26681eacd9444da6b6643c

Time to Solve: 20 minutes.

Scenario: "Chennai": Pull a Rabbit from a Hat

Level: Hard

Type: Fix

Access: Paid

Description: There is a RabbitMQ (RMQ) cluster defined in a docker-compose.yml file.

Bring this system up and then run the producer.py script in such a way that is able to send messages to RMQ. In particular you have to send the message "hello-lwc".

- RMQ is a queuing system: messages are put in the queue with a "producer" and they are taken out from the other side by a "consumer". The queue name has to be the same for both.

- To send the message "hello-lwc": python3 ~/producer.py hello-lwc. Should return Message sent to RabbitMQ. "IncompatibleProtocolError" means RMQ is not working properly.

- To test consuming it: python3 ~/consumer.py, this will retrieve the next message from the queue and print it. Once everything is working send more than one message so there's at least one in the queue when the validation runs.

- Do not change the consumer.py and producer.py files; if you do the Check My Solution will fail.

Test: python3 ~/consumer.py returns hello-lwc

See /home/admin/agent/check.sh for the exact test.

Time to Solve: 30 minutes.

Scenario: "Florence": Database Migration Hell

Level: Hard

Type: Fix

Access: Paid

Description: You are working as a DevOps Engineer in a company and another team member left the company and left the docker-compose.yml of a database-backed web application unfinished.

Generally, the problem revolves around the database migration and docker compose.

Additionally on front of the application there is an Nginx server and you need to fix the proper access to it as well.

The source of code is in /home/admin/app

Credit Kamil Błaż

Test: curl --cacert /etc/nginx/certs/sadserver.crt https://sadserver.local returns a message containing "ready to serve requests"

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute

Time to Solve: 30 minutes.

Scenario: "Zaragoza": Test changing critical files

Level: Hard

Type: Do

Access: Paid

Description: The goal is to make the script /home/admin/agent/check.sh return OK, without editing the original /etc/hosts file.

Think of testing changes in the critical directory /etc in a safe way. In this case, adding "127.0.0.1 my.local.test" to /etc/hosts .

There would be many ways of trying to do this with "sudo" access, like the usual procedure of making a copy of the config file, editing there and copying or renaming back to the original file. In our case, to avoid all those simple solutions, there is no general "sudo" privileges in this scenario (but there may be for some commands).

Test: The string my.local.test is in /etc/hosts

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Amygdala": Do you have enough insight to see the secrets?

Level: Hard

Type: Fix

Access: Paid

Description: Troubleshoot and fix a Kubernetes web application running in the app namespace. Make the deployment run successfully so that it returns Hello handsome! when you curl it.

Fix first your admin user access to the local Kubernetes cluster; the KUBECONFIG environment variable must be set to $HOME/.kube/config.

You have full admin access to a Vault server (containing the secrets you need) from the admin user. All the used manifests for the application are placed on the /home/admin/manifests directory.

Test: Running: POD_IP=$(kubectl get po -n app -l app=app -o jsonpath='{.items[0].status.podIP}') curl http://$POD_IP returns Hello handsome!.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Karakorum": WTFIT – What The Fun Is This?

Level: Hard

Type: Fix

Access: Paid

Description: (NOTE: this is not a new scenario but an existing Pro one temporarily available to all users as the last Advent of SysAdmin 2025 scenario).

There's a binary at /home/admin/wtfit that nobody knows how it works or what it does ("what the fun is this"). Someone remembers something about wtfit needing to communicate to a service in order to start.

Run this wtfit program so it doesn't exit with an error, fixing or working around things that you need but are broken in this server.

Test: Running /home/admin/wtfit returns OK.

Time to Solve: 20 minutes.