Scenario: "Saint John": what is writing to this log file?

Level: Easy

Type: Fix

Access: Public

Description: A developer created a testing program that is continuously writing to a log file /var/log/bad.log and filling up disk. You can check for example with tail -f /var/log/bad.log.
This program is no longer needed. Find it and terminate it. Do not delete the log file.

Test: The log file size doesn't change (within a time interval bigger than the rate of change of the log file).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Gitega": Find the Bad Git Commit

Level: Easy

Type: Do

Access: Paid

Description: The directory at /home/admin/git has a Git repository with a Golang program and a test for it.

To execute the test, from this "git" directory run: go test. The last (current HEAD) commit fails the test. Suppose the first commit passed the test.

Find the (long hash) commit that first broke the test and enter it in the /home/admin/solution file. For example: echo 9e80a7eb1b09385e93ab4a76cb2c93beec48fd9f > /home/admin/solution

Test: Doing md5sum /home/admin/solution returns f7db1bb6b7bfcd66a4eb66782804b39d.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: Linux Server Review - Guided Learning

Level: Easy

Type: Do

Access: Email

Description: This is a guided learning scenario. Follow this Linux Server Review Scenario Guide

The purpose of this scenario is to review a Linux server and be able to answer questions like:

• What's the purpose of the server?
• What's the hardware (CPU / RAM / disk / net) utilization of the server? is there a problem there?
• What is running and what's going on in the server?

Note: This challenge doesn't have a specific solution (there's no "Check My Solution")

Test: (there's no test)

Time to Solve: 30 minutes.

Scenario: "Kampot": A New Port

Level: Easy

Type: Do

Access: Email

Description: A Python app serving simulated bank data runs as root and listens on port 20280. The app is managed by supervisor and cannot be stopped or reconfigured to use a different port.

An internal legacy monitoring system expects the service to be available on port 80, but the app is hardcoded to 20280 for security and legacy reasons. Your task is to make the service accessible on port 80 locally.

Test: curl localhost:80/accounts returns [{"id":1,"name":"Alice","type":"Checking"},{"id":2,"name":"Bob","type":"Savings"},{"id":3,"name":"Charlie","type":"Business"}]

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Rio de Janeiro": Do we have another option?

Level: Easy

Type: Fix

Access: Paid

Description: This scenario server is dedicated to Jenkins, a Java application managed by systemd. Jenkins is failing to start. Troubleshoot and find the problem, then apply the solution so Jenkins runs properly.

Test: The service must return the string "Sign in - Jenkins" amongst some other html code. You can check with the command curl -s localhost:8888/login | grep Jenkins | head -n1

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Melbourne": WSGI with Gunicorn

Level: Medium

Type: Fix

Access: Paid

Description: There is a Python WSGI web application file at /home/admin/wsgi.py , the purpose of which is to serve the string "Hello, world!". This file is served by a Gunicorn server which is fronted by an nginx server (both servers managed by systemd). So the flow of an HTTP request is: Web Client (curl) -> Nginx -> Gunicorn -> wsgi.py . The objective is to be able to curl the localhost (on default port :80) and get back "Hello, world!", using the current setup.

Test: curl -s http://localhost returns Hello, world! (serving the wsgi.py file via Gunicorn and Nginx)

Time to Solve: 20 minutes.

Scenario: "Kihei": Surely Not Another Disk Space Scenario

Level: Medium

Type: Fix

Access: Email

Description: There is a /home/admin/kihei program. Make the changes necessary so it runs succesfully, without deleting the /home/admin/datafile file.

Test: Running /home/admin/kihei returns Done..

Time to Solve: 30 minutes.

Scenario: "Paris": Where is my webserver?

Level: Medium

Type: Hack

Access: Email

Description: A developer put an important password on his webserver localhost:5000 . However, he can't find a way to recover it. This scenario is easy to to once you realize the one "trick".

Find the password and save it in /home/admin/mysolution , for example: echo "somepassword" > ~/mysolution

Scenario credit: PuppiestDoggo

Test: md5sum ~/mysolution returns d8bee9d7f830d5fb59b89e1e120cce8e

Time to Solve: 15 minutes.

Scenario: "Buenos Aires": Kubernetes Pod Crashing

Level: Medium

Type: Fix

Access: Email

Description: There are two pods: "logger" and "logshipper" living in the default namespace. Unfortunately, logshipper has an issue (crashlooping) and is forbidden to see what logger is trying to say. Could you help fix Logshipper?

Do not change the K8S definition of the logshipper pod. Use "sudo".

Because k8s takes a minute or two to change the pod state initially, the check for the scenario is made to fail in the first two minutes.

Credit Srivatsav Kondragunta

Test: kubectl get pods -l app=logshipper --no-headers -o json | jq -r '.items[] | "\(.status.containerStatuses[0].ready)"' returns true

Time to Solve: 20 minutes.

Scenario: "Warsaw": Prometheus can't scrape the webserver

Level: Medium

Type: Fix

Access: Paid

Description: A developer created a golang application that is exposing the /metrics endpoint. They have a problem with scraping the metrics from the application. They asked you to help find the problem.

Full source code of the application is available at the /home/admin/app directory.

Credit Kamil Błaż

Test: The endpoint http://localhost:9000/metrics should return HTTP code 200.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Moyogalpa": Security Snag. The Trials of Mary and John

Level: Medium

Type: Fix

Access: Email

Description: Mary and John are working on a Golang web application, and the security team has asked them to implement security measures. Unfortunately, they have broken the application, and it no longer functions. They need your help to fix it.

The fixed application should be able to allow clients to communicate with the application over HTTPS without ignoring any checks. (eg: curl https://webapp:7000/users.html) and serve its static files.

Test: curl https://webapp:7000/users.html should return the content of file.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 30 minutes.

Scenario: "Bekasi": Supervisor is still around

Level: Medium

Type: Fix

Access: Paid

Description: There is an nginx service running on port 443, it is the main web server for the company and looks like a new employee has deployed some changes to the configuration of supervisor and now it is not working as expected.

If you try to access curl -k https://bekasi it should return Hello SadServers! but for some reason it is not.

You cannot modify files from the /home/admin/bekasi folder in order to pass the check.sh

You must find out what the issue is and fix it.

Test: curl -k https://bekasi returns Hello SadServers!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Tukaani": XZ LZMA Library Compromised

Level: Medium

Type: Fix

Access: Paid

Description: (You can learn about Linux Libraries before starting this scenario).

The Linux shared library liblzma.so has been compromised (the real compromised XZ Utils liblzma has not been used). The liblzma.so at the path /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5 is the good one. Consider the same library liblzma.so.5.2.5 at other paths as compromised or malicious (ideally we would have used other real versions with different checksums).

Find all instances of this "malicious" liblzma library (remember, it's the same library but in different directory locations) and make it so none of the running processes use it, while the applications "webapp" and "jobapp" (both of which managed by systemd) still run properly (eg, stopping those applications is not a solution).

Test: lsof | grep liblzma.so.5 returns only the liblzma in the path: /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Atrani": Modify a SQlite3 Database

Level: Medium

Type: Fix

Access: Paid

Description: A developer created a script /home/admin/readdb.py that tests access to a database. Without modifying the readdb.py file, change the database so that running the script returns the string "John Karmack".

Test: Running /home/admin/readdb.py returns "John Karmack".

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Batumi": Troubleshoot "A" cannot connect to "B"

Level: Medium

Type: Fix

Access: Paid

Description: (To learn the skills to solve this challenge, see Can't Connect to a Service: Linux Troubleshooting Guide)

There is a web server (Caddy) on HTTP port :80 but curl http://127.0.0.1 doesn't work. Find out what's wrong and make the necessary fixes so the web server returns a URL.

Note: as a limitation, the file /home/admin/db_connector.py must not be modified so that the challenge is considered solved properly.
The web server has to respond on the IP address 127.0.0.1; not only on "localhost".

Test: The command curl http://127.0.0.1 returns a URL address.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Bharuch": Lost in Translation

Level: Medium

Type: Fix

Access: Paid

Description: There's a Docker container that runs a web server on port 3000, but it's not working.

Using the tooling and resources provided in the server, make the container run correctly.

Test: curl http://localhost:3000 should return "Hello from sadservers!"

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Bizerte": The Slow Application

Level: Medium

Type: Fix

Access: Email

Description: A Python web application running on port 5000 from the /opt directory is experiencing severe performance issues; every request takes more than 5 seconds to complete.
The application is supposed to use the redis-server cache service for speed.

Your mission is to diagnose the performance bottleneck and restore the application to its normal, fast response time.

Do not change the Python application file slow_app.py.

Test: curl localhost:5000 returns Data from FAST cache!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Ruaka": Kubernetes pod in distress

Level: Medium

Type: Fix

Access: Paid

Description: A developer wants to deploy an open-source tool on Kubernetes. The tool unfortunately has limited documentation.

They built a helm chart and a container image. When the application is deployed, for some reason the server in Kubernetes doesn't seem to work but when the binary is started on their laptop/machine it works perfectly.

The application server is deployed by Helm. The command they used is: helm upgrade --install ruaka charts/ruaka.

Debug and help the developer find the issue. NOTE: Do not change or delete any current Helm field value in the chart, only add if needed.

Remember to give enough time to k8S after you apply a change before checking the solution.

Test: kubectl get pod shows the ruaka application pod up and running, while no Helm fields have been taken out from the applicaiton chart.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Atlantis": Not found

Level: Medium

Type: Fix

Access: Paid

Description: There is a small "C" application in the /home/admin/app directory. Create the Docker container "app" with a small footprint and minimalistic so you get a hello binary that returns a greeting in Atlantean (Docker multi-stage build). The binary application is automatically called when running docker run app

Test: docker run app returns SOO-puhk

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Toulon": Denied Lamp

Level: Medium

Type: Fix

Access: Business

Description: The security team has asked again Mary and John to implemente more security measures. Unfortunately, this time they have broken the LAMP stack (Apache with PHP) so the frontend is unable get an answer upstream, thus, they need your help again to fix it.

The fixed application should be able to serve the content from the webserver, the problem is a network connectivity, although the logs have valuable informatiion, it has nothing to do with the configuration of the apache server.

Test: curl localhost | head -n1 returns SadServers - LAMP Stack

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Marseille": Rocky security

Level: Medium

Type: Fix

Access: Paid

Description: As the Christmas shopping season approaches, the security team has asked Mary and John to implemente more security measures. Unfortunately, this time they have broken the LAMP stack; the frontend is unable get an answer from upstream, thus they need your help again to fix it.

The application should be able to serve the content from the webserver.

Note for Pro users: direct SSH access is not avaiable (yet) for this scenario.

Test: curl localhost | head -n1 returns SadServers - LAMP Stack

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Torino": Optimize grande Docker image

Level: Medium

Type: Do

Access: Paid

Description: A Torino Node.js application is located in the ~/torino-app directory.
You can run it directly with: nohup node app.js > app.log 2>&1 &. You can also verify that it works by running: curl localhost:3000

There is already a torino Docker image built with the Dockerfile in ~/torino-app, but the resulting image size is 916 MB.

Your task is to optimize the Docker image size:
1. Build a new Docker image for the Torino application, also called torino:latest but with a total size under 122 MB
2. Create and run a container using this optimized image.

NOTE: You can only use the existing Docker images in the server.
To build a Node application you need to COPY in your Dockerfile, besides the app.js , the package*.json files and without Internet access, the node_modules directory, since you cannot RUN npm install.

Test: The torino Docker image is less than 122 MB and curl http://localhost:3000 returns Hello from Torino!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Bern": Docker web container can't connect to db container.

Level: Hard

Type: Fix

Access: Email

Description: There are two Docker containers running, a web application (Wordpress or WP) and a database (MariaDB) as back-end, but if we look at the web page, we see that it cannot connect to the database. curl -s localhost:80 |tail -4 returns:

<body id="error-page"> <div class="wp-die-message"><h1>Error establishing a database connection</h1></div></body> </html>

This is not a Wordpress code issue (the image is :latest with some network utilities added). What you need to know is that WP uses "WORDPRESS_DB_" environment variables to create the MySQL connection string. See the ./html/wp-config.php WP config file for example (from /home/admin).

Test: sudo docker exec wordpress mysqladmin -h mysql -u root -ppassword ping . The wordpress container is able to connect to the database in the mariadb container and returns mysqld is alive.

Time to Solve: 20 minutes.

Scenario: "Singara": Docker and Kubernetes web app not working.

Level: Hard

Type: Fix

Access: Email

Description: There's a k3s Kubernetes install you can access with kubectl. The Kubernetes YAML manifests under /home/admin have been applied. The objective is to access from the host the "webapp" web server deployed and find what message it serves (it's a name of a town or city btw). In order to pass the check, the webapp Docker container should not be run separately outside Kubernetes as a shortcut.

Test: curl localhost:8888 returns a value from the webapp deployed Kubernetes pod.

Time to Solve: 20 minutes.

Scenario: "Belo-Horizonte": A Java Enigma

Level: Hard

Type: Fix

Access: Paid

Description: (Credit for the idea: fuero)

There is a one-class Java application in your /home/admin directory. Running the program will print out a secret code, or you may be able to extract the secret from the class file without executing it but I'm not providing any special tools for that.

Put the secret code in a /home/admin/solution file, eg echo "code" > /home/admin/solution.

Test: md5sum /home/admin/solution |awk '{print $1}' returns 9d2bd7aabb26681eacd9444da6b6643c

Time to Solve: 20 minutes.

Scenario: "Chennai": Pull a Rabbit from a Hat

Level: Hard

Type: Fix

Access: Paid

Description: There is a RabbitMQ (RMQ) cluster defined in a docker-compose.yml file.

Bring this system up and then run the producer.py script in such a way that is able to send messages to RMQ. In particular you have to send the message "hello-lwc".

- RMQ is a queuing system: messages are put in the queue with a "producer" and they are taken out from the other side by a "consumer". The queue name has to be the same for both.

- To send the message "hello-lwc": python3 ~/producer.py hello-lwc. Should return Message sent to RabbitMQ. "IncompatibleProtocolError" means RMQ is not working properly.

- To test consuming it: python3 ~/consumer.py, this will retrieve the next message from the queue and print it. Once everything is working send more than one message so there's at least one in the queue when the validation runs.

- Do not change the consumer.py and producer.py files; if you do the Check My Solution will fail.

Test: python3 ~/consumer.py returns hello-lwc

See /home/admin/agent/check.sh for the exact test.

Time to Solve: 30 minutes.

Scenario: "Monaco": Disappearing Trick

Level: Hard

Type: Hack

Access: Email

Description: There is a web server on :5000 with a form. POSTing the correct form password into this web service will return a secret.

Save this secret provided by the web page (not the password you sent to it) to /home/admin/mysolution, for example: echo "SecretFromWebSite" > ~/mysolution

TIP: a developer worked on the web server code in this VM, using the same 'admin' account.

Scenario credit: PuppiestDoggo

Test: md5sum /home/admin/mysolution returns a250aa19f16dda6f9fcef286f035ec4b

Time to Solve: 30 minutes.

Scenario: "Florence": Database Migration Hell

Level: Hard

Type: Fix

Access: Paid

Description: You are working as a DevOps Engineer in a company and another team member left the company and left the docker-compose.yml of a database-backed web application unfinished.

Generally, the problem revolves around the database migration and docker compose.

Additionally on front of the application there is an Nginx server and you need to fix the proper access to it as well.

The source of code is in /home/admin/app

Credit Kamil Błaż

Test: curl --cacert /etc/nginx/certs/sadserver.crt https://sadserver.local returns a message containing "ready to serve requests"

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute

Time to Solve: 30 minutes.