Scenario: "Saint John": what is writing to this log file?
Level: Easy
Type: Fix
Access: Public
Description: A developer created a testing program that is continuously writing to a log file /var/log/bad.log and filling up disk. You can check for example with tail -f /var/log/bad.log. This program is no longer needed. Find it and terminate it.
Test: The log file size doesn't change (within a time interval bigger than the rate of change of the log file).
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.
Description: There's a web server access log file at /home/admin/access.log. The file consists of one line per HTTP request, with the requester's IP address at the beginning of each line.
Find what's the IP address that has the most requests in this file (there's no tie; the IP is unique). Write the solution into a file /home/admin/highestip.txt. For example, if your solution is "1.2.3.4", you can do echo "1.2.3.4" > /home/admin/highestip.txt
Test: The SHA1 checksum of the IP address sha1sum /home/admin/highestip.txt is 6ef426c40652babc0d081d438b9f353709008e93 (just a way to verify the solution without giving it away.)
Description: There is a web server on port :80 protected with Port Knocking. Find the one "knock" needed (sending a SYN to a single port, not a sequence) so you can curl localhost.
Test: Executing curl localhost returns a message with md5sum fe474f8e1c29e9f412ed3b726369ab65. (Note: the resulting md5sum includes the new line terminator: echo $(curl localhost))
Scenario: "Resumable Server": Linux Upskill Challenge
Level: Easy
Type: Do
Access: Email
Description: This is a Debian 11 server without a challenge; it's for you to do as you please.
It's meant in principle for guided tutorials like the Linux Upskill Challenge, with some limitations (there's still no outgoing Internet access).
Once you are done with your task, you can stop (pause) your server from your dashboard.
From the dashboard at any time you can restart the stopped server to continue your tasks or you can stop the server if it's running. You can also destroy it at any time.
After 30 mins, the server will be stopped. You can restart it from the dashboard.
This is a "beta" feature and there can be some issues.
Description: There's a file /home/admin/scores.txt with two columns (imagine the first number is a counter and the second one is a test score for example).
Find the average (more precisely; the arithmetic mean: sum of numbers divided by how many numbers are there) of the numbers in the second column (find the average score).
Use exactly two digits to the right of the decimal point. i. e., use exaclty two "decimal digits" without any rounding. Eg: if average = 21.349 , the solution is 21.34. If average = 33.1 , the solution is 33.10.
Save the solution in the /home/admin/solution file, for example: echo "123.45" > ~/solution
Tip: There's bc, Python3, Golang and sqlite3 installed in this VM.
Description: A web application relies on the PostgreSQL 13 database present on this server. However, the connection to the database is not working. Your task is to identify and resolve the issue causing this connection failure. The application connects to a database named app1 with the user app1user and the password app1user.
Description: There's a Kubernetes Deployment with an Nginx pod and a Load Balancer declared in the manifest.yml file. The pod is not coming up. Fix it so that you can access the Nginx container through the Load Balancer.
There's no "sudo" (root) access.
Test: Running curl 10.43.216.196 returns the default Nginx Welcome page.
See /home/admin/agent/check.sh for the test that "Check My Solution" runs.
Description: In a directory /home/admin/data, there are multiple files, all of them with same content. One of these files has been modified, a word was added. You need to identify which word it is and put it in the solution file (both newline terminated or not are accepted).
Test:md5sum /home/admin/solution should return 55aba155290288b58e9b778c8f616560 or 2eeefea9fc4b16ea624bed5c67a49d80
Check My Solution: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can
Description: The directory at /home/admin/git has a Git repository with a Golang program and a test for it.
To execute the test, from this "git" directory run: go test. The last (current HEAD) commit fails the test. Suppose the first commit passed the test.
Find the (long hash) commit that first broke the test and enter it in the /home/admin/solution file. For example: echo 9e80a7eb1b09385e93ab4a76cb2c93beec48fd9f > /home/admin/solution
Description: Break the Comma Separated Valued (CSV) file data.csv in the /home/admin/ directory into exactly 10 smaller files of about the same size named data-00.csv, data-01.csv, ... , data-09.csv files in the same directory. All the files should have the same header (first line with column names) as data.csv. None of the smaller files should be bigger than 32KB.
Note: to simplify, disregard broken lines in your files (ie, you can break a file at any point, not just at a newline). The resulting files don't have to be proper CSV files.
Test: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.
Description: Join (merge) all the 338 files in /home/admin/polldayregistrations_enregistjourduscrutin?????.csv into one single /home/admin/all.csv file with the contents of all the CSV files in any order. There should be only one line with the names of the columns as a header.
Test: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.
Description: A spy has left a password in a file in /proc/sys . The contents of the file start with "secret:" (without the quotes).
Find the file and save the word after "secret:" to the file /home/admin/secret.txt with a newline at the end (e.g. if the file contents were "secret:password" do: echo "password" > /home/admin/secret.txt).
(Note there's no root/sudo access in this scenario).
Description: There's an Nginx web server running on this machine, configured to serve a simple "Hello, World!" page over HTTPS. However, the SSL certificate is expired.
Create a new SSL certificate for the Nginx web server with the same Issuer and Subject (same domain and company information).
Test: Certificate should not be expired: echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -dates and the subject of the certificate should be the same as the original one: echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -subject
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute
Scenario: "Manhattan": can't write data into database.
Level: Medium
Type: Fix
Access: Public
Description: Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don't need to know details about it (although it may help).
Helpful Postgres information: it's a service that listens to a port (:5432) and writes to disk in a data directory, the location of which is defined in the data_directory parameter of the configuration file /etc/postgresql/14/main/postgresql.conf. In our case Postgres is managed by systemd as a unit with name postgresql.
Description: There's a web server serving a file /var/www/html/index.html with content "hello sadserver" but when we try to check it locally with an HTTP client like curl 127.0.0.1:80, nothing is returned. This scenario is not about the particular web server configuration and you only need to have general knowledge about how web servers work.
Test:curl 127.0.0.1:80 should return: hello sadserver
Description: There's an Nginx web server installed and managed by systemd. Running curl -I 127.0.0.1:80 returns curl: (7) Failed to connect to localhost port 80: Connection refused , fix it so when you curl you get the default Nginx page.
Test:curl -Is 127.0.0.1:80|head -1 returns HTTP/1.1 200 OK
Description: There's a "dockerized" Node.js web application in the /home/admin/app directory. Create a Docker container so you get a web app on port :8888 and can curl to it. For the solution to be valid, there should be only one running Docker container.
Test:curl localhost:8888 returns Hello World! from a running container.
Description: Try and figure out if you are inside a container (like a Docker one for example) or inside a Virtual Machine (like in the other scenarios).
Test: This scenario doesn't have a test (hence also no "Check My Solution" either).
Description: There is a Python WSGI web application file at /home/admin/wsgi.py , the purpose of which is to serve the string "Hello, world!". This file is served by a Gunicorn server which is fronted by an nginx server (both servers managed by systemd). So the flow of an HTTP request is: Web Client (curl) -> Nginx -> Gunicorn -> wsgi.py . The objective is to be able to curl the localhost (on default port :80) and get back "Hello, world!", using the current setup.
Test:curl -s http://localhost returns Hello, world! (serving the wsgi.py file via Gunicorn and Nginx)
Description: There's an etcd server running on https://localhost:2379 , get the value for the key "foo", ie etcdctl get foo or curl https://localhost:2379/v2/keys/foo
Description: Given the CSV file /home/admin/table_tableau11.csv, find the Electoral District Name/Nom de circonscription that has the largest number of Rejected Ballots/Bulletins rejetés and also has a population of less than 100,000.
The initial CSV file may be corrupted or invalid in a way that can be fixed without changing its data.
Installed in the VM are: Python3, Go, sqlite3, miller directly and PostgreSQL, MySQL in Docker images.
Save the solution in the /home/admin/mysolution , with the name as it is in the file, for example: echo "Trois-Rivières" > ~/mysolution (the solution must be terminated by newline).
Description: A developer put an important password on his webserver localhost:5000 . However, he can't find a way to recover it. This scenario is easy to to once you realize the one "trick".
Find the password and save it in /home/admin/mysolution , for example: echo "somepassword" > ~/mysolution
Description: There are two pods: "logger" and "logshipper" living in the default namespace. Unfortunately, logshipper has an issue (crashlooping) and is forbidden to see what logger is trying to say. Could you help fix Logshipper?
Do not change the K8S definition of the logshipper pod. Use "sudo".
Because k8s takes a minute or two to change the pod state initially, the check for the scenario is made to fail in the first two minutes.
Description: There are three Docker containers defined in the docker-compose.yml file: an HAProxy accepting connetions on port :5000 of the host, and two nginx containers, not exposed to the host.
The person who tried to set this up wanted to have HAProxy in front of the (backend or upstream) nginx containers load-balancing them but something is not working.
Test: Running curl localhost:5000 several times returns both hello there from nginx_0 and hello there from nginx_1
Check /home/admin/agent/check.sh for the test that "Check My Solution" runs.
Description: Find in the file frankestein.txt the second most frequent word and save in UPPER (capital) case in the /home/admin/mysolution file.
A word is a string of characters separated by space or newlines or punctuation symbols .,:; . Disregard case ('The', 'the' and 'THE' is the same word) and for simplification consider the apostrophe as another character not as punctuation ("it's" would be a word, distinct from "it" and "is"). Also disregard plurals ("car" and "cars" are different words) and other word variations (don't do "stemming").
We are providing a shorter test.txt file where the second most common word in upper case is "WORLD", so we could save this solution as: echo "WORLD" > /home/admin/mysolution
This problem can be done with a programming language (Python, Golang and sqlite3) or with common Linux utilities.
Description: A developer created a database named 'main' but now some data is missing in the database. You need to restore the database using the the dump "/home/admin/backup.sql". The issue is that the developer forgot the root password for the MariaDB server. If you encounter an issue while restoring the database, fix it.
"Abaokoro": Restore MySQL Databases Spooked by a Ghost
Scenario: "Abaokoro": Restore MySQL Databases Spooked by a Ghost
Level: Medium
Type: Fix
Access: Paid
Description: There are three databases that need to be restored. You need to create three databases called "first", "second" and "third" and restore the databases using the file "/home/admin/dbs_to_restore.zip". If you encounter an issue while restoring the database, fix it.
Test: Doing curl on the default port (:80) of any nginx pod returns a Welcome SadServers page. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.
Description: You have been tasked with compressing the file /home/admin/names, which is 35147 bytes, to a size smaller than 9400 bytes. You can use any compressing tool at your disposal (there are many available in the server), also you can modify the file without deleting anything in it. Put the solution (compressed file) in the /home/user/admin/solution directory with the default extension used by the compression tool (example: ~/solution/names.gzip).
Test: The size of the compressed file is smaller than 9400 bytes.
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.
Scenario: "Warsaw": Prometheus can't scrape the webserver
Level: Medium
Type: Fix
Access: Paid
Description: A developer created a golang application that is exposing the /metrics endpoint. They have a problem with scraping the metrics from the application. They asked you to help find the problem.
Full source code of the application is available at the /home/admin/app directory.
"Moyogalpa": Security Snag. The Trials of Mary and John
Scenario: "Moyogalpa": Security Snag. The Trials of Mary and John
Level: Medium
Type: Fix
Access: Email
Description: Mary and John are working on a Golang web application, and the security team has asked them to implement security measures. Unfortunately, they have broken the application, and it no longer functions. They need your help to fix it.
The fixed application should be able to allow clients to communicate with the application over HTTPS without ignoring any checks. (eg: curl https://webapp:7000/users.html) and serve its static files.
Test:curl https://webapp:7000/users.html should return the content of file.
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.
"Helsingør": The first walls of postgres physical replication
Scenario: "Helsingør": The first walls of postgres physical replication
Level: Medium
Type: Fix
Access: Email
Description: You're setting up a PostgreSQL database with replication, you decided to use Docker along with Docker Compose to make it easier to manage and test, after a few hours of work you finished the job and the master database is up and running, but you're having trouble with the replica.
You need to figure out what's wrong with the replica and fix it.
Since you are using Docker Compose, you can check the status of the running containers using docker compose ps or docker ps will do the job too). You may also want to check the logs of the containers.
All definition for the containers are inside the docker-compose.yml file. You can stand up the environment by running docker compose up -d and set it down by running `docker compose down`.
If you make any change to the docker-compose.yml file, you can restart the containers by running docker compose up -d --force-recreate.
Test: Postgres replica container works.
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.
Description: There is an nginx service running on port 443, it is the main web server for the company and looks like a new employee has deployed some changes to the configuration of supervisor and now it is not working as expected.
If you try to access curl -k https://bekasi it should return Hello SadServers! but for some reason it is not.
You cannot modify files from the /home/admin/bekasi folder in order to pass the check.sh
Description: You are tasked to add compression to the company website. The website is running on an Nginx server, and you decide to add Brotli compression to it.
Brotli has became very popular these days because of its high compression ratio. It's a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding, and 2nd order context modeling.
For this purpose, you decided to compile the brotli modules yourself and add them to the Nginx server.
The location of the Brotli source code is at /home/admin/ngx_brotli. The nginx source code (needed to compile the modules) is located at /home/admin/nginx-1.18.0. From the ngx_brotli repository first you need to compile the brotli dependencies and then configure and make modules for Nginx. Afer that you need to add the modules to the Nginx configuration.
After installing the modules, you need to make sure the responses from the server are being server with compression.
Create a port-forward to port 80 from the server to your computer and check the header Content-Encoding, responses must return br for Brotli compression. You can also use curl -H "Accept-Encoding: br, gzip" -I http://localhost to check the header.
Something nice about Brotli is that it fails over to gzip if the client doesn't support Brotli, so curl -H "Accept-Encoding: gzip" -I http://localhost should return gzip instead.
Description: (You can learn about Linux Libraries before starting this scenario).
The Linux shared library liblzma.so has been compromised (the real compromised XZ Utils liblzma has not been used). The liblzma.so at the path /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5 is the good one. Consider the same library liblzma.so.5.2.5 at other paths as compromised or malicious (ideally we would have used other real versions with different checksums).
Find all instances of this "malicious" liblzma library (remember, it's the same library but in different directory locations) and make it so none of the running processes use it, while the applications "webapp" and "jobapp" (both of which managed by systemd) still run properly (eg, stopping those applications is not a solution).
Test:lsof | grep liblzma.so.5 returns only the liblzma in the path: /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.
Description: A developer created a script /home/admin/readdb.py that tests access to a database. Without modifying the readdb.py file, change the database so that running the script returns the string "John Karmack".
Scenario: "Minneapolis with a Vengeance": Break a CSV file
Level: Medium
Type: Do
Access: Email
Description: Break the Comma Separated Valued (CSV) file data.csv in the /home/admin/ directory into exactly 10 smaller files of about the same size named data-00.csv, data-01.csv, ... , data-09.csv files in the same directory. All the files should have the same header (first line with column names) as data.csv. None of the smaller files should be bigger than 32KB.
Note: unlike the original Minneapolis scenario, here the resulting files have to be proper CSV files.
As a helper tool, you can run the program check_csv.py to check if your data-??.cs files look like proper CSV files.
Test: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.
Description: The Hanoi office has a Linux server with a large number of user accounts and groups. The system administrators need to identify which users belong to multiple groups for better access management.
Given two files, `users.txt` and `groups.txt`, create a new file `/home/admin/multi-group-users.txt` containing the usernames of users who belong to more than one group, one username per line, sorted alphabetically.
The `users.txt` file contains a list of usernames, one per line. The `groups.txt` file contains group names and their members, in the format `group_name:user1,user2,user3`.
There is a web server (Caddy) on HTTP port :80 but curl http://127.0.0.1 doesn't work. Find out what's wrong and make the necessary fixes so the web server returns a URL.
Note: as a limitation, the file /home/admin/db_connector.py must not be modified so that the challenge is considered solved properly. The web server has to respond on the IP address 127.0.0.1; not only on "localhost".
Test: The command curl http://127.0.0.1 returns a URL address.
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.
Time to Solve: 20 minutes.
Hard
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Jakarta": it's always DNS.
Level: Hard
Type: Fix
Access: Public
Description: Can't ping google.com. It returns ping: google.com: Name or service not known. Expected is being able to resolve the hostname. (Note: currently the VMs can't ping outside so there's no automated check for the solution).
Test:ping google.com should return something like PING google.com (172.217.2.46) 56(84) bytes of data.
"Bern": Docker web container can't connect to db container.
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Bern": Docker web container can't connect to db container.
Level: Hard
Type: Fix
Access: Email
Description: There are two Docker containers running, a web application (Wordpress or WP) and a database (MariaDB) as back-end, but if we look at the web page, we see that it cannot connect to the database. curl -s localhost:80 |tail -4 returns:
<body id="error-page"> <div class="wp-die-message"><h1>Error establishing a database connection</h1></div></body> </html>
This is not a Wordpress code issue (the image is :latest with some network utilities added). What you need to know is that WP uses "WORDPRESS_DB_" environment variables to create the MySQL connection string. See the ./html/wp-config.php WP config file for example (from /home/admin).
Test:sudo docker exec wordpress mysqladmin -h mysql -u root -ppassword ping . The wordpress container is able to connect to the database in the mariadb container and returns mysqld is alive.
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Karakorum": WTFIT – What The Fun Is This?
Level: Hard
Type: Fix
Access: Paid
Description: There's a binary at /home/admin/wtfit that nobody knows how it works or what it does ("what the fun is this"). Someone remembers something about wtfit needing to communicate to a service in order to start. Run this wtfit program so it doesn't exit with an error, fixing or working around things that you need but are broken in this server. (Note that you can open more than one web "terminal").
"Singara": Docker and Kubernetes web app not working.
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Singara": Docker and Kubernetes web app not working.
Level: Hard
Type: Fix
Access: Email
Description: There's a k3s Kubernetes install you can access with kubectl. The Kubernetes YAML manifests under /home/admin have been applied. The objective is to access from the host the "webapp" web server deployed and find what message it serves (it's a name of a town or city btw). In order to pass the check, the webapp Docker container should not be run separately outside Kubernetes as a shortcut.
Test:curl localhost:8888 returns a value from the webapp deployed Kubernetes pod.
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Hong-Kong": can't write data into database.
Level: Hard
Type: Fix
Access: Email
Description: (Similar to "Manhattan" scenario but harder). Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don't need to know details about it (although it may help).
Postgres information: it's a service that listens to a port (:5432) and writes to disk in a data directory, the location of which is defined in the data_directory parameter of the configuration file /etc/postgresql/14/main/postgresql.conf. In our case Postgres is managed by systemd as a unit with name postgresql.
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Pokhara": SSH and other sshenanigans
Level: Hard
Type: Fix
Access: Paid
Description: A user client was added to the server, as well as their SSH public key. The objective is to be able to SSH locally (there's only one server) as this user client using their ssh keys. This is, if as root you change to this user sudo su; su client, you should be able to login with ssh: ssh localhost.
Test: As user admin: sudo -u client ssh client@localhost 'pwd' returns /home/client
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Roseau": Hack a Web Server
Level: Hard
Type: Hack
Access: Email
Description: There is a secret stored in a file that the local Apache web server can provide. Find this secret and have it as a /home/admin/secret.txt file.
Note that in this server the admin user is not a sudoer.
Also note that the password crackers Hashcat and Hydra are installed from packages and John the Ripper binaries have been built from source in /home/admin/john/run
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Belo-Horizonte": A Java Enigma
Level: Hard
Type: Fix
Access: Paid
Description: (Credit for the idea: fuero)
There is a one-class Java application in your /home/admin directory. Running the program will print out a secret code, or you may be able to extract the secret from the class file without executing it but I'm not providing any special tools for that.
Put the secret code in a /home/admin/solution file, eg echo "code" > /home/admin/solution.
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Chennai": Pull a Rabbit from a Hat
Level: Hard
Type: Fix
Access: Paid
Description: There is a RabbitMQ (RMQ) cluster defined in a docker-compose.yml file.
Bring this system up and then run the producer.py script in such a way that is able to send messages to RMQ. In particular you have to send the message "hello-lwc".
- RMQ is a queuing system: messages are put in the queue with a "producer" and they are taken out from the other side by a "consumer". The queue name has to be the same for both.
- To send the message "hello-lwc": python3 ~/producer.py hello-lwc. Should return Message sent to RabbitMQ. "IncompatibleProtocolError" means RMQ is not working properly.
- To test consuming it: python3 ~/consumer.py, this will retrieve the next message from the queue and print it. Once everything is working send more than one message so there's at least one in the queue when the validation runs.
- Do not change the consumer.py and producer.py files; if you do the Check My Solution will fail.
Test:python3 ~/consumer.py returns hello-lwc
See /home/admin/agent/check.sh for the exact test.
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Monaco": Disappearing Trick
Level: Hard
Type: Hack
Access: Email
Description: There is a web server on :5000 with a form. POSTing the correct form password into this web service will return a secret.
Save this secret provided by the web page (not the password you sent to it) to /home/admin/mysolution, for example: echo "SecretFromWebSite" > ~/mysolution
TIP: a developer worked on the web server code in this VM, using the same 'admin' account.
Interested in a career debugging servers? Want to help manage thousands of GPU servers? Send an email to careers@fluidstack.io!
Scenario: "Florence": Database Migration Hell
Level: Hard
Type: Fix
Access: Paid
Description: You are working as a DevOps Engineer in a company and another team member left the company and left the docker-compose.yml of a database-backed web application unfinished.
Generally, the problem revolves around the database migration and docker compose.
Additionally on front of the application there is an Nginx server and you need to fix the proper access to it as well.
