Scenario: "Cairo": Time for a Timer

Level: Easy

Type: Fix

Access: Email

Description: A critical health check script at /opt/scripts/health.sh is supposed to run every 10 seconds. This check is triggered by a systemd timer.
The script's job is to check the local Nginx server and write its status (e.g., "STATUS: OK") to the log file at /var/log/health.log.
The log file is not being updated, and it appears the health check is failing.

Find out why the health check system is broken and fix it. The check will pass once the /var/log/health.log file is being correctly updated by the timer with a STATUS: OK message.

Test: The /opt/scripts/health.sh script writes STATUS: OK to /var/log/health.log every 10 seconds.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Alexandria": The Vanishing Backups

Level: Easy

Type: Fix

Access: Email

Description: A critical backup cron job has silently stopped working 3 days ago. The backup script is located at /opt/backup/backup.sh and should create daily backups in /var/backups/daily/, but no new backups have been created recently.

Looking at the backup directory, you can see old backup files from a few days ago, proving the system used to work. However, there are no error emails, no obvious error logs, and the cron service appears to be running normally.

Fix ALL issues preventing the backups from running, so that backups are created successfully and reliably.

Test directory: /var/backups/daily/
Backup script: /opt/backup/backup.sh

Test: The solution will be validated by checking if a backup file has been created in the last 10 minutes.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 5 minutes.

Scenario: "Kilifi": Speculative Misallocation!

Level: Medium

Type: Fix

Access: Email

Description: A developer is having trouble deploying an application on a preconfigured cluster.

The application kilifi is to be deployed on kubernetes in the default namespace.
The application server is deployed by helm. The command they used is helm install kilifi charts/kilifi.
The application operates correctly with ~210 MB of memory, but 256 MB is recommended.

Swap should remain disabled in the cluster.

Debug and help the developer fix any issue with deployment.

Test: The kilifi application runs properly; it's Service on :3333/healthz returns "kilifi ready to serve".

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Solanea": ClickHouse mad house

Level: Medium

Type: Do

Access: Email

Description: You have a ClickHouse installation CHI running on a Kubernetes cluster and a set of requests (located at ~/data/requests.csv) that you must populate into the http_requests table in the monitoring database (table may not exist in all pod instances).
Do this insert in all pod instances of the database.
The user and password to connect to the database are default.
The keeper pods provide clickhouse replication services.

Test: You are able to query the database and see the data:

clickhouse-client -h --password default -q 'SELECT COUNT(*) FROM monitoring.http_requests'

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Tunis": Redis Replication Problem

Level: Medium

Type: Fix

Access: Email

Description: A Redis master-replica setup is running on this server, with the master on port 6379 and the replica on port 6380. Both instances show as "connected" when you check their status, but data synchronization has silently broken.

Recent writes to the master don't appear on the replica, even though there are no obvious errors in the logs and both Redis instances appear healthy.

Fix the replication issues so that data written to the master (port 6379) immediately appears on the replica (port 6380) without data loss.

Master: localhost:6379
Replica: localhost:6380
Password: masterpass123

A helper test script is available at /home/admin/test_replication.sh

Test: The solution will be validated by writing a test key to the master and verifying it appears on the replica within 2 seconds.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Tigoni": Patch and Pray!

Level: Hard

Type: Fix

Access: Email

Description: A developer wants to upgrade their stateful application. This application handles their archives/backups.

The application tigoni is deployed on kubernetes in the default namespace.
The application server is deployed by Helm. The command they used is helm install tigoni charts/tigoni.

Upgrade the tigoni application to v2.0.0. The image already exists in the local repository.
Debug and help the developer fix any issue with the upgrade.
Everytime v1.0.0 is launched the archiving code starts on a clean slate.

Test: The tigoni pod with version v2 runs correctly (its endpoint :3000/healthz displays serverVersion:v2)

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Cabedelo": Harbor full of issues

Level: Hard

Type: Fix

Access: Email

Description: You need to build and push a docker image without changing the Dockerfile to your company's Harbor registry, which is running at harbor.sadservers.local, with its home directory at /opt/harbor. You have full admin access with admin:Harbor12345 credential. The source code and the Dockerfile are in the ~/app directory. The image name must be harbor.sadservers.local/images/app:1.0.0. It is also expected that the application will be up and running at localhost:5000 in a container named app.

IMPORTANT. Do not:
1. Generate new internal certificates
2. Change the Dockerfile
3. Change the /opt/harbor.yml file

Test: You are able to pull the application image from Harbor:
docker rmi harbor.sadservers.local/images/app:1.0.0
docker pull harbor.sadservers.local/images/app:1.0.0


You can access the application; curl localhost:5000 returns Hello world!

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.