SadServers Linux & DevOps Troubleshooting Scenarios

Scenario: "Valladolid": Cleaner not cleaning

Level: Easy

Type: Fix

Access: Email

Description: The systemd service log-cleaner.service is supposed to be run manually (not a timer or cron job) and delete log files older than 7 days in the /var/log/app directory.

The service runs successfully (exit code 0), but no logs are ever deleted.

Fix the service and/or the script so that old_data.log (older than 7 days) is deleted, but recent_data.log is preserved.

If you accidentally delete the wrong files while debugging, run ~/reset_logs.sh to restore them.

Test: Running sudo systemctl restart log-cleaner deletes the file /var/log/app/old_data.log but not /var/log/app/recent_data.log

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Amsterdam": Cron Hijack

Level: Medium

Type: Hack

Access: Email

Description: You are logged in as the user admin. A cron job (not a systemd timer) appears to be running as root every minute, related to a health check.
This server has as root's path /home/admin:/usr/local/bin:/usr/bin:/bin

Your mission is to find the running cron job, and use it to exploit the server so you can read the secret file at /root/secret.txt

Save the secret string from the secret file to the file /home/admin/solution.txt.

Test: cat /home/admin/solution.txt displays the same string that is in /root/secret.txt, with md5sum /home/admin/solution.txt returning c6ef5d3ea5e937ae56f8635f91cc727a (the solution string without an ending newline is also accepted)

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Madrid": exploiting capabilities

Level: Hard

Type: Hack

Access: Email

Description: You are logged in as the admin user without sudo privileges.

A secret string is in the file /root/flag.txt and you don't have permission to read it directly.

However, a standard system binary has been misconfigured with a "hidden" capability that allows it to bypass file permissions.

Your mission is to find the misconfigured binary and use it to copy the content of /root/flag.txt into the file /home/admin/flag.txt.

Test: cat /home/admin/flag.txt displays the same string that is in /root/flag.txt, with md5sum /home/admin/flag.txt returning a43d338b0fc1dfb0c6425aa55e24c8c6 (the solution string without an ending newline is also accepted)

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "London": Ollama LLM troubles

Level: Hard

Type: Fix

Access: Paid

Description: An AI agent has been deployed to production as a container called ai-agent managed by the Docker Compose configuration /home/admin/app/docker-compose.yaml. This ai-agent container relies on an Ollama LLM backend to generate a report but hasn't generated any yet. Your mission is to restore the broken agent-to-LLM (Ollama) connectivity, and tune the agent configuration so it can produce a report in /home/admin/app/agent/report.json. Example of the expected output:

{
  "summary": "Nginx is failing to reach its upstream service",
  "root_causes": [
    {
      "service": "nginx",
      "error": "connection refused to upstream 127.0.0.1:9999",
      "severity": "high"
    }
  ],
  "recommended_actions": "Fix upstream port configuration"
}

Test: The command docker compose up -d agent under the directory /home/admin/app must create the report file /home/admin/app/agent/report.json. The format of the answer must be as specified in the description.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Anatolia": compromised server

Level: Hard

Type: Fix

Access: Paid

Description: This web server has been compromised and is not serving the home page anymore, those troubleshooting skills you have as DevOps are urgently needed to solve the mystery of the missed home page and restore the integrity of the server.

Note: The default configuration files under /etc/apache2 are not the problem.

This scenario is based on a real server that was "hacked". Ideally you'd recover from infrastrucrure as code playbooks and clean data backups on a new server with the vulnerabilities fixed. Instead, in this exercise you are asked to clean manually the compromised server, restore it to a working condition and ideally, find how the server was broken into. The solution test only checks that the web service is working.

Test: curl localhost must return SadServer - Anatolia

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.