SadServers Linux & DevOps Troubleshooting Scenarios

Scenario: "Lyon": Migrate Ingress-NGINX to Traefik

Level: Medium

Type: Do

Access: Email

Description: Ingress-NGINX is being retired. As the DevOps Engineer, you will replace it with Traefik on the production Kubernetes cluster in a private VPC. This scenario is a local proof-of-concept for that migration.

The current K8s cluster has a "Hello World" pod running, i.e.: curl hello.lyon.local returns "Hello world" (see note 1). You should be able to see the same content delivered via Traefik once the ingress-nginx is down.

Notes: 1: Wait at the start until k8s is fully up before doing curl, otherwise you get 503, you can check for ex with k get pod -n ingress-nginx
2: The k8s manifests are under the ~/app dir. 3: ingress-nginx was deployed with a Helm chart.
4: The Helm chart for traefik is available under /home/admin/traefik (The Traefik image is already loaded in k3s).
5: Traefik dashboard and probes/metrics port by default is :8080 but that's used by the system; use a different port or disable.
6: The domain hello.lyon.local is actually pointing to the localhost.
7: The ingress must be listening on port 80 for any IP so it can respond to localhost:80 or actually to *:80

TIP: You can use k as an alias for kubectl, and it has autocomplete enabled.

Test: When the command curl -i hello.lyon.local is executed, it returns the message Hello World, while only the traefik pod must be present (instead of ingress-nginx).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "London": Ollama LLM troubles

Level: Hard

Type: Fix

Access: Paid

Description: An AI agent has been deployed to production as a container called ai-agent managed by the Docker Compose configuration /home/admin/app/docker-compose.yaml. This ai-agent container relies on an Ollama LLM backend to generate a report but hasn't generated any yet. Your mission is to restore the broken agent-to-LLM (Ollama) connectivity, and tune the agent configuration so it can produce a report in /home/admin/app/agent/report.json. Example of the expected output:

{
  "summary": "Nginx is failing to reach its upstream service",
  "root_causes": [
    {
      "service": "nginx",
      "error": "connection refused to upstream 127.0.0.1:9999",
      "severity": "high"
    }
  ],
  "recommended_actions": "Fix upstream port configuration"
}

Test: The command docker compose up -d agent under the directory /home/admin/app must create the report file /home/admin/app/agent/report.json. The format of the answer must be as specified in the description.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Anatolia": compromised server

Level: Hard

Type: Fix

Access: Paid

Description: This web server has been compromised and is not serving the home page anymore, those troubleshooting skills you have as DevOps are urgently needed to solve the mystery of the missed home page and restore the integrity of the server.

Note: The default configuration files under /etc/apache2 are not the problem.

This scenario is based on a real server that was "hacked". Ideally you'd recover from infrastrucrure as code playbooks and clean data backups on a new server with the vulnerabilities fixed. Instead, in this exercise you are asked to clean manually the compromised server, restore it to a working condition and ideally, find how the server was broken into. The solution test only checks that the web service is working.

Test: curl localhost must return SadServer - Anatolia

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.