SadServers: Linux, DevOps & SRE Labs | Job Interview Prep

Troubleshooting Scenarios: Applications

Linux & Bash

- Linux commands, Bash scripting
- Systemd
- Networking, DNS
- Storage
- SSH, Firewall
- Libraries
- Cron and more...

Web Servers

- Nginx
- Apache
- HAProxy
- Caddy
- Gunicorn
- uWSGI
- HTTPS/TLS

Databases

- PostgreSQL
- MySQL
- SQLite
- Redis
- ClickHouse
- MongoDB
- etcd

Data Processing

- CSV
- JSON
- SQL queries

Docker

- Building images
- Multi-stage builds
- Volumes
- Networks
- Docker Compose
- Podman

Kubernetes

- kubectl
- Helm
- K8S Roles & Permissions
- Services
- Namespaces
- Deployments, StatefulSets
- ConfigMaps, Secrets

Tooling / Applications

- Git
- Rabbitmq
- Envoy
- Vault
- Harbor
- Prometheus
- Jenkins

Hacking

- Capture the Flag (CTF) Challenges
- Code Vulnerabilities
- Privilege Escalation

Languages

- Python
- Golang
- PHP
- Java
- Node.js
- C

envoy git harbor hashicorp vault jenkins prometheus rabbitmq

realistic / interviews new pro

Easy

#	Name	Time	Type
1	"Gitega": Find the Bad Git Commit	15 m	Do Pro	"Gitega": Find the Bad Git Commit Scenario: "Gitega": Find the Bad Git Commit Level: Easy Type: Do Access: Paid Description: The directory at /home/admin/git has a Git repository with a Golang program and a test for it. To execute the test, from this "git" directory run: `go test`. The last (current HEAD) commit fails the test. Suppose the first commit passed the test. Find the (long hash) commit that first broke the test and enter it in the /home/admin/solution file. For example: `echo 9e80a7eb1b09385e93ab4a76cb2c93beec48fd9f > /home/admin/solution` Test: Doing `md5sum /home/admin/solution` returns `f7db1bb6b7bfcd66a4eb66782804b39d`. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.
2	"Rio de Janeiro": Do we have another option?	15 m	Fix Pro	"Rio de Janeiro": Do we have another option? Scenario: "Rio de Janeiro": Do we have another option? Level: Easy Type: Fix Access: Paid Description: This scenario server is dedicated to Jenkins, a Java application managed by systemd. Jenkins is failing to start. Troubleshoot and find the problem, then apply the solution so Jenkins runs properly. Test: The service must return the string "Sign in - Jenkins" amongst some other html code. You can check with the command `curl -s localhost:8888/login \| grep Jenkins \| head -n1` The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.

Medium

#	Name	Time	Type
1	"Warsaw": Prometheus can't scrape the webserver	30 m	Fix Pro	"Warsaw": Prometheus can't scrape the webserver Scenario: "Warsaw": Prometheus can't scrape the webserver Level: Medium Type: Fix Access: Paid Description: A developer created a golang application that is exposing the /metrics endpoint. They have a problem with scraping the metrics from the application. They asked you to help find the problem. Full source code of the application is available at the /home/admin/app directory. Credit Kamil Błaż Test: The endpoint http://localhost:9000/metrics should return HTTP code 200. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 30 minutes.
2	"Bangalore": Envoy Panics	15 m	Fix	"Bangalore": Envoy Panics Scenario: "Bangalore": Envoy Panics Level: Medium Type: Fix Access: Email Description: You have an Envoy proxy routing traffic to two unhealthy backend services. When the number of healthy backends falls under a panic threshold (default 50%), Envoy enters a panic mode and it will either send traffic to all upstream hosts or to none at all. We are simulating this condition by returning an HTTP 503 status code from the /health endpoint in both backends. In our case Envoy is sending traffic to all upstream services. Your objective is to change the panic Envoy behaviour so that it does not route any traffic to unhealthy services and instead Envoy returns "no healthy upstream". (Do not change anything in the backend services). There can also be other Envoy configuration issues you need to fix. Test: Running `curl localhost:10000` should return `no healthy upstream`. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.
3	"Campina Grande": Give me my cert, Vault	15 m	Fix Pro	"Campina Grande": Give me my cert, Vault Scenario: "Campina Grande": Give me my cert, Vault Level: Medium Type: Fix Access: Paid Description: A web application running at https://nginx.example.com has an expired certificate. Issue a new certificate using the Hashicorp Vault running on the server. The Vault instance is already unsealed and initialized, and you have full admin access with the admin user. Test: Running `curl https://nginx.example.com` returns Hello!. The certificate presented by Nginx is issued by the Vault PKI (check using `openssl verify -CAfile /usr/local/share/ca-certificates/vault-pki-ca.crt /etc/nginx/ssl/cert.pem`). The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 15 minutes.

Hard

#	Name	Time	Type
1	"Chennai": Pull a Rabbit from a Hat	30 m	Fix Pro	"Chennai": Pull a Rabbit from a Hat Scenario: "Chennai": Pull a Rabbit from a Hat Level: Hard Type: Fix Access: Paid Description: There is a RabbitMQ (RMQ) cluster defined in a docker-compose.yml file. Bring this system up and then run the producer.py script in such a way that is able to send messages to RMQ. In particular you have to send the message "hello-lwc". - RMQ is a queuing system: messages are put in the queue with a "producer" and they are taken out from the other side by a "consumer". The queue name has to be the same for both. - To send the message "hello-lwc": `python3 ~/producer.py hello-lwc`. Should return `Message sent to RabbitMQ`. "IncompatibleProtocolError" means RMQ is not working properly. - To test consuming it: `python3 ~/consumer.py`, this will retrieve the next message from the queue and print it. Once everything is working send more than one message so there's at least one in the queue when the validation runs. - Do not change the consumer.py and producer.py files; if you do the Check My Solution will fail. Test: `python3 ~/consumer.py` returns `hello-lwc` See /home/admin/agent/check.sh for the exact test. Time to Solve: 30 minutes.
2	"Amygdala": Do you have enough insight to see the secrets?	20 m	Fix Pro	"Amygdala": Do you have enough insight to see the secrets? Scenario: "Amygdala": Do you have enough insight to see the secrets? Level: Hard Type: Fix Access: Paid Description: Troubleshoot and fix a Kubernetes web application running in the app namespace. Make the deployment run successfully so that it returns Hello handsome! when you curl it. Fix first your admin user access to the local Kubernetes cluster; the KUBECONFIG environment variable must be set to $HOME/.kube/config. You have full admin access to a Vault server (containing the secrets you need) from the admin user. All the used manifests for the application are placed on the /home/admin/manifests directory. Test: Running: `POD_IP=$(kubectl get po -n app -l app=app -o jsonpath='{.items[0].status.podIP}') curl http://$POD_IP` returns `Hello handsome!`. The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.
3	"Cabedelo": Harbor full of issues	20 m	Fix Pro	"Cabedelo": Harbor full of issues Scenario: "Cabedelo": Harbor full of issues Level: Hard Type: Fix Access: Paid Description: You need to build and push a docker image without changing the Dockerfile to your company's Harbor registry, which is running at harbor.sadservers.local, with its home directory at /opt/harbor. You have full admin access with admin:Harbor12345 credential. The source code and the Dockerfile are in the ~/app directory. The image name must be harbor.sadservers.local/images/app:1.0.0. It is also expected that the application will be up and running at localhost:5000 in a container named app. IMPORTANT. Do not: 1. Generate new internal certificates 2. Change the Dockerfile 3. Change the /opt/harbor.yml file Test: You are able to pull the application image from Harbor: `docker rmi harbor.sadservers.local/images/app:1.0.0` `docker pull harbor.sadservers.local/images/app:1.0.0` You can access the application; `curl localhost:5000` returns `Hello world!` The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute. Time to Solve: 20 minutes.