Bash Troubleshooting Scenarios

Scenario: "Saint John": what is writing to this log file?

Level: Easy

Type: Fix

Access: Public

Description: A developer created a testing program that is continuously writing to a log file /var/log/bad.log and filling up disk. You can check for example with tail -f /var/log/bad.log.
This program is no longer needed. Find it and terminate it. Do not delete the log file.

Test: The log file size doesn't change (within a time interval bigger than the rate of change of the log file).

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Saskatoon": counting IPs.

Level: Easy

Type: Do

Access: Email

Description: There's a web server access log file at /home/admin/access.log. The file consists of one line per HTTP request, with the requester's IP address at the beginning of each line (first column).

Find what's the IP address that has the most requests in this file (there's no tie; the IP is unique). Write the solution into a file /home/admin/highestip.txt. For example, if your solution is "1.2.3.4", you can do echo "1.2.3.4" > /home/admin/highestip.txt

NOTE: The solution IP shows 482 times, ie grep -c -F -f highestip.txt access.log returns 482, if your solution has a different (lower) number you got the wrong most common IP.

Test: The SHA1 checksum of the IP address sha1sum /home/admin/highestip.txt is 6ef426c40652babc0d081d438b9f353709008e93 (just a way to verify the solution without giving it away, we also accept the right IP with no ending newline in the file.)

The

Time to Solve: 15 minutes.

Scenario: "The Command Line Murders"

Level: Easy

Type: Do

Access: Public

Description: This is the Command Line Murders with a small twist as in the solution is different

Enter the name of the murderer in the file /home/admin/mysolution, for example echo "John Smith" > ~/mysolution

Hints are at the base of the /home/admin/clmystery directory. Enjoy the investigation!

Test: md5sum ~/mysolution returns 9bba101c7369f49ca890ea96aa242dd5

(You can always see /home/admin/agent/check.sh to see how the solution is evaluated).

Time to Solve: 20 minutes.

Scenario: "Resumable Server": Linux Upskill Challenge

Level: Easy

Type: Do

Access: Email

Description: This is a Debian 11 server without a challenge; it's for you to do as you please.

It's meant in principle for guided tutorials like the Linux Upskill Challenge, with some limitations (there's still no outgoing Internet access).

Once you are done with your task, you can stop (pause) your server from your dashboard.

From the dashboard at any time you can restart the stopped server to continue your tasks or you can stop the server if it's running. You can also destroy it at any time.

After 30 mins, the server will be stopped. You can restart it from the dashboard.

This is a "beta" feature and there can be some issues.

Test: None

Time to Solve: 30 minutes.

Scenario: "Lhasa": Easy Math

Level: Easy

Type: Do

Access: Email

Description: There's a file /home/admin/scores.txt with two columns (the first number is a line number and the second one is a test score for example).

Find the average (more precisely; the arithmetic mean: sum of numbers divided by how many numbers are there) of the numbers in the second column (find the average score).

Use exactly two digits to the right of the decimal point. i. e., use exaclty two "decimal digits" without any rounding. Eg: if average = 21.349 , the solution is 21.34. If average = 33.1 , the solution is 33.10.

Save the solution in the /home/admin/solution file, for example: echo "123.45" > ~/solution

Tip: There's bc, Python3, Golang and sqlite3 installed in this VM.

Test: md5sum /home/admin/solution returns 6d4832eb963012f6d8a71a60fac77168 solution

Time to Solve: 15 minutes.

Scenario: "Apia": Needle in a Haystack

Level: Easy

Type: Do

Access: Paid

Description: In a directory /home/admin/data, there are multiple files, all of them with same content. One of these files has been modified, a word was added. You need to identify which word it is and put it in the solution file (both newline terminated or not are accepted).

Test: md5sum /home/admin/solution should return 55aba155290288b58e9b778c8f616560 or 2eeefea9fc4b16ea624bed5c67a49d80

Check My Solution: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can

Time to Solve: 20 minutes.

Scenario: "Bata": Find in /proc

Level: Easy

Type: Do

Access: Email

Description: A spy has left a password in a file in /proc/sys . The contents of the file start with "secret:" (without the quotes).

Find the file and save the word after "secret:" to the file /home/admin/secret.txt with a newline at the end (e.g. if the file contents were "secret:password" do: echo "password" > /home/admin/secret.txt).

(Note there's no root/sudo access in this scenario).

Test: Running md5sum /home/admin/secret.txt returns a7fcfd21da428dd7d4c5bb4c2e2207c4

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: Linux Server Review - Guided Learning

Level: Easy

Type: Do

Access: Email

Description: This is a guided learning scenario. Follow this Linux Server Review Scenario Guide

The purpose of this scenario is to review a Linux server and be able to answer questions like:

• What's the purpose of the server?
• What's the hardware (CPU / RAM / disk / net) utilization of the server? is there a problem there?
• What is running and what's going on in the server?

Test: (there's no test)

Time to Solve: 30 minutes.

Scenario: "Tokamachi": Troubleshooting a Named Pipe

Level: Easy

Type: Fix

Access: Email

Description: There's a process reading from the named pipe /home/admin/namedpipe.

If you run this command that writes to that pipe:

/bin/bash -c 'while true; do echo "this is a test message being sent to the pipe" > /home/admin/namedpipe; done' &

And check the reader log with tail -f reader.log

You'll see that after a minute or so it works for a while (the reader receives some messages) and then it stops working (no more received messages are printed to the reader log or it takes a long time to process one). Troubleshoot and fix (for example changing the writer command) so that the writer keeps sending the messages and the reader is able to read all of them.

Test: There should be a process running where a message is being sent to the pipe and that while that is running, another message can be sent to the pipe and read back.
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Yokohama": Linux Users Working Together

Level: Easy

Type: Fix

Access: Paid

Description: There are four Linux users working together in a project in this server: abe, betty, carlos, debora.

First, they have asked you as the sysadmin, to make it so each of these four users can read the project files of the other users in the /home/admin/shared directory, but none of them can modify a file that belongs to another user. Users should be able modify their own files.

Secondly, they have asked you to modify the file shared/ALL so that any of these four users can write more content to it, but previous (existing) content cannot be altered.

Test: All users (abe, betty, carlos, debora) can write to their own files. None of them can write to another user's file.
All users can add more content (append)) to the shared/project_ALL file but none can change its current content.
The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Cairo": Time for a Timer

Level: Easy

Type: Fix

Access: Email

Description: A critical health check script at /opt/scripts/health.sh is supposed to run every 10 seconds. This check is triggered by a systemd timer.
The script's job is to check the local Nginx server and write its status (e.g., "STATUS: OK") to the log file at /var/log/health.log.
The log file is not being updated, and it appears the health check is failing.

Find out why the health check system is broken and fix it. The check will pass once the /var/log/health.log file is being correctly updated by the timer with a STATUS: OK message.

Test: The /opt/scripts/health.sh script writes STATUS: OK to /var/log/health.log every 10 seconds.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Alexandria": The Vanishing Backups

Level: Easy

Type: Fix

Access: Email

Description: A critical backup cron job has silently stopped working 3 days ago. The backup script is located at /opt/backup/backup.sh and should create daily backups in /var/backups/daily/, but no new backups have been created recently.

Looking at the backup directory, you can see old backup files from a few days ago, proving the system used to work. However, there are no error emails, no obvious error logs, and the cron service appears to be running normally.

Fix ALL issues preventing the backups from running, so that backups are created successfully and reliably.

Test directory: /var/backups/daily/
Backup script: /opt/backup/backup.sh

Test: The solution will be validated by checking if a backup file has been created in the last 10 minutes.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 5 minutes.

Scenario: "Kortenberg": Can't touch this!

Level: Easy

Type: Fix

Access: Paid

Description: Is "All I want for Christmas is you" already everywhere?. A bit unrelated, someone messed up the permissions in this server, the admin user can't list new directories and can't write into new files. Fix the issue.
NOTE: Besides solving the problem in your current admin shell session, you need to fix it permanently, as in a new login shell for user "admin" (like the one initiated by the scenario checker) should have the problem fixed as well.

Test: The admin user in a separate Bash login session should be able to create a new directory in your /home/admin directory, as well as being able to create a file into this new directory and add text into the new file.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Oaxaca": Close an Open File

Level: Medium

Type: Fix

Access: Email

Description: The file /home/admin/somefile is open for writing by some process. Close this file without killing the process.

Test: lsof /home/admin/somefile returns nothing.

Time to Solve: 15 minutes.

Scenario: "Marrakech": Word Histogram

Level: Medium

Type: Do

Access: Email

Description: Find in the file frankestein.txt the second most frequent word and save in UPPER (capital) case in the /home/admin/mysolution file.

A word is a string of characters separated by space or newlines or punctuation symbols .,:; . Disregard case ('The', 'the' and 'THE' is the same word) and for simplification consider the apostrophe as another character not as punctuation ("it's" would be a word, distinct from "it" and "is"). Also disregard plurals ("car" and "cars" are different words) and other word variations (don't do "stemming").

We are providing a shorter test.txt file where the second most common word in upper case is "WORLD", so we could save this solution as: echo "WORLD" > /home/admin/mysolution

This problem can be done with a programming language (Python, Golang and sqlite3) or with common Linux utilities.

Test: echo "SOLUTION" | md5sum returns 19bf32b8725ec794d434280902d78e18

See /home/admin/agent/check.sh for the test that "Check My Solution" runs.

Time to Solve: 20 minutes.

Scenario: "Hanoi": Find the Multitasking Users

Level: Medium

Type: Do

Access: Paid

Description: The Hanoi office has a Linux server with a large number of user accounts and groups. The system administrators need to identify which users belong to multiple groups for better access management.

Given two files, `users.txt` and `groups.txt`, create a new file `/home/admin/multi-group-users.txt` containing the usernames of users who belong to more than one group, one username per line, sorted alphabetically.

The `users.txt` file contains a list of usernames, one per line. The `groups.txt` file contains group names and their members, in the format `group_name:user1,user2,user3`.

Test: Running md5sum /home/admin/multi-group-users.txt returns dc0ae86caae7125d21df03a0ab29d8ae

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Kampala": Strange Script Error

Level: Medium

Type: Fix

Access: Email

Description: A developer has been working on Linux deployment scripts on their machine and then transferred the files to a Linux server. However, when they try to execute the scripts, they encounter the mysterious error:

-bash: cannot execute: required file not found

The scripts appear to be syntactically correct, but something is preventing them from executing properly. The developer needs your help to identify and fix the issue so the deployment can proceed.

There are several script files in /home/admin/deploy/ that need to be fixed before the deployment process can work correctly.

Test: All script files in /home/admin/deploy/ should execute without the cannot execute: required file not found error.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Zaragoza": Test changing critical files

Level: Hard

Type: Do

Access: Paid

Description: The goal is to make the script /home/admin/agent/check.sh return OK, without editing the original /etc/hosts file.

Think of testing changes in the critical directory /etc in a safe way. In this case, adding "127.0.0.1 my.local.test" to /etc/hosts .

There would be many ways of trying to do this with "sudo" access, like the usual procedure of making a copy of the config file, editing there and copying or renaming back to the original file. In our case, to avoid all those simple solutions, there is no general "sudo" privileges in this scenario (but there may be for some commands).

Test: The string my.local.test is in /etc/hosts

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.