Disk volumes Troubleshooting Scenarios

Scenario: "Manhattan": can't write data into database.

Level: Medium

Type: Fix

Access: Public

Description: Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don't need to know details about it (although it may help).

Helpful Postgres information: it's a service that listens to a port (:5432) and writes to disk in a data directory, the location of which is defined in the data_directory parameter of the configuration file /etc/postgresql/14/main/postgresql.conf. In our case Postgres is managed by systemd as a unit with name postgresql.

Test: (from default admin user) sudo -u postgres psql -c "insert into persons(name) values ('jane smith');" -d dt

Should return:INSERT 0 1

Time to Solve: 20 minutes.

Scenario: "Kihei": Surely Not Another Disk Space Scenario

Level: Medium

Type: Fix

Access: Email

Description: There is a /home/admin/kihei program. Make the changes necessary so it runs succesfully, without deleting the /home/admin/datafile file.

Test: Running /home/admin/kihei returns Done..

Time to Solve: 30 minutes.

Scenario: "Bondo": Split my pile!

Level: Medium

Type: Fix

Access: Email

Description: A developer wants to run a program that splits their pile of their data into compressed parts for efficient transport across their network. Unfortunately when the tool runs it never completes.

The application binary in question is called bondo located in /home/admin/bondo.

Run it, then debug and help the developer find the issue.

Test: Executing /home/admin/bondo as admin returns part files generation completed!.

The file /etc/fstab has not been modified and the solution would work on reboot.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Montevideo": restore test snapshot would clobber production

Level: Medium

Type: Fix

Access: Email

Description: This host runs a small app whose live data lives under /production/. Nightly snapshot backups land under /snapshots/ in an rsnapshot-style layout.

A recent bug in the snapshot job may have pointed some files in the latest rotation (/snapshots/daily-2026-06-01/) at the live tree instead of making independent copies.

Ops scheduled a dry-run restore from that snapshot into /production/. If any snapshot path still aliases live data, the restore would overwrite production in place.

Read /home/admin/backup-notes.txt. Find every incorrectly shared file between /production/ and /snapshots/daily-2026-06-01/, then repair the snapshot so it is safe to restore from. Do not damage live production data, and do not break intentional space-saving deduplication inside older snapshot rotations.

Note: the tools rsync and dd are available in this server.

Test: Every file in /snapshots/daily-2026-06-01/ mirroring a name in /production/ must be an independent copy: restoring the snapshot must not alias or overwrite live files.
Older rotations (snapshot-1–snapshot-3) must keep shared config.ini hard links.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Hong-Kong": can't write data into database.

Level: Hard

Type: Fix

Access: Paid

Description: (Similar to "Manhattan" scenario but harder). Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don't need to know details about it (although it may help).

Postgres information: it's a service that listens to a port (:5432) and writes to disk in a data directory, the location of which is defined in the data_directory parameter of the configuration file /etc/postgresql/14/main/postgresql.conf. In our case Postgres is managed by systemd as a unit with name postgresql.

Test: sudo -u postgres psql -c "insert into persons(name) values ('jane smith');" -d dt

Should return:INSERT 0 1

Time to Solve: 40 minutes.