SadServers Linux & DevOps Troubleshooting Scenarios

Scenario: "Trade": FIX drop reconciliation

Level: Medium

Type: Fix

Access: Email

Description: FIX (Financial Information eXchange) is a common electronic protocol for trading. Messages are tag=value fields separated by SOH (ASCII 0x01; logs here use |). A message starts with 8= (BeginString), 9= (BodyLength), 35= (MsgType) and ends with 10= (CheckSum). See FIX TagValue encoding.

Our order router sends FIX orders to the exchange gateway and gets fills back. Yesterday the session dropped for a few minutes: the gateway kept executing, the router kept sending, but some fills never made it back.

Both sides log raw FIX 4.2 under /home/admin/fix/: router.log (what the router saw) and gateway.log (authoritative). Useful tags:

35  MsgType: D = NewOrderSingle (order), 8 = ExecutionReport (fill),
             5 = Logout, A = Logon, 4 = SequenceReset (session markers)
11  ClOrdID: client order ID on orders
17  ExecID: fill ID, unique per fill (e.g. EX000431)

Test: fills.dat holds exactly the gateway's fill set: every 17=ExecID from gateway.log is present once, with no extras or duplicates.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute.

Time to Solve: 20 minutes.

Scenario: "Flip": algo symbol flips

Level: Medium

Type: Fix

Access: Email

Description: FIX (Financial Information eXchange) is an electronic protocol for trading. Messages are tag=value fields separated by SOH (ASCII 0x01; logs here use |). A message starts with 8= (BeginString), 9= (BodyLength), 35= (MsgType) and ends with 10= (CheckSum). See FIX encoding.

Our execution algo has a bug: on some fills it stamps the wrong symbol, e.g. an AAPL execution labeled DELL, GOOG labeled INTC.

You get three files: reference market data at /home/admin/market/BANDS.csv (symbol,min_px,max_px bands that never overlap), plus /home/admin/fix/orders.log (35=D orders with 11=ClOrdID and true 55=Symbol) and /home/admin/fix/executions.log (35=8 fills with 17=ExecID, 11=ClOrdID, stamped 55=Symbol, 31=LastPx).

FIX tag dictionary (lines look like 8=FIX.4.2|9=...|...|10=...|, | in place of the SOH delimiter):

35  MsgType: D = NewOrderSingle (order), 8 = ExecutionReport (fill)
11  ClOrdID: client order ID (links a fill back to its order)
17  ExecID: fill ID, unique per fill (e.g. EX000123)
55  Symbol: true symbol on orders, as-stamped (possibly wrong) on fills
31  LastPx: price of this fill

Test: corrections.csv lists exactly the flipped fills (~25 of ~960): every ExecID paired with its true symbol, no missing, extras, or duplicates.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute.

Time to Solve: 25 minutes.

Scenario: "Cork": The catalog won't open

Level: Medium

Type: Fix

Access: Email

Description: After last week's host reboot the product catalog Postgres database will not stay up. Inserts into the catalog database fail. Ops moved application data off the root disk onto a dedicated volume some time ago; root itself still looks healthy.

You do not need deep Postgres expertise. The server listens on :5432 when healthy. The data directory is set in /etc/postgresql/*/main/ (see data_directory). systemd manages the unit named postgresql.

Test: As admin: sudo -u postgres psql -d catalog -c "INSERT INTO products(name) VALUES ('jane smith');"

Should return: INSERT 0 1

The fix must still work after a reboot.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Bournemouth": Same job, three shells

Level: Medium

Type: Do

Access: Email

Description: This box is a small shell lab. Debian's /bin/sh is dash, your login shell is bash, and zsh is installed. Cron, Docker CMD lines, and most #!/bin/sh scripts do not run bash — and zsh script.sh does not honour a POSIX shebang.

Read /home/admin/lab/00-START-HERE.txt and run the demos with:
/home/admin/lab/compare.sh /home/admin/lab/demos/01-identity.sh
(there are ten snippets under demos/; run-all.sh runs every one).

Then write a script /home/admin/fleet-report.sh that prints a host status table from /home/admin/lab/inventory/ (hosts, disk.tsv, access.log).

Rules:
- One header line exactly: HOST DISK ERR STATUS
- Then one line per host from hosts, in that file's order: NAME DISK ERR STATUS (single spaces, no tabs)
- DISK is the integer from disk.tsv
- ERR is how many access.log lines have that host and code 500
- STATUS is crit if DISK >= 90 or ERR >= 5; else warn if DISK >= 80 or ERR >= 1; else ok

The same file must print that table with no extra lines when run as: sh /home/admin/fleet-report.sh, bash /home/admin/fleet-report.sh, and zsh /home/admin/fleet-report.sh.

demos/10-broken-report.sh is a bash sketch that is not a valid solution.

Test: /home/admin/fleet-report.sh exists. sh, bash, and zsh each print the same report for the inventory files, matching the rules above.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.

Scenario: "Pos": position manager down + reconcile

Level: Hard

Type: Fix

Access: Paid

Description: FIX (Financial Information eXchange) is a common electronic protocol for trading. Messages are tag=value fields separated by SOH (ASCII 0x01; logs here use |). A message starts with 8= (BeginString), 9= (BodyLength), 35= (MsgType) and ends with 10= (CheckSum). See FIX TagValue encoding.

Dev just called: a deployment of the position manager service (posmgr, a systemd service) went wrong in prod. They've asked you to take a look — posmgr show just errors. Get the service back to active first.

Once it is up, query the service for its CURRENT view (posmgr show). That view is stale: the authoritative trade history is /home/admin/fix/executions.log, raw FIX 4.2 35=8 fills carrying 55=Symbol, 54=Side (1=Buy, 2=Sell) and 32=LastQty.

FIX tag dictionary (lines look like 8=FIX.4.2|9=...|35=8|...|10=...|, | in place of the SOH delimiter):

35  MsgType: D = NewOrderSingle (order), 8 = ExecutionReport (fill)
11  ClOrdID: client order ID (links a fill back to its order)
17  ExecID: fill ID, unique per fill (e.g. EX000151)
55  Symbol: e.g. AAPL, MSFT, TSLA, NVDA, AMD
54  Side: 1 = Buy, 2 = Sell
32  LastQty: shares in this fill
31  LastPx: price of this fill

Test: posmgr.service is active and positions.csv holds exactly the nets from executions.log: every symbol with its true net qty, no missing, extras or dups.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute.

Time to Solve: 60 minutes.

Scenario: "Dublin": registration queue overload (HTTP 429)

Level: Hard

Type: Fix

Access: Paid

Description: Open enrollment is live and the student registration portal is rejecting traffic.

The app runs on this VM (no containers): Redis-backed background jobs, systemd workers, manual deploy. During the spike the portal at localhost:5000 returns HTTP 429 on POST /register e.g. curl -i -X POST http://127.0.0.1:5000/register (header and payload not needed to simplify).

Services live under systemd as registration-app, registration-worker@N, and registration-loadgen (the load generator simulates ongoing external enrollment traffic — treat it like real users arriving; do not "fix" the outage by turning that traffic off). Redis queue name: registration:jobs. Useful endpoints on port 5000:

GET  /          portal home
POST /register  enqueue a registration (429 when overloaded)
GET  /metrics   Redis queue lengths (queue_jobs, queue_failed, ...)

Test: registration-loadgen stays active, curl -s http://127.0.0.1:5000/metrics shows queue_jobs below 50, and curl -s -o /dev/null -w '%{http_code}\n' -X POST http://127.0.0.1:5000/register returns 202 (not 429) repeatedly.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute.

Time to Solve: 40 minutes.