Scenario: "Dublin": registration queue overload (HTTP 429)

Level: Hard

Type: Fix

Tags:

Access: Paid

Description: Open enrollment is live and the student registration portal is rejecting traffic.

The app runs on this VM (no containers): Redis-backed background jobs, systemd workers, manual deploy. During the spike the portal at localhost:5000 returns HTTP 429 on POST /register e.g. curl -i -X POST http://127.0.0.1:5000/register (header and payload not needed to simplify).

Services live under systemd as registration-app, registration-worker@N, and registration-loadgen (the load generator simulates ongoing external enrollment traffic — treat it like real users arriving; do not "fix" the outage by turning that traffic off). Redis queue name: registration:jobs. Useful endpoints on port 5000:

GET  /          portal home
POST /register  enqueue a registration (429 when overloaded)
GET  /metrics   Redis queue lengths (queue_jobs, queue_failed, ...)

Find why the backlog will not drain and restore capacity so registrations are accepted again while traffic continues. You do not need to change application Python under /opt/registration/.

Root (sudo) Access: True

Test: registration-loadgen stays active, curl -s http://127.0.0.1:5000/metrics shows queue_jobs below 50, and curl -s -o /dev/null -w '%{http_code}\n' -X POST http://127.0.0.1:5000/register returns 202 (not 429) repeatedly.

The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can read and execute.

Time to Solve: 40 minutes.