Data processing Troubleshooting Scenarios

Scenario: "Minneapolis": Break a CSV file

Level: Easy

Type: Do

Access: Email

Description: Break the Comma Separated Valued (CSV) file data.csv in the /home/admin/ directory into exactly 10 smaller files of about the same size named data-00.csv, data-01.csv, ... , data-09.csv files in the same directory. All the files should have the same header (first line with column names) as data.csv. None of the smaller files should be bigger than 32KB.

Note: to simplify, disregard broken lines in your files (ie, you can break a file at any point, not just at a newline). The resulting files don't have to be proper CSV files.

Test: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 10 minutes.

Scenario: "Saint Paul": Merge Many CSVs files

Level: Easy

Type: Do

Access: Email

Description: Join (merge) all the 338 files in /home/admin/polldayregistrations_enregistjourduscrutin?????.csv into one single /home/admin/all.csv file with the contents of all the CSV files in any order. There should be only one line with the names of the columns as a header.

Test: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 15 minutes.

Scenario: "Unimak Island": Fun with Mr Jason

Level: Medium

Type: Do

Access: Paid

Description: Using the file station_information.json , find the station_id where "has_kiosk" is false and "capacity" is greater than 30.

Save the station_id of the solution in the /home/admin/mysolution file, for example: echo "ec040a94-4de7-4fb3-aea0-ec5892034a69" > ~/mysolution

You can use the installed utilities jq, gron, jid as well as Python3 and Golang.

Test: md5sum /home/admin/mysolution returns 8d8414808b15d55dad857fd5aeb2aebc

Time to Solve: 15 minutes.

Scenario: "Ivujivik": Parlez-vous Français?

Level: Medium

Type: Do

Access: Paid

Description: Given the CSV file /home/admin/table_tableau11.csv, find the Electoral District Name/Nom de circonscription that has the largest number of Rejected Ballots/Bulletins rejetés and also has a population of less than 100,000.

The initial CSV file may be corrupted or invalid in a way that can be fixed without changing its data.

Installed in the VM are: Python3, Go, sqlite3, miller directly and PostgreSQL, MySQL in Docker images.

Save the solution in the /home/admin/mysolution , with the name as it is in the file, for example: echo "Trois-Rivières" > ~/mysolution (the solution must be terminated by newline).

Test: md5sum /home/admin/mysolution returns e399d171f21839a65f8f8ab55ed1e1a1

Time to Solve: 20 minutes.

Scenario: "Minneapolis with a Vengeance": Break a CSV file

Level: Medium

Type: Do

Access: Email

Description: Break the Comma Separated Valued (CSV) file data.csv in the /home/admin/ directory into exactly 10 smaller files of about the same size named data-00.csv, data-01.csv, ... , data-09.csv files in the same directory. All the files should have the same header (first line with column names) as data.csv. None of the smaller files should be bigger than 32KB.

Note: unlike the original Minneapolis scenario, here the resulting files have to be proper CSV files.

As a helper tool, you can run the program check_csv.py to check if your data-??.cs files look like proper CSV files.

Test: The "Check My Solution" button runs the script /home/admin/agent/check.sh, which you can see and execute.

Time to Solve: 20 minutes.