Postmortem: Duplicate badge number errors

[kitsuta]

Some background: MFF and, to a lesser extent, MAGFest have had intermittent issues with duplicate badge numbers being assigned to attendees for multiple years. Nowadays this manifests as an SQL error shown to attendees when registering, since the badge number column has the UNIQUE constraint set on it. Although the badge numbering code has been rewritten multiple times, a lack of ability to consistently reproduce the error has hindered efforts to actually fix the problem, and it was generally assumed that a race condition of some sort was involved.

We now have a way to potentially create a reproducible test case, as well as a few more interesting facts, thanks to the way we handled preassigned badge numbers for MFF this year and the problems it caused at-con (for just the possible reproduction steps, skip to the end):

We had originally set all badge types to be "preassigned," but when dealers started encountering the UNIQUE constraint violation while registering, we decided to set preassigned_badge_types to just 'staff_badge',. Then, after preregistration closed, we set all badge types to be preassigned and ran the resave_all_attendees_and_groups sep command to effectively mass-assign badge numbers to attendees. At this point we saw the system attempting to assign duplicate badge numbers to attendees while resaving them.

Assuming this was because all badge number assignments were being done inside a single transaction, we made the resaving function atomic (#515). This had an unusual result: the system now tried to assign the same already-assigned badge number to an attendee, repeatedly, each time we ran the command. A cursory review of the badge number in question didn't show anything particularly special about it, and unfortunately we didn't have time for a deeper dive.

When we forced the function to ignore the error (#517), the vast majority of attendees were successfully given badge numbers. Of the attendees who weren't assigned badge numbers (presumably because they tripped the UNIQUE constraint error and were skipped over), most were badges with the badge status INVALID_STATUS. Out of the ~100 attendees that were skipped, perhaps half a dozen had a valid badge. It is not clear why this was the case.

Below are what I hope are the steps to reproduce this on a local instance for testing:

1. Set preassigned_badge_types to 'staff_badge',.
2. Add a bunch of attendees to your system of various badge types, statuses, etc. MFF's print server creator Cheetah has a script/set of data to do this.
3. Set preassigned_badge_types to include most or all badge types, e.g., preassigned_badge_types = 'attendee_badge','staff_badge','sponsor_badge','shiny_badge','friday','saturday','sunday','guest_badge'
4. At the command line, run sep resave_all_attendees_and_groups. To run it inside the Docker container, run docker exec quickstart_web_1 /app/env/bin/python3 /app/sideboard/sep.py.
5. If this doesn't throw an error, read the story above for extra details that may affect the outcome -- for example, there may need to be some existing attendees with preassigned badge numbers.