Create a "cleaned spreadsheet"

[mikeymckay]

Invalid Shehia names are a problem when raw data is used for analysis. Particularly when looking at USSD or Case Notification data that has not had to pass validation to be saved. Shehia data, in particular, tends to become more accurate with each step of the case investigation. For instance, a shehia will be known definitively once the DMSO is at the household.

In the past, I have tried to encourage the team to use the built-in reports when possible. However, there has always been a tendency to generate reports manually in excel based on spreadsheet data that has been downloaded. Indeed, I expect there will always be a question that can only be answered by analysing the raw data. The problem is that the raw data doesn't include automated cleaning, data selection, etc, that the built-in reports do. For instance, in the absence of a 'Date of Positive Result' we use the creation date of the Case Notification as the 'Date of Positive Result'.

A solution is to provide a “cleaned” spreadsheet that includes these sort of automated data manipulations.

This will not fix all of the incorrect shehias in the system. Instead, we will create a list of invalid shehias and their case IDs from this new “cleaned” spreadsheet and then we will manually fix them where possible.

[mikeymckay]

Done and pushed