677 refactor expunge

[jsjiang]

@sfisher Hi Scott,

The like query is very heavy. The refactoring is focused on limiting the rows that will be scanned for each query.

The proc-expunge.py is the updated script. The original script is renamed and saved as proc-expunge_v1.py.

Here are the changes:

• add data range and pagesize as optional parameters to the command
• define default scan window to 3 days to reduce to be scanned records
• use the created time range to find the mapped ID range (since the records are created in time sequence)
• use the ID range to filter query set - filter on primary key is more efficient
• process each query set in pagesize batches
• increase default page size from 100 to 1000

Please review and let me know if you have quesitons.

Thank you

Jing

[sfisher]

Thanks for making these changes for logging and error handing. They look good to me.

[jsjiang]

I am closing this pull request. We will discuss other implementation options in the new year

• use cronjob instead of daemon
• add an index to the Identifier table on the isTest field to speed up the select query
• add date/time range options to the command
• add date/time range to the select query
• add error handling and error reporting