System to poll Airtable for changes

[alexquick]

When an intake volunteer marks a Request as "Send to Slack" / "Ready For Aid" it needs to get sent to Slack

Notes

Airflow doesn't have any simple push notifications, but we can poll. This will likely involve creating 2 hidden fields: meta and an auto last_modified column. The last_modified will let the system narrow down the number of rows to examine, and meta will be a hidden json-encoded text column of the last values observed. This way we can tell exactly which fields changed and to what. This is inspired as a generalized version of: https://github.com/bedstuystrong/firebase-functions/blob/master/functions/airtable.js#L78

The internal API could look something like:

const changes = new AirtableChanges(base);
changes.onChange((record, beforeValues, afterValues)=>{
   . . . callback . . .
});
changes.start();

Polling changes could look like:

1. Get all rows changed since last last_modified (maybe include a jitter or overlap)
2. Compare each row's non-meta fields vs cached values in meta (json). Filter out rows that have not had any field actually change vs meta
3. Dispatch change events for the affected fields (could use eventemitter or skip entirely and hard-code the send-to-slack logic for the time being)
4. Update meta with the current values and save. This means that the row will end up at 1. again unfortunately, but will get filtered out at 2.

Related: #1

[mjmaurer]

could we add a field last-processed field that is set a tiny bit in the future? then we could have an airtable formula like last_modified > last_processed

[mjmaurer]

I think notifying slack should be out of scope for this bug. Polling out be huge