Airtable Polling

[alexquick]

Polling

Creates a new worker that knows how to periodically poll Airtable. This Relies on 2 fields in the Airtable table and provides a 3rd [1]:

1. Meta: Stores the last values for the row. This is used to determine what has changed and lets you see what the changes were.
2. Last Modified: Set up to be an automatic Last Modified field on Airtable's side. This lets us easily poll for changed/new rows by keeping track of the max Last Modified we've seen with a small overlap. [2]
3. Last Processed: Date field that holds the last time this record was processed through the system [2].

The basic flow is to:

1. Find all records where lastModified > the last observed lastModified minus an overlap. This will return all rows that have changed and some that haven't (that fall in the overlap)
2. Check to make sure that non-meta columns have actually changed.
3. Update the record's lastProcessed and meta.lastValues to reflect what the state is currently.
4. Emit the records that have changed to the caller

Notes

• You can run it like so:

npm run local:airtable-sync

> mutual-aid-app@ local:airtable-sync /Users/alex/devel/slack-app
> NODE_ENV=dev node ./src/workers/airtable-sync/worker.js

Starting airtable-sync.requests
Found 1 changes in Requests
1 moved from Dispatch Started -> Delivery Needed
Found 0 changes in Requests

• I didn't end up going with the fancy API for responding to changes. I was playing around with it a bit and I think that we should get some use-cases implemented before we decide what that is going to be. I volunteer to refactor it when that time comes.
• I did add some helpers to the returned records that let you easily ask them about which fields have changed and what their values are, which makes it a little bit easier at least: https://github.com/crownheightsaid/mutual-aid-app/pull/10/files#diff-5998397ba9505af6ba775adb2b878334R42-R54
• You can also start the worker in the express process: AIRTABLE_SYNC=5000 npm start

## Slack 

I just sort of ... made something up. The block editor is pretty cool!

• Ultimately what do we want the slack message to look like?
  We are missing some fields in the airtable that are in the messages that are being manually posted now: name and a public-facing request description (vs. notes).

• We can rip out the slack part, I wanted to leave it in for the review you you could see the poller being used.

• Having @display name in the Volunteers base doesn't seem like enough to mention users without problems because names with spaces are handled poorly, so it looks up the slack user based on email address at the moment. This requires the users:read.email scope.

• If you want to send the message for a single Request you can run:
  node ./src/workers/airtable-sync/newDeliveryRequest.js 1, where in this case 1 is the Request ID of the Request.

---

[1]

[2] These need to be set up as With Time / 24HR / UTC in Airtable
Resolves: #5

[alexquick]

Any ideas about how to stem the tide of hardcoded Airtable column names? Everything I can think of feels over-engineered.

Maybe exporting an object out of the airtable module like:

requestFields = {
   requestId: "Request ID",
   status: "Status"
}
. . .  elsehwhere . . .
record.get(requestFields.requestId)

which is, like, verbose but not magical and will let us at least track this a bit easier?

[mjmaurer]

definitely agree. That's what we're doing with page names in openHome for the slack app

[mab253]

after this afternoon, i definitely see the need for this :)

[mjmaurer]

Another awesome pull request !!
I love this API, most comments are about documentation

[mjmaurer]

definitely agree. That's what we're doing with page names in openHome for the slack app

[mjmaurer]

Having @display name in the Volunteers base doesn't seem like enough to mention users without problems because names with spaces are handled poorly, so it looks up the slack user based on email address at the moment. This requires the users:read.email scope.

We can backfill Slack's UID with ChangeDetector

[mjmaurer]

I'd love to get this in and be able to play around with it. if you comment out the slack posting part, i think we should merge this ASAP

@mab253 thoughts?

[alexquick]

Alrighty, I took out slack and made the changes you suggested. Thanks for the feedback, I was a little foggy when writing some of those comments!

[mjmaurer]

This is an awesome addition and I can't wait to start working with it. Given urgency, I'm going to merge. After chatting with @mab253 I think she would agree.