Ingest emails from Gmail and send to Airtable #4

alexquick · 2020-03-27T20:52:06Z

Well I'm glad we all really hate zapier, because it was way more work than I thought it was going to be (isn't it always) to link this all together.

General flow

There's now a scheduled task in the app [1] that wakes up every 5 seconds and polls for changes to a label in a given gmail account
When it finds new mail in that label, it sends them to Requests base in airtable filling in:
- The email address (should we strip the name if given or leave it?)
- A message made up of the subject and a best-effort decoding of the body [2]
It then labels the message with a $labelname-tracked label so that it doesn't end up saving that message again if it's restarted (and to avoid having to use Airtable to de-duplicate data-- though that's definitely an option if we end up having issues).

Questions

How do these get labeled in the first place? What sort of workflow are volunteers going to have in the gmail inbox (if any)
If a volunteer has an exchange with someone to get more information, how should we handle that thread?
So, like, I really hate zapier. But is it enough of an issue that this (relatively) complicated part of the system needs to be run and maintained?
Are other mutual aid groups that we hope pick up this project going to be willing to go through the song/dance they need to to get the initial gmail tokens? [3]

Resolves: #3

[1] Could easily be a separate process: unset SYNC_GMAIL, add worker: npm run gmail-sync to Procfile, and scale the worker up with ps:scale
[2] Multipart, embedded content, and text/html are a pain!
[3] It's a bummer. . You need to make a new OAuth client for your account (individually) and then do a song and dance to get a refresh token. Alternatively, we could look into using IMAP instead of the Gmail API. The downside here is the credentials, IMAP will use the password of the gmail account (or if the account has 2FA enabled an app-specific password... but 2FA for 'shared' accounts is difficult..)

this may be a pyrric victory over zapier :/

alexquick

A pyrrhic victory over Zapier I think. Thoughts?

alexquick · 2020-03-27T21:13:10Z

src/gmail-sync-worker.js

+  } catch (e) {
+    console.error('Failed to poll gmail. Rescheduling\n %O', e);
+  }
+  setTimeout(syncGmail, interval);


Do you have any node scheduled task runners you like?

Nope. dealers choice

Ok in that case I think maybe leave this as the simplest possible implementation and then think about it when/if the system grows another recurring task?

src/gmail-sync-worker.js

alexquick · 2020-03-27T21:15:08Z

src/airtable.js

+   * so we would like a single place to map the name to an internal
+   * symbol: "Email Address" -> "emailAddress"
+   */
+  airtableBindings = {


On one hand it didn't feel good duplicating the hardcoded airtable field strings all over the place. On the other hand I feel like there needs to be... a better solution. Ideas?

yea, i feel like hardcoded strings might be our best bet. we need some normalization

Ok I'll get rid of this complexity and just hardcode some strings. When the app starts dealing with Airtable more comprehensively we can rethink it.

mjmaurer

is eslint working for you? (i think vscode might still be automagicing for me). The difference that popped out is single quotes instead of double quotes. feel free to change any eslint rules

src/gmail-sync-worker.js

mjmaurer · 2020-03-28T02:26:33Z

src/airtable.js

+   * so we would like a single place to map the name to an internal
+   * symbol: "Email Address" -> "emailAddress"
+   */
+  airtableBindings = {


yea, i feel like hardcoded strings might be our best bet. we need some normalization

mjmaurer · 2020-03-28T02:28:51Z

src/gmail-sync-worker.js

+  } catch (e) {
+    console.error('Failed to poll gmail. Rescheduling\n %O', e);
+  }
+  setTimeout(syncGmail, interval);


Nope. dealers choice

mjmaurer · 2020-03-28T02:45:58Z

This is a gorgeous pull request.

some general context for my thoughts: around monday of this week, I was pushing hard for anything but airtable. mainly for privacy concerns, but also because I knew issues like this would pop up; and also because I wanted to make something that other MA groups could easily deploy.

I stopped advocating because I wanted to onboard volunteers fast (airtable flow already exists), and move fast in general (airtable forms are great, schema changes easy). I think I still agree with this. We should work on quickly building the crown heights community instead of a more general approach. of course this may also come at the cost of testing and documentation as well. The dev environment should be as sleek as possible though.

we havent had a conversation about this yet, I'd love to hear thoughts

How do these get labeled in the first place? What sort of workflow are volunteers going to have in the gmail inbox (if any)

@mab253 what's the situation?

If a volunteer has an exchange with someone to get more information, how should we handle that thread?

If I understand, I think the answer is all info should be added by an intake volunteer into the airtable. We need to come up with a deletion policy as well.

So, like, I really hate zapier. But is it enough of an issue that this (relatively) complicated part of the system needs to be run and maintained?

how's the development environment for zapier? I think if we did go down that route, it should almost definitely be a different repo so we don't need to target another node version.

Are other mutual aid groups that we hope pick up this project going to be willing to go through the song/dance they need to to get the initial gmail tokens? [3]

See the context paragraph I posted above. Think maybe we should document as much as possible, but maybe relax the ease of setup requirement. if you have thoughts on moving out of a monorepo, I'd be interested, but the fewer api keys the better (this coming from someone who just added two api dependencies)

…tion

mab253 · 2020-03-29T18:14:40Z

hiiiii i finally have a chance to look @ all this - awesome, awesome work.

a few things:

we all hate Zapier (BOO!), but i hear Alex on the question of whether this is worth maintaining - apparently the Bed Stuy group fields < 10% of their requests via e-mail, many more coming via text + voicemail.
the filter/label process in gMail i think would have to be a human volunteer task, coordinated by the people who started that crownheightsmutualaid@gmail.com, mainly operating on Facebook. Maybe 1x or 2x per day they go through and add a label that's "Request"?
in terms of the vol flow of reaching out to the person: I think the very first incoming e-mail should be labeled with "Request," sent over to Airtable, and then all reaching out done by an intake volunteer via the Airtable. That way it's treated like any other ticket, pushed to the Slack, etc. If a human reading the e-mails wants to handle the request without pushing it to Airtable, they simply don't label it, perhaps!
re: dev environment in Zapier, I think we chatted about this a little in the Slack, but the gMail -> Airtable integration is built into their UI, we wouldn't have to code anything. A plus for making this replicable and easy to teach ... but then it's Zapier, grrr.
I'm still of the mind that Airtable has a pretty solid balance of user-friendliness for non-coders, and access to dev/integrations via the super simple API. There are going to be limitations for whatever we do, but also creative workarounds - ! still happy to discuss of course, but I think it says a lot that most of the other groups in NYC are using Airtable as well.

so: what do we think? is this 1 of the few times we use Zapier (..😱)?!

mjmaurer · 2020-03-30T15:44:14Z

TLDR: I agree with using Zapier for the gmail intake. @alexquick I'm so sorry to move back on this awesome PR. We can keep the saveRecords part of airtable.js tho correct?

re: dev environment in Zapier, I think we chatted about this a little in the Slack, but the gMail -> Airtable integration is built into their UI, we wouldn't have to code anything. A plus for making this replicable and easy to teach ... but then it's Zapier, grrr.

I think the UI-only actually bothers me the most. If we keep the Zapier work to less-used flows, then could we mirror instructions on setting up in a separate github repo? I think a good goal is easy setup for other groups. But overall I agree with Zapier for less-used flows that have high overhead

Related: I think it would be cool if we renamed the repos to what they do. So twilio becomes -> crownheightsaid/request-intake-twilio and gmail could become crownheightsaid/request-intake-gmail. Renaming repos is super easy

I'm still of the mind that Airtable has a pretty solid balance of user-friendliness for non-coders, and access to dev/integrations via the super simple API.

Totally agree! I think we're doing a good job of keeping privacy implications in mind

alexquick · 2020-03-30T15:47:25Z

The best code is code you get to delete! Git never forgets, so if this becomes an issue going forward we can resurrect this.

alexquick added 6 commits March 27, 2020 00:35

feat: first cut of service to poll gmail

b18003e

this may be a pyrric victory over zapier :/

feat: add npm test

7d7b039

feat: ability to save Requests to airtable

fad6f0d

feat: teach GmailEmailSource how to mark messages as completed

3927433

feat: add 'npm run gmail-sync' to run gmail sync worker

d8aed0e

feat: make gmail-sync run in the main app process

a841aaf

alexquick changed the title ~~Ingest emails from Gmail and send to airtable~~ Ingest emails from Gmail and send to Airtable Mar 27, 2020

alexquick added 2 commits March 27, 2020 16:53

chore: add new env to template.config.sh

c3adf39

fix: formatting

03c8109

alexquick commented Mar 27, 2020

View reviewed changes

alexquick requested review from mjmaurer and mab253 March 27, 2020 21:16

alexquick marked this pull request as ready for review March 27, 2020 21:17

mjmaurer requested changes Mar 28, 2020

View reviewed changes

alexquick added 4 commits March 28, 2020 11:16

fix: formatting/linting

3996831

feat: line up prettier autoformatting with eslint defaults

53de2db

Merge remote-tracking branch 'origin/master' into feature-gmail-inges…

119fb6d

…tion

fix: remove complicted airtable model thing

a4b8293

alexquick closed this Mar 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ingest emails from Gmail and send to Airtable #4

Ingest emails from Gmail and send to Airtable #4

alexquick commented Mar 27, 2020 •

edited

Loading

alexquick left a comment

alexquick Mar 27, 2020

mjmaurer Mar 28, 2020

alexquick Mar 28, 2020

alexquick Mar 27, 2020

mjmaurer Mar 28, 2020

alexquick Mar 28, 2020

mjmaurer left a comment

mjmaurer Mar 28, 2020

mjmaurer Mar 28, 2020

mjmaurer commented Mar 28, 2020

mab253 commented Mar 29, 2020

mjmaurer commented Mar 30, 2020

alexquick commented Mar 30, 2020

Ingest emails from Gmail and send to Airtable #4

Ingest emails from Gmail and send to Airtable #4

Conversation

alexquick commented Mar 27, 2020 • edited Loading

General flow

Questions

alexquick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjmaurer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjmaurer commented Mar 28, 2020

mab253 commented Mar 29, 2020

mjmaurer commented Mar 30, 2020

alexquick commented Mar 30, 2020

alexquick commented Mar 27, 2020 •

edited

Loading