Account Sync Engine

[FelixMalfait]

[WIP issue / do not pay attention to it]

Need to be able to connect accounts (gmail, outlook, imap, etc.) and sync them with internal APIs (messaging for now but later probably calendar, contacts).
I'm separating the logic to store message threads (in #2210) from the logic to sync accounts here.

Implementation

Allow users to add an e-mail account in settings

• Add a new tab "accounts" under profile settings
• Add a button "Add Google account" that redirects to Google with the right scopes
• Create a page to receive the connection tokens and store them in ExternalApiConnections / MessagingChannel
• Display the list of connected channels in the accounts tab we created

Note: we already implement a nodejs Passport + GoogleStrategy for login so we should be able to reuse the existing code and pass additional scopes. We don't want to ask for those scopes during login because it would hurt conversion (currently we ask for [profile, email] which is a 1-step process, asking for emails would add one more permission screen)

Scope: Full-stack
Time: 2.5 days

Setup queue and background job system

By the time we start working on this, this sub-task might have been completed already.
(Look for "pg-boss" or "bullmq")

Scope: backend
Time: 2 days

Upon connecting the account, dispatch a job to fetch emails

• Create a job ImportEmails(MessagingChannelId)
• Implement "googleapis" node package and authenticate user with stored token
• Call the api to retrieve all threads
• Iterate over all pages, iterate over every thread, iterate over every message, iterate over every recipient, save to db

Note: we will also import emails through webhook, so make sure we split the mass-import/pagination logic from the logic to import an email and map it into db which will be re-used.
Note2: use batch-requests to fetch the threads to avoid hitting api rate limits -> https://developers.google.com/gmail/api/reference/quota ; maybe need to add a slight delay too (it says "allow short bursts" but not sure how long is the moving average computation window)

Scope: backend
Time: 3 days

Polling sync

Take most recent historyId and fetch events that happened since then https://developers.google.com/gmail/api/reference/rest/v1/users.history/list

Create person and a company from email

When we create a MessageRecipient, we should already check if there's a corresponding person in the people table.
Now, if that person doesn't exist, we want to create it manually

Scope: backend
Time: 1 day

Discussion

Data model

ConnectedAccount

• id, createdAt, updatedAt
• ownerId - fk to users
• type - string enum: google, outlook, smtp, intercom, twilio
• accessToken - string
• refreshToken - string
• externalScopes - array
• hasEmailScope - boolean

Contact creation

Should we create a contact for every email recipient?
How do we differentiate newsletter, internal emails and spam from real interactions?
Sent mailbox only as v1?

Tasks

Preview Give feedback

1. Create connected-account data model #2809
   type: feature +1
   
2. [Timebox] Create a POC to test the gmail API #2814
   type: feature +1
   
3. Create IS_MESSAGING_ENABLED feature flag #2816
   scope: backend type: feature +2
   
4. ✉️ Settings/Accounts page #2884
   4 of 4
   scope: front type: feature +2
   
5. ✉️ Settings/Accounts/Emails page #2885
   4 of 4
   scope: front type: feature +2
   
6. ✉️ Settings/Accounts/New page #2890
   2 of 2
   scope: front type: feature +2
   
7. Show messages in a new message tab (Master Ticket) #2897
   4 of 4
   type: chore +1
   
8. ✉️ Settings/Accounts/Emails/Inbox Settings page #2999
   4 of 4
   scope: front type: feature +2
   
9. 📅 [Master] Settings/Accounts/Calendars page #4058
   7 of 7
   scope: front type: feature +2
   
10. 📅 [Master] Settings/Accounts/Calendars/Calendar settings page #4062
    6 of 6
    scope: front type: feature +2
    
11. 📅 [Master] Show Page/Activities Panel/Calendar tab #4069
    7 of 7
    scope: front type: feature +2