feat: Populate gCal calendar cache via webhooks #11928

zomars · 2023-10-17T02:53:40Z

fixes #11524

What does this PR do?

NEW Loom Demo
OLD Loom Demo
Inbound Webhooks Cache Refresh Demo
Adds a mechanism to listen to changes in Google Calendar
Adds an inbound webhook to pre-populate gCal cache
Adds a team feature flag so we can enable this progressively
Adds a cron job that handles calendar subscriptions and renewals before expiration.

Calendar Cache and Cron Jobs Enhancements:

apps/web/app/api/calendar-cache/cron/route.ts: Exported the GET handler and set cache revalidation to zero to ensure fresh data each minute.
apps/web/cron-tester.ts: Added dotenv configuration for environment variables and updated cron job endpoints to use the correct API key parameter. [1] [2]
apps/web/vercel.json: Added a new cron job entry for the calendar cache API.

API Handler Refactoring:

apps/web/pages/api/availability/calendar.ts: Refactored the API handler by splitting it into separate GET, POST, and DELETE handlers, and introduced shared authentication middleware. [1] [2] [3] [4]
packages/app-store/googlecalendar/api/webhook.ts: Added a new webhook handler to validate requests and update calendar availability cache.

Authentication and Security Improvements:

apps/web/pages/api/cron/calendar-cache-cleanup.ts: Added API key validation to ensure only authorized requests can access the cron job endpoint.

Calendar Cache and Google Calendar Integration:

packages/app-store/googlecalendar/lib/CalendarService.ts: Integrated the CalendarCache class for handling calendar cache operations and removed outdated cache handling functions. [1] [2] [3]
packages/app-store/googlecalendar/lib/CalendarService.test.ts: Updated tests to cover new calendar cache functionalities, including cache hits, misses, and calendar watch/unwatch operations. [1] [2] [3] [4] [5]

These changes aim to improve the efficiency, security, and reliability of calendar-related features in the application.

TODO (Follow up maybe)

Add a mechanism to auto-refresh about-to-expire webhooks
Add a mechanism to bypass cache either via request header or query parameter
Add a way to bulk enable for ALL users, once we feel confident enough to do so

Type of change

New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

How should this be tested?

Create an accessible tunnel from a service like tmole
Put the resulting URL .env like GOOGLE_WEBHOOK_URL="https://xxxxxxxxx.tunnelmole.net"
Make sure your running the cron using yarn dev:cron inside apps/web directory.
As a team member, Install a Google Calendar from the App Store
Once installed, enable a selected calendar
In the DB create a new TeamFeature entry with the proper teamId and a featureId named calendar-cache
You should receive a POST request populating your first cache
Add an event to your Google Calendar, you should receive another POST request, refreshing the cache.

Mandatory Tasks

Make sure you have self-reviewed the code. A decent size PR without self-review might be rejected.

vercel · 2023-10-17T02:53:50Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
api	❌ Failed (Inspect)			Nov 15, 2024 2:40am
dev	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 15, 2024 2:40am

6 Skipped Deployments

Name	Status	Preview	Updated (UTC)
ai	⬜️ Ignored (Inspect)	Visit Preview	Nov 15, 2024 2:40am
cal	⬜️ Ignored (Inspect)	Visit Preview	Nov 15, 2024 2:40am
cal-demo	⬜️ Ignored (Inspect)	Visit Preview	Nov 15, 2024 2:40am
calcom-web-canary	⬜️ Ignored (Inspect)	Visit Preview	Nov 15, 2024 2:40am
qa	⬜️ Ignored (Inspect)	Visit Preview	Nov 15, 2024 2:40am
ui	⬜️ Ignored (Inspect)	Visit Preview	Nov 15, 2024 2:40am

github-actions · 2023-10-17T02:53:55Z

Thank you for following the naming conventions! 🙏 Feel free to join our discord and post your PR link.

zomars · 2023-10-17T02:58:26Z

packages/trpc/server/routers/viewer/admin/_router.ts

Migrated procedures to new boilerplate syntax

github-actions · 2023-10-17T03:03:35Z

📦 Next.js Bundle Analysis for @calcom/web

This analysis was generated by the Next.js Bundle Analysis action. 🤖

New Page Added

The following page was added to the bundle from the code in this PR:

Page	Size (compressed)	First Load	% of Budget (`350 KB`)
`/settings/admin/calendar-cache`	`271.49 KB`	459.65 KB	131.33%

deploysentinel · 2023-10-17T03:04:25Z

Current Playwright Test Results Summary

✅ 447 Passing - ⚠️ 15 Flaky

Run may still be in progress, this comment will be updated as current testing workflow or job completes...

(Last updated on 02/14/2024 11:21:12pm UTC)

Run Details

Running Workflow PR Update on Github Actions

Commit: d77b02f

Started: 02/14/2024 11:12:29pm UTC

⚠️ Flakes

📄 apps/web/playwright/booking/multipleEmailQuestion.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Booking With Multiple Email Question and Each Other Question Booking With Multiple Email Question and checkbox group Question Multiple Email required and checkbox group required Retry 1 • Initial Attempt	0.40% (1) 1 / 253 run failed over last 7 days	4.74% (12) 12 / 253 runs flaked over last 7 days

📄 apps/web/playwright/booking/selectQuestion.e2e.ts • 3 Flakes

Top 1 Common Error Messages

`null`	3 Test Cases Affected

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Booking With Phone Question and Each Other Question Booking With Select Question and checkbox group Question Select and checkbox group not required Retry 1 • Initial Attempt	0% (0) 0 / 256 runs failed over last 7 days	5.47% (14) 14 / 256 runs flaked over last 7 days
Booking With Phone Question and Each Other Question Booking With Select Question and Radio group Question Select required and Radio group required Retry 1 • Initial Attempt	0% (0) 0 / 256 runs failed over last 7 days	3.91% (10) 10 / 256 runs flaked over last 7 days
Booking With Phone Question and Each Other Question Booking With Select Question and Short text question Select and Short text not required Retry 1 • Initial Attempt	0% (0) 0 / 256 runs failed over last 7 days	4.30% (11) 11 / 256 runs flaked over last 7 days

📄 apps/web/playwright/booking/responsiveBooking.e2e.ts • 2 Flakes

Top 1 Common Error Messages

`null`	2 Test Cases Affected

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Booking page with no questions Booking page with 1920x1080 resolution Retry 1 • Initial Attempt	0% (0) 0 / 256 runs failed over last 7 days	8.20% (21) 21 / 256 runs flaked over last 7 days
Booking page with no questions Booking page with 640x480 resolution Retry 1 • Initial Attempt	1.56% (4) 4 / 256 runs failed over last 7 days	6.64% (17) 17 / 256 runs flaked over last 7 days

📄 apps/web/playwright/login.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
user can login & logout succesfully -- future login flow user & logout using dashboard Retry 1 • Initial Attempt	0.78% (2) 2 / 256 runs failed over last 7 days	45.31% (116) 116 / 256 runs flaked over last 7 days

📄 apps/web/playwright/integrations-stripe.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Stripe integration Paid booking should be able to be rescheduled Retry 1 • Initial Attempt	0.76% (2) 2 / 262 runs failed over last 7 days	3.82% (10) 10 / 262 runs flaked over last 7 days

📄 apps/web/playwright/booking/addressQuestione2e/addressQuestion.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Booking With Address Question and Each Other Question Booking With Address Question and select Question Address required and select required Retry 1 • Initial Attempt	0% (0) 0 / 261 runs failed over last 7 days	4.21% (11) 11 / 261 runs flaked over last 7 days

📄 apps/web/playwright/event-types.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Event Types tests -- future user Different Locations Tests can add Attendee Phone Number location and book with it Retry 1 • Initial Attempt	0.74% (2) 2 / 271 runs failed over last 7 days	11.44% (31) 31 / 271 runs flaked over last 7 days

📄 apps/web/playwright/profile.e2e.ts • 2 Flakes

Top 1 Common Error Messages

`null`	2 Test Cases Affected

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Update Profile Cannot update a users email when existing user has same email (verification enabled) Retry 1 • Initial Attempt	0.42% (1) 1 / 239 run failed over last 7 days	50.63% (121) 121 / 239 runs flaked over last 7 days
Update Profile Can update a users email (verification enabled) Retry 1 • Initial Attempt	11.93% (29) 29 / 243 runs failed over last 7 days	52.67% (128) 128 / 243 runs flaked over last 7 days

📄 packages/embeds/embed-core/playwright/tests/preview.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Preview Preview - embed-core should load Retry 1 • Initial Attempt	0% (0) 0 / 273 runs failed over last 7 days	35.53% (97) 97 / 273 runs flaked over last 7 days

📄 packages/app-store/routing-forms/playwright/tests/basic.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Routing Forms Seeded Routing Form Router URL should work Retry 1 • Initial Attempt	0% (0) 0 / 272 runs failed over last 7 days	14.34% (39) 39 / 272 runs flaked over last 7 days

📄 packages/embeds/embed-core/playwright/tests/inline.e2e.ts • 1 Flake

Test Case Results

Test Case	Last 7 days Failures	Last 7 days Flakes
Inline Iframe Inline Iframe - Configured with Dark Theme Retry 1 • Initial Attempt	0.73% (2) 2 / 274 runs failed over last 7 days	42.34% (116) 116 / 274 runs flaked over last 7 days

View Detailed Build Results

zomars

Ready for review. Happy anniversary!

zomars · 2024-11-13T02:42:12Z

apps/web/app/api/calendar-cache/cron/route.ts

+ * This runs each minute and we need fresh data each time
+ * @see https://nextjs.org/docs/app/building-your-application/caching#opting-out-2
+ **/
+export const revalidate = 0;


So cron is always fresh

zomars · 2024-11-13T02:42:43Z

apps/web/app/settings/(admin-layout)/admin/calendar-cache/page.tsx

+    () => ""
+  );
+
+const Page = () => <h1>Calendar cache index</h1>;


This is a placeholder page where we might control cache entries or remove them if necessary.

zomars · 2024-11-13T02:43:00Z

apps/web/cron-tester.ts

@@ -20,7 +23,7 @@ try {
    "*/5 * * * * *",
    async function () {
      await Promise.allSettled([
-        fetchCron("/tasks/cron"),
+        fetchCron("/calendar-cache/cron"),


For testing locally

zomars · 2024-11-13T02:49:10Z

apps/web/pages/api/availability/calendar.ts

+  const user = req.userWithCredentials;
+  const { integration, externalId, credentialId } = selectedCalendarSelectSchema.parse(req.query);
+  const calendarCacheRepository = await CalendarCache.initFromCredentialId(credentialId);
+  await calendarCacheRepository.unwatchCalendar({ calendarId: externalId });


Even tho we already handle subscriptions in the cron, we make sure we purge cache when removing a selected calendar.

zomars · 2024-11-13T02:49:56Z

apps/web/vercel.json

@@ -8,6 +8,10 @@
      "path": "/api/tasks/cron",
      "schedule": "* * * * *"
    },
+    {
+      "path": "/api/calendar-cache/cron",
+      "schedule": "* * * * *"


Running each minute

zomars · 2024-11-13T02:56:29Z

packages/app-store/tests/__mocks__/OAuthManager.ts


 import type * as OAuthManager from "../../_utils/oauth/OAuthManager";

 vi.mock("../../_utils/oauth/OAuthManager", () => oAuthManagerMock);

 beforeEach(() => {
-  mockReset(oAuthManagerMock);
+  mockClear(oAuthManagerMock);


@hariombalhara this was causing errors. IDK if this can have unintended behaviors. Left more context on Campsite.

zomars · 2024-11-13T02:57:06Z

packages/core/getCalendarsEvents.ts

-    const passedSelectedCalendars = selectedCalendars.filter((sc) => sc.integration === type);
+    const passedSelectedCalendars = selectedCalendars
+      .filter((sc) => sc.integration === type)
+      // Needed to ensure cache keys are consistent


Again, to increase chance of cache hits.

zomars · 2024-11-13T02:57:55Z

packages/features/calendar-cache/api/cron.ts

+    handleCalendarsToUnwatch(),
+  ]);
+
+  // TODO: Credentials can be installed on a whole team, check for selected calendars on the team


cc @joeauyeung you might have some ideas here?

zomars · 2024-11-13T02:59:12Z

packages/features/calendars/CalendarSwitch.tsx

+        // new URLSearchParams does not accept numbers
+        credentialId: String(credentialId),


We use coerce on server side so we're good.

zomars · 2024-11-13T03:03:02Z

packages/prisma/sql/getSelectedCalendarsToWatch.sql

Useful query to have. Also we could use it in as typedRaw query once we upgrade prisma

apps/web/pages/api/availability/calendar.ts

keithwillcode · 2024-11-14T18:03:04Z

apps/web/pages/api/availability/calendar.ts

+
+export default defaultResponder(async (req: NextApiRequest, res: NextApiResponse) => {
+  await authMiddleware(req);
+  return defaultHandler({


Very nice changes in this file 🙏

packages/app-store/googlecalendar/api/webhook.ts

keithwillcode · 2024-11-14T18:13:26Z

packages/features/calendar-cache/calendar-cache.ts

+    const isCalendarCacheEnabledGlobally = await featureRepo.checkIfFeatureIsEnabledGlobally(
+      "calendar-cache"
+    );
+    if (isCalendarCacheEnabledGlobally) return new CalendarCacheRepository(calendar);


Very nice approach

This is a great example of pulling the "if" statements as high up in the app as possible and letting IoC take over.

packages/features/flags/features.repository.ts

packages/lib/server/defaultResponder.appDir.ts

packages/lib/server/repository/selectedCalendar.ts

packages/prisma/sql/getSelectedCalendarsToWatch.sql

zomars · 2024-11-15T02:10:56Z

@keithwillcode feedback addressed. 🙏

keithwillcode · 2024-11-15T17:58:12Z

packages/app-store/googlecalendar/api/webhook.ts

+  );
+
+  if (!selectedCalendar) {
+    throw new HttpError({


This is weird to throw an exception for a 200. Maybe in a follow-up we can refactor this.

I'm aware. But defaultResponder just handles the proper response and code.

keithwillcode

Great work, @zomars.

As discussed. This will be launched dark and tested from there.

PeerRich · 2024-11-15T18:01:17Z

yeah lets test with i.cal.com org for now

This reverts commit d294a74.

…17675) This reverts commit d294a74.

)" (…" This reverts commit 6272c88.

…17682)

zomars added the core area: core, team members only label Oct 17, 2023

vercel bot deployed to Preview – ai October 17, 2023 02:55 View deployment

vercel bot deployed to Preview – ui October 17, 2023 02:55 View deployment

vercel bot had a problem deploying to Preview – api October 17, 2023 02:55 Failure

zomars force-pushed the feat/calendar-cache-inbound branch from 3297976 to 6490d57 Compare October 17, 2023 02:55

zomars commented Oct 17, 2023

View reviewed changes

packages/trpc/server/routers/viewer/admin/_router.ts Outdated

Copy link

Member Author

zomars Oct 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Migrated procedures to new boilerplate syntax

vercel bot deployed to Preview – ai October 17, 2023 02:59 View deployment

vercel bot deployed to Preview – ui October 17, 2023 03:00 View deployment

vercel bot had a problem deploying to Preview – api October 17, 2023 03:00 Failure

vercel bot deployed to Preview – dev October 17, 2023 03:04 View deployment

vercel bot deployed to Preview – cal-demo October 17, 2023 03:07 View deployment

vercel bot deployed to Preview – ai October 17, 2023 03:54 View deployment

vercel bot had a problem deploying to Preview – api October 17, 2023 03:54 Failure

vercel bot deployed to Preview – dev October 17, 2023 03:57 View deployment

vercel bot deployed to Preview – cal-demo October 17, 2023 03:58 View deployment

github-actions bot added the 🚨 merge conflict This PR has a merge conflict that has to be addressed label Oct 17, 2023

vercel bot deployed to Preview – ui October 17, 2023 19:08 View deployment

vercel bot deployed to Preview – ai October 17, 2023 19:08 View deployment

vercel bot had a problem deploying to Preview – api October 17, 2023 19:09 Failure

vercel bot deployed to Preview – cal-demo October 17, 2023 19:12 View deployment

vercel bot deployed to Preview – dev October 17, 2023 19:12 View deployment

github-actions bot removed the 🚨 merge conflict This PR has a merge conflict that has to be addressed label Oct 17, 2023

vercel bot deployed to Preview – ai October 17, 2023 19:33 View deployment

vercel bot had a problem deploying to Preview – api October 17, 2023 19:34 Failure

vercel bot deployed to Preview – ui October 17, 2023 19:34 View deployment

vercel bot deployed to Preview – dev October 17, 2023 19:37 View deployment

vercel bot deployed to Preview – api November 13, 2024 02:42 View deployment

vercel bot deployed to Preview – dev November 13, 2024 02:51 View deployment

Update toggleFeatureFlag.handler.ts

9d3fc4b

zomars commented Nov 13, 2024

View reviewed changes

vercel bot deployed to Preview – api November 13, 2024 03:13 View deployment

vercel bot deployed to Preview – dev November 13, 2024 03:22 View deployment

This was referenced Nov 13, 2024

feat: Overlay Calendar v2 and Troubleshooter v2 #14693

Draft

feat: New onboarding flow #17251

Closed

keithwillcode requested changes Nov 14, 2024

View reviewed changes

zomars added 3 commits November 14, 2024 16:59

Merge branch 'main' into feat/calendar-cache-inbound

d67a534

fix: feedback

1cf5313

Update getSelectedCalendarsToWatch.sql

8de6dea

zomars requested a review from keithwillcode November 15, 2024 02:10

vercel bot deployed to Preview – api November 15, 2024 02:18 View deployment

vercel bot deployed to Preview – dev November 15, 2024 02:26 View deployment

Fix unit tests

aa64264

vercel bot deployed to Preview – api November 15, 2024 02:49 View deployment

vercel bot deployed to Preview – dev November 15, 2024 02:56 View deployment

keithwillcode reviewed Nov 15, 2024

View reviewed changes

keithwillcode approved these changes Nov 15, 2024

View reviewed changes

keithwillcode merged commit d294a74 into main Nov 15, 2024
38 of 39 checks passed

keithwillcode deleted the feat/calendar-cache-inbound branch November 15, 2024 17:59

keithwillcode added a commit that referenced this pull request Nov 15, 2024

revert: "feat: Populate gCal calendar cache via webhooks (#11928)"

a02eaf1

This reverts commit d294a74.

keithwillcode added a commit that referenced this pull request Nov 15, 2024

revert: "feat: Populate gCal calendar cache via webhooks (#11928)" (#…

6272c88

…17675) This reverts commit d294a74.

zomars added a commit that referenced this pull request Nov 16, 2024

Revert "revert: "feat: Populate gCal calendar cache via webhooks (#11928

b76d4f0

)" (…" This reverts commit 6272c88.

keithwillcode mentioned this pull request Nov 17, 2024

chore: Add ability to disable Sentry source maps #17699

Merged

3 tasks

zomars added a commit that referenced this pull request Nov 18, 2024

revert: "feat: Populate gCal calendar cache via webhooks (#11928)"" (#…

7c0ee9f

…17682)

zomars mentioned this pull request Dec 17, 2024

feat: allows forcing/skipping calendar cache serving #18224

Merged

3 tasks

		// new URLSearchParams does not accept numbers
		credentialId: String(credentialId),

feat: Populate gCal calendar cache via webhooks #11928

feat: Populate gCal calendar cache via webhooks #11928

Conversation

zomars commented Oct 17, 2023 • edited Loading

What does this PR do?

Calendar Cache and Cron Jobs Enhancements:

API Handler Refactoring:

Authentication and Security Improvements:

Calendar Cache and Google Calendar Integration:

TODO (Follow up maybe)

Type of change

How should this be tested?

Mandatory Tasks

vercel bot commented Oct 17, 2023 • edited Loading

github-actions bot commented Oct 17, 2023 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Oct 17, 2023 • edited Loading

📦 Next.js Bundle Analysis for @calcom/web

New Page Added

deploysentinel bot commented Oct 17, 2023 • edited Loading

Current Playwright Test Results Summary

⚠️ Flakes

Test Case Results

Top 1 Common Error Messages

Test Case Results

Top 1 Common Error Messages

Test Case Results

Test Case Results

Test Case Results

Test Case Results

Test Case Results

Top 1 Common Error Messages

Test Case Results

Test Case Results

Test Case Results

Test Case Results

zomars left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zomars commented Nov 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keithwillcode left a comment

Choose a reason for hiding this comment

PeerRich commented Nov 15, 2024

zomars commented Oct 17, 2023 •

edited

Loading

vercel bot commented Oct 17, 2023 •

edited

Loading

github-actions bot commented Oct 17, 2023 •

edited

Loading

github-actions bot commented Oct 17, 2023 •

edited

Loading

deploysentinel bot commented Oct 17, 2023 •

edited

Loading