stale data in production / race condition between Netlify and AWS when invalidating cache

[thescientist13]

Overview

It often seems that after a build triggered by a chance in Contentful CMS, the next Netlify builds are still stale, and inconsistently sow. I think the issue is that updating content in Contentful does two things:

1. Ping Netlify to do a build
2. Ping AWS to invalidate the /api/v2/events cache

Since the project build and pulls from the AWS API, if that invalidation hasn't finished yet, it will cause the Netlify build to use old data.

Details

It seems that these operations need to be sequential, not parallel. Not sure if that is possible via Contentful? (aka "step" functions). Or maybe have the project bypass the public API and go right API Gateway? That would expose the raw endpoint in the source code though, which is less than ideal. Or maybe it could be an environment variable?

---

In the meantime it will just mean I have to do a couple extra clicks, but would be nice to find a solution for this.

[thescientist13]

Honestly, I think this must be a Netlify issue. After updating content in Contentful, local dev shows the correct underline for the event link

And the main studio website shows all the updated content too on the event details page (built by Architect)

But of course, the main TT website doesn't show anything new, even after multiple revalidation and rebuilds 😫 😠 🤷‍♂️

[thescientist13]

Just rebuilt the site again and now everything is fine 🤷‍♂️
https://app.netlify.com/sites/laughing-kare-be3077/deploys/63a7071698831476890aec3c

(although technically the hand should not be showing now since it is after the performance date 😬 )

[thescientist13]

Now testing locally, no link data now 🤷‍♂️ 😡

{
  event: {
    id: 48,
    title: "Tuesday's Tunes - 12/20/2022",
    description: '<p>Our next show is coming soon!  Check back for more details.</p>',
    startTime: 1671584400,
    endTime: 1671584400,
    tags: [ 'tt' ]
  }
}

It's definitely there in the API call
https://www.analogstudios.net/api/v2/events?tag=tt

Even tried a cache busting query param

const events = (await fetch(`https://www.analogstudios.net/api/v2/events?tag=tt&t=${new Date().getTime()}`

---

So either an issue with

1. node-fetch
2. Node.js

Really annoying either way....

[thescientist13]

Just tried with native fetch and same thing :/

[thescientist13]

So maybe for now, need to do this

1. Convert invalidation to /api/v2/events*
2. Disable Netlify web hook in Contentful

Then going forward, the workflow would be

1. Publish content in Contentful
2. Validate cache invalidate in AWS
3. Manually rebuild the site in Netlify

Unless I can somehow "step" this sequence, we may be tempting the cache gods to much. 😬

[thescientist13]

Well, did as follows and things seem fine locally and in production at least.

Will need to think of another way to approach this, but will keep this one in ready for now. Will want to validate how the next content update holds and see if this sequence works. If so, maybe will need to have an AWS event trigger the Netlify build after the cache invalidation is complete. (would create a new issue for this if so)

[thescientist13]

OK, so after a couple tests now with only invalidating CloudFront, and then initiating a Netlify, everything appears to be working as expected. so basically it seems I need to figure out a way to make sure Netlify only builds after CloudFront invalidation is complete.