Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalidate CDN caches after uploading database dump #6656

Merged
merged 9 commits into from
Jun 22, 2023
Merged

Invalidate CDN caches after uploading database dump #6656

merged 9 commits into from
Jun 22, 2023

Conversation

jdno
Copy link
Member

@jdno jdno commented Jun 21, 2023

After uploading a new version of the database dump, the cached version needs to be invalidated in CloudFront and Fastly.

Fixes #6652

jdno added 2 commits June 21, 2023 14:13
@jdno
Implement cache invalidations on Fastly
2fc6b7c 
An implementation for cache invalidations on Fastly has been added.
Specific paths can be invalided using Fastly's cached URL endpoint,
but this does not support wildcard invalidations.

Documentation on Fastly's invalidation API can be found here:
https://developer.fastly.com/reference/api/purging/
@jdno
Invalidate CDN caches after uploading database dump
642383d 
After uploading a new version of the database dump, the cached version
needs to be invalidated in CloudFront and Fastly.
@jdno jdno requested a review from Turbo87 June 21, 2023 12:57
@Turbo87 Turbo87 added C-bug 🐞 Category: unintended, undesired behavior A-backend ⚙️ labels Jun 21, 2023
Turbo87
Turbo87 reviewed Jun 21, 2023
View reviewed changes
src/worker/dump_db.rs Outdated Show resolved Hide resolved
src/worker/dump_db.rs Outdated Show resolved Hide resolved
src/worker/dump_db.rs Outdated Show resolved Hide resolved
src/worker/fastly.rs Outdated Show resolved Hide resolved
jdno added 4 commits June 21, 2023 16:05
@jdno
Turn cache invalidation into a simple function
cecb3cd
@jdno
Warn instead of panicking when invalidations fail
7bd880d 
When invalidating a CDN fails, we want to log a warning and continue
execution. This ensures that invalidations against all CDNs are
attempted.
@jdno
Get Fastly instance from environment
8900804 
An instance of the Fastly struct has been added to the `Environment` so
that it can be shared between jobs.
@jdno
Optionally invalidate the Fastly cache
4e0443d 
If the API token for Fastly is missing in the environment, skip
invalidating the cache there. This makes it possible to opt-in to the
new feature.
Turbo87
Turbo87 reviewed Jun 21, 2023
View reviewed changes
src/worker/dump_db.rs Outdated Show resolved Hide resolved
src/worker/fastly.rs Outdated Show resolved Hide resolved
@alex
Copy link
Member

alex commented Jun 21, 2023

Thanks for jumping on this so quickly!

Turbo87
Turbo87 approved these changes Jun 22, 2023
View reviewed changes
Copy link
Member

@Turbo87 Turbo87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@Turbo87 Turbo87 marked this pull request as ready for review June 22, 2023 14:43
@Turbo87 Turbo87 merged commit c36b6d5 into rust-lang:master Jun 22, 2023
@jdno jdno deleted the db-dump-invalidations branch June 22, 2023 16:10
@Turbo87
Copy link
Member

Turbo87 commented Jul 15, 2023

Jul 15 11:03:01 app/background_worker.1 DEBUG invalidate: crates_io::worker::fastly: invalidation request to Fastly failed status=401 headers={"connection": "keep-alive", "content-length": "54", "content-type": "application/json", "x-served-by": "cache-iad-kcgs7200110"} body=Ok("{"msg":"Provided credentials are missing or invalid"}\n") path="db-dump.tar.gz"
Jul 15 11:03:01 app/background_worker.1 WARN crates_io::worker::dump_db: failed to invalidate Fastly cache: failed to invalidate db-dump.tar.gz on Fastly

looks like production might have the wrong fastly credentials? 🤔

@jdno
Copy link
Member Author

jdno commented Jul 18, 2023

As a quick status update: I was able to confirm the issue, but unable to resolve it. Messaged Fastly support, since it looks like the problem might be on their side. Will update here when the issue is resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Turbo87 Turbo87 Turbo87 approved these changes

Assignees
No one assigned
Labels
A-backend ⚙️ C-bug 🐞 Category: unintended, undesired behavior
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cloudfront cache not cleared after uploading new db-dump.tar.gz
3 participants
@jdno @alex @Turbo87