Extremely high amount of Data Cache writes

[sandervspl]

Summary

I just got a notification that I am over 50% of my monthly Data Cache writes and it's only day 4 of my billing cycle... I just checked and it seems like my website is doing over 10k Data Cache writes EVERY HOUR (95% are from Incremental Static Regeneration). This is odd because my website has a little over 1500 static pages that only revalidate once every 24 hours. 380 pages are statically generated every day, the other ~1100 pages are incrementally generated during the day when people visit them.

I just looked through my code and I can only find 1 revalidatePath function, which is the one that revalidates pages every 24 hours.

I don't understand the math here... how can I be doing over 10k Data Cache writes??? The dashboard doesn't tell me at all what is causing this.

[amyegan]

Hey @sandervspl. It sounds like there is a lot of revalidation and incrementally generated pages on your site that could be driving up the Data Cache writes depending on things like site traffic, configuration, and build frequency.

I recommend reading through this guide which has some tips for optimizing Data Cache writes: vercel.com/docs/pricing/data-cache#manage-and-optimize-usage-for-data-cache

[sandervspl]

Hey, thanks for replying. Yes that is all I can see indeed and I've read the page you have mentioned. But the website is completely static. I have two pages: homepage and 1 other dynamic URL page (this page has a generateStaticParams() to generate all pages as static on build). They both have the following configuration in their page.tsx

export const revalidate = false;
export const dynamic = 'force-static';
export const dynamicParams = true;

On Demand Revalidation happens only every 24 hours, not every hour.

I have just redeployed making sure every dynamic URL page is generated on build. None should be build on visit anymore.

Several hours later I still see 10k to 30k revalidations per hour! I just don't understand what is causing this? I don't do any revalidation!

This is the result of an all-search on revalidate

/api/revalidate is hit only 193 times today

The big spike was a new deployment, trying to fix this issue.

I'm lost on what more I can do.

[sandervspl]

I have resolved the issue. These are the steps I had to take in case anyone else encounters the same problem.

1. I've added this config to all my page.tsx AND route.tsx files (such as opengraph-image routes)

export const revalidate = false;
export const dynamic = 'force-static';
export const dynamicParams = false; // -> disables ISR

2. I generate ALL pages on build now, instead of some
3. I deleted all twitter-image.tsx files because for some reason the route config don't work on those routes, even though the docs say they do

Also applied these changes but I'm not sure if they're necessary

1. Added export const fetchCache = 'force-cache'; to the root layout
2. Removed unused pages/routes even though they started with an underscore

I would LOVE to see more in-depth data on WHAT is being revalidated, because this was super stressful. It was somewhat easy for me because I only have two routes, but it was very unclear to me what was being revalidated. And honestly, I still don't understand how 10k-30k revalidates per hour make sense. I get 200-500 page views per hour.

[lhguerra]

@leerob I opened a support ticket through my client's account, I'll post here anything that I can share about the solution.

[lhguerra]

I've been waiting for two days now with a severity 2 ticket open... Vercel SLA is always a mistery

[lhguerra]

Oh and the site url is https://aposta10.com/

[amyegan]

Thanks for sharing the URL. High ISR read and write units for your deployments could be caused by

• a large number of paths being revalidated
• a short revalidation interval
• large amounts of data being revalidated

From what I am able to see it looks like the project uses GraphQL. Since query responses can be quite large, that may be contributing to high data cache usage. Cache control options or a different revalidate period could help in that case.

When a page contains a mix of static and dynamic data, the dynamic data needs to be re-fetched when rendering the page. In this scenario, Vercel Data Cache is used to cache the static data to avoid slow origin fetches.

Both ISR and Vercel Data Cache support time-based revalidation, on-demand revalidation, and tag based revalidation.

https://vercel.com/docs/infrastructure/data-cache

Identifying paths

If you go to the Usage tab, click Data Cache in the sidebar, go to the Bandwidth tab, and toggle to Data written that should give you some more information about the paths contributing to Data Cache usage.

Support tickets

Tickets are processed in the order they are received, and when one of our support engineers is available they can help you with more details specific to your project. Thank you for waiting while the team works through a large number of cases.

https://vercel.com/guides/vercel-support-queue-time

[lhguerra]

That's one of the issues: since I use graphql everything runs in the same path, its not like I can filter through querystrings to understand the traffic in a segmented way.

But I have other ways to monitor that should bring some insight, for example in Axiom plugin I know exactly which route is the biggest consumer of resources.

But I get roughly 1 million requests a day (including everything, even images), and in this month I'm already being charged for nearly 200 million data cache writes. If I could only understand how a single request can generate multiple data cache writes, maybe I could have an insight on how to change the way I build pages or something.

[schnubor]

We're still on the page router but are experiencing the same issue, even after completely disabling ISR (removing all revalidate keys from all pages) we see millions of cache writes that are apparently caused by ISR. We created a support ticket (#00246926) 10 days (!) ago, but did not receive any information on what's causing all those writes or how to identify them.

It's becoming frustrating and expensive, since we don't even exactly know what we're paying for at this point. Observability on this horrible, even the support representative admitted this. @amyegan @leerob

[schnubor]

This helped: https://github.com/orgs/vercel/discussions/7108#discussioncomment-10376075

[g-rasse]

We're seeing the same thing and are also waiting for a response to the ticket.

We haven't completely disabled ISR, but changed the revalidation time to 18h (which made no difference). Still writing thousands of units every hour ramping up our costs without having any idea where it's coming from or why.

Similar issue here:
https://github.com/orgs/vercel/discussions/7337

EDIT:
To clarify, we're using the pages router with GraphQL.

[farhanulju]

Seeing the same issue at my E-commerce site with no revalidation (Only On-demand)
Getting 100k+ Data writes with less than 1000 views per day. There is no way to understand what is causing it.

The Data-cache usage page doesn't show any log points.

Got only 14 requests. Yet got 100k requests yesterday,

[wanjas]

This is because it's in "writing units".

One write unit equals 8 KB of data written to the cache (source)

I'm trying to evaluate this new pricing model. This cache pricing and "Fast Origin Transfer" are on the way to triple the coast for our project.

[tavux-ratkaisut]

Yes units and not amount of requests but the problem is still there. We have no way of checking what's actually causing high usage and try to optimise based on that.

Currently in a situation where costs are ramping up, but no way to try and fix it as it's unclear what's causing high unit amounts.

I would assume multiple people are sooner or later forced to move a project elsewhere if it's impossible to find out what's causing the issue.

[lhguerra]

Yet another way to make billing confusing and rip us off!

[andrewmumblebee]

For anyone who is still on version v14.2.2 or lower of NextJS.

Update NextJS to at least v14.2.3

There was a bug with cache tags, which ended up causing them to always revalidate, this caused my data-cache write usage to match my data-cache read usage.

This is the PR that fixes it vercel/next.js#64799

After updating my writes dropped from over a million a day to a more reasonable ~25,000 depending on how active our content writers/developers were.

Fortunately i got some usage refunded when that happened due to it being a Vercel framework issue, but I've now hit another issue with data-cache write/read usage matching eachother after changing CMS, but using the same revalidate tags 😆, so having to debug it again.

I really have a love hate relationship with the data cache, it's great when it works as expected, but terrifying when it doesn't due to the cost structure and not having visibility on what exactly is causing the revalidations.

[lhguerra]

Was this really added to version 14? I only see that PR merged in the canary branch which is for the 15 version afaik

[sdbarrington]

I have this same problem and have tried everything, to the point I may have to move to GCP as I'm worried if I start pulling in more traffic costs won't be manageable. How can a nextjs site with ~200 visitors a day possibly need this many data writes...

I've tried:

• update nextjs to latest.
• revalidate: false
• revalidate: 3600000000
• completely removed revalidate.

None of these seems to make any difference, my content is fairly static, so how can I just turn this off and manually trigger it once a week or something?

[amyegan]

Hi @sdbarrington. I don't know enough about your project to make specific recommendations. Based on your description, it sounds like on-demand revalidation is what you want.

There are also some other docs that can help you optimize data cache writes:

• Optimizing Data Cache reads and writes
  • On-demand revalidation
  • Time-based revalidation
• Data Cache writes chart

[schnubor]

Maybe this is of help for people that are still using the page router and are experiencing this as we did:

Check your usage of getStaticProps, especially on dynamic routes with unknown slugs that cannot be covered by using getStaticPaths, e.g. /product/[id].

In our case we were using getStaticProps on a dynamic page like e.g. /product/[id] (thousands of pages) -> every time this page got called it caused cache writes. We replaced it with getServerSideProps et voila, it drastically reduced our cache writes.

[sdbarrington]

Thanks! I think that must be it, I will give that a try.

Seems annoying though, as this content very rarely changes, so feels like the right use case for getStaticProps

[sandervspl]

For me I thought I had a similar case. In my case I'm using the app router, though. I tried generating all my dynamic pages statically (thousands of pages) and setting revalidate to false and adding force-static config to all pages. Still I see tens/hundreds of thousands of data cache writes. Less than when I made this post originally, but still many more than I would think with fully prebuilt static pages. Something seems very wrong with this data cache thing.

[sdbarrington]

I've deployed it with getstaticProps now, so will report back over the next couple of days. thansk for your help. Yes it's very confusing and gets expensive quickly.

[sdbarrington]

swithcing to getStaticProps seems to have fixed it for me, a performance hit, but worth it as the cache costs we're ballooning.

[victorcarvalhosp]

Trying to one today, hope it works. Our cost increased a lot with no apparent reason, most due to a really high Data Cache Writes, more than $1000 in just 8 days! It's a bummer changing from getStaticProps to getServerSideProps due to the costs, as the performance is worst.

[farhanulju]

I had the same issue until last month. It got fixed once we changed our API logic.

Before:
We loaded our All Product API endpoint on every product page to fetch review and other relevant data on 20 products that we will ultimately show on a product page.

After:
We load only 20 products on each product page from the API.
The Backend handles all the logic and provides the list of 20 Products we need.

Result: Page size reduced from 1.25 MiB to 23 KiB.

And our issue was resolved.

[itsjab]

We are facing a similar issue, where Data Cache Writes were skyrocketing after going live.

In our case, we believe that the cache writes were caused by a dynamic page that was accidentally cached. We thought that dynamic pages using search params would not be cached by default. This seems to be the case, but we forgot
to remove a cache tag in the fetch request retrieving data from the CMS.

So, what we see in the logs is the following (s. Screenshot):

1. Cache Miss for the initial request to the dynamic page (this is expected)
2. Found cache tag on request
3. Update data cache
   

My theory is that this happens on every request to the dynamic page. Looking forward to your feedback.

Finally, I have to agree with the general notion in this thread. The data cache usage is extremely obscure, it's difficult to debug issues, and quickly becomes a financial issue on projects with high traffic.

[github-actions[bot]]

This discussion was automatically locked because the community moved to a new site. Please join us at vercel.community