docs: simplify streaming metadata (#84253)

icyJoseph · vercel[bot] · web-flow · commit 7f25da71fb14 · 2025-10-03T17:16:39.000+02:00
Co-authored-by: vercel[bot] &lt;35613825+vercel[bot]@users.noreply.github.com&gt;
diff --git a/docs/01-app/01-getting-started/14-metadata-and-og-images.mdx b/docs/01-app/01-getting-started/14-metadata-and-og-images.mdx
@@ -13,6 +13,7 @@ related:
     - app/api-reference/file-conventions/metadata/opengraph-image
     - app/api-reference/file-conventions/metadata/robots
     - app/api-reference/file-conventions/metadata/sitemap
+    - app/api-reference/config/next-config-js/htmlLimitedBots
 ---
 
 The Metadata APIs can be used to define your application metadata for improved SEO and web shareability and include:
@@ -117,9 +118,15 @@ export default function Page({ params, searchParams }) {}
 
 ### Streaming metadata
 
-For dynamically rendered pages, if resolving `generateMetadata` might block rendering, Next.js streams the resolved metadata separately and injects it into the HTML as soon as it's ready.
+For dynamically rendered pages, Next.js streams metadata separately, injecting it into the HTML once `generateMetadata` resolves, without blocking UI rendering.
 
-Statically rendered pages don’t use this behavior since metadata is resolved at build time.
+Streaming metadata improves perceived performance by allowing visual content to stream first.
+
+Streaming metadata is **disabled for bots and crawlers** that expect metadata to be in the `<head>` tag (e.g. `Twitterbot`, `Slackbot`, `Bingbot`). These are detected by using the User Agent header from the incoming request.
+
+You can customize or **disable** streaming metadata completely, with the [`htmlLimitedBots`](/docs/app/api-reference/config/next-config-js/htmlLimitedBots#disabling) option in your Next.js config file.
+
+Statically rendered pages don’t use streaming since metadata is resolved at build time.
 
 Learn more about [streaming metadata](/docs/app/api-reference/functions/generate-metadata#streaming-metadata).
 
diff --git a/docs/01-app/03-api-reference/04-functions/generate-metadata.mdx b/docs/01-app/03-api-reference/04-functions/generate-metadata.mdx
@@ -41,6 +41,10 @@ export default function Page() {}
 
 Dynamic metadata depends on **dynamic information**, such as the current route parameters, external data, or `metadata` in parent segments, can be set by exporting a `generateMetadata` function that returns a [`Metadata` object](#metadata-fields).
 
+Resolving `generateMetadata` is part of rendering the page. If the page can be pre-rendered and `generateMetadata` doesn't introduce dynamic behavior, the resulting metadata is included in the page’s initial HTML.
+
+Otherwise the metadata resolved from `generateMetadata` [can be streamed](/docs/app/api-reference/functions/generate-metadata#streaming-metadata) after sending the initial UI.
+
 ```tsx filename="app/products/[id]/page.tsx" switcher
 import type { Metadata, ResolvingMetadata } from 'next'
 
@@ -104,8 +108,6 @@ For type completion of `params` and `searchParams`, you can type the first argum
 > - The `metadata` object and `generateMetadata` function exports are **only supported in Server Components**.
 > - You cannot export both the `metadata` object and `generateMetadata` function from the same route segment.
 > - `fetch` requests inside `generateMetadata` are automatically [memoized](/docs/app/guides/caching#request-memoization) for the same data across `generateMetadata`, `generateStaticParams`, Layouts, Pages, and Server Components.
-> - Resolving `generateMetadata` is part of rendering the page. If the page can be pre-rendered and `generateMetadata` doesn't introduce dynamic behavior, its result is included in the page’s initial HTML.
-> - `generateMetadata` [can stream results](/docs/app/api-reference/functions/generate-metadata#streaming-metadata) to improve page load performance.
 > - React [`cache` can be used](/docs/app/guides/caching#react-cache-function) if `fetch` is unavailable.
 > - [File-based metadata](/docs/app/api-reference/file-conventions/metadata) has the higher priority and will override the `metadata` object and `generateMetadata` function.
 
@@ -1170,33 +1172,33 @@ There are two default `meta` tags that are always added even if a route doesn't
 
 ### Streaming metadata
 
-Metadata returned by `generateMetadata` is streamed to the client. This allows Next.js to inject metadata into the HTML as soon as it's resolved.
+Streaming metadata allows Next.js to render and send the initial UI to the browser, without waiting for `generateMetadata` to complete.
 
-Streamed metadata is appended to the `<body>` tag. Since metadata mainly targets bots and crawlers, Next.js streams metadata for bots that can execute JavaScript and inspect the full DOM (e.g. `Googlebot`). We have verified that these bots interpret the metadata correctly.
+When `generateMetadata` resolves, the resulting metadata tags are appended to the `<body>` tag. We have verified that metadata is interpreted correctly by bots that execute JavaScript and inspect the full DOM (e.g. `Googlebot`).
 
-For **HTML-limited** bots that can’t run JavaScript (e.g. `Twitterbot`), metadata continues to block page rendering and is placed in the `<head>` tag.
+For **HTML-limited bots** that can’t execute JavaScript (e.g. `facebookexternalhit`), metadata continues to block page rendering. The resulting metadata will be available in the `<head>` tag.
 
-Next.js automatically detects the user agent of incoming requests to determine whether to serve streaming metadata or fallback to blocking metadata.
+Next.js automatically detects **HTML-limited bots** by looking at the User Agent header. You can use the [`htmlLimitedBots`](/docs/app/api-reference/config/next-config-js/htmlLimitedBots) option in your Next.js config file to override the default [User Agent list](https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/router/utils/html-bots.ts).
 
-If you need to customize this list, you can define them manually using the `htmlLimitedBots` option in `next.config.js`. Next.js will ensure user agents matching this regex receive blocking metadata when requesting your web page.
+To fully disable streaming metadata:
 
 ```ts filename="next.config.ts" switcher
 import type { NextConfig } from 'next'
 
 const config: NextConfig = {
-  htmlLimitedBots: /MySpecialBot|MyAnotherSpecialBot|SimpleCrawler/,
+  htmlLimitedBots: /.*/,
 }
 
 export default config
 ```
 
 ```js filename="next.config.js" switcher
 module.exports = {
-  htmlLimitedBots: /MySpecialBot|MyAnotherSpecialBot|SimpleCrawler/,
+  htmlLimitedBots: /.*/,
 }
 ```
 
-Specifying a `htmlLimitedBots` config will override the Next.js' default list, allowing you full control over what user agents should opt into this behavior.
+Streaming metadata improves perceived performance by reducing [TTFB](https://developer.mozilla.org/docs/Glossary/Time_to_first_byte) and can help lowering [LCP](https://developer.mozilla.org/docs/Glossary/Largest_contentful_paint) time.
 
 Overriding `htmlLimitedBots` could lead to longer response times. Streaming metadata is an advanced feature, and the default should be sufficient for most cases.
 
diff --git a/docs/01-app/03-api-reference/05-config/01-next-config-js/htmlLimitedBots.mdx b/docs/01-app/03-api-reference/05-config/01-next-config-js/htmlLimitedBots.mdx
@@ -23,9 +23,50 @@ module.exports = {
 
 ## Default list
 
-Next.js includes [a default list of HTML limited bots](https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/router/utils/html-bots.ts).
+Next.js includes a default list of HTML limited bots, including:
 
-Specifying a `htmlLimitedBots` config will override the Next.js' default list, allowing you full control over what user agents should opt into this behavior. However, this is advanced behavior, and the default should be sufficient for most cases.
+- Google crawlers (e.g. Mediapartners-Google, AdsBot-Google, Google-PageRenderer)
+- Bingbot
+- Twitterbot
+- Slackbot
+
+See the full list [here](https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/router/utils/html-bots.ts).
+
+Specifying a `htmlLimitedBots` config will override the Next.js' default list. However, this is advanced behavior, and the default should be sufficient for most cases.
+
+```ts filename="next.config.ts" switcher
+const config: NextConfig = {
+  htmlLimitedBots: /MySpecialBot|MyAnotherSpecialBot|SimpleCrawler/,
+}
+
+export default config
+```
+
+```js filename="next.config.js" switcher
+module.exports = {
+  htmlLimitedBots: /MySpecialBot|MyAnotherSpecialBot|SimpleCrawler/,
+}
+```
+
+## Disabling
+
+To fully disable streaming metadata:
+
+```ts filename="next.config.ts"
+import type { NextConfig } from 'next'
+
+const config: NextConfig = {
+  htmlLimitedBots: /.*/,
+}
+
+export default config
+```
+
+```js filename="next.config.js" switcher
+module.exports = {
+  htmlLimitedBots: /.*/,
+}
+```
 
 ## Version History
 
