server component prerendering at build time incompatible with runtime configuration through env vars

[bripkens]

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: darwin
      Arch: arm64
      Version: Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:43 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T8112
    Binaries:
      Node: 18.14.0
      npm: 9.3.1
      Yarn: N/A
      pnpm: 8.5.0
    Relevant packages:
      next: 13.4.5-canary.6
      eslint-config-next: 13.4.4
      react: 18.2.0
      react-dom: 18.2.0
      typescript: 5.1.3

Which area(s) of Next.js are affected? (leave empty if unsure)

App directory (appDir: true)

Link to the code that reproduces this issue or a replay of the bug

https://github.com/bripkens/env-var-reproducer

To Reproduce

To reproduce this issue, do the following:

$ git clone https://github.com/bripkens/env-var-reproducer.git
$ cd env-var-reproducer
$ npm i
$ npm run build

The build will fail with

npm run build

> env-var-reproducer@0.1.0 build
> next build

- info Creating an optimized production build
- info Compiled successfully
- info Linting and checking validity of types
- info Collecting page data
[    ] - info Generating static pages (0/5)TypeError: Failed to parse URL from undefined/api/users
    at new Request (node:internal/deps/undici/undici:6949:19)
    at r (/Users/ben/repos/env-var-reproducer/.next/server/chunks/464.js:5956:65)
    at doOriginalFetch (/Users/ben/repos/env-var-reproducer/node_modules/next/dist/server/lib/patch-fetch.js:273:24)
    at /Users/ben/repos/env-var-reproducer/node_modules/next/dist/server/lib/patch-fetch.js:394:20 {
  [cause]: TypeError [ERR_INVALID_URL]: Invalid URL
      at new NodeError (node:internal/errors:399:5)
      at URL.onParseError (node:internal/url:565:9)
      at new URL (node:internal/url:645:5)
      at new Request (node:internal/deps/undici/undici:6947:25)
      at r (/Users/ben/repos/env-var-reproducer/.next/server/chunks/464.js:5956:65)
      at doOriginalFetch (/Users/ben/repos/env-var-reproducer/node_modules/next/dist/server/lib/patch-fetch.js:273:24)
      at /Users/ben/repos/env-var-reproducer/node_modules/next/dist/server/lib/patch-fetch.js:394:20 {
    input: 'undefined/api/users',
    code: 'ERR_INVALID_URL'
  }
}

Error occurred prerendering page "/users". Read more: https://nextjs.org/docs/messages/prerender-error
TypeError: Failed to parse URL from undefined/api/users
    at new Request (node:internal/deps/undici/undici:6949:19)
    at r (/Users/ben/repos/env-var-reproducer/.next/server/chunks/464.js:5956:65)
    at doOriginalFetch (/Users/ben/repos/env-var-reproducer/node_modules/next/dist/server/lib/patch-fetch.js:273:24)
    at /Users/ben/repos/env-var-reproducer/node_modules/next/dist/server/lib/patch-fetch.js:394:20
- info Generating static pages (5/5)

> Export encountered errors on following paths:
        /users/page: /users

Please note: I know that the env var is not defined. The bug is exactly about cases where one cannot define the env var because the value depends on the runtime environment.

Describe the Bug

Pages (using app router) can now make fetch requests directly. For example, like this:

export default async function Users() {
  const resp = await fetch(`${process.env.API_ORIGIN}/api/users`);

  if (!resp.ok) {
    throw new Error('Failed to retrieve users.');
  }

  const users = await resp.json() as User[];

  return (
    <div>Got {users.length} users.</div>
  )
}

As part of the build process (next build), Next.js will attempt to pre-render the page. This all works fine, as long as the API endpoint is known and available at build time.

However, there are cases where an API is not available at build time. For example, when a web app is compiled and bundled into a Docker container image. Docker container images can be used in a variety of ways, and it is common to support configuration of API/service endpoints.

With the way prerendering currently works with app router, I don't see how I can make this work properly. Aside from these workarounds:

• Disable caching entirely for the page through export const dynamic = "force-dynamic"; or for the fetch request through {cache: "no-store"}. This stops prerendering at build time, but also disables any and all caching at runtime.
• (as suggested on the forum), just swallow these HTTP errors by wrapping the fetch call in try { /* fetch */ } catch (e) { /* ignore */ }

Expected Behavior

It seems to me like there are at least two ways to approach this:

1. Identify automatically when a non-existing env var is used at build-time and opt-out of prerendering for the page. Essentially marking this as a dynamic function code path.
2. Provide a configuration option to opt-out of build-time prerendering completely.

Note that in both cases the code/functions would be treated as regular dynamic functions similar to functions leveraging headers or cookies.

Which browser are you using? (if relevant)

not relevant

How are you deploying your application? (if relevant)

Vercel AND other platform/Docker

[jaredonline]

We ran into this over here too #44062

[bripkens]

Found a workaround that is making use of middleware and the identification of dynamic rendering when calling headers.

TLDR: Turn the env var into a request header in middleware.ts. Then read the var using headers in your page.tsx. Of course you would wanna hide this in some utility. Example:

middleware.ts

import { NextRequest } from "next/dist/server/web/spec-extension/request";
import { NextMiddlewareResult } from "next/dist/server/web/types";
import { NextResponse } from "next/server";

export default async function (request: NextRequest): Promise<NextMiddlewareResult> {
	const requestHeaders = new Headers(request.headers);
	requestHeaders.set("x-runtime-env-api-origin", process.env.API_ORIGIN ?? "");

	return NextResponse.next({
		request: {
			headers: requestHeaders,
		},
	});
}

export const config = {
	matcher: [
		/*
		 * Match all request paths except for the ones starting with:
		 * - _next/static (static files)
		 * - _next/image (image optimization files)
		 * - favicon.ico (favicon file)
		 */
		"/((?!_next/static|_next/image|favicon.ico).*)",
	],
};

page.tsx

import { headers } from "next/headers";

export default async function Users() {
	const apiOrigin = headers().get("x-runtime-env-api-origin");
	if (!apiOrigin) {
		throw new Error("Missing API origin discovered");
	}
	const resp = await fetch(`${apiOrigin}/api/users`);
}