Soft navigated page on refresh shows raw data

[nandap4790]

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

Binaries:
Node: 19.0.0
npm: 8.19.2
Relevant packages:
next: 13.2.0
react: 18.2.0
react-dom: 18.2.0

Which area(s) of Next.js are affected? (leave empty if unsure)

App directory (appDir: true), Routing (next/router, next/navigation, next/link)

Link to the code that reproduces this issue

To Reproduce

rec.mov

Describe the Bug

While navigating from one route to another, a soft navigation happens and the page loads correctly. On refreshing the just landed page, I get the data in:

0:[["children",["slug","technology/magazine-story","oc"],[["slug","technology/magazine-story","oc"],{"children":["",{}]}],"$L1",["$L2","$L3"]]]
4:I{"id":"8041","name":"","chunks":["272:webpack-0731a6b8a4bac69e","701:701-3dedbc3784b9ded1"],"async":false}
5:I{"id":"26","name":"","chunks":["272:webpack-0731a6b8a4bac69e","701:701-3dedbc3784b9ded1"],"async":false}
1:["$","$L4",null,{"parallelRouterKey":"children","segmentPath":["children",["slug","technology/magazine-story","oc"],"children"],"loading":["$","div",null,{"className":"loading-screen","children":["$","h1",null,{"children":"Loading..."}]}],"loadingStyles":[],"hasLoading":true,"template":["$","$L5",null,{}],"childProp":{"current":["$L6",null,null,[["$","link","0",{"rel":"stylesheet","href":"/_next/static/css/9a131fd76d91f625.css","precedence":"next.js"}],["$","link","1",{"rel":"stylesheet","href":"/_next/static/css/1c50b25a063b5e8d.css","precedence":"next.js"}],["$","link","2",{"rel":"stylesheet","href":"/_next/static/css/667f892cb33f1a48.css","precedence":"next.js"}],["$","link","3",{"rel":"stylesheet","href":"/_next/static/css/f1080765345e3100.css","precedence":"next.js"}]]],"segment":""}}]
3:null

format, and it is the same when i do a back/forward navigation.

Expected Behavior

Instead of showing the raw data, page should load the corresponding HTML

Which browser are you using? (if relevant)

Chrome 112.0.5615.49

How are you deploying your application? (if relevant)

No response

[Fredkiss3]

Please try to provide a simple reproduction repo/stackblitz so that we can try to help.
You could use this as your base : https://stackblitz.com/fork/github/vercel/next.js/tree/canary/examples/reproduction-template-app-dir

[steve-marmalade]

For what it's worth, I've seen this issue as well but have not been able to create a simple reproduction, so I have refrained from opening an issue.

[Fredkiss3]

@steve-marmalade i may have an idea, i got a similar issue when i was trying to use cloudfare cache with next app router, in cloudfare if you setup the cache to cache everything, it will try to cache all the resources marked with cache-control headers.

Next ISR (SSG) by default adds a corresponding cache-control to make it cacheable from the CDN, the only problem is that with app router, the same request to the same route is used whether your page is navigated via <Link> or via browser refresh, i've noticed that the only difference between those request is that next adds a RSC: 1 header to fetch the RSC payload for link navigations.

The problem here is that cloudfare cache (with the setup cache everything) by default caches requests per route and ignore request headers, so if cloudfare caches your payload before your page html content, the next time you access the page it will load the RSC payload instead and if cloudfare caches your page html before your payload, link navigations will not work.

To fix that i just removed the cache everything option of cloudfare cache so that cloudfare only caches static assets.

It should be noted however that this will only happen if you use the app router, in next pages dir, next load the payload for link navigations with another request, something like this :

GET /_next/data/q-1di4c_JMDaO2WkUSANX/path-to-your-page/your-page.json

Cloudfare docs for cache rules : https://developers.cloudflare.com/cache/how-to/create-page-rules/#cache-everything

[timneutkens]

To clarify App Router adds a few headers to the fetch when navigating client-side to get the RSC payload instead of HTML. Next.js on the server sets the Vary header to ensure that browsers / CDNs can cache based on the headers passed to the fetch: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Vary

Going to close this issue as the Vary header is provided so this is a CDN configuration problem.

[nandap4790]

Thank you for the response, will check and come back

[Fredkiss3]

Ok, after an intense testing, i found out that cloudfare CDN does not respect the Vary header to properly cache different responses types.

to quote from Cloudfare docs :

vary — Cloudflare does not consider vary values in caching decisions. Nevertheless, vary values are respected when Vary for images is configured and when the vary header is vary: accept-encoding.

To fix this i used a cloudfare worker to modify the request to the origin server by adding the vary headers as query params so that cloudfare would consider a request with RSC header different from one without, this is the script from my worker :

export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);

    const varyKeys = [
      'RSC', 
      'Next-Router-State-Tree', 
      'Next-Router-Prefetch', 
      'Accept-Encoding',
    ]

    // append these headers to the searchParams in order to distinguish requests with them or not
    for(const key of varyKeys) {
      if (request.headers.has(key) ) {
        url.searchParams.set(key, request.headers.get(key))
      }
    }

    const rewrittenRequest = new Request(url, request);
    let cache = true

    // Do not cache routes starting with `/api`
    if(url.pathname.startsWith('/api')) {
      cache = false;
    }

    let response = await fetch(rewrittenRequest, {
      cf: {
        cacheEverything: cache,
        // only cache 2xx status codes
        cacheTtlByStatus: { 404: 1, "500-599": 0, "300-399": 1 }
      },
    });
    // Reconstruct the Response object to make its headers mutable.
    response = new Response(response.body, response);
    return response;
  },
};

And i also added a middleware to strip the RSC query params before passing it to the pages :

import { NextResponse } from 'next/server';
import { NextRequest } from 'next/server';

export function middleware(request: NextRequest) {
    const url = new URL(request.nextUrl);

    // Check for vary headers appended by cloudfare worker
    const varyKeys = [
        'RSC',
        'Next-Router-State-Tree',
        'Next-Router-Prefetch',
        'Accept-Encoding',
    ];

    let varyFound = false;

    // remove these headers from the URL
    for (const key of varyKeys) {
        if (url.searchParams.has(key)) {
            url.searchParams.delete(key);
            varyFound = true;
        }
    }

    if (varyFound) {
        return NextResponse.rewrite(url);
    }
    return NextResponse.next();
}

export const config = {
    // exclude api, images & static paths (_next/static & _next/image)
    matcher: [
        '/((?!api|_next/static|_next/image|favicon.ico|robots.txt|sitemap.xml).*)',
    ],
};

Warning

• This is only for cloudfare, if you are using another CDN, you can verify if the CDN respects the Vary header
• Pay attention to also purge the cache on cloudfare when you deploy a new version of your app

[github-actions]

This closed issue has been automatically locked because it had no new activity for a month. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.