RFC: Crawler-Aware ISR Fallback

[steven-tey]

Update: This has shipped! https://nextjs.org/blog/next-12#bot-aware-isr-fallback

---

Goals

• Allow web crawlers to server-render ISR pages using fallback: true, while still serving the previous behavior of the fallback state to non-crawler User-Agents.

Background

Currently, ISR with fallback: true renders a fallback state before rendering the page contents. To block the page from loading (server-rendering), you need to use fallback: 'blocking'.

A current tradeoff of using fallback: true is SEO-dependent pages render a skeleton loader for both users and web crawlers. As a result, users have to resort to using fallback: 'blocking', which might not be ideal since it's always server-rendered.

Proposal

Distinguish between a user and web crawler request for fallback: true:

• User: Serve fallback state (loading skeleton), followed by loading content client-side
• Web Crawler: Block page loading (server-rendering) and load content directly

Client-Side Transitions

A related issue to this RFC is client-side transitions. For example, if a page uses fallback: true and is linked internally with next/link, the transition still behaves like fallback: "blocking". We should allow users to customize this behavior.

[nandorojo]

Super excited for this.

Would it work for iMessage and Twitter previews too?

A key part of blocking the fallback for me is ensuring that URL previews work when people share them. But when users are on the app, I want to show the skeleton.

[steven-tey]

Yes! We are planning to use this Crawler Regex as inspiration when we build out this behavior, and it includes both Twitterbot and Applebot – the latter of which is responsible for website previews on iMessage!

[sjelfull]

@steven-tey Sounds great. You might want to look at this open-source list for inspiration to see how it compares to the Crawler Regex you linked. The former list has 2x the amount of records and is regularly updated.

[steven-tey]

Beautiful, thanks for sharing that, it looks awesome! Will share this with the team!

[steven-tey]

Update: This feature has been implemented and is now available on canary – feel free to try it out!

[lucasbasquerotto]

Hi, is there a way to know in getStaticProps if a request came from a crawler or not?

Use case: we don't want to load server side when the user is not a crawler (never, it should only return skeletons and load client-side, but it should always load server-side for crawlers), and because this feature is Crawler Aware, I assume you somehow know if a request came from a crawler or not, so it should be possible to access it in getStaticProps, or not?

[DB-Alex]

Is it possible to extend the user agents. We want for example semrush also to load server side. But this is not included in the regex.