upgrade nextjs

[williamstein]

We are currently at next version 13.5.6, which works fine. This is the version right before the 14.x series.

I tried to update to 14.0.3, which as of today just involved updating the next page in packages/next and packages/hub, and no other package updates.

It fails to render anything in dev mode with

***
 ⨯ /projects/6b851643-360e-435e-b87e-f9a6ab64a8b1/cocalc/src/packages/node_modules/.pnpm/antd@5.11.3_react-dom@18.2.0_react@18.2.0/node_modules/antd/es/layout/index.js:3
import InternalLayout, { Content, Footer, Header } from './layout';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at internalCompileFunction (node:internal/vm:73:18)
    at wrapSafe (node:internal/modules/cjs/loader:1178:20)
    at Module._compile (node:internal/modules/cjs/loader:1220:27)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at mod.require (/projects/6b851643-360e-435e-b87e-f9a6ab64a8b1/cocalc/src/packages/node_modules/.pnpm/next@14.0.3_@babel+core@7.21.4_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/require-hook.js:64:28)
    at require (node:internal/modules/cjs/helpers:121:18)
    at antd/es/layout (/projects/6b851643-360e-435e-b87e-f9a6ab64a8b1/cocalc/src/packages/next/.next/server/pages/404.js:3801:18)
    at __webpack_require__ (/projects/6b851643-360e-435e-b87e-f9a6ab64a8b1/cocalc/src/packages/next/.next/server/webpack-runtime.js:33:43)
    at eval (webpack-internal:///./components/landing/footer.tsx:7:72)
    at ./components/landing/footer.tsx (/projects/6b851643-360e-435e-b87e-f9a6ab64a8b1/cocalc/src/packages/next/.next/server/pages/404.js:1951:1)
    at __webpack_require__ (/projects/6b851643-360e-435e-b87e-f9a6ab64a8b1/cocalc/src/packages/next/.next/server/webpack-runtime.js:33:43)
    at eval (webpack-internal:///./pages/404.tsx:13:83) {
  page: '/404'
}

I spent hours trying to debug this and got nowhere.

Obviously the problem is some antd compatibility issue.

I tried moving _document.tsx

packages/next$ cd pages/
~/cocalc/src/packages/next/pages$ mv _document.tsx _document0.tsx 
~/cocalc/src/packages/next/pages$ 

but that made no difference.

If you try

~/cocalc/src/packages/next$ pnpm exec next dev

which just directly runs next without the hub or that complexity, the problem is the same.

There are finally official directions to "try to use nextjs with antd" here -- https://ant.design/docs/react/use-with-next -- I had to figure out what we do in pages/_document.tsx before that. I tried those directions and it seemed to make no difference.

Maybe the problem has something to do with changes to the complicated babel configuration of nextjs. https://nextjs.org/docs/pages/building-your-application/configuring/babel

I tried to switch to using babble instead of SWC by creating this:

module.exports = function (api) {
  api.cache(true);

  const presets = ["next/babel", { "preset-typescript": {} }];
  const plugins = [];

  return {
    presets,
    plugins,
  };
};

but try as I may, I got a lot of typescript errors, because clearly the typescript isn't being parsed.

Disabling "swcMinify: true, // enable faster RUST-based minifier" doesn't help in next.config.js.

My guess is that people using and developing nextjs 14 have moved on to ESM modules, and that's that core reason for this problem appearing. We are still using commonjs to interface between our different modules, e.g., the hub imports the nextjs app. That's probably why this doesn't work. Obviously it would be nice to switch to ESM modules everywhere in cocalc, but that's likely a miserable herculean task.

[williamstein]

Update: I reverted the upgrade to 13.5.6 since that might have caused weird problems. I don't know.

[haraldschilly]

I was just investigating our next.js version situation. Here it says 13.5.6, but actually we're lower. I found d2ceac6 explaining that.

We should update that 13.4 line to 13.4.20-canary.13 or higher. I saw 13.4.20-canary.40 is the most recent one. I can try that, but I wonder what was broken in 13.5 or what to even look for. … or probably 13.4.19. I don't know why there is no proper 13.4.20 release.

[williamstein]

I'm taking this over because we shouldn't wait weeks on this and it's important.

[williamstein]

(also it's going to be too painful for @schrodingersket who would have to chase all over to realize, e.g., that nextjs is also in the hub, etc.)

[williamstein]

Everything works fine on kucalc and the next page, but some of the proxying does NOT work in dev mode. I'll dive in and try to fix this.

[williamstein]

Actually, I just fully tested things and dev mode does work fine. There's a harmless error in the log about websocket upgrading happening twice, but it all works. I have a cocalc-docker dev VM and that is what is not working, so maybe it's an issue just with cocalc-docker, or maybe I just need to be more careful in my build. Will investigate further. In any case:

• dev mode works fine with newest nextjs
• cocalc.com (i.e., kucalc) works fine with newest nextjs
• cocalc-docker: investigating

[williamstein]

OK, I fixed all the fallout caused by this, as far as I can tell. Again, it was subtle and not easy for me at least, but the fixes aren't hacks and I think make sense.

Now we are working with the latest nextjs version, which provides important security updates and should make development a little faster and more robust.