[⚡ Feature]: Mocking process.env.<BINDING> when running next dev

[tom-huntington]

Wrangler mocks D1 locally with from this file .wrangler/state/..../db.sqlite

So I was thinking it would be possible polymorphically get the binding to query that file when developing using next dev.

That way you could get the same code i.e.

const { results } = await process.env.<BINDING_NAME>.prepare(
        "SELECT * FROM Customers WHERE CompanyName = ?"
      )
      .bind("Bs Beverages")
      .all();

to produce the same result regardless whether you are developing from wrangler or next.

Or rather use a getter

function getDB(): D1Database {
  if (process?.env?.DB) return process.env.DB
  // get D1Database for next dev
  return ...
}

Are you interested in providing an out of the box solution for this?

Or maybe just developing with using Turso then switching to D1 would be easier

[james-elicx]

Heya Tom, funnily enough, I've actually been working on a proxy for bindings this past couple of days that is intended to ease this problem.

It's a utility function that you use that tries to go through a dev server, which you would run in conjunction (--dev CLI flag), when you're in development mode with next dev, and falls back to process.env otherwise.

So far, I have tests passing for what I believe is full functionality with KV and D1, and partial functionality with R2.

PR coming soon 🙂 See https://github.com/james-elicx/cf-bindings-proxy

[tom-huntington]

Sounds good!

[dario-piotrowicz]

@tom-huntington if you want you can keep the issue open so that we can track progress here 😄

[dario-piotrowicz]

This is sort of in progress by @james-elicx, however I am not sure if we are going to go with this as there are conflicting opinions internally as to wether we should leave the current dev server experience as is and improve wrangler instead 😓

[tom-huntington]

conflicting opinions internally as to wether we should leave the current dev server experience as is and improve wrangler instead 😓

nah, I need hot code reloading

The problem is that npx @cloudflare/next-on-pages --watch takes 35+ seconds to rebuild every time a change is made.

Maybe try to unblock this

[dario-piotrowicz]

@tom-huntington yes I understand, you're right, it's too cumbersome right now 😓 I'll try to see what we can do 😢

[vassbence]

This would be awesome!

Currently I have a monorepo in which there is a separate Next.js application and a JSON REST API running on Workers. With a monorepo setup I can start both the dev servers locally and everything works. With the unstable_dev() API I can even run e2e tests on the backend and all my bindings (KV, Queue, D1) work flawlessly.

The downsides of this setup are the following:

• Hassle to deploy, Next.js is deployed to Vercel and the Worker is deployed to CF.
• When using server-side only getServerSideProps and communicating with the API there is an extra HTTP fetch (browser -> vercel -> cloudflare worker)

It would be really awesome if this package could bridge the gap and provide a great experience and all capabilites of workers on next dev too.

[carlreid]

Also watching this closely, and agree with all the points outlined by others. Was wondering though if anyone has a some way to work around this issue currently with using next dev/hot reload?

My only idea was to split things, so that the API parts over in Workers which are run with wrangler. Least until this is resolved and can move things into API routes for Next.js

[vassbence]

I have the following setup:

• Turborepo monorepo
• /apps/api -> run of the mill cf worker setup with wrangler
• /apps/web -> next.js app

The only thing you have to do to make things smooth is to add a rewrite into your next.config.js file:

async rewrites() {
  return [
    {
      source: "/api/:path*",
      destination: `${process.env.NEXT_PUBLIC_BACKEND_API_URL}/:path*`,
    },
  ];
},

where NEXT_PUBLIC_BACKEND_API_URL is your worker's URL, e.g. http://localhost:8787

Just make sure to only use these rewrites on dev/preview environments as they can add quite a few miliseconds to response times.

[AlecKriebel]

+1 to this! Bindings in Next.js apps would be huge.

I chose Cloudflare workers over Vercel & others because I believe Cloudflare has a more complete suite of developer services (Queues, Storage, KV store, etc.) so actually getting to use those in my primary Next.js app would be absolutely incredible.

[james-elicx]

Hey folks,

Following up on my previous message, I've just open-sourced the experimental proxy for bindings that I started writing a while back 🙂. I would note that it is rather experimental, but there are unit tests in place for KV and D1 functionality, and partial R2 functionality.

It works by deconstructing the function calls and sending them to an instance of Wrangler that is running the proxy. In the proxy, it reconstructs the deconstructed function calls and executes them, returning the response back to the client. It defaults to using process.env in production builds. It does of course require running the proxy in a separate terminal window to the Next.js dev server.

You can read more in the repo - https://github.com/james-elicx/cf-bindings-proxy

If you do try it out, please do share your experience with it, or any problems you run into, over on that repo. Contributions are also welcome.

[Rei-x]

Hello, if someone wants a starting repository with working local development with D1 and KV (thanks to cf-bindings-proxy), then here you go https://github.com/Rei-x/next-on-cloudflare

[JanThiel]

I absolutely have to add my +1 to this one.
Took us two days to figure out that it's not actually us being incompetent while trying to get CF Pages with NextJS and D1 to run locally with wrangler. The DB binding kept being undefined...

So I gave @james-elicx code a try - integrating it using the help of @Rei-x template. That's now the devXP we expected initially :-). The code works locally in wrangler as well as on pages with the remote D1.

Thanks for the amazing work.

You should add a note here that the bindings currently do not work locally out of the box though:
https://developers.cloudflare.com/pages/framework-guides/deploy-a-nextjs-site/#use-bindings-in-your-nextjs-application

[dario-piotrowicz]

Hi all 🙂

Sorry it's been taking so long, anyways I've been working on an implementation that looks promising and that the team is happy to move forward with.

My solution from a user point of view just involves the following.

In your next.config.js you need to import and run a function in which you declare the bindings that you want to have access to in dev mode, something along the lines of:

// next.config.js
/** @type {import('next').NextConfig} */
const nextConfig = {}

module.exports = nextConfig

if(process.env.NODE_ENV === 'development') {
  const { setupDevBindings } = require('@cloudflare/next-on-pages/next-dev');

  setupDevBindings({
     r2Buckets: ['MY_R2'],
  });
}

(the process.env.NODE_ENV check is totally optional but recommended)

Then you run next dev as you normally would and get your bindings as you currently do in process.env (e.g. process.env.MY_R2), and that's basically it 🙂

This has zero impact on the production build.

It anyways just provides the bindings for next dev, it does not build the actual next-on-pages worker, so it's still highly advised to run wrangler pages dev to check your application before deploying it. But this would at least make the feedback loop during standard development much smoother (this solution is not unlike @james-elicx in this regard, in both the real source of truth is always wrangler pages dev).

My implementation is being introduced in #486 if anyone would have time and would like to give it a try and provide some feedback that would be highly appreciated 🙂, the PR is close to being done I think, I just need to polish the code a bit and add documentation so hopefully things should not change too much.

Note
In the PR description you'll find some minimal examples that you can clone to try out this implementation if you want 🙂

[lyc8503]

Is it possible to also implement a local version of mock? (e.g. mock KV storage with local filesystem, mock D1 with local sqlite)

So that we can develop without internet or deploy without cloudflare.

[dario-piotrowicz]

Hi @lyc8503 🙂👋 , I am not sure what you mean with "local version of mock" here, as everything with my implementation (and also wrangler pages dev) runs locally (and it should not require an internet connection).

Besides that, are you asking if you could provide an implementation for the KV, D1, etc... APIs so that you could used them in an application that you deploy somewhere else rather then Cloudflare?
If you are, these are very Cloudflare specific APIs and I can't really see much benefit in using them anywhere besides on the Cloudflare network, besides that this project is very Cloudflare specific and the output that the next-on-pages CLI generates is built appositely for Cloudflare Pages, so here as well I can't really imagine any use case of next-on-pages that doesn't involve deploying to Cloudflare.
Maybe I am misunderstanding things please let me know if I got the above wrong, if you could provide any use case you have in mind that would be helpful to clarify things 🙂

[lyc8503]

Hi @lyc8503 🙂👋 , I am not sure what you mean with "local version of mock" here, as everything with my implementation (and also wrangler pages dev) runs locally (and it should not require an internet connection).

Besides that, are you asking if you could provide an implementation for the KV, D1, etc... APIs so that you could used them in an application that you deploy somewhere else rather then Cloudflare? If you are, these are very Cloudflare specific APIs and I can't really see much benefit in using them anywhere besides on the Cloudflare network, besides that this project is very Cloudflare specific and the output that the next-on-pages CLI generates is built appositely for Cloudflare Pages, so here as well I can't really imagine any use case of next-on-pages that doesn't involve deploying to Cloudflare. Maybe I am misunderstanding things please let me know if I got the above wrong, if you could provide any use case you have in mind that would be helpful to clarify things 🙂

Thanks for your detailed reply! I tried to investigate more and found I may have misunderstood this project. I am not very familiar with this, sorry for that.

For example, I am now developing a Next.js project and I want this project can be deployed both locally (on my own Linux servers) AND serverlessly (on Cloudflare Workers).
One solution is that I could write two different logics to store data, one for the local builds using Redis or some other KV storages and the other one for the Cloudflare Workers using KV, but this requires more code & test.

So if there could be a local version of mapping from KV storage API to local redis or filesystem, those NextJs projects using KV / D1 can run locally.

But this may not be the target for this project / the correct way to do this. Maybe a local deployment of workerd is what I need?

[dario-piotrowicz]

@lyc8503 no problem, thanks for checking out this project in any case 😄

Anyways thanks for clarifying your use case, that is indeed quite outside the scope of this project and something quite particular that I haven't heard before (I assume you have your reasons from wanting to do that?), so it's not something we can really facilitate 😓

Regarding your use case I can see a few alternatives that you could try:

• having two different storing logic as you mentioned, not too ideal but if you make them both adhere to the same interface you could just need to implement the read/write operations and have everything on top abstracted, making it really not that bad in my opinion
• introduce a new storage solution that you deploy in one place (like your linux servers) and connect to via http, so that your Next.js application just always interacts with that and works regardless on when you run it
• trying to use workerd to locally run the next-on-pages application as you mentioned (but I would really avoid that as it sounds like the most complex solution of them all)

[lyc8503]

@lyc8503 no problem, thanks for checking out this project in any case 😄

Anyways thanks for clarifying your use case, that is indeed quite outside the scope of this project and something quite particular that I haven't heard before (I assume you have your reasons from wanting to do that?), so it's not something we can really facilitate 😓

Regarding your use case I can see a few alternatives that you could try:

• having two different storing logic as you mentioned, not too ideal but if you make them both adhere to the same interface you could just need to implement the read/write operations and have everything on top abstracted, making it really not that bad in my opinion
• introduce a new storage solution that you deploy in one place (like your linux servers) and connect to via http, so that your Next.js application just always interacts with that and works regardless on when you run it
• trying to use workerd to locally run the next-on-pages application as you mentioned (but I would really avoid that as it sounds like the most complex solution of them all)

Thanks for your advice!

I have no specific reason for that and I am just exploring the possibilities. After all, more choices on platform should not be a bad thing.

I used to write something for vercel and it just uses Redis for storage. The NextJs project has good and seamless support for both local and vercel deployment, but it seems that the situation is a little bit different here for Cloudflare as it's "edge" runtime
has more restrictions and uses its own APIs. But that's ok as I also benefits from running on edge, I'll just try alternatives above 🙂

[dario-piotrowicz]

@lyc8503 I see, I didn't know Vercel that was so simple with Vercel, it's definitely different with Cloudflare as I think that self hosting various things would sort of defeat their whole purpose/nullify their biggest strengths (as for example replication and caching over the Cloudflare network) 🤔

Sorry it's not as straightforward to do that here, and thanks for the understanding 🙂, I hope you can find success with some of the above 😃

[dario-piotrowicz]

The PR has been merged, so that changes are now in the latest beta release 🙂

The API might slightly change before we do a proper release though

[dario-piotrowicz]

Let's reopen this until we finalize the API (#527) and make this stable

[merill]

@dario-piotrowicz ❤️ this amazing PR!!!

This makes the dev experience so much better. Thank you! 🙏

[dario-piotrowicz]

Thanks @merill I'm glad you like it 😄

Sorry it's been marked as experimental for a bit, but we should just finalize the API and make it stable soon 🙂

[gcascio]

@dario-piotrowicz thanks for the great work in improving the DX!

While initially everything worked smoothly, I encountered an issue while using NextAuth V5 which I believe is related to following patch:

next-on-pages/internal-packages/next-dev/src/index.ts

Line 191 in 0a2008e

runtimeContext['Response'] = Response;

I created a minimal reproduction here. The auth flow works when setupDevBindings is commented out but as soon as setupDevBindings is used the following error occurs:

TypeError: "response" must be an instance of `Response`

The origin of this error lies in the oauth4webapi package which is used by NextAuth and does the following check in multiple places:

if (!(response instanceof Response)) {
  throw new TypeError('"response" must be an instance of Response')
}

I assume there is a mismatch between the patched and unpatched Response somewhere. I am confused though as why this happens since my understanding is that the patched Response should be in the global scope and therefore used by all participating code of the auth flow.

Maybe this is too much of an edge case but I thought I bring it up in case someone else encounters this.

[dario-piotrowicz]

@gcascio thanks a bunch for trying next-dev I'm glad you're enjoying it 😄

And thanks a lot for raising the issue, I did have a look at your minimal reproduction (by the way, awesome minimal reproduction with very clear instructions, I really appreciate it! ❤️)

I have a suspicion that I know what's going wrong there, as you surmised we do patch Response in the global scope, but this only for the edge runtime, I suspect that here we might have a situation in which nodejs runtime code (like that of a middleware) generates the response and such is passed on to the edge runtime code which is erroring (so basically the mismatch happens because the response is generated in one runtime and passed to the other, at least that's my suspicion/best guess right now).

It might be or not an edge case, I'd say that it'd be worth to investigate further and I hope that we can do something about it, could you please open a new issue for it so that we can discuss and handle it there? 🙏

(PS: in the terminal I also see an immutable error thrown because of Headers.delete that doesn't sound promising 😓)

[gcascio]

@dario-piotrowicz thanks for the rapid response and the insights!
I opened the new issue as requested: #614

[davorinrusevljan]

I have existing next.js project and I am able to get it running by publishing it into git. Now I trying to get my D1 bindings working locally.

So first I do not have next.config.js, I guess I should do changes inside next.config.mjs?

[davorinrusevljan]

I have existing next.js project and I am able to get it running by publishing it into git. Now I trying to get my D1 bindings working locally.

So first I do not have next.config.js, I guess I should do changes inside next.config.mjs?

ok, I have removed next.config.mjs, and instead added next.config.js as described.

Now, how would I get that my db is connected to the same preview db as when running on cloud flare worker in the edge? Or if I have to work with local db is it persistent and how can I fill it with necessary data?

Same for KV

thanks!

[dario-piotrowicz]

Hi @davorinrusevljan sorry for the late reply 🙂

You can use a next.config.mjs file without issues, in the following way for example:

import { setupDevBindings } from '@cloudflare/next-on-pages/next-dev';

/** @type {import('next').NextConfig} */
const nextConfig = {};

if (process.env.NODE_ENV === 'development') {
  await setupDevBindings({
    bindings: {
        MY_KV: {
          type: 'kv',
          id: 'xxx',
        },
        MY_D1: {
          type: 'd1',
          databaseName: '???'
        },
    }
  });
}

export default nextConfig;

Warning

Note that the databaseName is actually the databaseId, that's a bug, from the next release (#639) you can use actually specific a databaseId instead

Regarding the data, the db you get is a local only one, there is currently no way to get access to the remote/edge one

The db (as well as the other bindings) does have persistence, the data getting saved in .wrangler/state/v3/**, which is the same location wrangler v3 used.

Regarding populating the db (as well as the other bindings) I'm sorry but currently we don't have a clear solution for that (that's something that we've thought about and that we might address in the future). But since the data persistence location is the same as wrangler you can implement workers that populates the data or use wrangler commands (like for example wrangler d1 execute as presented here: https://developers.cloudflare.com/d1/how-to/importing-data/)

I hope this helps 🙂

[davorinrusevljan]

@dario-piotrowicz it surely helps, thank you!

I guess I made dumb decision to learn 2 new things (pages & next.js) at the same time, which rarely goes smooth especially if you are not sharpest pencil like me. But it is permeating slowly.

So if I am running locally with next, I need to enter binding information into the next.config.js, and if I am running locally with wrangler I have to put it into the wrangler command line or .toml.

One last question if I can abuse your kindness: do ids that I put into the local bindings have whatsoever relation to the ids that I enter into the pages website? Or I just poke some values from the blue sky, and then stick to them?

I guess question with database content is being used locally will grow in importance with time - with all next.js rendering options and versions, it is getting kind of important which data gets used when since it is getting more and more in prerender phase.

[seivad]

Hey guys I cannot get a fetch() call to work with my bindings, it's a range of errors but typically it is "cannot parse url from object object". I have an image resizing worker and the binding is setup as "resizer" but when I do the following it does not work. I'm not 100% sure how to make the right request object as this is being called from within a server action process the images.

I tried to use the direct RPC method but it complains that the payload is over 1mb and fails, images can be many mb's so was trying the fetch() but I feel like NextJS is overriding it or doing something odd to the env.BINDING.fetch() call.

const { env } = getRequestContext()
// Not sure how to put my image, width, height, format variables into this...
// I have them in a FormData object that but that didn't work either when passed in as the variable
const response = env.RESIZER.fetch(new Request('/'))
const image = await response.json()