Next.js API routes (and pages) should support reading files

[Timer]

Feature request

Is your feature request related to a problem? Please describe.

It's currently not possible to read files from API routes or pages.

Describe the solution you'd like

I want to be able to call fs.readFile with a __dirname path and have it "just work".

This should work in Development and Production mode.

Describe alternatives you've considered

This may need to integrate with @zeit/webpack-asset-relocator-loader in some capacity. This plugin handles these types of requires.

However, it's not a necessity. I'd be OK with something that only works with __dirname and __filename (no relative or cwd-based paths).

Additional context

Example:

// pages/api/test.js
import fs from 'fs'
import path from 'path'

export default (req, res) => {
  const fileContent = fs.readFileSync(
    path.join(__dirname, '..', '..', 'package.json'), 
    'utf8'
  )
  // ...
}

Note: I know you can cheat the above example ☝️ with require, but that's not the point. 😄

[amytych]

Just wanted to second that, trying to implement file uploading using API routes. I can get the file to upload but then need to be able to access it again to upload it to S3 bucket.

[ScottSmith95]

I second this! Also, being able to read directories is very important for my company's usage as we keep our data like team members and blog posts in a content directory so we're looking for a way to require all files in the directory.

[Timer]

The above PR will fix this! ☝️ 🙏

[marlonmarcello]

How about fs.writeFile is that possible? For example, create and save a JSON file based on a webhook that was posted on an /api/route

[huv1k]

Hey @marlonmarcello, this is going to be possible. Stay tuned 😊

[NicolasHz]

It's this already solved?

[huv1k]

Not yet, you can subscribe for #8334

[NicolasHz]

@huv1k Many thanks!

[BrunoBernardino]

Is there a way to help this move forward more quickly?

[bitjson]

Worth noting: if you're using TypeScript, you can already import a JSON file as a module directly (make sure resolveJsonModule is true in tsconfig.json). E.g.:

import myJson from '../../../some/path/my.json';

The shape of the JSON object is also automatically used as its type, so autocomplete is really nice.

[jkjustjoshing]

Workaround I'm using:

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

and in the location you need the path

import fs from 'fs'
import path from 'path'
import getConfig from 'next/config'
const { serverRuntimeConfig } = getConfig()

fs.readFile(path.join(serverRuntimeConfig.PROJECT_ROOT, './path/to/file.json'))

I know this doesn't solve the need to reference files with paths relative to the current file, but this solves my very related use case (reading image files from a /public/images folder).

[IanMitchell]

Saw in the PR this has changed a bit - any update on what the current plans are (or aren't)? Sounds like there are some strategies you don't want pursued, mind listing them + why so contributors can give this a shot?

[Svish]

@ijjk Hope there will be some good documentation on how to use this properly too 👍

[kachkaev]

@ijjk that’s great news! I have a couple possibly stupid questions, but it’d be great if you could clarify them for everyone 🙂

1. Does process.cwd() work both in client and server code, just like process.env.NEXT_PUBLIC_XYZ? I.e. is it a webpack replacement trick meaning that it stops working if I do const x = process; console.log(x.cwd())?

2. Do I need to worry about the actual working directory or will the value be ‘stable’ regardless of what real cwd is? Here is a somewhat artificial example to illustrate what I mean:

cd /path/to/project
cd node_modules/.bin
./next 
## Is process.cwd() in Next.js pages and API routes
##   /path/to/project
## or 
##   /path/to/project/node_modules/.bin
## ?

[rodrigoKulb]

Hi, this is being updated in the latest canary of Next.js and can be tested by enabling experimental: { nftTracing: true } in your next.config.js. Note: process.cwd() should be used instead of __dirname and will point to the current working directory of your Next.js app.

for me it worked

[ijjk]

@kachkaev this is specific to reading files inside server-side methods like API routes or getStaticProps/getServerSideProps. For client-side file loading new URL() should be used with webpack 5.

The process.cwd() value isn't modified during the build and will be set to where next start is run which should be the Next.js project's base directory.

[elrumordelaluz]

In case could be useful for someone to try, for me worked a combination of some comments, after lots of deploys.

// next.config.js
module.exports = {
  target: 'serverless',
  experimental: { nftTracing: true },
}

The folder with files called by the function inside /public, and invoked like:

resolve('./public/my-folder/my-file.txt')

[pgrodrigues]

After some tries, this works for me on next 11.1.3-canary.32.

Accessing template.mjml in email/ dir:

// next.config.js
module.exports = {
   experimental: { outputFileTracing: true }
}

// vercel.json
{
   "functions": {
      "pages/api/contact.js": {
         "includeFiles": "email/template.mjml"
      }
   }
}

// pages/api/contact.js
const { join, resolve } = require("path");
const { readFileSync } = require("fs");

export default async (req, res) => {
   const templateDirectory = resolve(process.cwd(), "email");
   const emailTemplate = readFileSync(join(templateDirectory, "template.mjml"), "utf8");

  // ...
};

[lazlothemonkey]

You can do this:

Install copy-webpack-plugin and copy desired files to serverless local filesystem. Then you can locate your files at path.join(process.cwd(), ".next/server/chunks")

next.config.js

const path = require("path")
const CopyPlugin = require("copy-webpack-plugin")

module.exports = {
    target: "serverless",
    future: {
        webpack5: true,
    },
    webpack: function (config, { dev, isServer }) {
        // Fixes npm packages that depend on `fs` module
        if (!isServer) {
            config.resolve.fallback.fs = false
        }
        // copy files you're interested in
        if (!dev) {
            config.plugins.push(
                new CopyPlugin({
                    patterns: [{ from: "content", to: "content" }],
                })
            )
        }

        return config
    },
}

utils.js

import path from "path"
import { promises as fs } from "fs"


export async function getPostBySlug(slug) {
    let basePath = process.cwd()
    if (process.env.NODE_ENV === "production") {
        basePath = path.join(process.cwd(), ".next/server/chunks")
    }

    const filePath = path.join(basePath, `content/${slug}.md`)
    const fileContent = await fs.readFile(filePath, "utf8")
    return fileContent

Thanks @sroussey . You saved me a lot of time.

Hello @tvavrys ,

I tried this approach. It works when I use npm run build locally but I when I deploy to vercel I keep getting this error:

2021-09-25T14:19:26.441Z	41890285-cb02-4c0d-9521-41cd89e9e6b6	ERROR	[Error: ENOENT: no such file or directory, open '/var/task/.next/serverless/chunks/textbooks/boledu-hls-textbook-main/markdown/1-introduction/README.md'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/var/task/.next/serverless/chunks/textbooks/boledu-hls-textbook-main/markdown/1-introduction/README.md'
}

I did change the base path to this:

if (process.env.NODE_ENV === "production") {
  basePath = path.join(process.cwd(), ".next/serverless/chunks")
}

so server to serverless (otherwise it doesn't work on npm run build)

and the next.config:

const path = require("path");
const CopyPlugin = require("copy-webpack-plugin");

module.exports = {
  target: "serverless",

  webpack5: true,

  webpack: function (config, { dev, isServer }) {
    // Fixes npm packages that depend on `fs` module
    if (!isServer) {
      config.resolve.fallback.fs = false;
    }
    // copy files you're interested in
    if (!dev) {
      config.plugins.push(
        new CopyPlugin({
          patterns: [{ from: "textbooks", to: "textbooks" }],
        })
      );
    }

    return config;
  },
};

Would you have any idea what could be breaking it? Thanks.

[masstapel]

@ijjk I'm using something similar in our project right now: During build time, we receive data from a CMS which we write in a JSON file to be read during build and run time. The directory structure is ./json/<locale_for_website>/<actual_JSON_file> (so located in root of project).

Everything seems to work during build and runtime both locally and deployed, except for revalidation: For some reason, process.cwd() returns a different value when revalidation of a ISR page is happening.

During revalidation when a new file needs to be generated, we first use fs.unlink to remove previous files and then we create and write the new one. In this step, process.cwd() returns /var/task/json as a path while during build time this returns /vercel/path0/json which is the actual location of the cached files when they got created during build time. Therefore, the content never gets revalidated correctly and revalidation fails.

Do you have a good suggestion for this use case? Many thanks!

[revskill10]

So, the question is, how to access current directory of a page ?
For example, i'm in the page /pages/[postId]/index.js, how to get this with process.cwd() ?

[sroussey]

That's a different issue, and really, the directory does not exist as the JS. Ode gets bundled into one file.

[Richard87]

Hi, I have the same error, with @grpc/client unable to load *.proto* files when target is serverless.

I created a basic repo to reproduce the error: https://github.com/richard87/test-grpc
Basically, running in dev "works" (get a error that it cant connect to the server), but running build && start, returning a autopilot.proto file not found error... check the /api/hello to test it out.

[bmadsen-ownrs]

I'm on Next.js 12 and I previously used @tvavrys's suggestion but this seems to be broken now. I have a serverless function that reads script files relative to the project's root and runs them when invoked from a Next.js page via http calls.

It seems that exporting a unstable_includeFiles config prop for the serverless api file works:

// in /pages/api/some-route.ts
export const config = {
  unstable_includeFiles: ['folder-relative-to-root'],
};

and then a path to the file is simply

const basePath = process.cwd();
const scriptsDir = path.join(basePath, 'folder-relative-to-root');

🤷 works but seems to be a temporary hack ?

[curly210102]

#8251 (comment)

It works with outputFileTracing in nextjs v12.0.2

// next.config.js
module.exports = {
   outputFileTracing: true
}

But, it doesn't work when file is "README.md", it's really weird. @ijjk

Code as below:

// vercel.json
{
   "functions": {
      "pages/api/contact.js": {
         "includeFiles": "extensions/**/README.md"
      }
   }
}

// pages/api/contact.js
import type { NextApiRequest, NextApiResponse } from "next";
import { resolve, join } from "path";
import { readFileSync } from "fs";

type Data = {
  content: string;
};

type Error = {
  error: string;
};

export default function handler(
  req: NextApiRequest,
  res: NextApiResponse<Data | Error>
) {
  if (req.method === "GET") {
    const { slugs } = req.query;
    const slug = typeof slugs === "string" ? slugs : slugs[0];

    const templateDirectory = resolve(process.cwd(), "extensions");
    const emailTemplate = readFileSync(
      join(templateDirectory, slug, "README.md"),
      "utf8"
    );
    return res.status(200).json({
      content: emailTemplate,
    });
  } else {
    return res.status(404).json({
      error: "Not Support Method",
    });
  }
}

• Reproduction Demo Repo: https://github.com/curly210102/next-js-issue
• Test URL: https://next-js-issue.vercel.app/api/contact?slugs=a
• Vercel Function Log:

[ijjk]

@curly210102 includeFiles in vercel.json isn't used by Next.js. It looks like the readFile call in the above snippet is using the wrong path it's reading from extensions/README.md but the readme is actually located at extensions/a in the repro https://github.com/curly210102/next-js-issue/tree/main/extensions/a

[curly210102]

@curly210102 includeFiles in vercel.json isn't used by Next.js. It looks like the readFile call in the above snippet is using the wrong path it's reading from extensions/README.md but the readme is actually located at extensions/a in the repro https://github.com/curly210102/next-js-issue/tree/main/extensions/a

@ijjk Oh sorry, the code snippet pasted in the above with a accidental omission, please take the repo as the prevailing.

To illustrate the specific behavior of "README.md", I added "index.md" demo as a comparison. And remove the vercel.json.

• Reproduction Demo Repo: https://github.com/curly210102/next-js-issue
• README.md Test URL: https://next-js-issue.vercel.app/api/contact?slugs=a
• index.md Test URL: https://next-js-issue.vercel.app/api/info?slugs=a

[QB3L]

Is there a solution for this? We need to be able to read some template files as well in an endpoint under pages/api

@curly210102 did you find anything that could resolve the issue in your example?

[mtimofiiv]

I have a slightly different permutation of the same problem. I reproduced it here:

https://github.com/mtimofiiv/vercel-fonts-demo

And it's deployed here: https://vercel-fonts-demo.vercel.app

Readme file in the repo has all the things I tried – but basically I followed a bunch of different instructions from this thread plus also this one:

lovell/sharp#2499

[lonniev]

When the fs.readFileSync ENOENT` (file not found) issue occurs, it may happen for primarily two reasons here:

1. The path to the sought file is incorrect
2. The file truly is absent at the path as provided

For (1), this may be due to relocation of the files in the process of deployment from development locations to operations locations.

For (2), this can be due to pilot error (of course) or to incomplete Next.js configuration.

When I encountered this issue, I was trying to read from the file system a GraphQL schema file. Those files are lightly-stylized blends of JSON and Javascript/Typescript - but they use a .graphql suffix.

My local development build and run was fine but the schema.graphql file that is imported during configuration of the Apollo GraphQL Server was reported as ENOENT when the project was deployed to Vercel.

Apparently, the *.graphql file was not getting deployed to the server-less location because it uses an atypical suffix.

See https://nextjs.org/docs/api-reference/next.config.js/custom-page-extensions

Here’s the crucial sample advice:

Open next.config.js and add the pageExtensions config:

module.exports = {
  pageExtensions: ['mdx', 'md', 'jsx', 'js', 'tsx', 'ts'],
}

Note: The default value of pageExtensions is ['tsx', 'ts', 'jsx', 'js']. 

Also be advised that when we modify the set of file extensions to be considered, we have to restate the default extensions - or else the files which were deployed will then get excluded - leaving a broken mess.

(now to test this in my repo. If this is all true, I can remove some workaround stuff that might have worked but seems to have broken Dynamic Routing.)

Update: this may help some people - but it isn't the right change for me. If I add qraphql as a page extension, then next.js attempts to parse the file, on its own, as legitimate code.

Update: I gave up on attempting to get Next.js and Vercel to be able to deploy the non-code file to the pages/api area and to run the server less functions with next. For now, I simply store the content of the graphql.schema file as a string in the Javascript file that was trying to load the file.

This was an educational diversion into the details of Next.js and Vercel but isn't the major goal of what I am prototyping.

I'll monitor for updates to this issue and refactor the bandaid area later.

[mdstroebel]

So I have had this same issue and I recently realised that I can just use Webpack Copy Plugin to enforce copying the required files to the bundle (within the .next folder for NextJS).

However, on deployment to Vercel, it is not adding/hosting those files when I check the Source -> Output UI in Vercel. See screenshot of that UI without my custom templates folder (which is bundled correctly within the .next folder locally when I run npm run build).