Occasional error when reading session data from file system

[midgleyc]

What version of Remix are you using?

1.6.5

Steps to Reproduce

1. Use file session storage (createFileSessionStorage)
2. Create a large amount of session data (>1MB), saved and read each page load.
3. Visit lots of pages quickly.

Expected Behavior

No errors.

Actual Behavior

Occasionally, the page fails to load:

SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at readData (/srv/app/node_modules/@remix-run/node/dist/sessions/fileStorage.js:91:28)
    at getSession (/srv/app/node_modules/@remix-run/server-runtime/dist/sessions.js:113:25)
    at loader10 (/srv/app/build/index.js:2229:17)
    at Object.callRouteLoader (/srv/app/node_modules/@remix-run/server-runtime/dist/data.js:77:14)
    at handleDataRequest (/srv/app/node_modules/@remix-run/server-runtime/dist/server.js:113:18)
    at requestHandler (/srv/app/node_modules/@remix-run/server-runtime/dist/server.js:34:18)
    at /srv/app/node_modules/@remix-run/express/dist/server.js:39:22

[kiliman]

The file session storage implementation doesn't work well since loaders can run in parallel. There is no file-locking on the current one so it's very possible to corrupt session data.

[kiliman]

I've got a working version with file locks, but still trying to figure out performance issues.

[raskyer]

Hello, do you know if any progress could be made around this issue ?

[dusty]

I just made one to run synchronously and it doesn't corrupt - we only use this in local dev - we use redis in prod.

// https://remix.run/docs/en/v1/api/remix#createsessionstorage
import type { Cookie } from '@remix-run/node'
import { createSessionStorage } from '@remix-run/node'
import crypto from 'crypto'
import fs from 'fs'
import path from 'path'

interface FileSessionStorageOptions {
  cookie: Cookie
  dir: string
}

// sync version so the file isn't corrupted on concurrent writes
export function createFileSessionStorage({ cookie, dir }: FileSessionStorageOptions) {
  return createSessionStorage({
    cookie,
    async createData(data, expires) {
      const content = JSON.stringify({ data, expires })
      // eslint-disable-next-line no-constant-condition
      while (true) {
        const id = crypto.randomUUID({ disableEntropyCache: true })
        try {
          const file = getFile(dir, id)
          fs.mkdirSync(path.dirname(file), { recursive: true })
          fs.writeFileSync(file, content, { encoding: 'utf-8', flag: 'wx' })
          return id
        } catch (error) {
          if (error.code !== 'EEXIST') throw error
        }
      }
    },
    async readData(id) {
      try {
        const file = getFile(dir, id)
        const content = JSON.parse(fs.readFileSync(file, 'utf-8'))
        if (!isExpired(content.expires)) return content.data
        fs.unlinkSync(file)
        return null
      } catch (error) {
        if (error.code !== 'ENOENT') throw error
        return null
      }
    },
    async updateData(id, data, expires) {
      const content = JSON.stringify({ data, expires })
      const file = getFile(dir, id)
      fs.mkdirSync(path.dirname(file), { recursive: true })
      fs.writeFileSync(file, content, 'utf-8')
    },
    async deleteData(id) {
      try {
        fs.unlinkSync(getFile(dir, id))
      } catch (error) {
        if (error.code !== 'ENOENT') throw error
      }
    },
  })
}

// Divide the session id up into a directory (first 2 bytes) and filename
// (remaining 6 bytes) to reduce the chance of having very large directories,
// which should speed up file access. This is a maximum of 2^16 directories,
// each with 2^48 files.
function getFile(dir: string, id: string) {
  return path.join(dir, id.slice(0, 4), id.slice(4))
}

function isExpired(date?: string) {
  const expires = typeof date === 'string' ? new Date(date) : null
  return expires ? expires < new Date() : false
}

[akomm]

Oddly I get this error only when my FF dev tools are open. I assumed some side-effect of opening it. It is though not the disabled cache, I've tested it.

Using the 2.3.1 vite plugin remix version.

[devnomic]

I think i hit same issue, this may need to be documented in the docs.
Can you share @kiliman ?

[kiliman]

@devnomic I would probably use the solution above by @dusty. I don't use file sessions, so I never got around to fixing the performance issues on my spike.