WebSockets

[zubricks]

WebSockets are a powerful feature that we plan to bring to Payload. With WebSockets you’ll be able to subscribe to when documents are created, updated, or deleted over the same connection.

This means you’d be able to build things like data driven dashboards, notification systems, and more.

[matthijs166]

Duplicate of #2534

Would love this feature!

[willviles]

Considering the move to Next.js and the likeliness many people will be deploying Payload in the same app as their frontend on Vercel (or other serverless environments), how would this work?

Would a WebSocket service be a standalone feature of Payload Cloud, perhaps?

[matthijs166]

https://youtu.be/zQdcAYmkvGs?t=731

[jnovak-SM2Dev]

Is there any timing on this? Just trying to plan for a project I have coming up in the next couple months that will need this feature.

[bormm]

@zubricks WebSockets are not a feature as described here, it's a protocol.
Therefor I wonder if the description is about supporting GraphQL subscriptions and of course support websockets as transport protocol for that. Otherwise, I could imagine that you want to reinvent the wheel and build some other notification or subscription mechanism in parallel to GraphQL that also uses WebSockets, but I don't think that's a good idea.

So please clarify, as its essential from my perspective that such a discussion uses the correct terms especially if a feature is flagged with priority 1.

[cgilly2fast]

Is there a branch that has been start for this feature? Looking to contribute to this feature

[carlosbensant]

PayloadCMS and FeathersJS are my go-to Node.js frameworks, the last for Real-time Apps. I'd love to see how this will be implemented and the API you'll provide to interact with events and subscriptions.

[ccssmnn]

I'm having a good time using socket.io with payload. You can use Payloads local API in the socket.io handlers and the socket.io instance in Payload CMS hooks. With socket.io client libraries in your frontend, you can do whatever you want. With adapters, you can connect socket.io to your DB to scale this approach to multiple server instances, leveraging MongoDB change streams.

I wonder if that gets more complicated when moving to Payload CMS v3.

In my case, my client app needs to refresh its data on changes. So I emit poke events after relevant change or delete events. This was very straightforward, since socket.io is served from the same server as I can access the auth cookie. Here is the server-side setup.

// example for hooks, that send poke events into organization rooms
import { extractId } from "./utils";
import { io } from "../../socket-io";
import {
  CollectionAfterChangeHook,
  CollectionAfterDeleteHook,
} from "payload/types";

export const sendOrganizationPokeAfterChange: CollectionAfterChangeHook =
  async ({ doc }) => {
    if (!doc.organization) return;
    const orgID = extractId(doc.organization);
    io.to(orgID).emit("poke");
  };

export const sendOrganizationPokeAfterDelete: CollectionAfterDeleteHook =
  async ({ doc }) => {
    if (!doc.organization) return;
    const orgID = extractId(doc.organization);
    io.to(orgID).emit("poke");
  };

// setting up socket.io
import { Server as HTTPServer } from "node:http";
import payload from "payload";
import { Socket, Server as SocketIOServer } from "socket.io";
import { z } from "zod";
import { extractId } from "./utils";
import { createAdapter } from "@socket.io/mongo-adapter";
import { MongoClient } from "mongodb";

export let io: SocketIOServer;

export async function initSocketIO(server: HTTPServer) {
  io = new SocketIOServer(server, {
    /* options */
  });

  io.on("connection", hanleOnConnection);

  await initMongoDBConnection(io);

  return io;
}

/**
 * when the request comes from a logged in user,
 * add this socket into every organization room that
 * this user is member of
 */
async function hanleOnConnection(socket: Socket) {
  const { user } = await checkAuthCookie(socket.request.headers.cookie);
  if (!user) {
    socket.disconnect(true);
    return;
  }
  const organizationIDs = user?.organizations?.map((v) =>
    extractId(v.organization),
  );
  for (const orgID of organizationIDs ?? []) {
    socket.join(orgID);
  }
}

// this is where socket.io can leverage mongodb to support messaging accross multiple server instances
async function initMongoDBConnection(io: SocketIOServer) {
  const mongoClient = new MongoClient(process.env.DATABASE_URI as string);
  await mongoClient.connect();
  const mongoCollection = mongoClient
    .db()
    .collection("socket.io-adapter-events");
  await mongoCollection.createIndex(
    { createdAt: 1 },
    { expireAfterSeconds: 3600, background: true },
  );
  io.adapter(
    createAdapter(mongoCollection as any, {
      addCreatedAtField: true,
    }),
  );
}

// when starting your server, attach socket.io to the http server
import { createServer } from "node:http";

import "dotenv/config";

import { initSocketIO } from "./socket-io";
import { initPayload } from "./payload";
import { initExpress } from "./express";

const main = async () => {
  const app = initExpress();
  const httpServer = createServer(app);
  await initPayload(app);
  initSocketIO(httpServer);
  httpServer.listen(3000);
};

main();

[brankodencic]

This looks as a very nice solution. DId you try making a version of it for 3.0?

[ccssmnn]

I will migrate my project as soon as 3.0 is stable and share my solution

[cbratschi]

Next.js has no WebSocket support. There are a couple of tickets open and discussions. Next.js upgrades the HTTP connection itself and does not support a custom handling there.

I hope they will add WebSocket support to routes. Would be great to use export WS = message => ...

[pianomansam]

Next.js has no WebSocket support. There are a couple of tickets open and discussions. Next.js upgrades the HTTP connection itself and does not support a custom handling there.

I hope they will add WebSocket support to routes. Would be great to use export WS = message => ...

The built-in server for Next.js does not, but a custom server allows you to. That is how trpc supports websockets w/ Next.

[cbratschi]

Next.js has no WebSocket support. There are a couple of tickets open and discussions. Next.js upgrades the HTTP connection itself and does not support a custom handling there.
I hope they will add WebSocket support to routes. Would be great to use export WS = message => ...

The built-in server for Next.js does not, but a custom server allows you to. That is how trpc supports websockets w/ Next.

Well, using a custom server is the current Payload 2.x setup to integrate it with Next.js but Payload 3.x is based on Next.js and a better solution for Websockets is needed without having to use custom servers.

Here is the Next.js discussion about WebSockets:

vercel/next.js#58698

Here is the example from TRPC for reference:

https://github.com/trpc/examples-next-prisma-websockets-starter/blob/main/src/server/prodServer.ts

[JeffreyArts]

@ccssmnn Have you already tried to upgrade your implementation to 3.0?

[ccssmnn]

i didn't see a satisfying way to create a socket.io instance, use nextjs with a custom server and make the socket.io instance available where i need it. you could deploy socket.io on the same nodejs server as your nextjs, but you need to treat it like an external server. nextjs code is bundled and there is no intended way of passing references to it.

currently migrating away from payload for my project 💔

[JeffreyArts]

I came across this article, which seems to be a solution, but I haven't tested it myself yet. https://medium.com/@kumarsehdev600/real-time-web-applications-with-next-js-and-socket-io-629706516597

I hope to find time for it in the next two weeks to give it a shot myself, if found. I'll let you know how it went.

[carlosbensant]

@ccssmnn What about using an external service like Pusher or Abby?

[ccssmnn]

@carlosbensant of course you can do that. My use case was relying on PayloadCMS as an application backend. I liked the idea of having a nodejs server where I can plug payload, socketio, as well as other frameworks like remix to have a reasonable setup.
It should be easily deployable on any setup, so clients can host it in their networks if they need to. Socket.io does even scale to a nodejs cluster. Websockets are not an option when deploying to vercel though.

Now that payload is so integrated with nextjs makes it easier to deploy wherever nextjs is deployable but cuts of my usecase.

[HarleySalas]

I exclusively host Next/Payload in standalone. In any case, the sort of thing I'd hope to see is some sort of adapter pattern, where realtime is optional and such adapter pattern should enable us to handle realtime in a similar way that storage providers do. We should be able to spin up a simple Hono instance, hook up party kit, or even things like Soketi, which can also be self-hosted.

Are there any ideas for a perfect scenario that fits inside of NextJS itself? I don't think so. I've already used Payload 3, with the Local API inside of a Hono application that uses the local api. It felt a little hacky, but perhaps a bit more time spent on it could yield a better result. It worked though. I've yet to experiment with things like Soketi, or Partykit, but I think it seems reasonable.

Most importantly, I just hope that realtime doesn't lock anyone in to having to host Nextjs a certain way, or that it doesn't force us to use any one particular provider.