draft: Cloudflare works ! 🎉 #618

wackfx · 2023-06-23T23:23:44Z

Hey there !

Amazing work with your lib. It's so awesome that I was really sad to see that it didn't work on cloudflare 😢

I've spent some time to make it work on Cloudflare. Tested on my own project - it's working for both Cloudflare pages workers and regular workers.

This PR is not ready to merge - but since I've spent hours trying to deploy this library on Cloudflare based on your work - I guess you may want to take some pointers to what's missing from your implementation to successfully deploy it.

Here's what changed:

Rebased your current work on master
polyfill: Crypto.subtle.digest(MD5) didn't work as md5 is not supported anymore. Added Joseph's Myers implementation to follow your pattern of 'no external libs'
polyfill: Added dynamic import of cloudflare:socket to please bundlers
transpile: Replacing net.Socket() with async net.Socket() to allow for dynamic import on CF code
polyfill: os fs polyfills that doesn't respect the original contract - but since we're on cloudflare there is not much more we can do
polyfill: process.hrtime
transpile: Buffer being undefined on cloudflare - added import on build:cf
transpile: added node: prefixes

How to use ?

Simply import postgres from 'postgres' and instantiate it like you would do in a regular node env.

I use this module with Qwik and the adapter cloudflare-pages.
You'll need to tell bundler (through vite.config.ts on Qwik) to resolve workerd and set the node library (but compatible with cf workers) as external like this:

That's it. Feel free to take whatever you want and close this PR, I won't take it personally, I know it's pretty hacky ;)
Thanks again for your initial work.

porsager · 2023-06-24T04:04:37Z

Oh my- very very cool, but did you not see #599 ? 😬

wackfx · 2023-06-24T08:44:18Z

Oh yes, I saw that, the cloudflare branch was actually the base work for this (your commits are in my branch!), but it didn't work when deployed on Cloudflare 😞

porsager · 2023-06-24T09:50:39Z

Did you use node-compat? The CloudFlare workers team tested that branch with success

porsager · 2023-06-24T17:50:08Z

Ok, back at computer. I should have read your entire description, I just didn't had time when I answered ;) I'll look closer later or tomorrow.

porsager · 2023-06-24T17:50:54Z

Even so, still curious if you tried with node-compat?

wackfx · 2023-06-25T18:13:01Z

Hey, sorry I missed the notification !

Yes, I did try it to run it with wranger and node_compat, with a simple JS test like this:

import postgres from './cf/src/index'
const DB_URI = "..." // URI
const sql = postgres(DB_URI)
export function onRequest(context) {
  console.log(await sql`SELECT * FROM users`)
  return new Response("Hello, world!")
}

Unfortunately, tried on both Railway and Supabase, it didn't work. It blocked at the authentication.

I did solve Railway by patching the md5 function (because the authentication sent by the server was a standard unprotected one)

Later on, I solved Supabase by patching the sha256 method because Supabase asked for a SASL authentication.

Main blocking issue I've found was because of data types: digest would wait for a Uint8Array (from decode/encode) and I was giving it a string - which made the whole thing freeze. Also, the connection.js logic required to get back an ArrayBuffer (and avoid any other type) to be functional.

Also, Cloudflare complained once I tried to deploy that some globals or modules was missing (because polyfill wouldn't add them). You really have to try to deploy something to make sure it works: sometimes it worked perfectly locally, but then Cloudflare would complain at deploy time.

I've my project rn running on Cloudflare Pages (with some workers functions) - I can give you the link if you want 😄

EDIT: Also my configuration: I'm using Qwik, which uses Vite / Rollup under the hood to pack the worker script, so some modifications I've made are only to please those bundlers, like the dynamic import of cloudflare:sockets in net.Socket() which Vite wouldn't let go unless it was dynamically imported.

EDIT 2: I believe this comment had the same issues as I had. I'm building for Cloudflare pages which looks more restrictive than regular workers.
In order to debug I did this:

Link the local package with npm link
Go to my Qwik project (which will use pages) and npm link postgres there to have my local code used
Build the project with bun run build
Test the bundled codewith bunx wrangler pages dev ./dist --compatibility-date=2023-06-20 --log-level=debug --compatibility-flag=nodejs_compat

porsager · 2023-06-25T18:55:52Z

Very cool you're getting these last things in order. I wanted to merge mine quite some time ago, but haven't had the time (luckily).

Mind rebasing your changes on top of my cloudflare branch to get a better view of what changed? (just rebased my cloudflare branch on master now)

Also wrt. md5, I'm not gonna include that part, but instead just not support md5 auth on cf then.

wackfx · 2023-06-26T05:37:40Z

Hey !

I've rebased on top of your cloudflare branch - sorry about the formatting issues.

Let me know if I can be of any more help 🙂

Thank you

porsager · 2023-06-26T07:31:13Z

I don't know why it still complains. Mind resolving the conflicts, and removing the md5 implementation?

superboss224 · 2023-06-26T14:54:47Z

Hello, I've tested today's commits on the cloudflare branch, and it's not enough for Cloudflare Pages

I may be enough for Cloudflare Workers, which has the node_compat flag that polyfills some native node modules, but it doesn't work for the new, up to date, nodejs_compat flag that is the only one usable with Cloudflare Pages (for full stack projects)

I'm still getting this error log :

16:33:51.546 | > Using @sveltejs/adapter-auto
16:33:51.852 | ✘ [ERROR] Could not resolve "os"
16:33:51.852 |  
16:33:51.853 | node_modules/postgres/src/index.js:1:15:
16:33:51.853 | 1 │ import os from 'os'
16:33:51.854 | ╵                ~~~~
16:33:51.854 |  
16:33:51.854 | The package "os" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
16:33:51.854 |  
16:33:51.854 | ✘ [ERROR] Could not resolve "fs"
16:33:51.855 |  
16:33:51.855 | node_modules/postgres/src/index.js:2:15:
16:33:51.855 | 2 │ import fs from 'fs'
16:33:51.855 | ╵                ~~~~
16:33:51.855 |  
16:33:51.855 | The package "fs" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
16:33:51.855 |  
16:33:51.856 | ✘ [ERROR] Could not resolve "stream"
16:33:51.858 |  
16:33:51.858 | node_modules/postgres/src/large.js:1:19:
16:33:51.858 | 1 │ import Stream from 'stream'
16:33:51.858 | ╵                    ~~~~~~~~
16:33:51.858 |  
16:33:51.858 | The package "stream" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
16:33:51.859 |  
16:33:51.865 | ✘ [ERROR] Could not resolve "net"
16:33:51.866 |  
16:33:51.866 | node_modules/postgres/src/connection.js:1:16:
16:33:51.866 | 1 │ import net from 'net'
16:33:51.866 | ╵                 ~~~~~
16:33:51.867 |  
16:33:51.867 | The package "net" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
16:33:51.867 |  
16:33:51.867 | ✘ [ERROR] Could not resolve "tls"
16:33:51.867 |  
16:33:51.867 | node_modules/postgres/src/connection.js:2:16:
16:33:51.867 | 2 │ import tls from 'tls'
16:33:51.867 | ╵                 ~~~~~
16:33:51.867 |  
16:33:51.868 | The package "tls" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
16:33:51.868 |  
16:33:51.868 | ✘ [ERROR] Could not resolve "crypto"
16:33:51.868 |  
16:33:51.869 | node_modules/postgres/src/connection.js:3:19:
16:33:51.869 | 3 │ import crypto from 'crypto'
16:33:51.869 | ╵                    ~~~~~~~~
16:33:51.869 |  
16:33:51.869 | The package "crypto" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
16:33:51.869 |  
16:33:51.869 | ✘ [ERROR] Could not resolve "stream"
16:33:51.869 |  
16:33:51.870 | node_modules/postgres/src/connection.js:4:19:
16:33:51.870 | 4 │ import Stream from 'stream'
16:33:51.870 | ╵                    ~~~~~~~~
16:33:51.870 |  
16:33:51.871 | The package "stream" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
16:33:51.871

So, we can't consider that Postgres.js is yet truly working with cloudflare

I think that this new, more restrictive nodejs_compat flag requires the module to have no nodejs native module imported (like os, fs, etc..) and that it will need further polyfilling/modifications to make it work

To test if it works, simply deploy a project that uses Postgres.js with Functions on pages, and use the nodejs_compat flag

Please let me know if you get a different result than mine on Cloudflare Pages, I'm really looking forward to using Postgres.js on it

porsager · 2023-06-26T14:56:28Z

Thanks a lot for testing @superboss224 .. I'll take a look tonight and try to deploy myself

wackfx · 2023-06-26T22:27:20Z

I believe @superboss224 had the same issues that I've fixed (looking at the import errors he gave). Definitely a disprecancy between Cloudflare Worker's and Page's workers.
Although depending on the database he uses, he may need some tinkering with the code.

wackfx · 2023-06-26T22:31:21Z

Sorry I'm not on my computer right now to fix the issues. May be able to take a look tomorrow. I've got this code running actually on Cloudflare Pages with Qwik (https://shwink.wackfx.com) so I do believe it's working.

superboss224 · 2023-06-27T12:27:07Z

I believe @superboss224 had the same issues that I've fixed (looking at the import errors he gave). Definitely a disprecancy between Cloudflare Worker's and Page's workers. Although depending on the database he uses, he may need some tinkering with the code.

Yes exactly !
Cloudflare Workers has a more powerful node_compat = true flag that also includes polyfills for native module, but that flag is getting more or less legacy, and I don't think that Cloudflare wants to continue supporting it actively in the future.

They prefer the new compatibility_flags = [ "nodejs_compat" ]

More info about the two different flags here :
https://developers.cloudflare.com/workers/wrangler/configuration/#add-polyfills-using-wrangler

The difference in the doc may seem subtle, but it is what is causing the discrepancy between Worker and Pages

(I'm not sure that the new nodejs_compat is necessary if everything can be polyfilled, but that new flag should work in Pages

porsager · 2023-06-27T12:39:14Z

Things are moving fast.. Deprecating things within months 😂 Well good thing we didn't even release yet so we can get over on the new thing 😬

superboss224 · 2023-06-28T01:36:41Z

Yep, giga fast, there's a recent switch to push to that truly "javascript edge runtime" that has nothing to do with node.js or native modules (on Vercel Edge, Cloudflare and other platforms)

But we can now use a real tcp connexion in those runtimes and it's awesome because they are closer to the end user and have almost no cold start (meanwhile Vercel Functions has 2-3 seconds cold start and an user will feel it)

And being able to use a DBs in those edge runtimes was the last piece missing to run entire Full Stacks apps on it and avoid downtimes/scaling issues/latency for good, so having Postgres.js working with it will be awesome !

porsager · 2023-07-02T21:28:08Z

@wackfx been looking at this, and was quite hard to diff with the formatting changes. I've updated my branch with the things I could glance was necessary from your changes. Do you see anything missing, or perhaps just want to try it out?

Also, it would be great to have sample projects for both workers and pages to do proper tests. Do you have some minimal setups to test with?

feat: reran transpile fix linter feat: final touches + test files squashed 2 commits fix: Polyfills bulk (to please linter) fix: Removed MD5 + put back SHA in the digest() squashed 5 commits fix: cloudflare workers deployment feat: fixed auth fix: encrypt not found in worker :( fix: postgres SASL fix: linting

wackfx

Added some PR comments

wackfx · 2023-07-04T22:16:51Z

cf/polyfills.js

-    `(?:${v6Seg}:){1}(?:(:${v6Seg}){0,4}:${v4Str}|(:${v6Seg}){1,6}|:)|` +
-    `(?::((?::${v6Seg}){0,5}:${v4Str}|(?::${v6Seg}){1,7}|:))` +
-    ')(%[0-9a-zA-Z-.:]{1,})?$'
+  `(?:${v6Seg}:){7}(?:${v6Seg}|:)|` +


Just some linting issues - I've went ahead and used eslint on this file but no kudos :(

wackfx · 2023-07-04T22:17:01Z

cf/polyfills.js

+        false,
+        ['deriveBits']
+      ),
+      keylen * 8,


Nothing changed here

wackfx · 2023-07-04T22:17:07Z

cf/polyfills.js

-        return type === 'sha256'
-          ? Crypto.subtle.digest('SHA-256', x)
-          : Crypto.subtle.digest('MD5', x)
+        if (type !== 'sha256')


Removed MD5 as you asked

wackfx · 2023-07-04T22:17:14Z

cf/polyfills.js

-        await Crypto.subtle.importKey('raw', key, { name: 'HMAC', hash: 'SHA-256' }, false, ['sign']),
-        textEncoder.encode(x)
-      ))
+    update: (x) => ({


Linting only

wackfx · 2023-07-04T22:17:32Z

cf/polyfills.js

    })
  })
 }

-export const process = {


added at the end of the file because of hrtime polyfill

wackfx · 2023-07-04T22:23:25Z

cf/test-pages.js

@@ -0,0 +1,22 @@
+// Add your database url and run this file with


This file gives some instructions to test the code on pages wrangler (notice the pages in the command)

wackfx · 2023-07-04T22:23:38Z

cf/test-worker.js

@@ -0,0 +1,15 @@
+// Add your database url and run this file with


Regular wrangler test

wackfx · 2023-07-04T22:23:46Z

transpile.cf.js

@@ -1,38 +1,47 @@
 import fs from 'fs'
 import path from 'path'

-const empty = x => fs.readdirSync(x).forEach(f => fs.unlinkSync(path.join(x, f)))
-    , ensureEmpty = x => !fs.existsSync(x) ? fs.mkdirSync(x) : empty(x)
+const empty = (x) =>


Are you sure you're using the .eslint from this repo?

wackfx · 2023-07-04T22:23:54Z

transpile.cf.js

    , root = 'cf'
    , src = path.join(root, 'src')

 ensureEmpty(src)

-fs.readdirSync('src').forEach(name =>
+fs.readdirSync('src').forEach((name) =>


wackfx · 2023-07-04T22:24:05Z

transpile.cf.js

-  const process = x.includes('process.')
-    ? 'import { process } from \'../polyfills.js\'\n'
-    : ''
+  const polyfills = [


Reworked transpile.cf.js to allow for a single ../polyfills.js import (eslint was not happy about the multiple imports)

cf/polyfills.js

superboss224 · 2023-08-12T20:22:57Z

Hello,
I use SvelteKit and looking again at the logs :

21:15:48.711 | > Using @sveltejs/adapter-auto
-- | --
21:15:49.005 | ✘ [ERROR] Could not resolve "os"
21:15:49.005 |  
21:15:49.005 | node_modules/postgres/src/index.js:1:15:
21:15:49.005 | 1 │ import os from 'os'

It seems like that it doesn't want to import from /cf/src folder, but rather directly from /src, so it can't use the polyfilled code at all, so it crashes

I tried to update my vite.config.js with export conditions :

import { sveltekit } from '@sveltejs/kit/vite';

/** @type {import('vite').UserConfig} */
const config = {
  resolve: {
    conditions: ['workerd', 'webworker', 'worker', 'browser', 'import', 'module', 'default']
  },
  plugins: [sveltekit()],
  server: {
    port: 3000
  }
}

export default config;

However, it is still doing the same thing when building on cloudflare, and not importing from /cf/src

Maybe it's a bug from SvelteKit's building process, because even in local, I didn't find any way of using /cf/src

Is there something I could do better about that ? or another way to "force import" from /cf/src without having to fork the package to a version that only have /cf/src ? I tried import postgres from 'postgres/cf/src/index.js' but it gives an error

Maybe there is a way to tell cloudflare or vite to choose /cf/src ?

It would be awesome if I could get some help to use the cloudflare version with SvelteKit,
Best regards

jahir9991 · 2023-08-23T14:00:54Z

it's not working for Cloudflare pages when using with sveltekit
any update on that plz?

JustMrMendez · 2023-08-23T16:12:30Z

Still having this issue on sveltekit, is there something i've to setup?

wackfx · 2023-08-23T17:48:51Z

@superboss224 @jahir9991 @JustMrMendez

According to your logs @superboss224, I believe you all have the same issues.
You need to use the workerd entrypoint, not the 'regular' node_modules/postgres/src/index.js

You may need to specify a resolve key in the vite configuration, like this:

resolve: {
      conditions: ['workerd', 'webworker', 'worker', 'browser', 'import', 'module', 'default'],
},

Please refer to Vite / Sveltekit documentation, this is part of my code (not using SvelteKit) that actually works on Cloudflare Pages.

JustMrMendez · 2023-08-23T17:56:30Z

@superboss224 @jahir9991 @JustMrMendez

According to your logs @superboss224, I believe you all have the same issues. You need to use the workerd entrypoint, not the 'regular' node_modules/postgres/src/index.js

You may need to specify a resolve key in the vite configuration, like this:
resolve: {
      conditions: ['workerd', 'webworker', 'worker', 'browser', 'import', 'module', 'default'],
},
Please refer to Vite / Sveltekit documentation, this is part of my code (not using SvelteKit) that actually works on Cloudflare Pages.

tried that too, did not worked:

export default defineConfig({
	plugins: [sveltekit()],
	resolve: {
		conditions: ['workerd', 'webworker', 'worker', 'browser', 'import', 'module', 'default']
	}
	// build: {
	// 	rollupOptions: {
	// 		external: ['cloudflare:sockets', 'cloudflare:events', 'node:stream', 'node:buffer']
	// 	}
	// }
});

wackfx · 2023-08-23T18:02:29Z

@JustMrMendez
i can see rollupOptions in your code, try with this:

    build: {
      rollupOptions: {
        external: [
          'cloudflare:sockets',
          'node:events', // not cloudflare
          'node:stream',
          'node:buffer'
        ]
   }

JustMrMendez · 2023-08-23T18:11:22Z

external: [
'cloudflare:sockets',
'node:events', // not cloudflare
'node:stream',
'node:buffer'
]

I also have the compatibility flags, wrangler.toml etc...

Same error, this are my logs:

> Using @sveltejs/adapter-cloudflare
--
14:10:03.207 | ✘ [ERROR] Could not resolve "os"
14:10:03.207 |  
14:10:03.208 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/index.js:1:15:
14:10:03.208 | 1 │ import os from 'os'
14:10:03.210 | ╵                ~~~~
14:10:03.211 |  
14:10:03.211 | The package "os" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
14:10:03.211 |  
14:10:03.214 | ✘ [ERROR] Could not resolve "fs"
14:10:03.215 |  
14:10:03.215 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/index.js:2:15:
14:10:03.215 | 2 │ import fs from 'fs'
14:10:03.216 | ╵                ~~~~
14:10:03.216 |  
14:10:03.216 | The package "fs" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
14:10:03.216 |  
14:10:03.218 | ✘ [ERROR] Could not resolve "stream"
14:10:03.218 |  
14:10:03.218 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/large.js:1:19:
14:10:03.218 | 1 │ import Stream from 'stream'
14:10:03.218 | ╵                    ~~~~~~~~
14:10:03.218 |  
14:10:03.219 | The package "stream" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
14:10:03.219 |  
14:10:03.235 | ✘ [ERROR] Could not resolve "net"
14:10:03.236 |  
14:10:03.236 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/connection.js:1:16:
14:10:03.236 | 1 │ import net from 'net'
14:10:03.236 | ╵                 ~~~~~
14:10:03.236 |  
14:10:03.237 | The package "net" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
14:10:03.237 |  
14:10:03.245 | ✘ [ERROR] Could not resolve "tls"
14:10:03.246 |  
14:10:03.246 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/connection.js:2:16:
14:10:03.246 | 2 │ import tls from 'tls'
14:10:03.246 | ╵                 ~~~~~
14:10:03.246 |  
14:10:03.247 | The package "tls" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
14:10:03.247 |  
14:10:03.247 | ✘ [ERROR] Could not resolve "crypto"
14:10:03.247 |  
14:10:03.247 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/connection.js:3:19:
14:10:03.247 | 3 │ import crypto from 'crypto'
14:10:03.247 | ╵                    ~~~~~~~~
14:10:03.247 |  
14:10:03.248 | The package "crypto" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
14:10:03.248 |  
14:10:03.248 | ✘ [ERROR] Could not resolve "stream"
14:10:03.248 |  
14:10:03.248 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/connection.js:4:19:
14:10:03.248 | 4 │ import Stream from 'stream'
14:10:03.248 | ╵                    ~~~~~~~~
14:10:03.249 |  
14:10:03.249 | The package "stream" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
14:10:03.249 |  
14:10:03.286 | error during build:
14:10:03.286 | Error: Build failed with 7 errors:
14:10:03.287 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/connection.js:1:16: ERROR: Could not resolve "net"
14:10:03.287 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/connection.js:2:16: ERROR: Could not resolve "tls"
14:10:03.287 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/connection.js:3:19: ERROR: Could not resolve "crypto"
14:10:03.287 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/connection.js:4:19: ERROR: Could not resolve "stream"
14:10:03.288 | node_modules/.pnpm/postgres@3.3.5/node_modules/postgres/src/index.js:1:15: ERROR: Could not resolve "os"
14:10:03.288 | ...
14:10:03.288 | at failureErrorWithLog (/opt/buildhome/repo/node_modules/.pnpm/esbuild@0.18.20/node_modules/esbuild/lib/main.js:1649:15)
14:10:03.288 | at /opt/buildhome/repo/node_modules/.pnpm/esbuild@0.18.20/node_modules/esbuild/lib/main.js:1058:25
14:10:03.288 | at /opt/buildhome/repo/node_modules/.pnpm/esbuild@0.18.20/node_modules/esbuild/lib/main.js:1003:52
14:10:03.288 | at buildResponseToResult (/opt/buildhome/repo/node_modules/.pnpm/esbuild@0.18.20/node_modules/esbuild/lib/main.js:1056:7)
14:10:03.289 | at /opt/buildhome/repo/node_modules/.pnpm/esbuild@0.18.20/node_modules/esbuild/lib/main.js:1085:16
14:10:03.289 | at responseCallbacks.<computed> (/opt/buildhome/repo/node_modules/.pnpm/esbuild@0.18.20/node_modules/esbuild/lib/main.js:703:9)
14:10:03.289 | at handleIncomingPacket (/opt/buildhome/repo/node_modules/.pnpm/esbuild@0.18.20/node_modules/esbuild/lib/main.js:762:9)
14:10:03.289 | at Socket.readFromStdout (/opt/buildhome/repo/node_modules/.pnpm/esbuild@0.18.20/node_modules/esbuild/lib/main.js:679:7)
14:10:03.289 | at Socket.emit (node:events:514:28)
14:10:03.289 | at addChunk (node:internal/streams/readable:343:12)
14:10:03.319 | ELIFECYCLE  Command failed with exit code 1.
14:10:03.359 | Failed: Error while executing user command. Exited with error code: 1
14:10:03.370 | Failed: build command exited with code: 1
14:10:09.844 | Failed: error occurred while running build command

wackfx · 2023-08-23T21:49:29Z

Interestingly enough, it seems that using npm install postgres doesn't actually install the version containing the cloudflare compatible code, ./cf folder is nowhere to be found (cc @porsager).

I have something that works with sveltekit / cloudflare / postgres

Option 1 - Request changes on official `adapter-cloudflare` plugin

The official plugin is missing some really important options to make this work.

Can't override external. This is important you may want to use node modules available on Pages with the node-compatibility flag. It's missing the node flags we require
Can't override conditions. Same thing, it doesn't cost that much to let advanced user the import conditions. As it is, it's missing the workerd condition that this project uses.

They basically already have an option object that is really under used. I don't think those changes would be hard for them to accept (those are just regular build options).

for @porsager: It looks like the official module uses worker instead of workerd entrypoint. Not sure what's the standard and which project could change here, but wanted to make you notice this difference that might cause user issues.

Option 2 - Use the custom adapter I made

Basically took the official adapter - and adapted it to be compatible with node / worker (source code).

Run npm install https://github.com/porsager/postgres to install Postgres from GitHub (and have the cf folder)
Run npm install -D https://github.com/wackfx/adapter-cloudflare-node to install my custom adapter
Change svelte.config.js to this

// change this line
// import adapter from '@sveltejs/adapter-cloudflare';

// with this line (only add -node at the end)
import adapter from '@sveltejs/adapter-cloudflare-node';

// It should work with the default kit configuration
/** @type {import('@sveltejs/kit').Config} */
const config = {
	kit: {
		adapter: adapter(),
	}
};

Please, if you have any issue with this custom adapter, post it on my repository. Not here

JustMrMendez · 2023-08-23T22:57:38Z

man, what a nightmare this has been:

 ERR_PNPM_PREPARE_PACKAGE  Failed to prepare git-hosted package fetched from
 "https://codeload.github.com/porsager/postgres/tar.gz/838c8daa89568e60161d6cee7d14c2ac26b696f1":
 postgres@3.3.5 prepublishOnly: `npm run lint`
spawn ENOENT

porsager · 2023-08-24T06:17:36Z

@JustMrMendez what did you expect playing with bleeding edge stuff? 😅 Your package manager (pnpm) shouldn't run the prepublishOnly script on install. It couldn't be more clearly named, but for some reason they might have thought that would be necessary when installing from github 🫠 I've moved on from pnpm myself because of annoyances like these, but if I recall correctly you can add --ignore-scripts to the install command to avoid any scripts running.

@wackfx I haven't published cf support to npm yet so that's why 😉

porsager · 2023-08-24T06:22:41Z

you may be didn't build the master as there is no cf folder when we install postgres from npmjs

cf support is not released in the npm published version yet so you have to install from porsager/postgres to try it out.

porsager · 2023-08-24T06:41:52Z

It looks like the official module uses worker instead of workerd entrypoint. Not sure what's the standard and which project could change here, but wanted to make you notice this difference that might cause user issues.

@wackfx I've asked cloudflare what the recommended usage is, so let's see what they say :)

JustMrMendez · 2023-08-24T14:53:33Z

@JustMrMendez what did you expect playing with bleeding edge stuff? 😅 Your package manager (pnpm) shouldn't run the prepublishOnly script on install. It couldn't be more clearly named, but for some reason they might have thought that would be necessary when installing from github 🫠 I've moved on from pnpm myself because of annoyances like these, but if I recall correctly you can add --ignore-scripts to the install command to avoid any scripts running.

@wackfx I haven't published cf support to npm yet so that's why 😉

It installed, but getting back the ERROR: Could not resolve "*", and yes, this is what you get for living in the edge, i was just frustrated, because i was getting errors/problems on non-related things, like pnpm

wackfx · 2023-08-24T15:49:44Z

@JustMrMendez we're all about edges here, like our beloved Cloudflare friends 💛

Is your problem fixed now at least ? Is it working? You forgot to tell us about this.
Sure I can understand the frustration, believe me when I was debugging this thing yesterday, it was a pain (and I don't even use SvelteKit).

But in the end, when it works (and if it works?) - don't you feel relieved ? 🙂

JustMrMendez · 2023-08-24T15:51:52Z

@JustMrMendez we're all about edges here, like our beloved Cloudflare friends 💛

Is your problem fixed now at least ? Is it working? You forgot to tell us about this.
Sure I can understand the frustration, believe me when I was debugging this thing yesterday, it was a pain (and I don't even use SvelteKit).

But in the end, when it works (and if it works?) - don't you feel relief ? 🙂

Nope, just removed postgres packages, am trying to make my setup work without it for the moment being, I'll update soon

porsager · 2023-08-25T10:54:03Z

Ok, I got word from Cloudflare it should be exports.worker instead of exports.workerd, so I''ve changed that now.. Let me know if that solves the issues..

jahir9991 · 2023-08-25T13:55:36Z

not sure if i am missing anything or not ?
using "@sveltejs/adapter-cloudflare-node": "https://github.com/wackfx/adapter-cloudflare-node",
"postgres": "https://github.com/porsager/postgres",

build failed in prod....
should i need to add anything in vite.config ?

Using @sveltejs/adapter-cloudflare
19:52:17.809 | ✘ [ERROR] Could not resolve "postgres"
19:52:17.809 |
19:52:17.809 | .svelte-kit/output/server/chunks/hooks.server.js:4:21:
19:52:17.809 | 4 │ import postgres from "postgres";
19:52:17.810 | ╵ ~~~~~~~~~~
19:52:17.810 |
19:52:17.810 | The module "./cf/src/index.js" was not found on the file system:
19:52:17.810 |
19:52:17.810 | node_modules/postgres/package.json:9:14:
19:52:17.810 | 9 │ "worker": "./cf/src/index.js",
19:52:17.811 | ╵ ~~~~~~~~~~~~~~~~~~~
19:52:17.811 |
19:52:17.811 | You can mark the path "postgres" as external to exclude it from the bundle, which will remove this error.
19:52:17.811

superboss224 · 2023-08-25T16:12:01Z

@porsager @wackfx
Hello, I've just tested with exports.worker and it now solves the issue, specifying resolve conditions isn't even necessary anymore, it works and now loads from /cf

However, there is new errors now :

18:00:34.662 | ✘ [ERROR] Could not resolve "node:events"
18:00:34.662 |  
18:00:34.663 | node_modules/postgres/cf/polyfills.js:1:29:
18:00:34.663 | 1 │ import { EventEmitter } from 'node:events'
18:00:34.663 | ╵                              ~~~~~~~~~~~~~
18:00:34.663 |  
18:00:34.663 | The package "node:events" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
18:00:34.663 |  
18:00:34.663 | ✘ [ERROR] Could not resolve "node:buffer"
18:00:34.664 |  
18:00:34.664 | node_modules/postgres/cf/polyfills.js:2:23:
18:00:34.664 | 2 │ import { Buffer } from 'node:buffer'
18:00:34.664 | ╵                        ~~~~~~~~~~~~~
18:00:34.664

In the vite.config.js of my SvelteKit project, i'm using:

  build: {
		rollupOptions: {
			external: ['cloudflare:sockets', 'cloudflare:events', 'node:stream', 'node:buffer']
		}
	},

but it still doesn't resolve the two modules with node:

Maybe there's a cloudflare flag to use on the project to load those two modules ? or something else you can do to solve the issue on your side ?

wackfx · 2023-08-25T18:06:03Z

@superboss224

The answer is in my previous message, you should look at the code I linked.

Can't override external. This is important you may want to use node modules available on Pages with the node-compatibility flag. It's missing the node flags we require

esbuild is called by the cloudflare adapter, which doesn't allow for any options. You're only left with either asking them to add the option, or use a custom adapter like I made.

wackfx · 2023-08-25T18:09:53Z

@jahir9991
I don't know how you installed the packages, but I believe it's an installation issue, my package should still work as it doesn't override the worker usage.

Did you ran the npm install I linked ?

superboss224 · 2023-08-25T20:48:33Z

@wackfx
Okay so, the flag exists, and node:buffer / node:stream should work on Cloudflare Pages but esbuild in the cloudflare adapter just don't let those node: modules work properly, so it nullifies the flag's effect, and I should ask Cloudflare to take their flag into account into their own adapter to fix the bug

Is that it?

wackfx · 2023-08-25T21:37:54Z

@superboss224

esbuild in the cloudflare adapter just don't let those node: modules work properly, so it nullifies the flag's effect, and I should ask Cloudflare to take their flag into account into their own adapter to fix the bug

Yes. You can have the flag on Cloudflare, if your build system locally doesn't allow node: modules because esbuild is configured to target non-node environments, it won't work.

EDIT: About your message, Cloudflare is not the issue here. The package @sveltekit/cloudflare-page is. You should ask the maintainer of the package to solve this (or temporarily use my package I linked previously to make this issue go away).

superboss224 · 2023-08-26T02:31:06Z

@wackfx
Okay so, it looks like it's already going to get fixed :
sveltejs/kit#10544

And he also already has his temporary driver to "pre-fix" that

So it should work seamlessly (if there's no other errors) when the pull request is done, all we have to do is wait a little more,
thank you for your help

karlhorky · 2023-09-28T15:39:02Z

ohh nice looks like WinterCG is doing some specification work on connect(), maybe it becomes a cross-runtime standard 😮

Switchable Runtime for Middleware (Allow Node.js APIs in Middleware) vercel/next.js#46722 (reply in thread)

@porsager what do you think about publishing a new Postgres.js version with the changes in this PR?

porsager force-pushed the cloudflare branch from 7c39e0c to e0b8aa5 Compare June 25, 2023 18:50

karlhorky mentioned this pull request Jun 26, 2023

Support for Cloudflare Workers TCP connect()? #598

Closed

wackfx force-pushed the cloudflare branch 2 times, most recently from 51a8eb8 to ca8fd16 Compare June 26, 2023 05:27

porsager force-pushed the cloudflare branch from e0b8aa5 to 4f1d670 Compare June 26, 2023 07:18

porsager changed the base branch from cloudflare to master June 26, 2023 07:22

porsager changed the base branch from master to cloudflare June 26, 2023 07:22

porsager changed the base branch from cloudflare to master July 2, 2023 21:28

porsager changed the base branch from master to cloudflare July 2, 2023 21:28

wackfx force-pushed the cloudflare branch from b5429b5 to 6414063 Compare July 4, 2023 22:15

fix: merge cleanup

4dd6874

wackfx commented Jul 4, 2023

View reviewed changes

mattbishop reviewed Jul 5, 2023

View reviewed changes

cf/polyfills.js Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

superboss224 mentioned this pull request Aug 26, 2023

Mark cloudflare:email as external for email routing api sveltejs/kit#10404

Merged

5 tasks

karlhorky mentioned this pull request Sep 28, 2023

Usage in Vercel edge functions #631

Closed

bllchmbrs mentioned this pull request Dec 31, 2023

Error: Cannot resolve "cloudflare:sockets" from "...\node_modules\.pnpm\postgres@3.4.0\node_modules\postgres\cf\polyfills.js" and externals are not allowed! #727

Closed

porsager mentioned this pull request Mar 23, 2024

chore: update export conditions #819

Merged

		@@ -0,0 +1,22 @@
		// Add your database url and run this file with

		@@ -0,0 +1,15 @@
		// Add your database url and run this file with

draft: Cloudflare works ! 🎉 #618

draft: Cloudflare works ! 🎉 #618

Conversation

wackfx commented Jun 23, 2023

Here's what changed:

How to use ?

porsager commented Jun 24, 2023

wackfx commented Jun 24, 2023

porsager commented Jun 24, 2023 via email • edited Loading

porsager commented Jun 24, 2023

porsager commented Jun 24, 2023

wackfx commented Jun 25, 2023 • edited Loading

porsager commented Jun 25, 2023 • edited Loading

wackfx commented Jun 26, 2023 • edited Loading

porsager commented Jun 26, 2023

superboss224 commented Jun 26, 2023

porsager commented Jun 26, 2023

wackfx commented Jun 26, 2023

wackfx commented Jun 26, 2023 • edited Loading

superboss224 commented Jun 27, 2023

porsager commented Jun 27, 2023

superboss224 commented Jun 28, 2023

porsager commented Jul 2, 2023 • edited Loading

wackfx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wackfx Jul 4, 2023 • edited Loading

Choose a reason for hiding this comment

superboss224 commented Aug 12, 2023

jahir9991 commented Aug 23, 2023 • edited Loading

JustMrMendez commented Aug 23, 2023

wackfx commented Aug 23, 2023

JustMrMendez commented Aug 23, 2023

wackfx commented Aug 23, 2023

JustMrMendez commented Aug 23, 2023 • edited Loading

wackfx commented Aug 23, 2023 • edited Loading

Option 1 - Request changes on official adapter-cloudflare plugin

Option 2 - Use the custom adapter I made

JustMrMendez commented Aug 23, 2023

porsager commented Aug 24, 2023

This comment was marked as resolved.

porsager commented Aug 24, 2023

porsager commented Aug 24, 2023

JustMrMendez commented Aug 24, 2023

wackfx commented Aug 24, 2023 • edited Loading

JustMrMendez commented Aug 24, 2023

porsager commented Aug 25, 2023

jahir9991 commented Aug 25, 2023

superboss224 commented Aug 25, 2023

wackfx commented Aug 25, 2023 • edited Loading

wackfx commented Aug 25, 2023

superboss224 commented Aug 25, 2023

wackfx commented Aug 25, 2023 • edited Loading

superboss224 commented Aug 26, 2023

karlhorky commented Sep 28, 2023

porsager commented Jun 24, 2023 via email •

edited

Loading

wackfx commented Jun 25, 2023 •

edited

Loading

porsager commented Jun 25, 2023 •

edited

Loading

wackfx commented Jun 26, 2023 •

edited

Loading

wackfx commented Jun 26, 2023 •

edited

Loading

porsager commented Jul 2, 2023 •

edited

Loading

wackfx Jul 4, 2023 •

edited

Loading

jahir9991 commented Aug 23, 2023 •

edited

Loading

JustMrMendez commented Aug 23, 2023 •

edited

Loading

wackfx commented Aug 23, 2023 •

edited

Loading

Option 1 - Request changes on official `adapter-cloudflare` plugin

wackfx commented Aug 24, 2023 •

edited

Loading

wackfx commented Aug 25, 2023 •

edited

Loading

wackfx commented Aug 25, 2023 •

edited

Loading