-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot read property 'handleCommandComplete' of null #1872
Comments
Either way, it does make sense to implement these guards 👍 |
FWIW, I regularly see a very similar-looking crash when running pg 7.17.1 with an AWS Aurora Serverless database:
I haven't got a waterproof way to reproduce it, but the app rarely runs for more than 5 or 10 minutes without crashing. The same app runs without issues on a non-Aurora postgres. |
This feels like a reproducer but maybe I'm missing something obvious. If the query's rowCount is close to the stream's highWaterMark, and the async iterator breaks early, and the client is reused, then the type error is thrown. Not every time. More often when the highWaterMark is close to the result set row count. Workaround is not reusing a client after it's used with a QueryStream. Versions used: const pg = require('pg');
const QueryStream = require('pg-query-stream');
async function main() {
// result set has rowCount == highWaterMark
const queryText = 'SELECT * FROM generate_series(0, 100) numb';
// default highWaterMark is 100
const stream = new QueryStream(queryText, [] );
const client = new pg.Client();
await client.connect();
const query = client.query(stream);
for await (const row of query) {
if (row.numb > 50)
break;
}
const now = await client.query('select now()');
console.log( now.rows );
const ce = await client.end();
}
main(); |
I think the issue there might be not closing the |
I had the same idea and added |
I am not sure if this helps reproducing the issue, but I am hitting these with a connection that is run through WireGuard. What I generally have seen with the WireGuarded connections is that there are cases where the connection breaks and a few packets get dropped. These did not exist when I ran it over SSH tunnel - dunno if it has something to do that WireGuard uses UDP while I believe SSH runs TCP. It seems that for me, it appears together with Probably even with SSH the connection drops, but that signal probably arrives a lot later, and hence the pg client has had enough time to prepare to recover (or something similar). |
I have a reliable way of reproducing this in my app. Although I haven't managed to capture it in a small script yet. My app runs the following queries:
It seems that the way node-pg works is that once a query is completed, Now, this issue is happening because I run 3 commands, but 4 Basically this is what happens:
The problem is in step 6. CommandComplete for Im still investigating to understand what's happening, but it seems that the buffer is not correctly emptied after step 4. I still haven't managed to capture this in a small script, which means that my application is triggering it. Maybe my app is doing something wrong or just triggering a corner case. @brianc let me know if im on the right path here or if there's any way i can help moving this conversation forward. |
Just to add a little bit more clues just in case anyone is following, this is the content of Hex: UTF-8: |
Alright, I think I figured it out, at least in my case. This script can reproduce the issue easily: const pg = require('pg')
var domain = require('domain')
var d = domain.create()
d.on('error', function(err) {
console.error(err);
})
const pool = new pg.Pool({
connectionString: process.argv[2]
})
d.run(() => {
pool.connect((err, conn) => {
conn.query('START TRANSACTION', () => {
conn.query('ROLLBACK')
throw 'foo'
})
})
})
The problem is here pg-protocol will call the handler back as soon as it receives a message. Before clearing it's internal buffer. Now, if you have error handlers (like a domain handler in this sample script), that means the server will continue interacting with this connection in next ticks. But since the internal queue of node-pg is not cleared out, the next time you query, pg-protocol's internal buffer still holds the previous buffer which was never cleared. And that means your callback will be called for the first query again. @brianc A simple solution would be to call callback using a |
as for me the problem is still here... pg 8.7.3, Postgres 14.2, Node.js 16.10 |
Edit: not relevant, sorry |
@matthieusieben no, I reverted pg to 7.4.1, as somebody adviced in another place, works ok for now |
I've hit the following error coming from node-postgres 7.9.0 (latest) which crashes the node process
I'm using a connection pool via
new pg.Pool(config)
but as far as I'm aware no other modules that would interact with node-postgres internals at allI haven't got a reproduction as I've only seen it coupled with a read replica that is falling behind the primary and causing
Client has encountered a connection error and is not queryable
andterminating connection due to conflict with recovery
errors to be emitted and would expect that to be what's causing theactiveQuery
to be nulled. A simple fix would be to add a guard to https://github.com/brianc/node-postgres/blob/v7.9.0/lib/client.js#L293-L295 but I'm not sure if you'd accept that as a patch and if/where you would want to emit it as an event instead - if you could offer guidance on how you'd like to see this solved then I'm very happy to send a PRThe text was updated successfully, but these errors were encountered: