Race Condition With sendAndConfirmNonceTransaction

[0xIchigo]

I sometimes get an ephemeral error when using sendAndConfirmNonceTransaction that the nonce is invalid and has already been advanced despite the transaction being successful. This is the error in question:

SolanaError: The nonce `3myBceoXbBzN7cAY5kaymDuJTNpmd8tZPuhsfEPU8QDS` is no longer valid. It has advanced to `BMkGEQPkyJWXTwAKywcZB6DUaK2Z2zNy39bLJtLXJF6w`
    at E:\HeliusCoding\solana-web3.js2.0-experiments\node_modules\@solana\transaction-confirmation\src\confirmation-strategy-nonce.ts:72:27
    at processTicksAndRejections (node:internal/process/task_queues:105:5) {
  cause: undefined,
  context: {
    __code: 2,
    actualNonceValue: 'BMkGEQPkyJWXTwAKywcZB6DUaK2Z2zNy39bLJtLXJF6w',
    expectedNonceValue: '3myBceoXbBzN7cAY5kaymDuJTNpmd8tZPuhsfEPU8QDS'
  }
}

It seems that the notification handler is bugged. The notification of the nonce advancing arrives before the notification of the confirmed signature. Upon discovering that the nonce has advanced, the confirmation algorithm should determine if it advanced because the transaction landed or before the transaction landed.

I posted the issue on the Solana Stack Exchange with the full code snippet I was using to get this error

[steveluscher]

It looks like this is because the signature notifications are driven by the CompletedDataSetsService which runs in the solComplDataSet thread, and RpcSubscriptions::process_notifications() that runs in a series of solRpcNotifier threads. I don't think these notifiers are synchronized in any way, which implies that the confirmation algorithm sometimes hears about the nonce advancing before it hears about the transaction signature being confirmed, and vice versa.