Duplicate transactions of newBlockHeaders subscription

[Pogooo]

Description

With the client I subscribe to the newBlockHeaders event, which works correctly, but sometimes
I get duplicate error for the transaction hash, when it tries to save to the database. Also when I check the transactions which went wrong those are contract call transactions (I don't know it's a coincidence or not), but mostly no difference between the old inserted transaction data in my db and the result of etherscan.io.

Expected behavior

Every transaction of the new blocks can be saved to the database without duplicate error.

Actual behavior

Sometimes appears hash duplication at insertion.

Steps to reproduce the behavior

this.web3.eth.subscribe("newBlockHeaders", null, async (error: Error, blockHeader: BlockHeader) => {
    if(error) {
        console.log("\t[ETH] Error at new block headers event", error);
    } else {
        const currentBlock = await this.web3.eth.getBlock(blockHeader.hash, true);
        // console.log("transaction docs", currentBlock.transactions);
        try {
            this.storage.NormalTransaction.create(currentBlock.transactions);
        } catch(e) {
            console.log("the transaction is in the database already, but in the new block we've got once more", e);
        }
    }
});

Error Logs

{ MongoError: E11000 duplicate key error collection: ethereum.normaltransactions index: hash_1 dup key: { : "0x159938f7729d90d3488685def0bb0c43354055082bf54140085ee6b05e7981fe" }
at Function.create (.../node_modules/mongodb-core/lib/error.js:43:12)
at toError (.../node_modules/mongodb/lib/utils.js:149:22)
at coll.s.topology.insert (.../node_modules/mongodb/lib/operations/collection_ops.js:859:39)
at .../node_modules/mongodb-core/lib/connection/pool.js:532:18
at process._tickCallback (internal/process/next_tick.js:61:11)
driver: true,
name: 'MongoError',
index: 0,
code: 11000,
errmsg:
'E11000 duplicate key error collection: ethereum.normaltransactions index: hash_1 dup key: { : "0x159938f7729d90d3488685def0bb0c43354055082bf54140085ee6b05e7981fe" }',
[Symbol(mongoErrorContextSymbol)]: {} }
unhandledRejection MongoError: E11000 duplicate key error collection: ethereum.normaltransactions index: hash_1 dup key: { : "0x159938f7729d90d3488685def0bb0c43354055082bf54140085ee6b05e7981fe" }

Versions

• web3.js: 1.0.18
• nodejs: 10.11
• ethereum node: geth

[nivida]

As described here can this happen because of a chain re-organization.

Fix:

const blockNumbers = [];
this.web3.eth.subscribe("newBlockHeaders", null, async (error: Error, blockHeader: BlockHeader) => {
    if (error) {
        console.log("\t[ETH] Error at new block headers event", error);

        return;
    } 

    const currentBlock = await this.web3.eth.getBlock(blockHeader.hash, true);
        
    try {
        if (!blockNumbers.includes(currentBlock.number)) {
          this.storage.NormalTransaction.create(currentBlock.transactions);
          blockNumbers.push(currentBlock.number);
        }
    } catch(e) {
        console.log("the transaction is in the database already, but in the new block we've got once more", e);
    }
});

[nivida]

@Pogooo Btw.: Feel free to add a new subscription type called newBlock with extending of the NewHeadsSubscription class. You would have to do the above steps in the onNewSubscriptionItem method of the newly created NewBlockSubscription class.

[Pogooo]

Thank you a lot for your advice! It seems better than I excepted if it notifies also when reorganization happens, because I must not recheck the transactions on the blockchain when I list from the DB. But instead of the code above I should remove all transactions from the DB when duplication happens in that block and save the transactions again, is it correct? (because I see I get notification about every block that reorganized).

[nivida]

I should remove all transactions from the DB when duplication happens in that block and save the transactions again, is it correct?

Yes, this is correct.

[Pogooo]

Unfortunately I still get duplicate key errors. First I only tried to delete the blocks, that I get again from the event and insert the new transactions, but I received duplications continuously. So I delete also all transactions that should be inserted by hash, but after a while I also get conflicts. So it means I don't get event from any block changes on the chain in practice? Because in this code the error only can come from when I get new block which contains transactions that I already saved once at a previous block.

this.web3.eth.subscribe("newBlockHeaders", null, async (error: Error, blockHeader: BlockHeader) => {
    if(error) {
        console.log("\t[ETH] Error at new block headers event", error);
        return;
    }

    const currentBlock = await this.web3.eth.getBlock(blockHeader.hash, true);

    if (!this.syncedBlockNumbers.includes(currentBlock.number)) {
        this.syncedBlockNumbers.push(currentBlock.number);
        this.storage.NormalTransaction.create(currentBlock.transactions);
    } else { //reorganization
        const duplicateTxIds: Array<string> = [];
        currentBlock.transactions.forEach((transaction) => {
            duplicateTxIds.push(transaction.hash);
        })
        await this.storage.NormalTransaction.deleteMany({hash: { "$in": duplicateTxIds }}); //delete every transactions which the reorganized block has.
        await this.storage.NormalTransaction.deleteMany({blockNumber: blockHeader.number}); //we delete the older version transactions of the reorganized block.
        this.storage.NormalTransaction.create(currentBlock.transactions);
    }
});

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions