mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position

[crebsy]

System information

Erigon version: ./erigon --version
2.59.3-088fd8ef

OS & Version: Windows/Linux/OSX
Ubuntu 22.04.3 LTS

Commit hash:
088fd8e

Erigon Command (with flags/config):

command: |
      --chain=bor-mainnet
      --datadir=/data
      --bor.heimdall=http://heimdall-rest:1317
      --http
      --http.addr=0.0.0.0
      --http.vhosts=any
      --http.api=eth,erigon,engine,web3,net,ots,debug
      --maxpeers=200
      --torrent.download.rate=300mb

Chain/Network:
polygon

Expected behaviour

Completes stage [5/15 Bodies] successfully

Actual behaviour

Crashes with the mdbx cursor error and starts over at the beginning of the stage:

err="[5/15 Bodies] WriteRawBodyIfNotExists: WriteRawTransactions: txId=3488038762, baseTxId=3488038762, 
label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position"

Steps to reproduce the behaviour

Sync node

Backtrace

Output of print_stages

$ ./build/bin/integration print_stages --datadir=/data/polygon --chain bor-mainnet
INFO[06-16|03:12:47.782] logging to file system                   log dir=/data/polygon/logs file prefix=integration log level=info json=false
INFO[06-16|03:12:47.784] [db] open                                lable=chaindata sizeLimit=12TB pageSize=8192
INFO[06-16|03:12:50.106] [snapshots:all] Blocks Stat              blocks=51300k indices=51300k alloc=2.6GB sys=2.7GB
INFO[06-16|03:12:50.233] [snapshots:all] Blocks Stat              blocks=51300k indices=51300k alloc=2.7GB sys=2.8GB
Note: prune_at doesn't mean 'all data before were deleted' - it just mean stage.Prune function were run to this block. Because 1 stage may prune multiple data types to different prune distance.

                                 stage_at        prune_at     
Snapshots                        51299999        0                                                    
Headers                          57834183        0
BorHeimdall                      57834183        0
BlockHashes                      57834183        0
Bodies                           51299999        0
Senders                          51299999        0
Execution                        45997039        45997039
Translation                      0               0
HashState                        45997039        0
IntermediateHashes               45997039        0
AccountHistoryIndex              45997039        0
StorageHistoryIndex              45997039        0
LogIndex                         45997039        0
CallTraces                       45997039        0
TxLookup                         45997039        51299999
Finish                           45997039        0
--
prune distance: 

blocks.v2: true, blocks=51299999, segments=51299999, indices=51299999
blocks.bor.v2: segments=51299999, indices=51299999

history.v3: false,  idx steps: 0.00, lastBlockInSnap=0, TxNums_Index(0,1)

sequence: EthTx=3488038761, NonCanonicalTx=0

in db: first header 51300000, last header 57834183, first body 51300000, last body 54612309
--

output of mdbx_stat:

$ ./build/bin/mdbx_stat -e /data/polygon/chaindata/mdbx.dat                                                                                                                
mdbx_stat v0.12.0-71-g1cac6536 (2022-07-28T09:57:31+07:00, T-9a6d7e5b917e5fbd14dc51835fa749d092aa1d72)                                                                                                      
Running for /data/polygon/chaindata/mdbx.dat...                                                                                                                                                             
Environment Info                                                                                                                                                                                            
  Pagesize: 8192                                                                                                                                                                                            
  Dynamic datafile: 24576..13194139533312 bytes (+16777216/-33554432), 3..1610612736 pages (+2048/-4096)                                                                                                    
  Current mapsize: 13194139533312 bytes, 1610612736 pages                                                                                                                                                   
  Current datafile: 6107611267072 bytes, 745558016 pages
  Last transaction ID: 282               
  Latter reader transaction ID: 282 (0)
  Max readers: 116      
  Number of reader slots uses: 1
Status of Main DB
  Pagesize: 8192
  Tree depth: 2                                                                                       
  Branch pages: 1
  Leaf pages: 2                                                                                       
  Overflow pages: 0                                                                                   
  Entries: 141

[crebsy]

just found this #10269
will try with v2.60.1 and report back if that fixed it for me

[AskAlexSharov]

if it doesn't help. try: upgrade and integration stage_headers --reset

[AskAlexSharov]

feel free to reopen if need