SPDD differences in epoch 216/217

[sandtreader]

The SPDD checks diverge at epoch 217

2025-12-15T12:04:51.824819Z ERROR acropolis_module_accounts_state::verifier: SPDD verification epoch 217 total active stake difference: reference 15657991508072959 != computed 15657991008072959
2025-12-15T12:04:51.824831Z ERROR acropolis_module_accounts_state::verifier: SPDD verification epoch 217, 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79: ref 974848249106 != comp 974348249106
2025-12-15T12:04:51.824834Z ERROR acropolis_module_accounts_state::verifier: Verification of SPDD, end of epoch 217: Failed

Difference is 500A = 1 SPO deposit

Definition of done

SPDD check matches in epoch 217

[sandtreader]

Actually the error happens at 216 - must have missed this before

2025-12-17T11:07:33.568684Z ERROR acropolis_module_accounts_state::verifier: SPDD verification epoch 216 total active stake difference: reference 14987431249067863 != computed 14987430749067863
2025-12-17T11:07:33.568694Z ERROR acropolis_module_accounts_state::verifier: SPDD verification epoch 216, 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79: ref 973144831805 != comp 972644831805
2025-12-17T11:07:33.568699Z ERROR acropolis_module_accounts_state::verifier: Verification of SPDD, end of epoch 216: Failed

[sandtreader]

Difference is we are 500 ADA too low - therefore most likely we haven't paid a pool deposit refund that we should have.

However this particular pool has never retired (although it's inactive):

https://beta.cexplorer.io/pool/pool13zh79cqv3y36cgknzagwegp8y0kuzel2j046jmup97hhjwk5myw?tab=about

One of its delegators must be the reward account for a different pool. There aren't many:

2025-12-17T11:42:19.991456Z DEBUG acropolis_module_accounts_state::state: Delegation of e14db762458e59e9790786eaaf1f9fc39227216480405368841318a03d to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:19.993630Z DEBUG acropolis_module_accounts_state::state: Delegation of e1a3e71716dd93f487166a749e7079db529386cf7dc8b79a36bbcbb13c to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:20.029944Z DEBUG acropolis_module_accounts_state::state: Delegation of e1fd56324d72068abc09c88d671278ed36bef994f6702cc22bfa0a80a5 to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:20.104940Z DEBUG acropolis_module_accounts_state::state: Delegation of e15c202c348e68fc81a174fc5bb9692a95a8af31a827001e74a7aa9513 to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:20.105253Z DEBUG acropolis_module_accounts_state::state: Delegation of e12b272fc2e522ebe98f1e76a58d2b0ab4381844fa0d4ece8f219ffd61 to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:20.106538Z DEBUG acropolis_module_accounts_state::state: Delegation of e137df19fa7f377c2e22318e2b73b5b598c007613692830e74b92afb5a to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:20.393475Z DEBUG acropolis_module_accounts_state::state: Delegation of e1d11f56b78d6367a3cd19bc4dc0240ae493bef473d342637900480d10 to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:20.395997Z DEBUG acropolis_module_accounts_state::state: Delegation of e127fc29c8324f05629c010e2efc9ca57521c8f178fb3619927b0d13cf to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:20.557095Z DEBUG acropolis_module_accounts_state::state: Delegation of e130c6658e4e070f9a3e50cf971f9b9582a9b03802645da0a8ce202d7a to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:20.907773Z DEBUG acropolis_module_accounts_state::state: Delegation of e1dc56e06a860f828f356184d0191c1f2109be417bf81fdc93af75ebdd to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:21.534884Z DEBUG acropolis_module_accounts_state::state: Delegation of e143e6b46d1e36885a286e67f374f3c164b3746b9c8dd8a81dc2b75513 to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79
2025-12-17T11:42:26.166144Z DEBUG acropolis_module_accounts_state::state: Delegation of e1cc4e0e5527fd7893a19436f39295e26f4ca769bd9603f4c71984bfd3 to 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79

Will try to track if any of these are pool reward accounts for other pools

[sandtreader]

$ grep 88afe2e00c8923ac /tmp/acropolis.log | grep Delegation | cut -d" " -f6 | grep -f - /tmp/acropolis.log | grep reward

2025-12-17T11:42:21.833170Z DEBUG acropolis_module_accounts_state::state: Registered new SPO 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79 epoch=215 pledge=2000000000 cost=340000000 margin=Ratio { numerator: 1, denominator: 40 } reward=e14db762458e59e9790786eaaf1f9fc39227216480405368841318a03d
2025-12-17T11:42:26.279385Z DEBUG acropolis_module_accounts_state::state: Updated parameters for SPO 88afe2e00c8923ac22d31750eca02723edc167ea93eba96f812faf79 epoch=216 pledge=15000000000 cost=340000000 margin=Ratio { numerator: 1, denominator: 40 } reward=e14db762458e59e9790786eaaf1f9fc39227216480405368841318a03d

So e14db is actually the reward account for the errant pool, which is the normal case.

This did indeed receive 500A reward (refund) in epoch 216:
https://beta.cexplorer.io/stake/stake1u9xmwcj93ev7j7g8sm4278ulcwfzwgtyspq9x6yyzvv2q0gjslq5u?tab=rewards&page=4

But why?

[sandtreader]

Here's the refund being paid, to pool 591:

cexplorer=> select * from reward r join stake_address a on a.id = r.addr_id where encode(a.hash_raw, 'hex') like 'e14db7624%' and r.type='refund';
 addr_id |  type  |  amount   | spendable_epoch | pool_id | earned_epoch |  id   |                           hash_raw                           |                            view                             | script_hash 
---------+--------+-----------+-----------------+---------+--------------+-------+--------------------------------------------------------------+-------------------------------------------------------------+-------------
   87622 | refund | 500000000 |             216 |     591 |          216 | 87622 | \xe14db762458e59e9790786eaaf1f9fc39227216480405368841318a03d | stake1u9xmwcj93ev7j7g8sm4278ulcwfzwgtyspq9x6yyzvv2q0gjslq5u | 
(1 row)
cexplorer=> select * from pool_hash where id = 591;
 id  |                          hash_raw                          |                           view                           
-----+------------------------------------------------------------+----------------------------------------------------------
 591 | \x54e85031e5655b9f63cf8dbbe3136a4203d2d1efc4dd05d32cdfa0bd | pool12n59qv09v4de7c703ka7xym2ggpa9500cnwst5evm7st6anvaj4
(1 row)

This pool does indeed retire in epoch 216:

cexplorer=> select * from pool_retire where hash_id = 591;
 id  | hash_id | cert_index | announced_tx_id | retiring_epoch 
-----+---------+------------+-----------------+----------------
 132 |     591 |          0 |         2678745 |            216

Here are the pool updates for it:

cexplorer=> select * from pool_update where hash_id=591;
  id  | hash_id | cert_index |                            vrf_key_hash                            |   pledge    | active_epoch_no | meta_id | margin | fixed_cost | registered_tx_id | reward_addr_id |  deposit  
------+---------+------------+--------------------------------------------------------------------+-------------+-----------------+---------+--------+------------+------------------+----------------+-----------
  999 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |   500000000 |             210 |     998 |   0.05 |  340000000 |          2452685 |          39476 | 500000000
 1828 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |   600000000 |             212 |    1827 |  0.025 |  340000000 |          2479761 |          39476 |          
 2570 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |   650000000 |             213 |    2569 |  0.025 |  340000000 |          2520482 |          39476 |          
 3002 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |   750000000 |             214 |    3001 |      0 |  340000000 |          2548413 |          39476 |          
 3004 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |   745000000 |             214 |    3003 |      0 |  340000000 |          2548436 |          39476 |          
 3048 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |   745000000 |             214 |    3047 |      0 |  340000000 |          2552904 |          39476 |          
 3308 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 | 12100000000 |             214 |    3307 |      0 |  340000000 |          2570855 |          39476 |          
 4423 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |     2000000 |             218 |    4421 |   0.02 |  340000000 |          2673609 |          87622 |          
 4424 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |     2000000 |             218 |    4422 |   0.02 |  340000000 |          2673655 |          87622 |          
 4425 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |  1000000000 |             218 |    4423 |   0.02 |  340000000 |          2673661 |          87622 |          
 4427 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |  1000000000 |             218 |    4425 |   0.02 |  340000000 |          2673676 |          87622 |          
 4429 |     591 |          0 | \x6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4 |  2030000000 |             218 |    4427 |   0.02 |  340000000 |          2673963 |          87622 |          
(12 rows)

Note how it switches to 87622 = e14db7624 in epoch "218" (taking DBSync's epochs with a big pinch of salt, since it will have retired by then)

The old reward address 39476 is weird:

cexplorer=> select * from stake_address where id=39476;
  id   |                           hash_raw                           |                            view                             | script_hash 
-------+--------------------------------------------------------------+-------------------------------------------------------------+-------------
 39476 | \xe154e85031e5655b9f63cf8dbbe3136a4203d2d1efc4dd05d32cdfa0bd | stake1u92ws5p3u4j4h8mre7xmhccndfpq85k3alzd6pwn9n06p0gqwy5x3 | 
(1 row)

It has the same hash as the pool!

Our logs confirm this is where we paid the refund to:

2025-12-17T11:42:21.833308Z DEBUG acropolis_module_accounts_state::state: SPO 54e85031e5655b9f63cf8dbbe3136a4203d2d1efc4dd05d32cdfa0bd has retired - refunding their deposit to e154e85031e5655b9f63cf8dbbe3136a4203d2d1efc4dd05d32cdfa0bd

This address remains delegated to this pool so will disappear out of the SPDD - hence explaining why the refund is lost rather than just moving.

So it's looking like we don't pick up reward address changes quickly enough for them to affect the refund payment.

[sandtreader]

Confirmation:

During 215:

2025-12-17T14:03:23.514406Z DEBUG acropolis_module_spo_state::state: New pending SPO update 54e85031e5655b9f63cf8dbbe3136a4203d2d1efc4dd05d32cdfa0bd PoolRegistration { operator: PoolId(Hash<28>(
"54e85031e5655b9f63cf8dbbe3136a4203d2d1efc4dd05d32cdfa0bd")), vrf_key_hash: VrfKeyHash(Hash<32>("6495541186b2e53698244e231274a0754678664d2655d0e233aa3b9a03d21ef4")), pledge: 2030000000, cost: 34
0000000, margin: Ratio { numerator: 1, denominator: 50 }, reward_account: StakeAddress { network: Mainnet, credential: AddrKeyHash(Hash<28>("4db762458e59e9790786eaaf1f9fc39227216480405368841318a
03d")) }, pool_owners: [StakeAddress { network: Mainnet, credential: AddrKeyHash(Hash<28>("4db762458e59e9790786eaaf1f9fc39227216480405368841318a03d")) }], relays: [SingleHostAddr(SingleHostAddr 
{ port: Some(3001), ipv4: Some(84.232.231.225), ipv6: None })], pool_metadata: Some(PoolMetadata { url: "http://charmada.io/poolmeta.json", hash: [22, 44, 51, 142, 212, 33, 79, 227, 19, 25, 211,
 152, 241, 104, 116, 105, 42, 42, 64, 66, 82, 43, 146, 177, 249, 155, 59, 0, 175, 119, 215, 158] }) } epoch=215 block=4649288
...
2025-12-17T14:03:24.238578Z DEBUG acropolis_module_spo_state::state: SPO 54e85031e5655b9f63cf8dbbe3136a4203d2d1efc4dd05d32cdfa0bd wants to retire at the end of epoch 216 (cert in block number 4653052, tx 4653052:2)

At epoch 216 boundary:

2025-12-17T14:03:26.000838Z DEBUG acropolis_module_spo_state::state: Retiring SPO 54e85031e5655b9f63cf8dbbe3136a4203d2d1efc4dd05d32cdfa0bd
2025-12-17T14:03:26.081666Z DEBUG acropolis_module_accounts_state::state: SPO 54e85031e5655b9f63cf8dbbe3136a4203d2d1efc4dd05d32cdfa0bd has retired - refunding their deposit to e154e85031e5655b9f63cf8dbbe3136a4203d2d1efc4dd05d32cdfa0bd

Reward account should have been updated to 4db7 before the retirement kicks in.