Revert UTxOIdentifier to use TxHash

[lowhung]

Context

The tx_unpacker maintains a UTxORegistry that tracks the live UTxO set, mapping TxOutRef → TxIdentifier. This enables converting transaction inputs into UTxOIdentifier (block_number + tx_index + output_index) for downstream modules like utxo_state.

When processing a transaction, the tx_unpacker:

1. Looks up inputs in its registry to get the TxIdentifier (to create UTxOIdentifier)
2. Removes those inputs from the registry (they're spent)
3. Adds the new outputs to the registry

Problem

When bootstrapping from a snapshot, we bypass tx_unpacker entirely:

• utxo_state gets the full UTxO set from the snapshot
• tx_unpacker's registry remains empty

When new blocks arrive post-snapshot, tx_unpacker can't resolve inputs to UTxOIdentifier because they never passed through it. The tx_unpacker doesn't crash—it silently drops unresolvable inputs and logs an error. The result is incomplete delta messages where spent UTxOs never get marked as spent in utxo_state.

We cannot bootstrap the UTxORegistry from the snapshot because it doesn't contain block_number or tx_index—only TxOutRef (tx_hash + output_index).

Options Discussed

1. Revert to TxHash-based identifiers — Simpler and safer; removes dependency on block_number/tx_index. Need to assess memory impact on other modules (e.g., Address State)

2. Invent synthetic serial numbers — Pass snapshot through tx_unpacker with invented identifiers. Unclear if other components construct UTxOIdentifier from block_number independently.

3. Bootstrap the registry somehow — Not feasible without the required data in snapshots.

Additional Consideration

The /tx/utxos endpoint requires data not immediately available in the requested transaction:

• Input UTxO values (chain_store could look these up)
• Hash of tx that consumed outputs (currently falls through the gaps)

Acceptance Criteria

• Snapshot bootstrap populates utxo_state directly without breaking input resolution
• Post-snapshot blocks correctly mark spent UTxOs
• Determine memory impact of TxHash-based approach on affected modules
• Clarify ownership of consumed_by_tx lookup between chain_store and utxo_state

[lowhung]

Would love someone like @alexwoods or @sandtreader to review this ticket.

[sandtreader]

Many thanks for capturing that!

More thoughts:

The fundamental problem is that UTXOIdentifier contains a block number and tx_index in that block, neither of which are available when we want to create a UTXO from snapshot. The reason for doing this is to save space - a UTXOIdentifier is 8 bytes, while a (tx_hash, tx_index, output_index) is 36. If this was only stored once (as was the case, pre BF) it wouldn't be worth doing, but if there are multiple places UTXOs are stored (e.g. in the BF historical state) then it is.

I've been thinking about Option (2) - invent an internal serial number for each UTXO. It would allow us to retain the compressed UTXOIdentifier for storage, but causes problems if anyone wants to look up the Tx hash to report externally. The ChainStore currently provides this facility through GetUTxOHashes queries, and the rest_blockfrost uses it for accounts and addresses.

I did an experiment to remove UTXOIdentifier::block_number() to see who uses it, and apart from some logging, the ChainStore's GetUTxOHashes handler is the only thing that does. But it's also the only thing that has the full historical state and can look up by block number - the TxUnpackers UTXORegistry is only live UTXOs and maps from hash to block number+index, not vice versa. We would have to have this reverse map for every UTXO ever created, which would be huge.

So regrettably I think Option (2) isn't viable, unless I've missed something (shout if so!). So we're left with Option (1)...

This is definitely possible but may be tricky given the layers of history on top of it - it certainly can't be automated, and will increase memory footprint in the 'historical data node' mode (we need to quantify how much). It would however simplify a bunch of stuff:

• No need for the UTXORegistry in TxUnpacker, which would go back to being nice and stateless
• No need for the GetUTxOHashes query

I think the simplest change is to redefine UTXOIdentifier back to being a tx_hash and output_index - essentially the same as a TxOutRef. Actually since TxOutRef is primarily used in the UTXORegistry and things that feed it, it can simply be replaced with UTXOIdentifier. I think the only things that will break are tests and the things we want to get rid of!

[sandtreader]

We would have to have this reverse map for every UTXO ever created, which would be huge.

I've realised that AddressState and possibly HistoricalAccountsState will effectively have exactly that, and it will be BIG. If we guesstimate 350m UTXOs have ever existed, that's 11GB just for the hashes! This of course is why we did this in the first place.

But given the constraints it places on bootstrapping and the complexity, I think we need to kick that problem to those modules. If the GetUTxOHashes query remains (I think this means it should, but based on explicit block number+index) then these modules can store the UTXOs as a CompressedUTxOIdentifier (renamed from the current UTXOIdentifier) - and use the query to look these up as now.

TL;DR:

1. Rename UTXOIdentifier as CompressedUTXOIdentifier
2. Rename TxOutRef as UTXOIdentifier
3. Fix up the mess.