change: Track the balance of the deferred chain value pool

[upbqdn]

Motivation

Close #8688.

Depends-On: #8694.

Solution

There are two main changes in this PR:

• Calculate and store deferred value balances for each block in the non-finalized state, and
• track the total amount of the deferred chain value pool in the finalized state.

This PR also bumps the major database format version.

High-level Description

• The amount of the deferred balance per block is calculated in zebra-consensus during the semantic validation of both checkpointed and non-checkpointed blocks.
• The deferred balances are passed to zebra-state in the validated blocks.
• The balances are added to the chain value pool of each chain in the non-finalized state upon committing the block, and subtracted upon removing the block.
• Zebra stores the balances of individual blocks in the non-finalized state but doesn't do so in the finalized one. It only tracks the total amount in the deferred chain value pool of the tip of the finalized state by adding the deferred balances to the pool upon committing blocks coming from the non-finalized state. Therefore, retrieving the deferred values for individual blocks from the finalized state doesn't currently work. If we ever need this functionality, I think it will be better to calculate it dynamically based on height and network.
• The ValueBalance struct has a new field, so it has a new serialization format for storing the struct in the database. This new format is backward-compatible but not forward-compatible with the currently supported Zebra versions, so this PR bumps the major database format version from 25.3.0 to 26.0.0.

Tests

Current tests should suffice.

PR Author's Checklist

• The PR name will make sense to users.
• The PR provides a CHANGELOG summary.
• The solution is tested.
• The documentation is up to date.
• The PR has a priority label.

PR Reviewer's Checklist

• The PR Author's checklist is complete.
• The PR resolves the issue.

[oxarbitrage]

Can we rebase to get rid of extra commits ?

[arya2]

Can we rebase to get rid of extra commits ?

I'd prefer that we avoid a rebase since there would be a lot of major conflicts with a relatively high chance of one being resolved incorrectly. We could try squashing the changes? (Though I think this is fine as-is).

[arya2]

Thank you for this change!

The full sync on Mainnet with almost all of the changes succeeded here, and we just checked that the Testnet sync passed as well in GCP (though Github is having trouble updating the test's status).

It looks like the full sync tests that are still pending in Github aren't required for merging into the main branch anyway, so this should merge soon.