ingest: Add support for ledger entry change type "restore" for Protocol 23

[urvisavla]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've reviewed the changes in this PR and if I consider them worthwhile for being mentioned on release notes then I have updated the relevant CHANGELOG.md within the component folder structure. For example, if I changed horizon, then I updated (services/horizon/CHANGELOG.md. I add a new line item describing the change and reference to this PR. If I don't update a CHANGELOG, I acknowledge this PR's change may not be mentioned in future release notes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

This PR adds support for a new ledger entry change type called restore in the ingest library, introduced in Protocol 23. Additionally, the meta for update and remove changes will now change if an entry was restored in the same transaction. For example, if an entry is restored and removed within the same transaction, the meta will be [restored, removed] and if restored and updated in the same transaction, the meta will be [restored, updated]. This differs from pre-Protocol 32, where updated and removed entries always appear as [state, updated] and [state, removed].

Main Changes

1. Store the Change Type directly in the Change struct

Previously, the Change struct in the ingest library didn't store the change type explicitly but inferred it from the Pre and Post values. However, with the introduction of restore, distinguishing change types purely based on Pre and Post is no longer possible.

2. Support for restore change type in change compactor

The compaction process in the ingest library combines multiple changes within the same ledger for a single change to reduce the number of DB operations. With the introduction of the restore change type, the compaction logic is updated to:

• define how changes occurring after a restore change should be compacted.
• define how a restore change should be compacted when it precedes other change types.

3. Additionally, a new Change Compactor config EmitArchivedEntryRemovalChange has been added to control the handling REMOVED change for archived entries. By default, REMOVED changes are not emitted for entries that are created and removed within the same ledger. However, if an entry is restored and then removed in the same ledger, it is unclear whether a REMOVED change should be emitted, as this depends on the downstream system's strategy for storing archived entries. This setting allows REMOVED changes to be emitted for such entries which may be necessary for systems that store them. Systems that do not store archived entries (e.g. those running periodic garbage collection of archived entries) can set this to false to suppress the REMOVED change.

Why

#5421

Known limitations

• Support for submitting the restore change type required for integration tests is currently unavailable so this change is only unit-tested.
• This change updates the ingest library only. Updating downstream services such as Horizon and RPC are outside the scope of this PR.

[tamirms]

@urvisavla the PR looks good, I think the only remaining task is to add a check to ensure that the Change Compactor instance is only used to compact changes from a single ledger

[urvisavla]

@urvisavla the PR looks good, I think the only remaining task is to add a check to ensure that the Change Compactor instance is only used to compact changes from a single ledger

@tamirms
To ensure the change compactor processes changes from only one ledger sequence, I initially tried checking for mismatches like this:

for each existingChange in cache:
    if existingChange.LedgerSequenceNumber ≠ newChange.LedgerSequenceNumber:
        return "Ledger sequence mismatch: Cannot compact changes from different ledgers"

    break // Only check the first entry

However, I'm having trouble reliably getting the ledger sequence number from a Change. I've looked at Change::LedgerEntry::LastModifiedLedgerSeq (Pre & Post), Change::Transaction, Change::Ledger etc but none of them consistently provide the correct sequence number for all LedgerEntry changes.

What's the most reliable way to determine the current ledger sequence number for a Change without storing it in the ChangeCompactor object and then passing it with every AddChange call?

Also, it seems the new Change struct fields added in #5536 are not designed to be used with the Change Compactor which wasn't immediately obvious to me and caused some confusion.

[tamirms]

@urvisavla you can get it from change.Ledger or, if that's not present you can check change.Transaction.Ledger, or you can fix #5578 and then it should always be present in change.Ledger

Also, it seems the new Change struct fields added in #5536 are not designed to be used with the Change Compactor which wasn't immediately obvious to me and caused some confusion.

The fields are not preserved after the compaction process but the changes fed into the change reader should still have valid fields

[urvisavla]

The fields are not preserved after the compaction process but the changes fed into the change reader should still have valid fields

These fields must be preserved during compaction if we are to compare the ledger sequence number of incoming changes with existing ones in the cache. However, there are alternative solutions, such as storing the ledger sequence in the ChangeCompactor object. I have created #5607 to evaluate potential solutions and implement it separately to unblock this PR.

[tamirms]

However, there are alternative solutions, such as storing the ledger sequence in the ChangeCompactor object.

yes, that's what I had in mind

I have created #5607 to evaluate potential solutions and implement it separately to unblock this PR.

👍