Add incremental snapshot utils

[brooksprumo]

This commit adds high-level functions (bank_to_incremental_snapshot() and bank_from_incremental_snapshot()), and additionally
refactors snapshot_utils and serde_snapshot to use a common code path for
snapshots. New tests were added to snapshot_utils for incremental snapshots.

This PR grabs bits from PR #17875, and incorporated feedback from that PR review. This PR is another in the pursuit of Incremental Snapshots (#17088), and this PR finally/actually adds the ability to both load-from and store-to an incremental snapshot. Yay!

This PR is quite big. A lot of the deserialization code was touched to support incremental snapshots and avoid code duplication. The diff may be easier to read when ignoring whitespace.

The next PR in this series will add new integration-style tests to core/tests/, and will address new issues those tests uncover (likely clean_accounts()).

Additional Bits

Care has been taken to rename (almost) everything around snapshots.

• Wherever specific, all variables and functions will be full_ or incremental_ unless they work for both.
• The term "snapshot" was overloaded in the code; sometimes called a bank snapshot, sometimes a slot snapshot, and sometimes referring to the snapshot package/snapshot archive. Outside of Bank, it is now a "bank snapshot".

Outstanding Questions

• Naming: Should it be "snapshot" and "incremental snapshot" , or "full snapshot" and "incremental snapshot" There's quite a bit of existing non-code that uses just "snapshot"; should that be updated too? More discussion here
  • resolved by renaming
• Assert vs error when storages overlap. Thread 1 here, thread 2 here, and comment 3 here
  • resolved by error-ing instead of panic-ing
• Should I remove deserialize_snapshot_data_file_capped()? Thread here
• Should I create another struct for common parameters? Thread here

New Issues Created

• [Incremental Snapshots] Add field for maximum number of incremental snapshots to retain #18639
• [Incremental Snapshots] Handle duplicate AppendVec IDs #18641
• [Incremental Snapshots] Filter account_storage_entries when serializing accounts_db_fields for an incremental snapshot #18813
• Move snapshot package TempDir creation & hardlinking into ctor for AccountsPackagePre #18815
• Check file size of snapshot_version when unarchiving snapshot #18841

[codecov]

Codecov Report

Merging #18504 (a83e0b2) into master (dfde1de) will decrease coverage by 0.0%.
The diff coverage is 87.8%.

@@            Coverage Diff            @@
##           master   #18504     +/-   ##
=========================================
- Coverage    82.8%    82.8%   -0.1%     
=========================================
  Files         443      443             
  Lines      125343   125957    +614     
=========================================
+ Hits       103895   104389    +494     
- Misses      21448    21568    +120     

[brooksprumo]

@lijunwangs @ryoqun @carllin Checks have passed, and all review comments have been addressed. I've replied with answers/questions to any outstanding review comments. PR is ready for another round of reviewing. Thanks in advance!

[ryoqun]

• Assert vs error when storages overlap. Thread 1 here, and thread 2 here

@brooksprumo as a context, we want to support untrusted snapshots eventually in the (distant) future. so, panic! shouldn't be reachable when consuming (= untaring) any downloaded snapshots. #7167

That context might help to decide panic!/error?

Also, well-groomed pr description. :)

[brooksprumo]

@brooksprumo as a context, we want to support untrusted snapshots eventually in the (distant) future. so, panic! shouldn't be reachable when consuming (= untaring) any downloaded snapshots. #7167

That context might help to decide panic!/error?

@ryoqun Yes, thank you. Makes sense!

Also, well-groomed pr description. :)

Thanks :)

[lijunwangs]

Looks good to me.

Pull request has been modified.

[brooksprumo]

@carllin @lijunwangs I had to fix a merge conflict, so your approvals were removed. Nothing else changed, but let me know if you'd like me to wait for you to re-review before I merge.

@ryoqun Would you like to re-review before I merge?

[ryoqun]

I'd say a masterpiece of plumbing pr ever. xD

I only left some minor comments. merge anytime without needing another review round.

Pull request has been modified.