Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(accountsdb): Generate snapshots, fix UAF, improve bincode #179

Merged
merged 37 commits into from
Jul 16, 2024
Merged

feat(accountsdb): Generate snapshots, fix UAF, improve bincode #179

merged 37 commits into from
Jul 16, 2024

Commits on Jul 16, 2024

  1. Add utility for identifying arraylist and hashmap

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    65639e8 View commit details
    Browse the repository at this point in the history

  2. Make use of less brittle stdlib type check 

    The string returned by `@typeName` has no guaranteed nor reliable
format, and is not technically guaranteed to be unique to all types.
Using these specific helpers to identify arraylists and hashmaps is
less brittle.
    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    a7b6c68 View commit details
    Browse the repository at this point in the history

  3. Update our zstd dependency

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    413e54b View commit details
    Browse the repository at this point in the history

  4. Make the backing int of FileId a decl

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    132fc05 View commit details
    Browse the repository at this point in the history

  5. Clean up some types & use ArrayHashMap

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    be8279c View commit details
    Browse the repository at this point in the history

  6. Add writeSnapshotTar initial implementation

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    272fb7c View commit details
    Browse the repository at this point in the history

  7. Make tar also accept EOF in absence of sentinel

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    83e3955 View commit details
    Browse the repository at this point in the history

  8. Stop using @typeName in bincode.free as well 

    also make it support unmanaged data structures properly
    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    ee2271a View commit details
    Browse the repository at this point in the history

  9. Improve incremental bank fields & type refactors

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    9ed2cf7 View commit details
    Browse the repository at this point in the history

  10. Store slice instead of arraylist in file_map

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    6242997 View commit details
    Browse the repository at this point in the history

  11. LogLevel CLI arg improvement 

    Remove consequently unused `enumFromName` utility function & amend
some imports in the process.
    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    b02a387 View commit details
    Browse the repository at this point in the history

  12. Turn the fields field into a parameter

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    032b0b5 View commit details
    Browse the repository at this point in the history

  13. Add utils.fmt.boundedFmt

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    b3046be View commit details
    Browse the repository at this point in the history

  14. Some bincode additions & improvements 

    * add `arraylist.defaultArrayListUnmanagedOnEOFConfig`
* Handle hashmap size overflow
* Free hashmap key value pair on error
* Error on duplicate hash map entries
* Add `skip_write_fn` config predicate: kind of a duct-tape solution,
  but it'll serve us while we use this bincode model.
    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    84a0d9d View commit details
    Browse the repository at this point in the history

  15. Publicize ReferenceMemory

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    27f2b7a View commit details
    Browse the repository at this point in the history

  16. Implement writeSnapshotTarTo method, & more 

    This commit brings an initial high level implementation for the
generation of snapshots based off of currently derivable data in
accountsdb. It also makes a number of changes to accommodate testing,
and improve code robustness, namely a switch from byte slice paths to
directory handles in related & auxiliary code.
Also this specifically modifies loadAndVerifyAccountsFiles to take
the task indexes in order to slice into the file map data, instead of
the file name list.

Another notable change is the fixup for the `largest_root_slot` field
not actually being assigned the largest available root slot.

There are also various other improvements and refactors.
    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    7096c0f View commit details
    Browse the repository at this point in the history

  17. Structure testWriteSnapshot better for clean cwd

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    0c3f0a8 View commit details
    Browse the repository at this point in the history

  18. bincode improvements & hashmap config 

    * Make the deserialize allocator parameter non-optional
* Move the free function after the write function
* Add the optional & list namespaces
* Add `readIntAsLength`
* Remove `getSerializedSize`
* Don't return error.EOF instead of error.EndOfStream
* Reorganize struct type reading and writing
* Make `write` use the type detection instead of `@typeName`
* Add dedicated hash map config
    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    deaa226 View commit details
    Browse the repository at this point in the history

  19. Various accountsdb improvements & fixes 

    * Pass fields directly instead of passing whole structs
* Store a only a single AccountFileInfo per hashmap entry,
  whilst serializing and deserializing it as a slice.
* Fix copy paste error:
  @"!bincode-config:incremental_snapshot_persistence"
  @"!bincode-config:snapshot_persistence"
    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    60ad87e View commit details
    Browse the repository at this point in the history

  20. Note probably-temporary parameters & alias types

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    597ce20 View commit details
    Browse the repository at this point in the history

  21. Fix UAF with slightly hacky-ish code

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    c984ff9 View commit details
    Browse the repository at this point in the history

  22. Note purpose of duration of lock in function

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    0bc6b6f View commit details
    Browse the repository at this point in the history

  23. Small renames

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    0b2f8c7 View commit details
    Browse the repository at this point in the history

  24. Move tar functions to tar module

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    972e0f4 View commit details
    Browse the repository at this point in the history

  25. Better snapshot generation names

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    e9644cd View commit details
    Browse the repository at this point in the history

  26. run zig fmt

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    6ccce17 View commit details
    Browse the repository at this point in the history

  27. Rename & assign count value to local variable 

    * Renames `fields_file_map` to `file_info_map`
* Assign `file_info_map.count()` to `n_account_files`
    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    519bad0 View commit details
    Browse the repository at this point in the history

  28. Eliminate unused AccountStorage type

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    6db142d View commit details
    Browse the repository at this point in the history

  29. Some bincode refactors 

    * Import `sig.utils.types.arrayListInfo`.
* Clarify control flow.
* Get rid of `MaybeHashMapConfig`.
* Document the `skip_write_fn` predicate.
    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    e0392ce View commit details
    Browse the repository at this point in the history

  30. Remove blocks

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    3230959 View commit details
    Browse the repository at this point in the history

  31. Bincode simplifications

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    c71539c View commit details
    Browse the repository at this point in the history

  32. Enhance & limit boundedFmt

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    f9f5c2b View commit details
    Browse the repository at this point in the history

  33. Correct the account file filtering filtering logic

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    8a097c7 View commit details
    Browse the repository at this point in the history

  34. Improve file_map iteration & use more FileId 

    * Iterate over `file_info_map` instead of `file_map`, locking only
  account files which are definitely going to be written based on the
  former.
* Concretize the fact that we're only properly supporting full snapshot
  generation at the moment, incremental snapshot generation is future
  work.
* Replace various instances of `id: usize` with `id: FileId` and amend
  related usage sites. In `AccountFileInfo`, this is also amended to
  override the bincode field config to serialize and deserialize it
  as a `usize`.
    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    e294dce View commit details
    Browse the repository at this point in the history

  35. Add section for snapshot generation to readme

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    ab93e50 View commit details
    Browse the repository at this point in the history

  36. Delete dead code, rename local gens to Ss

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    08572b9 View commit details
    Browse the repository at this point in the history

  37. Extract dedicated bincode.readInt function

    @InKryption
    InKryption committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    551eab3 View commit details
    Browse the repository at this point in the history