Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store the address book in the database #796

Closed
steven-sheehy opened this issue May 29, 2020 · 0 comments · Fixed by #894
Closed

Store the address book in the database #796

steven-sheehy opened this issue May 29, 2020 · 0 comments · Fixed by #894
Assignees
@Nana-EC
Labels
enhancement Type: New feature P1 parser Area: File parsing
Milestone
Mirror 0.17.0

Comments

@steven-sheehy
Copy link
Member

steven-sheehy commented May 29, 2020 •
edited
Loading

Problem
We currently store the address book on the filesystem. This is sub-optimal because:

  • Eventually we'd like to remove the dependency on the filesystem
  • If we want to parallelize the processing of records (e.g. simultaneously processed catch up and current), those records may have a different address book in use at that particular point in time and we only store the latest on disk
  • If we have HA importers with leader election, only one will get the updated address book and the other will fail once it becomes primary
  • We may want to serve the address book via an API

Solution

  • Store the address book in the database so it's shared among all instances
  • Create table address_book with columns for start and end consensus timestamps indicating what time range it was valid
  • Create table address_book_entry with similar fields as NodeAddress proto
  • Consider watching both 0.0.101 and 0.0.102 address books and combining into the same table
  • Add raw address book bytes

Alternatives
Store multiple address books on disk, sync them to other instances through other means

Additional Context
@steven-sheehy steven-sheehy added enhancement Type: New feature parser Area: File parsing labels May 29, 2020
@Nana-EC Nana-EC self-assigned this Jul 15, 2020
@Nana-EC Nana-EC mentioned this issue Jul 20, 2020
Merged
2 tasks
Nana-EC added a commit that referenced this issue Jul 31, 2020
@Nana-EC
Store address book in DB (#894)
e7051a6 
We currently store the address book on the filesystem which is sub optimal for reasons highlighted in #796 
We want to be able to persist the address book details in the db for multiple future features

- Added V1_28_0 sql migration to create `address_book` and `address_book_entry `. Also updates `file_data` to have entityId and transaction types.
- Added `address_book` table to store address book files with columns for start and end consensus timestamps indicating what time range it was valid and raw bytes
- Added `address_book_entry ` tables to store node addresses with similar fields as NodeAddress proto
- Added `AddressBookService` interface
- Added `AddressBookServiceImpl` class that will check db for a valid address book. If not found it w
- Removed file system storage and reading of address book with support for loading bootstrap address book from file system or classpath
- Updated `AddressBookServiceImpl.update()` to handle `fileData` objects. Logic will store in `file_data`. If file is of append operation it will retrieve previous create/update and append operations bytes and concatenate them to create an addressBook. Object is then stored in `address_book` and `address_book_entry` tables
- Updated RecordEntityListener to support addressbook vs non-addressbook file parsing
- Updated `AddressBookServiceImpl` to handle update start and end timestamps of address book after network restart
- Added Java migration to parse all previous file_data rows for potential address books

Signed-off-by: Nana-EC <56320167+Nana-EC@users.noreply.github.com>
@steven-sheehy steven-sheehy added this to the Mirror 0.17.0 milestone Aug 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Nana-EC Nana-EC

Labels
enhancement Type: New feature P1 parser Area: File parsing
Projects
None yet
Milestone
Mirror 0.17.0
Development

Successfully merging a pull request may close this issue.

Store address book in DB hashgraph/hedera-mirror-node
2 participants
@steven-sheehy @Nana-EC