BM-60: Database schema ER diagram #45

aakoshh · 2022-07-04T14:00:56Z

Created an Entity Relationship diagram for our modest database schema.

The colours were generated by https://coolors.co/79addc-ffc09f-ffee93-fcf5c7-adf7b6

SebastianElvis · 2022-07-05T00:05:29Z

docs/diagrams/database_schema.puml

+ * outcome: success or failure
+ * events: emitted by staking execution
+ * logs: emitted by staking execution
+ }


Why would the epoching module store the events and logs emitted by the staking module in DB, rather than returning it to the user and other modules directly?

Yeah so this is one of the instances where this logical model might be different from the physical one. I will try to add a note.

I am actually a bit confused as to where these are physically stored. Events say they do not impact consensus - I thought that was only true about Logs. Although Header contains LastResultHash, and it indeed says these events are ignored. Not sure how you can build an IBC solution on subscribing to events then.

Anyway, they must be stored somewhere. I just wanted to highlight that we leave some sort of tombstone behind about the execution, eventually.

gitferry

Very cool ER diagram, thanks @aakoshh! I left several comments mainly surrounding the checkpointing module. Please check.

gitferry · 2022-07-05T03:02:23Z

docs/diagrams/database_schema.puml

+}
+
+
+package "checkpointing" #FFEE93 {


Maybe we also need an entity for LastCommitHash which is drawn from Tendermint Block.

We need this to create bls-sig transaction

I added

babylon/docs/diagrams/database_schema.puml

Line 173 in 6eca9c6

raw_ckpt |o..|| block : commit hash from

to show this relationship, that it's coming from the Tendermint block.

I don't know the best way to implement this, but I commented about doing aggregation incrementally, why would allow the creation of a raw checkpoint with 0 signatures, aggregate them until the threshold is reached, then freeze. If we did that, then the LastCommitHash can be part of the raw checkpoint from the beginning.

Yeah. I agree. I will do that in future PRs. Thanks for the reminder!

gitferry · 2022-07-05T03:05:49Z

docs/diagrams/database_schema.puml

+ | SIGNED: has +1/3 sigs, await submit
+ | SUBMITTED: included on BTC
+ | CONFIRMED: k-deep on BTC
+ * effective_timestamp: BTC timestamp of k-deep


Have we decided to add effective_timestamp in the Checkpointing module? @fishermanymc What's it for?

I'm not exactly sure about which timestamps (if any) need to be copied here, but they could be useful for someone to see the lifecycle of the checkpoint: when was it included first on BTC, when did we consider it confirmed, etc. But now that we don't allow multiple checkpoints and we don't even allow submissions without a prior checkpoint, the effective timestamp is not needed for example to order checkpoints.

Removed these for now, but you have use for it, let me know.

gitferry · 2022-07-05T03:11:02Z

docs/diagrams/database_schema.puml

+ * PoP: Proof-of-Possession
+ }
+
+ entity "BLS Signature" as bls_sig {


I'm assuming this is like a logical model. We don't need this to be stored because it is processed right away and accumulated as aggr_bls_sig, right?

Exactly, I added this as an illustration to show that there are multiple signatures, and what relationships they have. I think these are currently explicitly stored, but @gitferry said he might remove them in a future PR.

Yes. Thanks for the reminder!

gitferry · 2022-07-05T03:16:18Z

docs/diagrams/database_schema.puml

+ | ACCUMULATING: await sigs
+ | SIGNED: has +1/3 sigs, await submit
+ | SUBMITTED: included on BTC
+ | CONFIRMED: k-deep on BTC


I thought CONFIRMED is w-deep rather than k-deep and w>k?

Oh, I'm not actually sure, I haven't internalised that model. I thought w is something like a timeout, and if a checkpoint doesn't appear within w it means Babylon has potential issues with censoring. But I'm not sure how it affects confirmation.

A k-deep checkpoint surely achieved something in that it won't be rolled back from Bitcoin. At best there can be an alternative submission to show up, but that just means someone else should have received the reward.

I'm not sure either. Do @fishermanymc and @KonradStaniec have any opinion on this?

Left this at k-deep and added a FINALIZED state with w-deep.

Removed the timestamps as it's not clear exactly what timestamp that should be, since the last block is still volatile.

Thanks for the info. I guess I should update the status in the Checkpointing module accordingly. Will do in a future PR.

docs/diagrams/database_schema.puml

gitferry · 2022-07-05T03:21:39Z

docs/diagrams/database_schema.puml

+ * effective_timestamp: BTC timestamp of k-deep
+ }
+
+ entity "BLS Key" as bls_key {


Maybe we need to add MsgCreateValidator in this entity since the message is stored and forwarded to the staking module at the end of the epoch.

Thanks for the suggestion. I added a MsgWrappedCreateValidator and MsgWrappedStaking messages with some loose association to BLS key and the queued message to show that they create these, and move PoP over as well.

I just realized that the checkpointing module does not need to store MsgWrappedCreateValidator messages, but forward them to the epoching module. So, I would say that they are here as a logical model.

Yes, these are totally here as logical entries. I used enum for them instead of entity and dashed lines to associate them with the others on the diagram to differentiate.

docs/diagrams/database_schema.puml

vitsalis

Thanks for the awesome diagram! Added some minor comments about the btclightclient module.

docs/diagrams/database_schema.puml

vitsalis · 2022-07-07T09:12:41Z

docs/diagrams/database_schema.puml

+ entity "Tip" as btc_tip {
+ <<singleton>>
+ --
+ * hash <<FK>>: block hash


Actually, we store the entire header bytes in order to simplify retrieval. By storing the hash, in order to get the header bytes we would need to retrieve the height using the hash->height storage and then retrieve the header bytes using the (height, hash) -> header storage.

Thanks, I think that falls under the difference between a logical and the physical schema. The point I was trying to make here is that every tip is also header, but at most one header is a tip.

We could attempt to do some other diagram that fully captures the KV store indexes we use.

docs/diagrams/database_schema.puml

aakoshh · 2022-07-08T15:26:17Z

Are there any more comments on this? Could somebody approve, if not?

SebastianElvis · 2022-07-08T15:34:36Z

Are there any more comments on this? Could somebody approve, if not?

Perhaps adding the DB schema for the validator set and the slashed validator set in the epoching module, which are introduced in #35?

aakoshh · 2022-07-08T17:18:36Z

@SebastianElvis thanks, added those 👍

gitferry

LGTM! Thanks for multiple rounds of modification. No blocker from my side.

Database schema ER diagram

6cc797a

aakoshh force-pushed the bm-60-er-diagram branch from b1ab5c3 to 6cc797a Compare July 4, 2022 14:16

aakoshh requested review from fishermanymc, KonradStaniec, vitsalis, gitferry and SebastianElvis July 4, 2022 14:17

BM-60: Add trusted header.

394dc04

aakoshh force-pushed the bm-60-er-diagram branch from d5ca0e9 to 394dc04 Compare July 4, 2022 14:31

aakoshh added 3 commits July 4, 2022 21:25

BM-60: Annotations on relationships.

d7048ee

BM-60: Reference last commit hash.

3372f2b

BM-60: Next validators

87e21fd

SebastianElvis reviewed Jul 5, 2022

View reviewed changes

gitferry reviewed Jul 5, 2022

View reviewed changes

aakoshh added 7 commits July 5, 2022 10:48

BM-60: Tip.

9acccef

BM-60: Note about not storing delayed results.

6eca9c6

BM-60: Ed25519

e61cf34

BM-60: Add some messages.

76b537a

BM-60: Note about messages.

b843b35

BM-60: Add w-deep

9ac5a6d

BM-60: Keep track of accumulated power.

a1da7a9

vitsalis reviewed Jul 7, 2022

View reviewed changes

aakoshh added 3 commits July 7, 2022 11:25

BM-60: Note about starting total PoW

56f3c2b

BM-60: Add version to BTC header.

39cd65a

BM-60: Indicate that no real FK between epoch and QM

695bc45

BM-60: Validator snapshot and slashed

ffcbd9c

gitferry approved these changes Jul 9, 2022

View reviewed changes

aakoshh merged commit 8a003fa into main Jul 9, 2022

aakoshh deleted the bm-60-er-diagram branch July 9, 2022 07:22

KonradStaniec pushed a commit that referenced this pull request Oct 25, 2023

feat: Add query for block at the given height (#45)

b3db59d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BM-60: Database schema ER diagram #45

BM-60: Database schema ER diagram #45

aakoshh commented Jul 4, 2022

SebastianElvis Jul 5, 2022

aakoshh Jul 5, 2022

gitferry left a comment

gitferry Jul 5, 2022

aakoshh Jul 5, 2022

gitferry Jul 5, 2022

gitferry Jul 5, 2022

aakoshh Jul 5, 2022

aakoshh Jul 6, 2022

gitferry Jul 5, 2022

aakoshh Jul 5, 2022

gitferry Jul 5, 2022

gitferry Jul 5, 2022

aakoshh Jul 5, 2022

gitferry Jul 5, 2022

aakoshh Jul 6, 2022

gitferry Jul 6, 2022

gitferry Jul 5, 2022

aakoshh Jul 5, 2022

gitferry Jul 5, 2022

aakoshh Jul 5, 2022

vitsalis left a comment

vitsalis Jul 7, 2022

aakoshh Jul 7, 2022

aakoshh commented Jul 8, 2022

SebastianElvis commented Jul 8, 2022

aakoshh commented Jul 8, 2022

gitferry left a comment

BM-60: Database schema ER diagram #45

BM-60: Database schema ER diagram #45

Conversation

aakoshh commented Jul 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gitferry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vitsalis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aakoshh commented Jul 8, 2022

SebastianElvis commented Jul 8, 2022

aakoshh commented Jul 8, 2022

gitferry left a comment

Choose a reason for hiding this comment