-
Notifications
You must be signed in to change notification settings - Fork 158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make the overlay schedule abstract #1742
Conversation
dc027f5
to
3a752b0
Compare
This is just a first step, making the way for the actual optimisation. I'm on vacation next week, so I just wanted to open this PR so it's not forgotten about. It would be good if somebody from the ledger team takes this over. |
This will allow us to replace its implementation/representation by something more compact and efficient. On mainnet, the epoch size is 432,000, which means we're creating a map of nearly half a million of entries just to implement a spaced round-robin scheme. When writing ledger snapshots, we're also writing this map, albeit in a slightly more compact representation (see `compactOverlaySchedule`), to disk. Note that the size of this map will shrink when `d` goes down. It should be enough to keep the arguments passed to `overlayScheduleHelper` in memory and implement the current API of `OverlaySchedule` as a function instead of a mapping. Similarly for serialising an `OverlaySchedule`, just de/serialise these arguments. To test this approach, the function-based implementation can be compared against the current map-based implementation (e.g., by implementing `overlayScheduleToMap`).
3a752b0
to
2a9a243
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, good to move it to a separate module 👍
-- * OBftSlot (for testing) | ||
OBftSlot (..), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not just for testing, it's mentioned in the public API, i.e., lookupInOverlaySchedule
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, my search didn't spot it, then I realised I had a broken search...
@@ -3,6 +3,7 @@ | |||
{-# LANGUAGE FlexibleContexts #-} | |||
{-# LANGUAGE FlexibleInstances #-} | |||
{-# LANGUAGE MultiParamTypeClasses #-} | |||
{-# LANGUAGE RankNTypes #-} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably not needed.
emptyOverlaySchedule :: OverlaySchedule crypto | ||
emptyOverlaySchedule = OverlaySchedule Map.empty |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we intend this API to be qualified when imported, we could have shorter names like empty
, lookup
, isEmpty
, new/create/...
, toMap
, compact
, decompact
, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, considered that but couldn't be bothered to change things around yet...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@@ -137,7 +138,7 @@ getLeaderSchedule globals ss cds poolHash key = Set.filter isLeader epochSlots | |||
where | |||
isLeader slotNo = | |||
let y = VRF.evalCertified () (mkSeed seedL slotNo epochNonce) key | |||
in Map.notMember slotNo overlaySched | |||
in isNothing (lookupInOverlaySchedule slotNo overlaySched) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe better to use not $ isOverlaySlot
now
-- * OBftSlot (for testing) | ||
OBftSlot (..), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, my search didn't spot it, then I realised I had a broken search...
@@ -3,6 +3,7 @@ | |||
{-# LANGUAGE FlexibleContexts #-} | |||
{-# LANGUAGE FlexibleInstances #-} | |||
{-# LANGUAGE MultiParamTypeClasses #-} | |||
{-# LANGUAGE RankNTypes #-} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not needed
@@ -129,7 +126,12 @@ coreNodeKeysBySchedule :: | |||
AllIssuerKeys c 'GenesisDelegate | |||
coreNodeKeysBySchedule = coreNodeKeysForSlot . fullOSched | |||
where | |||
fullOSched pp = Map.unions $ [overlayScheduleFor e pp | e <- [0 .. 10]] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can probably do this a better way.
'LedgerState' is already a giant grab-bag of functionality. This is a good excuse to extract some of it.
This PR now collects the `OverlaySchedule` directly in `BBodyEnv`, removing the need for overlaySlots and hopefully avoiding an expensive call to `Map.keysSet`. It should be functionally equivalent to #1775 whilst allowing us to optimise further in making the overlay schedule non-explicit.
7a219b3
to
1a5f10e
Compare
Highlights: * IntersectMBO/cardano-ledger#1784 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1779 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1742 * IntersectMBO/ouroboros-network#2512 * Remove unused source-repository-package cardano-sl-x509 (this package required the `ip < 1.5` constraint which conflicted with the `primitive >= 0.7.1.0` lower bound in `cardano-ledger-specs`). * Remove unused http-client dependency from stack.yaml * Remove duplicate from cabal.project
1696: Update dependencies r=mrBliss a=mrBliss Highlights: * IntersectMBO/cardano-ledger#1784 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1779 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1742 * IntersectMBO/ouroboros-network#2512 Co-authored-by: Thomas Winant <thomas@well-typed.com> Co-authored-by: Samuel Leathers <samuel.leathers@iohk.io>
1696: Update dependencies r=mrBliss a=mrBliss Highlights: * IntersectMBO/cardano-ledger#1784 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1779 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1742 * IntersectMBO/ouroboros-network#2512 Co-authored-by: Thomas Winant <thomas@well-typed.com> Co-authored-by: Samuel Leathers <samuel.leathers@iohk.io>
This will allow us to replace its implementation/representation by something
more compact and efficient.
On mainnet, the epoch size is 432,000, which means we're creating a map of
nearly half a million of entries just to implement a spaced round-robin scheme.
When writing ledger snapshots, we're also writing this map, albeit in a slightly
more compact representation (see
compactOverlaySchedule
), to disk.Note that the size of this map will shrink when
d
goes down.It should be enough to keep the arguments passed to
overlayScheduleHelper
inmemory and implement the current API of
OverlaySchedule
as a function insteadof a mapping. Similarly for serialising an
OverlaySchedule
, just de/serialisethese arguments.
To test this approach, the function-based implementation can be compared against
the current map-based implementation (e.g., by implementing
overlayScheduleToMap
).