[CT-647] construct the initial orderbook snapshot #1147
Conversation
WalkthroughThe update introduces enhancements to the Central Limit Order Book (CLOB) system, focusing on off-chain updates for orderbook snapshots and insurance fund balance management. Specifically, it adds a parameter to improve insurance fund queries and introduces a new functionality for generating and testing off-chain updates within a gRPC streaming context. These changes aim to enhance the system's efficiency and reliability in handling orderbook snapshots and insurance fund balances. Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review Status
Actionable comments generated: 2
Configuration used: CodeRabbit UI
Files selected for processing (5)
- protocol/mocks/ClobKeeper.go (1 hunks)
- protocol/mocks/MemClob.go (1 hunks)
- protocol/x/clob/memclob/memclob_grpc_streaming.go (1 hunks)
- protocol/x/clob/memclob/memclob_grpc_streaming_test.go (1 hunks)
- protocol/x/clob/types/memclob.go (1 hunks)
Additional comments: 5
protocol/x/clob/memclob/memclob_grpc_streaming_test.go (2)
- 15-52: The test
TestGetOffchainUpdatesForOrderbookSnapshot_Buycorrectly sets up a scenario to test the generation of off-chain updates for buy orders. Observations:
Correctness: The test correctly creates an orderbook, adds orders to it, and then retrieves the off-chain updates for the orderbook snapshot. The expected updates are correctly defined based on the order of the buy orders.
Testing Best Practices: The test covers a positive scenario for buy orders. Consider adding more test cases to cover edge cases, such as when there are no buy orders or when the orderbook does not exist for the given
clobPairId.Mocking: The use of mocks for the
clobKeeperis appropriate, ensuring that the test focuses on the functionality of theGetOffchainUpdatesForOrderbookSnapshotmethod without external dependencies.The test is well-implemented but consider adding more test cases for comprehensive coverage.
- 54-91: The test
TestGetOffchainUpdatesForOrderbookSnapshot_Sellis similar to the buy orders test but focuses on sell orders. Key points:
Correctness & Completeness: This test follows the same pattern as the buy orders test and correctly verifies the generation of off-chain updates for sell orders. The expected updates are accurately defined based on the order of the sell orders.
Additional Test Cases: As with the buy orders test, consider adding tests for scenarios where there are no sell orders or the orderbook does not exist. This will ensure the method behaves correctly under all conditions.
The test is correctly implemented. Expanding the test coverage would further ensure the reliability of the functionality.
protocol/x/clob/types/memclob.go (1)
- 136-139: The addition of the
GetOffchainUpdatesForOrderbookSnapshotmethod to theMemClobinterface is a necessary update to reflect the new functionality introduced in theMemClobimplementation. This ensures that any implementation of theMemClobinterface will need to provide this functionality.
- Consistency & Architecture: Adding this method to the interface maintains consistency across the codebase and adheres to the interface segregation principle, ensuring that the
MemClobinterface accurately represents the capabilities of its implementations.The update to the interface is correctly implemented and aligns with best practices for interface design.
protocol/mocks/MemClob.go (1)
- 156-170: The addition of the
GetOffchainUpdatesForOrderbookSnapshotmethod to theMemClobmock struct is correctly implemented. It follows Go conventions and makes appropriate use of the testify/mock package for simulating method behavior in tests. This method will facilitate testing of functionalities that rely on obtaining offchain updates for the orderbook snapshot, aligning with the PR's objectives.protocol/mocks/ClobKeeper.go (1)
- 267-273: The addition of the
perpetualIdparameter to theGetInsuranceFundBalancefunction is a significant change. This update likely aims to enhance the specificity of insurance fund balance queries by associating them with a particular perpetual ID. Ensure that all calls to this function throughout the codebase have been updated to include the newperpetualIdargument to avoid runtime errors. Additionally, consider adding unit tests or updating existing ones to cover this change, ensuring that the function behaves as expected with the new parameter.
| // GetOffchainUpdatesForOrderbookSnapshot returns the offchain updates for the orderbook snapshot. | ||
| // This is used by the gRPC streaming server to send the orderbook snapshot to the client. | ||
| func (m *MemClobPriceTimePriority) GetOffchainUpdatesForOrderbookSnapshot( | ||
| ctx sdk.Context, | ||
| clobPairId types.ClobPairId, | ||
| ) (offchainUpdates *types.OffchainUpdates) { | ||
| offchainUpdates = types.NewOffchainUpdates() | ||
|
|
||
| if orderbook, exists := m.openOrders.orderbooksMap[clobPairId]; exists { | ||
| // Generate the offchain updates for buy orders. | ||
| // Updates are sorted in descending order of price. | ||
| buyPriceLevels := lib.GetSortedKeys[lib.Sortable[types.Subticks]](orderbook.Bids) | ||
| for i := len(buyPriceLevels) - 1; i >= 0; i-- { | ||
| subticks := buyPriceLevels[i] | ||
| level := orderbook.Bids[subticks] | ||
|
|
||
| // For each price level, generate offchain updates for each order in the level. | ||
| level.LevelOrders.Front.Each( | ||
| func(order types.ClobOrder) { | ||
| offchainUpdates.Append( | ||
| m.GetOffchainUpdatesForOrder(ctx, order.Order), | ||
| ) | ||
| }, | ||
| ) | ||
| } | ||
|
|
||
| // Generate the offchain updates for sell orders. | ||
| // Updates are sorted in ascending order of price. | ||
| sellPriceLevels := lib.GetSortedKeys[lib.Sortable[types.Subticks]](orderbook.Asks) | ||
| for i := 0; i < len(sellPriceLevels); i++ { | ||
| subticks := sellPriceLevels[i] | ||
| level := orderbook.Asks[subticks] | ||
|
|
||
| // For each price level, generate offchain updates for each order in the level. | ||
| level.LevelOrders.Front.Each( | ||
| func(order types.ClobOrder) { | ||
| offchainUpdates.Append( | ||
| m.GetOffchainUpdatesForOrder(ctx, order.Order), | ||
| ) | ||
| }, | ||
| ) | ||
| } | ||
| } | ||
|
|
||
| return offchainUpdates | ||
| } |
There was a problem hiding this comment.
The GetOffchainUpdatesForOrderbookSnapshot method efficiently generates off-chain updates for both buy and sell orders in the orderbook. A few observations and suggestions:
-
Correctness & Performance: The method correctly checks for the existence of the orderbook for the given
clobPairIdand iterates over buy and sell orders to generate off-chain updates. The use oflib.GetSortedKeysto sort price levels is a good practice for ensuring the updates are in the correct order (descending for buys, ascending for sells). -
Maintainability: The code is well-structured and follows a clear logic flow, making it easy to understand and maintain. However, consider extracting the logic for generating off-chain updates for buy and sell orders into separate private methods to reduce the method's length and improve readability.
-
Error Handling: Currently, the method does not handle or return any errors. Ensure that all external calls, especially those that might fail (e.g.,
m.GetOffchainUpdatesForOrder), are properly handled. Consider modifying the method signature to return an error as well and handle these gracefully. -
Testing: Ensure comprehensive tests are written to cover various scenarios, including cases where the orderbook does not exist for the given
clobPairId, and cases with different combinations of buy and sell orders.
Consider refactoring to improve readability and error handling.
| // GetOffchainUpdatesForClobOrder returns the offchain updates for the clob order. | ||
| func (m *MemClobPriceTimePriority) GetOffchainUpdatesForOrder( | ||
| ctx sdk.Context, | ||
| order types.Order, | ||
| ) (offchainUpdates *types.OffchainUpdates) { | ||
| offchainUpdates = types.NewOffchainUpdates() | ||
| orderId := order.OrderId | ||
|
|
||
| // Generate a order place message. | ||
| if message, success := off_chain_updates.CreateOrderPlaceMessage( | ||
| ctx, | ||
| order, | ||
| ); success { | ||
| offchainUpdates.AddPlaceMessage(orderId, message) | ||
| } | ||
|
|
||
| // Get the current fill amount of the order. | ||
| fillAmount := m.GetOrderFilledAmount(ctx, orderId) | ||
|
|
||
| // Generate an update message updating the total filled amount of order. | ||
| if message, success := off_chain_updates.CreateOrderUpdateMessage( | ||
| ctx, | ||
| orderId, | ||
| fillAmount, | ||
| ); success { | ||
| offchainUpdates.AddUpdateMessage(orderId, message) | ||
| } | ||
|
|
||
| return offchainUpdates | ||
| } |
There was a problem hiding this comment.
The GetOffchainUpdatesForOrder method is well-implemented, focusing on generating off-chain updates for a single order. Key points:
-
Correctness: The method correctly generates a place message and an update message for the order, using the
off_chain_updates.CreateOrderPlaceMessageandoff_chain_updates.CreateOrderUpdateMessagefunctions, respectively. -
Error Handling & Robustness: Similar to the previous method, this method does not handle potential errors from external calls. It's crucial to handle these scenarios to avoid partial updates or inconsistencies in the off-chain updates.
-
Performance: The method is efficient for generating updates for a single order. However, if used in a loop for many orders, consider the overall performance impact, especially regarding the repeated calls to
m.GetOrderFilledAmount.
Recommend adding error handling for external calls and evaluating the performance impact when used for multiple orders.
| return offchainUpdates | ||
| } | ||
|
|
||
| // GetOffchainUpdatesForClobOrder returns the offchain updates for the clob order. |
There was a problem hiding this comment.
nit: returns a place order offchain message and a update order offchain message used to construct an order for the orderbook snapshot grpc stream.
There was a problem hiding this comment.
Review Status
Actionable comments generated: 5
Configuration used: CodeRabbit UI
Files selected for processing (8)
- protocol/mocks/ClobKeeper.go (2 hunks)
- protocol/streaming/grpc/grpc_streaming_manager.go (5 hunks)
- protocol/streaming/grpc/noop_streaming_manager.go (1 hunks)
- protocol/streaming/grpc/types/manager.go (1 hunks)
- protocol/x/clob/abci.go (1 hunks)
- protocol/x/clob/keeper/keeper.go (1 hunks)
- protocol/x/clob/keeper/orders.go (1 hunks)
- protocol/x/clob/types/clob_keeper.go (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- protocol/mocks/ClobKeeper.go
Additional comments: 7
protocol/streaming/grpc/noop_streaming_manager.go (2)
- 31-31: The addition of the
snapshotparameter to theSendOrderbookUpdatesmethod in theNoopGrpcStreamingManageris consistent with the interface change. This is a good practice to ensure that the no-op implementation remains compatible with the interface it implements. However, since this is a no-op implementation, the parameter is not used within the method body, which is expected.- 35-37: The implementation of the
GetUninitializedClobPairIdsmethod in theNoopGrpcStreamingManagerreturns an empty slice, which is appropriate for a no-op scenario. This ensures that the method complies with the interface without affecting the functionality of this particular implementation. It's a clean way to handle cases where the streaming manager functionality is disabled or not needed.protocol/streaming/grpc/grpc_streaming_manager.go (3)
- 8-8: The addition of the
libpackage import is necessary for the new functionality introduced in this file, specifically for theGetSortedKeysfunction used in theGetUninitializedClobPairIdsmethod. It's important to ensure that this import is used effectively and that there are no unused imports to maintain code cleanliness.- 26-28: The introduction of the
initializefield of typesync.Oncein theOrderbookSubscriptionstruct is a thoughtful design choice. It ensures that the initialization logic for each subscription, likely involving sending an initial snapshot of the orderbook, is executed exactly once. This is crucial for avoiding duplicate initializations and ensuring the correct setup of each subscription.- 128-145: The addition of the
GetUninitializedClobPairIdsfunction is a key enhancement. It retrieves the IDs of CLOB pairs that have not been initialized, which is essential for setting up new gRPC streams with the correct state. The use of thelib.GetSortedKeysfunction to return the IDs in a sorted manner is a good practice, as it ensures consistency in the order of IDs, which can be important for deterministic processing. This method effectively leverages theinitializefield in theOrderbookSubscriptionstruct to determine which subscriptions need initialization.protocol/x/clob/keeper/keeper.go (1)
- 223-240: The implementation of the
InitializeNewGrpcStreamsfunction in theKeeperstruct is a crucial addition. It initializes new gRPC streams for all uninitialized CLOB pairs by sending the corresponding orderbook snapshots. This function plays a key role in ensuring that new streams are correctly initialized with the current state of the orderbook, which is essential for maintaining data consistency and accuracy. The method effectively leverages theGetUninitializedClobPairIdsandSendOrderbookUpdatesmethods to achieve this. Ensure that this function is called at the appropriate time during the application's lifecycle to initialize new streams as needed.protocol/x/clob/abci.go (1)
- 248-250: The addition of the call to
keeper.InitializeNewGrpcStreams(ctx)in thePrepareCheckStatefunction is a strategic placement. It ensures that new GRPC streams are initialized with orderbook snapshots at the beginning of the check state preparation. This is crucial for ensuring that any new streams created during the block processing have the correct initial state. This addition effectively leverages the new functionality introduced in theKeeperstruct to maintain the accuracy and consistency of the orderbook data across all streams.
| @@ -1312,7 +1312,7 @@ func (k Keeper) SendOffchainMessages( | |||
| k.GetIndexerEventManager().SendOffchainData(update) | |||
| } | |||
|
|
|||
| k.GetGrpcStreamingManager().SendOrderbookUpdates(offchainUpdates) | |||
| k.GetGrpcStreamingManager().SendOrderbookUpdates(offchainUpdates, false) | |||
There was a problem hiding this comment.
The addition of the SendOrderbookUpdates call with the false boolean argument in the SendOffchainMessages function introduces a new behavior for sending off-chain orderbook updates. This change seems to be part of the enhancements for handling off-chain updates more efficiently or in a specific manner. However, it's crucial to ensure that the false argument aligns with the intended behavior, especially regarding how updates are processed or filtered before being sent. If the false value disables a specific processing step or filtering, confirm that this behavior is desired in all scenarios where SendOffchainMessages is called.
Consider verifying the impact of this change on the overall system, especially in scenarios where different behaviors for sending off-chain updates might be required. If the false argument significantly alters the behavior in a way that might not be universally applicable, it might be worth exploring the possibility of making this behavior configurable or ensuring that the function's callers are aware of the implications.
- k.GetGrpcStreamingManager().SendOrderbookUpdates(offchainUpdates, false)
+ // Ensure the boolean argument passed to SendOrderbookUpdates aligns with the intended behavior.
+ // If the `false` value has specific implications, consider making this behavior configurable or
+ // clearly documenting its effects to avoid unintended consequences.
+ k.GetGrpcStreamingManager().SendOrderbookUpdates(offchainUpdates, desiredBehavior)Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| k.GetGrpcStreamingManager().SendOrderbookUpdates(offchainUpdates, false) | |
| // Ensure the boolean argument passed to SendOrderbookUpdates aligns with the intended behavior. | |
| // If the `false` value has specific implications, consider making this behavior configurable or | |
| // clearly documenting its effects to avoid unintended consequences. | |
| k.GetGrpcStreamingManager().SendOrderbookUpdates(offchainUpdates, desiredBehavior) |
| @@ -14,5 +14,9 @@ type GrpcStreamingManager interface { | |||
| ) ( | |||
| err error, | |||
| ) | |||
| SendOrderbookUpdates(*clobtypes.OffchainUpdates) | |||
| GetUninitializedClobPairIds() []uint32 | |||
There was a problem hiding this comment.
Adding the GetUninitializedClobPairIds method to the GrpcStreamingManager interface is a significant change. It implies that all implementations of this interface must now provide a mechanism to retrieve IDs of uninitialized CLOB pairs. This is crucial for ensuring that new streams can be initialized with the correct state. Ensure that all existing implementations are updated accordingly to avoid compilation errors.
| SendOrderbookUpdates( | ||
| offchainUpdates *clobtypes.OffchainUpdates, | ||
| snapshot bool, | ||
| ) |
There was a problem hiding this comment.
The modification to the SendOrderbookUpdates method to include a snapshot boolean parameter is a thoughtful addition. It allows the method to differentiate between regular updates and snapshot updates, which is essential for initializing new streams with the current state of the orderbook. This change enhances the flexibility and functionality of the gRPC streaming service. However, ensure that all calls to this method throughout the codebase are updated to include the new parameter to maintain compatibility.
protocol/x/clob/types/clob_keeper.go
Outdated
| @@ -131,4 +131,5 @@ type ClobKeeper interface { | |||
| clobPair ClobPair, | |||
| ) error | |||
| UpdateLiquidationsConfig(ctx sdk.Context, config LiquidationsConfig) error | |||
| InitializeNewGrpcStreams(ctx sdk.Context) | |||
There was a problem hiding this comment.
The addition of the InitializeNewGrpcStreams method to the ClobKeeper interface is a significant change. It implies that all implementations of this interface must now provide a mechanism to initialize new gRPC streams, likely for uninitialized CLOB pairs. This is crucial for ensuring that the system can dynamically handle new streams and maintain an up-to-date state. Ensure that all existing implementations are updated accordingly to avoid compilation errors.
| @@ -68,6 +75,7 @@ func (sm *GrpcStreamingManagerImpl) Subscribe( | |||
| // sends messages to the subscribers. | |||
| func (sm *GrpcStreamingManagerImpl) SendOrderbookUpdates( | |||
| offchainUpdates *clobtypes.OffchainUpdates, | |||
| snapshot bool, | |||
There was a problem hiding this comment.
The update to the SendOrderbookUpdates method to include a snapshot boolean parameter aligns with the interface change and enhances the method's functionality. This allows the method to distinguish between regular updates and initial snapshot updates, which is essential for initializing new streams with the correct state. Ensure that all calls to this method are updated to include the new parameter.
There was a problem hiding this comment.
Review Status
Actionable comments generated: 0
Configuration used: CodeRabbit UI
Files selected for processing (1)
- protocol/x/clob/memclob/memclob_grpc_streaming.go (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- protocol/x/clob/memclob/memclob_grpc_streaming.go
* [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (dydxprotocol#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments Signed-off-by: Eric <eric.warehime@gmail.com>
commit d98f859 Author: Eric <eric.warehime@gmail.com> Date: Mon Mar 11 12:46:53 2024 -0700 Update sample pregenesis Signed-off-by: Eric <eric.warehime@gmail.com> commit 7f178fe Author: Mohammed Affan <affanmd@nyu.edu> Date: Mon Mar 11 13:46:08 2024 -0400 [OTE-209] Emit metrics gated through execution mode (dydxprotocol#1157) Signed-off-by: Eric <eric.warehime@gmail.com> commit 47e365d Author: dydxwill <119354122+dydxwill@users.noreply.github.com> Date: Mon Mar 11 13:43:16 2024 -0400 add aggregate comlink response code stats (dydxprotocol#1162) Signed-off-by: Eric <eric.warehime@gmail.com> commit 7774ad9 Author: shrenujb <98204323+shrenujb@users.noreply.github.com> Date: Fri Mar 8 17:30:49 2024 -0500 [TRA-70] Add state migrations for isolated markets (dydxprotocol#1155) Signed-off-by: Shrenuj Bansal <shrenuj@dydx.exchange> Signed-off-by: Eric <eric.warehime@gmail.com> commit 89c7405 Author: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com> Date: Thu Mar 7 17:28:06 2024 -0500 [CT-517] E2E tests batch cancel (dydxprotocol#1149) * more end to end test * extraprint * more e2e test Signed-off-by: Eric <eric.warehime@gmail.com> commit 41a3a41 Author: Teddy Ding <teddy@dydx.exchange> Date: Thu Mar 7 15:42:30 2024 -0500 [OTE-200] OIMF protos (dydxprotocol#1125) * OIMF protos * add default genesis value, modify methods interface * fix genesis file * fix integration test * lint Signed-off-by: Eric <eric.warehime@gmail.com> commit 2a062b1 Author: Teddy Ding <teddy@dydx.exchange> Date: Thu Mar 7 15:24:15 2024 -0500 Add `v5` upgrade handler and set up container upgrade test (dydxprotocol#1153) * wip * update preupgrade_genesis * fix preupgrade_genesis.json * nit * setupUpgradeStoreLoaders for v5.0.0 Signed-off-by: Eric <eric.warehime@gmail.com> commit b7942b3 Author: jayy04 <103467857+jayy04@users.noreply.github.com> Date: Thu Mar 7 13:43:48 2024 -0500 [CT-647] construct the initial orderbook snapshot (dydxprotocol#1147) * [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (dydxprotocol#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments Signed-off-by: Eric <eric.warehime@gmail.com> commit c67a3c6 Author: shrenujb <98204323+shrenujb@users.noreply.github.com> Date: Thu Mar 7 12:40:37 2024 -0500 [TRA-84] Move SA module address transfers to use perpetual based SA accounts (dydxprotocol#1146) Signed-off-by: Shrenuj Bansal <shrenuj@dydx.exchange Signed-off-by: Eric <eric.warehime@gmail.com> commit dba23e0 Author: Mohammed Affan <affanmd@nyu.edu> Date: Thu Mar 7 10:34:11 2024 -0500 update readme link to point to the right page (dydxprotocol#1151) Signed-off-by: Eric <eric.warehime@gmail.com> commit b5870d5 Author: Tian <tian@dydx.exchange> Date: Wed Mar 6 16:43:04 2024 -0500 [TRA-86] scaffold x/vault (dydxprotocol#1148) * scaffold x/vault Signed-off-by: Eric <eric.warehime@gmail.com> commit 0eca041 Author: jayy04 <103467857+jayy04@users.noreply.github.com> Date: Wed Mar 6 10:48:42 2024 -0500 [CT-652] add command line flag for full node streaming (dydxprotocol#1145) Signed-off-by: Eric <eric.warehime@gmail.com> commit b319cb8 Author: jayy04 <103467857+jayy04@users.noreply.github.com> Date: Tue Mar 5 21:58:35 2024 -0500 [CT-646] stream offchain updates through stream manager (dydxprotocol#1138) * [CT-646] stream offchain updates through stream manager * comments * fix lint * get rid of finished * comments * comments Signed-off-by: Eric <eric.warehime@gmail.com> commit 1c54620 Author: shrenujb <98204323+shrenujb@users.noreply.github.com> Date: Tue Mar 5 16:34:19 2024 -0500 [TRA-78] Add function to retrieve collateral pool addr for a subaccount (dydxprotocol#1142) Signed-off-by: Shrenuj Bansal <shrenuj@dydx.exchange> Signed-off-by: Eric <eric.warehime@gmail.com> commit b8c1d62 Author: dydxwill <119354122+dydxwill@users.noreply.github.com> Date: Tue Mar 5 15:03:28 2024 -0500 [OTE-141] implement post /compliance/geoblock (dydxprotocol#1129) Signed-off-by: Eric <eric.warehime@gmail.com> commit ab8c570 Author: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com> Date: Tue Mar 5 11:19:53 2024 -0500 Fix mock-gen dydxprotocol#1140 Signed-off-by: Eric <eric.warehime@gmail.com> commit 12506a1 Author: shrenujb <98204323+shrenujb@users.noreply.github.com> Date: Mon Mar 4 21:33:28 2024 -0500 [TRA-64] Use market specific insurance fund for cross or isolated markets (dydxprotocol#1132) Signed-off-by: Shrenuj Bansal <shrenuj@dydx.exchange> Signed-off-by: Eric <eric.warehime@gmail.com> commit 929f09e Author: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com> Date: Mon Mar 4 13:48:04 2024 -0800 [CT-514] Clob `MsgBatchCancel` functionality (dydxprotocol#1110) * wip implementation * use new cometbft * Revert "use new cometbft" This reverts commit e5b8a03. * go mod tidy * basic e2e test * more msgBatchCancels in code * repeated fixed32 -> uint32 * remove debug prints * update cometbft replace go.mod sha * one more debug print * typo * regen indexer protos * update comment on proto * proto comment changes * extract stateful validation into own fn * pr format comments * clean up test file * new return type with success and failure Signed-off-by: Eric <eric.warehime@gmail.com> commit 41de83e Author: dydxwill <119354122+dydxwill@users.noreply.github.com> Date: Mon Mar 4 12:22:16 2024 -0500 add index to address read replica lag (dydxprotocol#1137) Signed-off-by: Eric <eric.warehime@gmail.com> commit 735d9a8 Author: dydxwill <119354122+dydxwill@users.noreply.github.com> Date: Mon Mar 4 11:56:59 2024 -0500 rename (dydxprotocol#1136) Signed-off-by: Eric <eric.warehime@gmail.com> commit 86617dd Author: jayy04 <103467857+jayy04@users.noreply.github.com> Date: Mon Mar 4 10:43:31 2024 -0500 [CT-644] instantiate grpc stream manager (dydxprotocol#1134) * [CT-644] instantiate grpc stream manager * update type * update channel type Signed-off-by: Eric <eric.warehime@gmail.com> commit 32afd64 Author: Eric <eric.warehime@gmail.com> Date: Mon Mar 11 12:41:06 2024 -0700 Update go version in Dockerfile Signed-off-by: Eric <eric.warehime@gmail.com> commit ba27204 Author: Eric <eric.warehime@gmail.com> Date: Fri Mar 8 09:44:04 2024 -0800 Add slinky utils, use that to convert between market and currency pair commit 667a804 Author: Eric <eric.warehime@gmail.com> Date: Wed Mar 6 20:43:40 2024 -0800 Update error messages commit d53292c Author: Eric <eric.warehime@gmail.com> Date: Wed Mar 6 20:16:01 2024 -0800 Update docstrings, rename OracleClient commit daad125 Author: Eric <eric.warehime@gmail.com> Date: Mon Mar 4 10:51:23 2024 -0800 VoteExtension slinky logic
* [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments
* [CT-645] Move off chain updates and v1 to a different package (#1131) * [CT-645] Add protos for orderbook stream query service * move removal reasons to a separate package * [CT-645] Add protos for orderbook stream query service (#1133) * [CT-645] Add protos for orderbook stream query service * make update not nullable * fix build * [CT-644] instantiate grpc stream manager (#1134) * [CT-644] instantiate grpc stream manager * update type * update channel type * [CT-646] stream offchain updates through stream manager (#1138) * [CT-646] stream offchain updates through stream manager * comments * fix lint * get rid of finished * comments * comments * [CT-652] add command line flag for full node streaming (#1145) * [CT-647] construct the initial orderbook snapshot (#1147) * [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments * fix test
* [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments
* [CT-645] Move off chain updates and v1 to a different package (#1131) * [CT-645] Add protos for orderbook stream query service * move removal reasons to a separate package * [CT-645] Add protos for orderbook stream query service (#1133) * [CT-645] Add protos for orderbook stream query service * make update not nullable * fix build * [CT-644] instantiate grpc stream manager (#1134) * [CT-644] instantiate grpc stream manager * update type * update channel type * [CT-646] stream offchain updates through stream manager (#1138) * [CT-646] stream offchain updates through stream manager * comments * fix lint * get rid of finished * comments * comments * [CT-652] add command line flag for full node streaming (#1145) * [CT-647] construct the initial orderbook snapshot (#1147) * [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments * [CT-700] separate indexer and grpc streaming events (#1209) * [CT-700] separate indexer and grpc streaming events * fix tests * comments * update * [CT-700] only send response when there is at least one update (#1216) * [CT-712] send order update when short term order state fill amounts are pruned (#1241) * [CT-712] send fill amount updates for reverted operations (#1240) * [CT-723] add block number + stage to grpc updates (#1252) * [CT-723] add block number + stage to grpc updates * add indexer changes * [CT-727] avoid state reads when sending updates (#1261)
* [CT-645] Move off chain updates and v1 to a different package (#1131) * [CT-645] Add protos for orderbook stream query service * move removal reasons to a separate package * [CT-645] Add protos for orderbook stream query service (#1133) * [CT-645] Add protos for orderbook stream query service * make update not nullable * fix build * [CT-644] instantiate grpc stream manager (#1134) * [CT-644] instantiate grpc stream manager * update type * update channel type * [CT-646] stream offchain updates through stream manager (#1138) * [CT-646] stream offchain updates through stream manager * comments * fix lint * get rid of finished * comments * comments * [CT-652] add command line flag for full node streaming (#1145) * [CT-647] construct the initial orderbook snapshot (#1147) * [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments * [CT-700] separate indexer and grpc streaming events (#1209) * [CT-700] separate indexer and grpc streaming events * fix tests * comments * update * [CT-700] only send response when there is at least one update (#1216) * [CT-712] send order update when short term order state fill amounts are pruned (#1241) * [CT-712] send fill amount updates for reverted operations (#1240) * [CT-723] add block number + stage to grpc updates (#1252) * [CT-723] add block number + stage to grpc updates * add indexer changes * [CT-727] avoid state reads when sending updates (#1261)
* [OTE-221] Add query for PendingSendPacket (backport #1176) (#1221) --------- Co-authored-by: Teddy Ding <teddy@dydx.exchange> (cherry picked from commit e545bbf) # Conflicts: # indexer/packages/v4-protos/src/codegen/dydxprotocol/bundle.ts # indexer/packages/v4-protos/src/codegen/gogoproto/bundle.ts # indexer/packages/v4-protos/src/codegen/google/bundle.ts # protocol/go.mod * fix protos * update go.mod --------- Co-authored-by: Mohammed Affan <affanmd@nyu.edu> Co-authored-by: affan <affan@dydx.exchange> * [Backport v4.x] backport full node streaming to v4.x branch (#1270) * [CT-645] Move off chain updates and v1 to a different package (#1131) * [CT-645] Add protos for orderbook stream query service * move removal reasons to a separate package * [CT-645] Add protos for orderbook stream query service (#1133) * [CT-645] Add protos for orderbook stream query service * make update not nullable * fix build * [CT-644] instantiate grpc stream manager (#1134) * [CT-644] instantiate grpc stream manager * update type * update channel type * [CT-646] stream offchain updates through stream manager (#1138) * [CT-646] stream offchain updates through stream manager * comments * fix lint * get rid of finished * comments * comments * [CT-652] add command line flag for full node streaming (#1145) * [CT-647] construct the initial orderbook snapshot (#1147) * [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments * [CT-700] separate indexer and grpc streaming events (#1209) * [CT-700] separate indexer and grpc streaming events * fix tests * comments * update * [CT-700] only send response when there is at least one update (#1216) * [CT-712] send order update when short term order state fill amounts are pruned (#1241) * [CT-712] send fill amount updates for reverted operations (#1240) * [CT-723] add block number + stage to grpc updates (#1252) * [CT-723] add block number + stage to grpc updates * add indexer changes * [CT-727] avoid state reads when sending updates (#1261) * [CT-712] send updates for both normal order matches and liquidation (#1280) (#1281) * Fix lib.ErrorLogWithError: properly pass in args (#1306) (#1310) (cherry picked from commit a91c1ca) Co-authored-by: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com> * fix broken tests (#1312) (#1316) (cherry picked from commit 5ec37d2) Co-authored-by: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com> --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Mohammed Affan <affanmd@nyu.edu> Co-authored-by: affan <affan@dydx.exchange> Co-authored-by: jayy04 <103467857+jayy04@users.noreply.github.com> Co-authored-by: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com>
Changelist
[Describe or list the changes made in this PR]
Test Plan
[Describe how this PR was tested (if applicable)]
Author/Reviewer Checklist
state-breakinglabel.indexer-postgres-breakinglabel.PrepareProposalorProcessProposal, manually add the labelproposal-breaking.feature:[feature-name].backport/[branch-name].refactor,chore,bug.