-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CT-652] add command line flag for full node streaming #1145
Conversation
WalkthroughThe recent updates focus on integrating GRPC streaming capabilities into the application. A new flag, Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Review Status
Actionable comments generated: 1
Configuration used: CodeRabbit UI
Files selected for processing (3)
- protocol/app/app.go (2 hunks)
- protocol/app/flags/flags.go (7 hunks)
- protocol/app/flags/flags_test.go (5 hunks)
Additional comments: 9
protocol/app/flags/flags.go (6)
- 23-24: The addition of the
GrpcStreamingEnabled
field to theFlags
struct is clear and follows the existing structure. This change correctly introduces the new flag into the application's configuration.- 38-39: The definition of the
GrpcStreamingEnabled
constant is consistent with the naming conventions used for other flags in the application. This ensures a uniform approach to flag naming.- 49-49: The introduction of
DefaultGrpcStreamingEnabled
as a constant for the default value of theGrpcStreamingEnabled
flag is a good practice. It enhances maintainability by centralizing the default value, making it easier to update if needed.- 78-82: Adding the
GrpcStreamingEnabled
flag to the CLI flags within theAddFlagsToCmd
function is implemented correctly. The description provided for the flag is clear and informative, helping users understand its purpose.- 92-101: The updates to the
Validate
function to handle theGrpcStreamingEnabled
flag are logically sound. The validation ensures that GRPC streaming can only be enabled when bothGrpcEnable
is true and the node is a non-validating full node. This prevents potential misconfigurations that could lead to unexpected behavior.- 121-121: The addition of the
GrpcStreamingEnabled
flag to theGetFlagValuesFromOptions
function correctly retrieves the flag value from the command-line options and sets it in theFlags
struct. This ensures that the application can correctly interpret and use the flag value based on user input.protocol/app/flags/flags_test.go (3)
- 32-34: The addition of a test case for the
GrpcStreamingEnabled
flag in theTestAddFlagsToCommand
function is correctly implemented. This test ensures that the flag is properly added to the command-line interface.- 70-85: The new test cases added to the
TestValidate
function for theGrpcStreamingEnabled
flag are comprehensive. They cover scenarios where the flag is enabled under different conditions, ensuring that the validation logic works as expected.- 107-121: > 📝 NOTE
This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [110-134]
The updates to the
TestGetFlagValuesFromOptions
function to include theGrpcStreamingEnabled
flag are correctly implemented. These tests verify that the flag's value is correctly retrieved from the options and set in theFlags
struct, ensuring the application can use the flag as intended.
if appFlags.GrpcStreamingEnabled { | ||
logger.Info("GRPC streaming is enabled") | ||
return streaming.NewGrpcStreamingManager() | ||
} | ||
return streaming.NewNoopGrpcStreamingManager() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The gRPC server is initialized without specifying credentials, which could lead to insecure communication. It's important to ensure that communication between the client and server is encrypted to prevent potential man-in-the-middle attacks. Consider using grpc.Creds()
with credentials derived from an SSL/TLS certificate to create a secure gRPC connection. For example:
import "google.golang.org/grpc/credentials"
creds, err := credentials.NewServerTLSFromFile("cert.pem", "cert.key")
if err != nil {
log.Fatalf("Failed to generate credentials %v", err)
}
grpcServer := grpc.NewServer(grpc.Creds(creds))
This change will help in securing the gRPC server by enabling TLS encryption.
…1145) Signed-off-by: Eric <eric.warehime@gmail.com>
commit d98f859 Author: Eric <eric.warehime@gmail.com> Date: Mon Mar 11 12:46:53 2024 -0700 Update sample pregenesis Signed-off-by: Eric <eric.warehime@gmail.com> commit 7f178fe Author: Mohammed Affan <affanmd@nyu.edu> Date: Mon Mar 11 13:46:08 2024 -0400 [OTE-209] Emit metrics gated through execution mode (dydxprotocol#1157) Signed-off-by: Eric <eric.warehime@gmail.com> commit 47e365d Author: dydxwill <119354122+dydxwill@users.noreply.github.com> Date: Mon Mar 11 13:43:16 2024 -0400 add aggregate comlink response code stats (dydxprotocol#1162) Signed-off-by: Eric <eric.warehime@gmail.com> commit 7774ad9 Author: shrenujb <98204323+shrenujb@users.noreply.github.com> Date: Fri Mar 8 17:30:49 2024 -0500 [TRA-70] Add state migrations for isolated markets (dydxprotocol#1155) Signed-off-by: Shrenuj Bansal <shrenuj@dydx.exchange> Signed-off-by: Eric <eric.warehime@gmail.com> commit 89c7405 Author: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com> Date: Thu Mar 7 17:28:06 2024 -0500 [CT-517] E2E tests batch cancel (dydxprotocol#1149) * more end to end test * extraprint * more e2e test Signed-off-by: Eric <eric.warehime@gmail.com> commit 41a3a41 Author: Teddy Ding <teddy@dydx.exchange> Date: Thu Mar 7 15:42:30 2024 -0500 [OTE-200] OIMF protos (dydxprotocol#1125) * OIMF protos * add default genesis value, modify methods interface * fix genesis file * fix integration test * lint Signed-off-by: Eric <eric.warehime@gmail.com> commit 2a062b1 Author: Teddy Ding <teddy@dydx.exchange> Date: Thu Mar 7 15:24:15 2024 -0500 Add `v5` upgrade handler and set up container upgrade test (dydxprotocol#1153) * wip * update preupgrade_genesis * fix preupgrade_genesis.json * nit * setupUpgradeStoreLoaders for v5.0.0 Signed-off-by: Eric <eric.warehime@gmail.com> commit b7942b3 Author: jayy04 <103467857+jayy04@users.noreply.github.com> Date: Thu Mar 7 13:43:48 2024 -0500 [CT-647] construct the initial orderbook snapshot (dydxprotocol#1147) * [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (dydxprotocol#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments Signed-off-by: Eric <eric.warehime@gmail.com> commit c67a3c6 Author: shrenujb <98204323+shrenujb@users.noreply.github.com> Date: Thu Mar 7 12:40:37 2024 -0500 [TRA-84] Move SA module address transfers to use perpetual based SA accounts (dydxprotocol#1146) Signed-off-by: Shrenuj Bansal <shrenuj@dydx.exchange Signed-off-by: Eric <eric.warehime@gmail.com> commit dba23e0 Author: Mohammed Affan <affanmd@nyu.edu> Date: Thu Mar 7 10:34:11 2024 -0500 update readme link to point to the right page (dydxprotocol#1151) Signed-off-by: Eric <eric.warehime@gmail.com> commit b5870d5 Author: Tian <tian@dydx.exchange> Date: Wed Mar 6 16:43:04 2024 -0500 [TRA-86] scaffold x/vault (dydxprotocol#1148) * scaffold x/vault Signed-off-by: Eric <eric.warehime@gmail.com> commit 0eca041 Author: jayy04 <103467857+jayy04@users.noreply.github.com> Date: Wed Mar 6 10:48:42 2024 -0500 [CT-652] add command line flag for full node streaming (dydxprotocol#1145) Signed-off-by: Eric <eric.warehime@gmail.com> commit b319cb8 Author: jayy04 <103467857+jayy04@users.noreply.github.com> Date: Tue Mar 5 21:58:35 2024 -0500 [CT-646] stream offchain updates through stream manager (dydxprotocol#1138) * [CT-646] stream offchain updates through stream manager * comments * fix lint * get rid of finished * comments * comments Signed-off-by: Eric <eric.warehime@gmail.com> commit 1c54620 Author: shrenujb <98204323+shrenujb@users.noreply.github.com> Date: Tue Mar 5 16:34:19 2024 -0500 [TRA-78] Add function to retrieve collateral pool addr for a subaccount (dydxprotocol#1142) Signed-off-by: Shrenuj Bansal <shrenuj@dydx.exchange> Signed-off-by: Eric <eric.warehime@gmail.com> commit b8c1d62 Author: dydxwill <119354122+dydxwill@users.noreply.github.com> Date: Tue Mar 5 15:03:28 2024 -0500 [OTE-141] implement post /compliance/geoblock (dydxprotocol#1129) Signed-off-by: Eric <eric.warehime@gmail.com> commit ab8c570 Author: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com> Date: Tue Mar 5 11:19:53 2024 -0500 Fix mock-gen dydxprotocol#1140 Signed-off-by: Eric <eric.warehime@gmail.com> commit 12506a1 Author: shrenujb <98204323+shrenujb@users.noreply.github.com> Date: Mon Mar 4 21:33:28 2024 -0500 [TRA-64] Use market specific insurance fund for cross or isolated markets (dydxprotocol#1132) Signed-off-by: Shrenuj Bansal <shrenuj@dydx.exchange> Signed-off-by: Eric <eric.warehime@gmail.com> commit 929f09e Author: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com> Date: Mon Mar 4 13:48:04 2024 -0800 [CT-514] Clob `MsgBatchCancel` functionality (dydxprotocol#1110) * wip implementation * use new cometbft * Revert "use new cometbft" This reverts commit e5b8a03. * go mod tidy * basic e2e test * more msgBatchCancels in code * repeated fixed32 -> uint32 * remove debug prints * update cometbft replace go.mod sha * one more debug print * typo * regen indexer protos * update comment on proto * proto comment changes * extract stateful validation into own fn * pr format comments * clean up test file * new return type with success and failure Signed-off-by: Eric <eric.warehime@gmail.com> commit 41de83e Author: dydxwill <119354122+dydxwill@users.noreply.github.com> Date: Mon Mar 4 12:22:16 2024 -0500 add index to address read replica lag (dydxprotocol#1137) Signed-off-by: Eric <eric.warehime@gmail.com> commit 735d9a8 Author: dydxwill <119354122+dydxwill@users.noreply.github.com> Date: Mon Mar 4 11:56:59 2024 -0500 rename (dydxprotocol#1136) Signed-off-by: Eric <eric.warehime@gmail.com> commit 86617dd Author: jayy04 <103467857+jayy04@users.noreply.github.com> Date: Mon Mar 4 10:43:31 2024 -0500 [CT-644] instantiate grpc stream manager (dydxprotocol#1134) * [CT-644] instantiate grpc stream manager * update type * update channel type Signed-off-by: Eric <eric.warehime@gmail.com> commit 32afd64 Author: Eric <eric.warehime@gmail.com> Date: Mon Mar 11 12:41:06 2024 -0700 Update go version in Dockerfile Signed-off-by: Eric <eric.warehime@gmail.com> commit ba27204 Author: Eric <eric.warehime@gmail.com> Date: Fri Mar 8 09:44:04 2024 -0800 Add slinky utils, use that to convert between market and currency pair commit 667a804 Author: Eric <eric.warehime@gmail.com> Date: Wed Mar 6 20:43:40 2024 -0800 Update error messages commit d53292c Author: Eric <eric.warehime@gmail.com> Date: Wed Mar 6 20:16:01 2024 -0800 Update docstrings, rename OracleClient commit daad125 Author: Eric <eric.warehime@gmail.com> Date: Mon Mar 4 10:51:23 2024 -0800 VoteExtension slinky logic
* [CT-645] Move off chain updates and v1 to a different package (#1131) * [CT-645] Add protos for orderbook stream query service * move removal reasons to a separate package * [CT-645] Add protos for orderbook stream query service (#1133) * [CT-645] Add protos for orderbook stream query service * make update not nullable * fix build * [CT-644] instantiate grpc stream manager (#1134) * [CT-644] instantiate grpc stream manager * update type * update channel type * [CT-646] stream offchain updates through stream manager (#1138) * [CT-646] stream offchain updates through stream manager * comments * fix lint * get rid of finished * comments * comments * [CT-652] add command line flag for full node streaming (#1145) * [CT-647] construct the initial orderbook snapshot (#1147) * [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments * fix test
* [CT-645] Move off chain updates and v1 to a different package (#1131) * [CT-645] Add protos for orderbook stream query service * move removal reasons to a separate package * [CT-645] Add protos for orderbook stream query service (#1133) * [CT-645] Add protos for orderbook stream query service * make update not nullable * fix build * [CT-644] instantiate grpc stream manager (#1134) * [CT-644] instantiate grpc stream manager * update type * update channel type * [CT-646] stream offchain updates through stream manager (#1138) * [CT-646] stream offchain updates through stream manager * comments * fix lint * get rid of finished * comments * comments * [CT-652] add command line flag for full node streaming (#1145) * [CT-647] construct the initial orderbook snapshot (#1147) * [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments * [CT-700] separate indexer and grpc streaming events (#1209) * [CT-700] separate indexer and grpc streaming events * fix tests * comments * update * [CT-700] only send response when there is at least one update (#1216) * [CT-712] send order update when short term order state fill amounts are pruned (#1241) * [CT-712] send fill amount updates for reverted operations (#1240) * [CT-723] add block number + stage to grpc updates (#1252) * [CT-723] add block number + stage to grpc updates * add indexer changes * [CT-727] avoid state reads when sending updates (#1261)
* [CT-645] Move off chain updates and v1 to a different package (#1131) * [CT-645] Add protos for orderbook stream query service * move removal reasons to a separate package * [CT-645] Add protos for orderbook stream query service (#1133) * [CT-645] Add protos for orderbook stream query service * make update not nullable * fix build * [CT-644] instantiate grpc stream manager (#1134) * [CT-644] instantiate grpc stream manager * update type * update channel type * [CT-646] stream offchain updates through stream manager (#1138) * [CT-646] stream offchain updates through stream manager * comments * fix lint * get rid of finished * comments * comments * [CT-652] add command line flag for full node streaming (#1145) * [CT-647] construct the initial orderbook snapshot (#1147) * [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments * [CT-700] separate indexer and grpc streaming events (#1209) * [CT-700] separate indexer and grpc streaming events * fix tests * comments * update * [CT-700] only send response when there is at least one update (#1216) * [CT-712] send order update when short term order state fill amounts are pruned (#1241) * [CT-712] send fill amount updates for reverted operations (#1240) * [CT-723] add block number + stage to grpc updates (#1252) * [CT-723] add block number + stage to grpc updates * add indexer changes * [CT-727] avoid state reads when sending updates (#1261)
* [OTE-221] Add query for PendingSendPacket (backport #1176) (#1221) --------- Co-authored-by: Teddy Ding <teddy@dydx.exchange> (cherry picked from commit e545bbf) # Conflicts: # indexer/packages/v4-protos/src/codegen/dydxprotocol/bundle.ts # indexer/packages/v4-protos/src/codegen/gogoproto/bundle.ts # indexer/packages/v4-protos/src/codegen/google/bundle.ts # protocol/go.mod * fix protos * update go.mod --------- Co-authored-by: Mohammed Affan <affanmd@nyu.edu> Co-authored-by: affan <affan@dydx.exchange> * [Backport v4.x] backport full node streaming to v4.x branch (#1270) * [CT-645] Move off chain updates and v1 to a different package (#1131) * [CT-645] Add protos for orderbook stream query service * move removal reasons to a separate package * [CT-645] Add protos for orderbook stream query service (#1133) * [CT-645] Add protos for orderbook stream query service * make update not nullable * fix build * [CT-644] instantiate grpc stream manager (#1134) * [CT-644] instantiate grpc stream manager * update type * update channel type * [CT-646] stream offchain updates through stream manager (#1138) * [CT-646] stream offchain updates through stream manager * comments * fix lint * get rid of finished * comments * comments * [CT-652] add command line flag for full node streaming (#1145) * [CT-647] construct the initial orderbook snapshot (#1147) * [CT-647] construct the initial orderbook snapshot * [CT-647] initialize new streams and send orderbook snapshot (#1152) * [CT-647] initialize new streams and send orderbook snapshot * use sync once * comments * [CT-700] separate indexer and grpc streaming events (#1209) * [CT-700] separate indexer and grpc streaming events * fix tests * comments * update * [CT-700] only send response when there is at least one update (#1216) * [CT-712] send order update when short term order state fill amounts are pruned (#1241) * [CT-712] send fill amount updates for reverted operations (#1240) * [CT-723] add block number + stage to grpc updates (#1252) * [CT-723] add block number + stage to grpc updates * add indexer changes * [CT-727] avoid state reads when sending updates (#1261) * [CT-712] send updates for both normal order matches and liquidation (#1280) (#1281) * Fix lib.ErrorLogWithError: properly pass in args (#1306) (#1310) (cherry picked from commit a91c1ca) Co-authored-by: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com> * fix broken tests (#1312) (#1316) (cherry picked from commit 5ec37d2) Co-authored-by: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com> --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Mohammed Affan <affanmd@nyu.edu> Co-authored-by: affan <affan@dydx.exchange> Co-authored-by: jayy04 <103467857+jayy04@users.noreply.github.com> Co-authored-by: Jonathan Fung <121899091+jonfung-dydx@users.noreply.github.com>
Changelist
[Describe or list the changes made in this PR]
Test Plan
[Describe how this PR was tested (if applicable)]
Author/Reviewer Checklist
state-breaking
label.indexer-postgres-breaking
label.PrepareProposal
orProcessProposal
, manually add the labelproposal-breaking
.feature:[feature-name]
.backport/[branch-name]
.refactor
,chore
,bug
.