Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove excessive logging with SegRep using Remote store #8452

Closed
ankitkala opened this issue Jul 5, 2023 · 1 comment · Fixed by #8767 or #9323
Closed

Remove excessive logging with SegRep using Remote store #8452

ankitkala opened this issue Jul 5, 2023 · 1 comment · Fixed by #8767 or #9323
Assignees
@dreamer-89
Labels
bug Something isn't working distributed framework Indexing:Replication Issues and PRs related to core replication framework eg segrep Storage Issues and PRs relating to data and metadata storage

Comments

@ankitkala
Copy link
Member

For segment replication using remote store, with each refresh, we log the

  1. segment files present on remote store
  2. segment files present on local shard
  3. segment fails downloaded/skipped

This create a lots of logs per shard per few seconds(with refresh). I think we should keep this enabled while the feature is behind experimental flag. This issue is to track the change to clean up these logs(or move to debug) before GA.
@ankitkala ankitkala added bug Something isn't working untriaged labels Jul 5, 2023
@dreamer-89 dreamer-89 linked a pull request Jul 19, 2023 that will close this issue
[Segment Replication] Refactor RemoteStoreReplicationSource #8767
Merged
6 tasks
@dreamer-89 dreamer-89 mentioned this issue Jul 19, 2023
Merged
6 tasks
@dreamer-89
Copy link
Member

PR #8767 refactors the RemoteStoreReplicationSource and does not call IndexShard.syncSegmentsFromRemoteSegmentStore anymore which contains the log statements. PR #8767 relies on replication source to perform this copy operation.

@dreamer-89 dreamer-89 self-assigned this Jul 20, 2023
@anasalkouz anasalkouz moved this from Todo to In Progress in Segment Replication Jul 27, 2023
@Bukhtawar Bukhtawar added Indexing:Replication Issues and PRs related to core replication framework eg segrep Storage Issues and PRs relating to data and metadata storage labels Jul 27, 2023
@github-project-automation github-project-automation bot moved this from In Progress to Done in Segment Replication Aug 2, 2023
@mch2 mch2 mentioned this issue Aug 14, 2023
Merged
6 tasks
@mch2 mch2 reopened this Aug 14, 2023
@github-project-automation github-project-automation bot moved this from Done to In Progress in Segment Replication Aug 14, 2023
@mch2 mch2 removed the untriaged label Aug 14, 2023
@github-project-automation github-project-automation bot moved this from In Progress to Done in Segment Replication Aug 15, 2023
@mch2 mch2 mentioned this issue Aug 18, 2023
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dreamer-89 dreamer-89

Labels
bug Something isn't working distributed framework Indexing:Replication Issues and PRs related to core replication framework eg segrep Storage Issues and PRs relating to data and metadata storage
Projects
Segment Replication
Status: Done
Milestone
No milestone
5 participants
@mch2 @ankitkala @Bukhtawar @anasalkouz @dreamer-89