Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log more info with max seq no on upload to and download from remote translog #10973

Merged
merged 4 commits into from
Oct 30, 2023

Conversation

sachinpkale
Copy link
Member

@sachinpkale sachinpkale commented Oct 28, 2023

Description

  • In order to debug data consistency issues with remote store, we need to log info related to last successful upload to remote translog and the download from remote translog per shard level.
  • In this PR, we track this info using sequence numbers.

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ranslog

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Oct 28, 2023

Compatibility status:

Checks if related components are compatible with change 32f631d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@sachinpkale sachinpkale requested a review from Bukhtawar October 30, 2023 06:45
@ashking94
Copy link
Member

@sachinpkale Pull Request Checks is failing.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@codecov
Copy link

codecov bot commented Oct 30, 2023

Codecov Report

Merging #10973 (32f631d) into main (fcbec24) will increase coverage by 0.02%.
Report is 6 commits behind head on main.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main   #10973      +/-   ##
============================================
+ Coverage     71.20%   71.22%   +0.02%     
- Complexity    58702    58731      +29     
============================================
  Files          4870     4870              
  Lines        276539   276545       +6     
  Branches      40190    40191       +1     
============================================
+ Hits         196903   196976      +73     
+ Misses        63190    63164      -26     
+ Partials      16446    16405      -41     
Files Coverage Δ
...rg/opensearch/index/translog/RemoteFsTranslog.java 76.30% <100.00%> (+0.69%) ⬆️

... and 453 files with indirect coverage changes

@sachinpkale sachinpkale merged commit 7c917c5 into opensearch-project:main Oct 30, 2023
16 of 17 checks passed
@sachinpkale sachinpkale added the backport 2.x Backport to 2.x branch label Oct 30, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 30, 2023
…ranslog (#10973)

* Log more info with max seq no on upload to and download from remote translog

Signed-off-by: Sachin Kale <kalsac@amazon.com>

* Change log level to debug

Signed-off-by: Sachin Kale <kalsac@amazon.com>

* Spotless fixes

Signed-off-by: Sachin Kale <kalsac@amazon.com>

* Fix UTs

Signed-off-by: Sachin Kale <kalsac@amazon.com>

---------

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
(cherry picked from commit 7c917c5)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
gbbafna pushed a commit that referenced this pull request Oct 30, 2023
…ranslog (#10973) (#10995)

* Log more info with max seq no on upload to and download from remote translog



* Change log level to debug



* Spotless fixes



* Fix UTs



---------



(cherry picked from commit 7c917c5)

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…ranslog (opensearch-project#10973)

* Log more info with max seq no on upload to and download from remote translog

Signed-off-by: Sachin Kale <kalsac@amazon.com>

* Change log level to debug

Signed-off-by: Sachin Kale <kalsac@amazon.com>

* Spotless fixes

Signed-off-by: Sachin Kale <kalsac@amazon.com>

* Fix UTs

Signed-off-by: Sachin Kale <kalsac@amazon.com>

---------

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants