Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-12165. Refactor VolumeInfoMetrics to use getCurrentUsage #7784

Merged
merged 1 commit into from
Feb 1, 2025

Conversation

adoroszlai
Copy link
Contributor

What changes were proposed in this pull request?

  1. Refactor VolumeInfoMetrics to fetch volume usage with a single call, instead of getting individual values separately.
  2. Fix wrong Capacity and TotalCapacity. Used + Available != Capacity, because Used only includes the space used by the HDDS data dir. Capacity is already queried from the filesystem, so we should display that.

https://issues.apache.org/jira/browse/HDDS-12165

How was this patch tested?

Started ozone compose cluster, created keys using Freon.

Checked Datanode web UI, verified that volume info is present.

Before:

before

After:

after

Checked metrics in JMX endpoint, verified that metrics keys are the same as before. Values are similar, except capacity, which is correct with the patch (TotalCapacity matches the value reported by df (times 1024, since it reports 1K blocks)).

Before:

{
  "name" : "Hadoop:service=HddsDatanode,name=VolumeInfoMetrics-/data/hdds",
  "modelerType" : "VolumeInfoMetrics-/data/hdds",
  "tag.Context" : "ozone",
  "tag.StorageType" : "DISK",
  "tag.DatanodeUuid" : "9ab8037b-59f4-4f99-b762-8a33649c6f11",
  "tag.StorageDirectory" : "/data/hdds/hdds",
  "tag.VolumeState" : "NORMAL",
  "tag.VolumeType" : "DATA_VOLUME",
  "tag.Hostname" : "0f71994a6169",
  "DbCompactLatencyNumOps" : 0,
  "DbCompactLatencyAvgTime" : 0.0,
  "TotalCapacity" : 49547334340,
  "Capacity" : 49497374720,
  "Available" : 49392513024,
  "Containers" : 1,
  "LayoutVersion" : 1,
  "Used" : 104861696,
  "Committed" : 968884224,
  "Reserved" : 49959620
}

After:

{
  "name" : "Hadoop:service=HddsDatanode,name=VolumeInfoMetrics-/data/hdds",
  "modelerType" : "VolumeInfoMetrics-/data/hdds",
  "tag.Context" : "ozone",
  "tag.StorageType" : "DISK",
  "tag.DatanodeUuid" : "1bbe31d4-9dd9-4fcf-93aa-fd4717876414",
  "tag.VolumeState" : "NORMAL",
  "tag.VolumeType" : "DATA_VOLUME",
  "tag.StorageDirectory" : "/data/hdds/hdds",
  "tag.Hostname" : "f1c5efee2f91",
  "DbCompactLatencyNumOps" : 0,
  "DbCompactLatencyAvgTime" : 0.0,
  "Committed" : 968884224,
  "LayoutVersion" : 1,
  "Containers" : 1,
  "Capacity" : 499546271036,
  "Available" : 49390276608,
  "Used" : 104861696,
  "Reserved" : 49959620,
  "TotalCapacity" : 499596230656
}

Copy link
Contributor

@ashishkumar50 ashishkumar50 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adoroszlai, Thanks for the refactor, LGTM.

@adoroszlai adoroszlai merged commit 80426c5 into apache:master Feb 1, 2025
42 checks passed
@adoroszlai
Copy link
Contributor Author

Thanks @ashishkumar50 for the review.

@adoroszlai adoroszlai deleted the HDDS-12165 branch February 1, 2025 07:23
errose28 added a commit to errose28/ozone that referenced this pull request Feb 5, 2025
* master: (168 commits)
  HDDS-12112. Fix interval used for Chunk Read/Write Dashboard (apache#7724)
  HDDS-12212. Fix grammar in decommissioning and observability documentation (apache#7815)
  HDDS-12195. Implement skip() in OzoneFSInputStream (apache#7801)
  HDDS-12200. Fix grammar in OM HA, EC and Snapshot doc (apache#7806)
  HDDS-12202. OpsCreate and OpsAppend metrics not incremented (apache#7811)
  HDDS-12203. Initialize block length before skip (apache#7809)
  HDDS-12183. Reuse cluster across safe test classes (apache#7793)
  HDDS-11714. resetDeletedBlockRetryCount with --all may fail and can cause long db lock in large cluster. (apache#7665)
  HDDS-12186. (addendum) Avoid array allocation for table iterator (apache#7799)
  HDDS-12186. Avoid array allocation for table iterator. (apache#7797)
  HDDS-11508. Decouple delete batch limits from Ratis request size for DirectoryDeletingService. (apache#7365)
  HDDS-12073. Don't show Source Bucket and Volume if null in DU metadata (apache#7760)
  HDDS-12142. Save logs from build check (apache#7782)
  HDDS-12163. Reduce number of individual getCapacity/getAvailable/getUsedSpace calls (apache#7790)
  HDDS-12176. Trivial dependency cleanup.(apache#7787)
  HDDS-12181. Bump jline to 3.29.0 (apache#7789)
  HDDS-12165. Refactor VolumeInfoMetrics to use getCurrentUsage (apache#7784)
  HDDS-12085. Add manual refresh button for DU page (apache#7780)
  HDDS-12132. Parameterize testUpdateTransactionInfoTable for SCM (apache#7768)
  HDDS-11277. Remove dependency on hadoop-hdfs in Ozone client (apache#7781)
  ...

Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java
hadoop-ozone/dist/src/main/smoketest/admincli/container.robot
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ClosedContainerReplicator.java
errose28 added a commit to errose28/ozone that referenced this pull request Feb 6, 2025
* master: (168 commits)
  HDDS-12112. Fix interval used for Chunk Read/Write Dashboard (apache#7724)
  HDDS-12212. Fix grammar in decommissioning and observability documentation (apache#7815)
  HDDS-12195. Implement skip() in OzoneFSInputStream (apache#7801)
  HDDS-12200. Fix grammar in OM HA, EC and Snapshot doc (apache#7806)
  HDDS-12202. OpsCreate and OpsAppend metrics not incremented (apache#7811)
  HDDS-12203. Initialize block length before skip (apache#7809)
  HDDS-12183. Reuse cluster across safe test classes (apache#7793)
  HDDS-11714. resetDeletedBlockRetryCount with --all may fail and can cause long db lock in large cluster. (apache#7665)
  HDDS-12186. (addendum) Avoid array allocation for table iterator (apache#7799)
  HDDS-12186. Avoid array allocation for table iterator. (apache#7797)
  HDDS-11508. Decouple delete batch limits from Ratis request size for DirectoryDeletingService. (apache#7365)
  HDDS-12073. Don't show Source Bucket and Volume if null in DU metadata (apache#7760)
  HDDS-12142. Save logs from build check (apache#7782)
  HDDS-12163. Reduce number of individual getCapacity/getAvailable/getUsedSpace calls (apache#7790)
  HDDS-12176. Trivial dependency cleanup.(apache#7787)
  HDDS-12181. Bump jline to 3.29.0 (apache#7789)
  HDDS-12165. Refactor VolumeInfoMetrics to use getCurrentUsage (apache#7784)
  HDDS-12085. Add manual refresh button for DU page (apache#7780)
  HDDS-12132. Parameterize testUpdateTransactionInfoTable for SCM (apache#7768)
  HDDS-11277. Remove dependency on hadoop-hdfs in Ozone client (apache#7781)
  ...

Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java
hadoop-ozone/dist/src/main/smoketest/admincli/container.robot
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ClosedContainerReplicator.java
errose28 added a commit to errose28/ozone that referenced this pull request Feb 7, 2025
…ee-improvements

* HDDS-10239-container-reconciliation: (168 commits)
  HDDS-12112. Fix interval used for Chunk Read/Write Dashboard (apache#7724)
  HDDS-12212. Fix grammar in decommissioning and observability documentation (apache#7815)
  HDDS-12195. Implement skip() in OzoneFSInputStream (apache#7801)
  HDDS-12200. Fix grammar in OM HA, EC and Snapshot doc (apache#7806)
  HDDS-12202. OpsCreate and OpsAppend metrics not incremented (apache#7811)
  HDDS-12203. Initialize block length before skip (apache#7809)
  HDDS-12183. Reuse cluster across safe test classes (apache#7793)
  HDDS-11714. resetDeletedBlockRetryCount with --all may fail and can cause long db lock in large cluster. (apache#7665)
  HDDS-12186. (addendum) Avoid array allocation for table iterator (apache#7799)
  HDDS-12186. Avoid array allocation for table iterator. (apache#7797)
  HDDS-11508. Decouple delete batch limits from Ratis request size for DirectoryDeletingService. (apache#7365)
  HDDS-12073. Don't show Source Bucket and Volume if null in DU metadata (apache#7760)
  HDDS-12142. Save logs from build check (apache#7782)
  HDDS-12163. Reduce number of individual getCapacity/getAvailable/getUsedSpace calls (apache#7790)
  HDDS-12176. Trivial dependency cleanup.(apache#7787)
  HDDS-12181. Bump jline to 3.29.0 (apache#7789)
  HDDS-12165. Refactor VolumeInfoMetrics to use getCurrentUsage (apache#7784)
  HDDS-12085. Add manual refresh button for DU page (apache#7780)
  HDDS-12132. Parameterize testUpdateTransactionInfoTable for SCM (apache#7768)
  HDDS-11277. Remove dependency on hadoop-hdfs in Ozone client (apache#7781)
  ...
nandakumar131 pushed a commit to nandakumar131/ozone that referenced this pull request Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants