[DO NOT MERGE] Backport some fixes from master to ozone-1.4

[myskov]

What changes were proposed in this pull request?

Backport the following fixes from master to ozone-1.4 for 1.4.1:

HDDS-10231. ContainerStateManager should not finalize the OPEN containers without a Pipeline. (#6123)
HDDS-10367. Fix possible NullPointerException in listKeysLight method (#6221)
HDDS-10472. Audit log should include EC replication config (#6338)
HDDS-10503. Bump jgrapht to 1.4.0 (#6364)
HDDS-10526. Bump follow-redirects to 1.15.6 (#6382)
HDDS-10554. Bump Zookeeper to 3.8.4 (#6407)
HDDS-10566. Bump commons-configuration2 to 2.10.1 (#6417)
HDDS-10562. Fix endless loop in ozone fs -ls (#6416)
HDDS-10583. Thread name prefix in ReplicationSupervisor is null (#6430)
HDDS-10547. Fix the buffer for the datanode checksum calculation (#6402)
HDDS-10588. Bump hadoop-shaded-guava to 1.2.0 (#6440)
HDDS-10594. File encryption info is not properly set when overwrite a file (#6445)
HDDS-10600. Bump nimbus-jose-jwt to 9.37.2 (#6454)
HDDS-10524. [Snapshot] Invalidate the cache entry from snapshotInfoTable cache in OMSnapshotPurgeRequest (#6443)
HDDS-10408. NPE causes OM crash in Snapshot Purge request. (#6250)
HDDS-10475. Refine audit logging for bucket creation operation (#6366)
HDDS-10460. Refine audit logging for bucket property updation operations (#6329)

How was this patch tested?

Ci run https://github.com/myskov/ozone/actions/runs/8554460237

[myskov]

@nandakumar131 @ivanzlenko @adoroszlai @symious @Cyrill @vtutrinov @ChenSammi @hemantk-12 @aswinshakil @tanvipenumudy please check that your commits are correctly cherry-picked

[hemantk-12]

Thanks, @myskov for the backport.

I reviewed snapshot changes #6250 and #6443 and the changes look good to me.

[ivandika3]

@myskov Thank you for the backport.

Since ETag feature already landed on ozone-1.4 branch. I propose to include ETag related fixes / improvements in version 1.4.1.

HDDS-9680. Use md5 hash of multipart object part's content as ETag (#5668)
HDDS-10395. Fix eTag compatibility issues during MPU (#6235)
HDDS-10403. CopyObject should set ETag based on the key content (#6251)
HDDS-10521. ETag field should not be returned during GetObject if the key does not contain ETag field (#6377)
HDDS-10587. Reset the thread-local MessageDigest instance during exception (#6435)

All these fixes have been backported to our internal cluster and currently they are working well.

What do you think?

cc: @vtutrinov @xichen01

[adoroszlai]

Thanks @myskov for working on this. I checked dependabot's and my own commits, they look good.

[adoroszlai]

/pending DO NOT MERGE (must be pushed manually)

[github-actions]

Marking this issue as un-mergeable as requested.

Please use /ready comment when it's resolved.

Please note that the PR will be closed after 21 days of inactivity from now. (But can be re-opened anytime later...)

DO NOT MERGE (must be pushed manually)

[myskov]

@ivandika3 I agree that ETag-related fixes should be ported. However, I suggest doing it in a separate PR.

[xichen01]

@myskov
These commits can significantly improve the performance of small EC files, we can consider cherry-picking them.
Detail:
https://issues.apache.org/jira/browse/HDDS-9911

• HDDS-9913. Reduce number of times configuration is loaded in Ozone client #5789 [had be cherry-picked to the branch ozone-1.4]
• HDDS-10384. EC client Reusing thread resources #6270 and #6326
• HDDS-10383. Introduce a Provider for client-side thread resources passing #6222
  • HDDS-10564 (Bug fix)Make Outputstream writeExecutor daemon threads #6418

cc: @adoroszlai

---

A Bug fix:
HDDS-9976. Ozone StateContext Memory leak for DeleteBlocksCommand when queue is full #5845

[kerneltime]

cc @duongkame

[tanvipenumudy]

HDDS-10475. Refine audit logging for bucket creation operation (#6366)
HDDS-10460. Refine audit logging for bucket property updation operations (#6329)

Thank you @myskov, I have taken a look at both my patches and they look good.

[myskov]

@xichen01 thanks for the suggestion. Let's do it in a separate PR.

[xichen01]

@myskov Thank you for the backport.

Since ETag feature already landed on ozone-1.4 branch. I propose to include ETag related fixes / improvements in version 1.4.1.

HDDS-9680. Use md5 hash of multipart object part's content as ETag (#5668)
HDDS-10395. Fix eTag compatibility issues during MPU (#6235)
HDDS-10403. CopyObject should set ETag based on the key content (#6251)
HDDS-10521. ETag field should not be returned during GetObject if the key does not contain ETag field (#6377)
HDDS-10587. Reset the thread-local MessageDigest instance during exception (#6435)

@ivandika3 will you create a new PR to merge above these PRs to the branch ozone-1.4?

[xichen01]

@xichen01 thanks for the suggestion. Let's do it in a separate PR.

@myskov OK, I will create a new PR to merge these.

[xichen01]

@xichen01 thanks for the suggestion. Let's do it in a separate PR.

@myskov Create a PR #6512