Throw any error from MinIO with ListObject APIs #214

mocsharp · 2023-04-04T23:15:51Z

Description

PR to throw any errors received from MinIO when listing objects instead of silently log the errors.
Adds the following configuration:
- timeout: sets the MinIO client API call timeout; defaults to 2500ms
Updated API doc to recommend the use of cancellation token to control the ListObjectsAsync, VerifyObjectsExistAsync, VerifyObjectExistsAsync, ListObjectsWithCredentialsAsync calls.

Status

Ready

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
All tests passed locally.
Documentation comments included/updated.

Signed-off-by: Victor Chang <vicchang@nvidia.com>

codecov · 2023-04-04T23:22:43Z

Codecov Report

❗ No coverage uploaded for pull request base (develop@e7f640e). Click here to learn what that means.
The diff coverage is 72.75986%.

@@              Coverage Diff              @@
##             develop        #214   +/-   ##
=============================================
  Coverage           ?   78.88075%           
=============================================
  Files              ?          38           
  Lines              ?        1501           
  Branches           ?          78           
=============================================
  Hits               ?        1184           
  Misses             ?         278           
  Partials           ?          39

Flag	Coverage Δ
unittests	`78.88075% <72.75986%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/Plugins/MinIO/Tests/Unit/MinIoHealthCheckTest.cs	`100.00000% <ø> (ø)`
src/Storage/API/ListObjectTimeoutException.cs	`0.00000% <0.00000%> (ø)`
src/Storage/API/StorageObjectNotFoundException.cs	`0.00000% <0.00000%> (ø)`
src/Storage/API/VerifyObjectsException.cs	`0.00000% <0.00000%> (ø)`
src/Storage/API/ListObjectException.cs	`23.07692% <23.07692%> (ø)`
src/Storage/API/StorageServiceException.cs	`25.00000% <25.00000%> (ø)`
src/Storage/API/StorageConnectionException.cs	`27.27273% <27.27273%> (ø)`
src/Plugins/MinIO/MinIoClientFactory.cs	`42.85714% <66.66667%> (ø)`
src/Plugins/MinIO/MinIoStorageService.cs	`68.63469% <85.45455%> (ø)`
src/Plugins/MinIO/ConfigurationKeys.cs	`100.00000% <100.00000%> (ø)`
... and 1 more

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e7f640e...531c0d9. Read the comment docs.

Signed-off-by: Victor Chang <vicchang@nvidia.com>

…r manage it via cancellation token. Signed-off-by: Victor Chang <vicchang@nvidia.com>

Signed-off-by: Victor Chang <vicchang@nvidia.com>

+semver: patch * Bump actions/cache from 3.2.2 to 3.3.0 (#190) Bumps [actions/cache](https://github.com/actions/cache) from 3.2.2 to 3.3.0. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v3.2.2...v3.3.0) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump gittools/actions from 0.9.15 to 0.10.2 (#189) Bumps [gittools/actions](https://github.com/gittools/actions) from 0.9.15 to 0.10.2. - [Release notes](https://github.com/gittools/actions/releases) - [Commits](GitTools/actions@v0.9.15...v0.10.2) --- updated-dependencies: - dependency-name: gittools/actions dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump AWSSDK.SecurityToken from 3.7.101.22 to 3.7.101.24 (#194) Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.101.22 to 3.7.101.24. - [Release notes](https://github.com/aws/aws-sdk-net/releases) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump AWSSDK.S3 from 3.7.103.24 to 3.7.103.27 (#199) Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.103.24 to 3.7.103.27. - [Release notes](https://github.com/aws/aws-sdk-net/releases) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Remove reference of .NET packages Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update license approvals Signed-off-by: Victor Chang <vicchang@nvidia.com> * Remove duplicate section of CodeQL scan Signed-off-by: Victor Chang <vicchang@nvidia.com> * fix up ci Signed-off-by: Neil South <neil.south@answerdigital.com> * Throw any error from MinIO with ListObject APIs (#214) * Throw any error from MinIO when listing objects * Capture ListObjectsAsync exceptions in VerifyObjectExistsAsync * Configure minio client timeout * Throw VerifyObjectsException on error * Convert MinIO exception with custom exceptions * Update API doc Signed-off-by: Victor Chang <vicchang@nvidia.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Victor Chang <vicchang@nvidia.com> Signed-off-by: Neil South <neil.south@answerdigital.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Neil South <neil.south@answerdigital.com> Co-authored-by: Neil South <104848880+neildsouth@users.noreply.github.com>

github-actions · 2023-04-06T15:17:26Z

🎉 This issue has been resolved in version 0.2.16 🎉

The release is available on:

GitHub Release

+semver: patch * Bump actions/cache from 3.2.2 to 3.3.0 (#190) Bumps [actions/cache](https://github.com/actions/cache) from 3.2.2 to 3.3.0. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v3.2.2...v3.3.0) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump gittools/actions from 0.9.15 to 0.10.2 (#189) Bumps [gittools/actions](https://github.com/gittools/actions) from 0.9.15 to 0.10.2. - [Release notes](https://github.com/gittools/actions/releases) - [Commits](GitTools/actions@v0.9.15...v0.10.2) --- updated-dependencies: - dependency-name: gittools/actions dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump AWSSDK.SecurityToken from 3.7.101.22 to 3.7.101.24 (#194) Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.101.22 to 3.7.101.24. - [Release notes](https://github.com/aws/aws-sdk-net/releases) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump AWSSDK.S3 from 3.7.103.24 to 3.7.103.27 (#199) Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.103.24 to 3.7.103.27. - [Release notes](https://github.com/aws/aws-sdk-net/releases) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Remove reference of .NET packages Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update license approvals Signed-off-by: Victor Chang <vicchang@nvidia.com> * Remove duplicate section of CodeQL scan Signed-off-by: Victor Chang <vicchang@nvidia.com> * fix up ci Signed-off-by: Neil South <neil.south@answerdigital.com> * Throw any error from MinIO with ListObject APIs (#214) * Throw any error from MinIO when listing objects * Capture ListObjectsAsync exceptions in VerifyObjectExistsAsync * Configure minio client timeout * Throw VerifyObjectsException on error * Convert MinIO exception with custom exceptions * Update API doc Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update dependencies (#241) +semver: patch * Update dependencies: AWSSDK, GuardClauses, FluentAssertions, Moq, xunit, coverlet * Update MinIo * Update .NET6 libs * Update licenses and dependency decisions Signed-off-by: Victor Chang <vicchang@nvidia.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Victor Chang <vicchang@nvidia.com> Signed-off-by: Neil South <neil.south@answerdigital.com> Signed-off-by: Victor Chang <mocsharp@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Neil South <neil.south@answerdigital.com> Co-authored-by: Neil South <104848880+neildsouth@users.noreply.github.com>

* Throw any error from MinIO when listing objects * Capture ListObjectsAsync exceptions in VerifyObjectExistsAsync * Configure minio client timeout * Throw VerifyObjectsException on error * Convert MinIO exception with custom exceptions * Update API doc Signed-off-by: Victor Chang <vicchang@nvidia.com>

+semver: major * Update license approvals Signed-off-by: Victor Chang <vicchang@nvidia.com> * Throw any error from MinIO with ListObject APIs (#214) * Throw any error from MinIO when listing objects * Capture ListObjectsAsync exceptions in VerifyObjectExistsAsync * Configure minio client timeout * Throw VerifyObjectsException on error * Convert MinIO exception with custom exceptions * Update API doc * Upgrade to .NET 8 (#281) * Upgrade to .NET 8 Signed-off-by: Victor Chang <vicchang@nvidia.com>

* Bump actions/cache from 3.2.2 to 3.3.0 (#190) Bumps [actions/cache](https://github.com/actions/cache) from 3.2.2 to 3.3.0. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v3.2.2...v3.3.0) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump gittools/actions from 0.9.15 to 0.10.2 (#189) Bumps [gittools/actions](https://github.com/gittools/actions) from 0.9.15 to 0.10.2. - [Release notes](https://github.com/gittools/actions/releases) - [Commits](GitTools/actions@v0.9.15...v0.10.2) --- updated-dependencies: - dependency-name: gittools/actions dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump AWSSDK.SecurityToken from 3.7.101.22 to 3.7.101.24 (#194) Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.101.22 to 3.7.101.24. - [Release notes](https://github.com/aws/aws-sdk-net/releases) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump AWSSDK.S3 from 3.7.103.24 to 3.7.103.27 (#199) Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.103.24 to 3.7.103.27. - [Release notes](https://github.com/aws/aws-sdk-net/releases) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Remove reference of .NET packages Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update license approvals Signed-off-by: Victor Chang <vicchang@nvidia.com> * Remove duplicate section of CodeQL scan Signed-off-by: Victor Chang <vicchang@nvidia.com> * fix up ci Signed-off-by: Neil South <neil.south@answerdigital.com> * Throw any error from MinIO with ListObject APIs (#214) * Throw any error from MinIO when listing objects * Capture ListObjectsAsync exceptions in VerifyObjectExistsAsync * Configure minio client timeout * Throw VerifyObjectsException on error * Convert MinIO exception with custom exceptions * Update API doc Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update dependencies (#241) +semver: patch * Update dependencies: AWSSDK, GuardClauses, FluentAssertions, Moq, xunit, coverlet * Update MinIo * Update .NET6 libs * Update licenses and dependency decisions Signed-off-by: Victor Chang <vicchang@nvidia.com> * Remove unused references (#243) * Remove unused references * Cleanup licenses * Update Moq Signed-off-by: Victor Chang <vicchang@nvidia.com> * Upgrade to .NET 8 (#281) * Upgrade to .NET 8 Signed-off-by: Victor Chang <vicchang@nvidia.com> * Bump codecov/codecov-action from 3.1.1 to 3.1.4 (#232) Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 3.1.1 to 3.1.4. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v3.1.1...v3.1.4) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump apache/skywalking-eyes from 0.4.0 to 0.5.0 (#245) Bumps [apache/skywalking-eyes](https://github.com/apache/skywalking-eyes) from 0.4.0 to 0.5.0. - [Release notes](https://github.com/apache/skywalking-eyes/releases) - [Changelog](https://github.com/apache/skywalking-eyes/blob/main/CHANGES.md) - [Commits](apache/skywalking-eyes@v0.4.0...v0.5.0) --- updated-dependencies: - dependency-name: apache/skywalking-eyes dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/checkout from 3 to 4 (#246) Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/cache from 3.3.0 to 3.3.2 (#247) Bumps [actions/cache](https://github.com/actions/cache) from 3.3.0 to 3.3.2. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v3.3.0...v3.3.2) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/upload-artifact from 3.1.2 to 4.0.0 (#278) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.2 to 4.0.0. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v3.1.2...v4.0.0) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update mc commands (#328) * Update mc commands Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update dependencies Signed-off-by: Victor Chang <vicchang@nvidia.com> * Remove windows build Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update minio client for testing Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update dependency versions Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update 3rd party licenses and copyright Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update licenses Signed-off-by: Victor Chang <vicchang@nvidia.com> --------- Signed-off-by: Victor Chang <vicchang@nvidia.com> * Update codecov/codecov-action@v4.4.1 Signed-off-by: Victor Chang <mocsharp@users.noreply.github.com> * Fix codecov copyright Signed-off-by: Victor Chang <vicchang@nvidia.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Victor Chang <vicchang@nvidia.com> Signed-off-by: Neil South <neil.south@answerdigital.com> Signed-off-by: Victor Chang <mocsharp@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Neil South <neil.south@answerdigital.com> Co-authored-by: Neil South <104848880+neildsouth@users.noreply.github.com>

Throw any error from MinIO when listing objects

520c9e3

Signed-off-by: Victor Chang <vicchang@nvidia.com>

mocsharp added the bug Something isn't working label Apr 4, 2023

mocsharp added this to the 0.2.16 milestone Apr 4, 2023

mocsharp requested review from neildsouth and woodheadio April 4, 2023 23:15

mocsharp self-assigned this Apr 4, 2023

neildsouth approved these changes Apr 5, 2023

View reviewed changes

mocsharp added 9 commits April 5, 2023 09:15

Capture ListObjectsAsync exceptions in VerifyObjectExistsAsync

cf0e7ca

Signed-off-by: Victor Chang <vicchang@nvidia.com>

Configure minio client timeout

2e3b571

Signed-off-by: Victor Chang <vicchang@nvidia.com>

Time limit on list objects call

eb0de25

Signed-off-by: Victor Chang <vicchang@nvidia.com>

Separate list object timeout

dbee41b

Signed-off-by: Victor Chang <vicchang@nvidia.com>

Throw VerifyObjectsException on error

b4f0aa6

Signed-off-by: Victor Chang <vicchang@nvidia.com>

Convert MinIO exception with custom exceptions

7133284

Signed-off-by: Victor Chang <vicchang@nvidia.com>

Revert previous change that adds a listObjectTimeout; instead let use…

1a7deaf

…r manage it via cancellation token. Signed-off-by: Victor Chang <vicchang@nvidia.com>

Update API doc

89bd977

Signed-off-by: Victor Chang <vicchang@nvidia.com>

Exclude XML to avoid additional errors

531c0d9

Signed-off-by: Victor Chang <vicchang@nvidia.com>

mocsharp requested a review from neildsouth April 6, 2023 00:51

neildsouth approved these changes Apr 6, 2023

View reviewed changes

woodheadio approved these changes Apr 6, 2023

View reviewed changes

mocsharp merged commit 1757165 into develop Apr 6, 2023

mocsharp deleted the vchang/list-object-error branch April 6, 2023 15:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Throw any error from MinIO with ListObject APIs #214

Throw any error from MinIO with ListObject APIs #214

Uh oh!

mocsharp commented Apr 4, 2023 •

edited

Loading

Uh oh!

codecov bot commented Apr 4, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Apr 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Throw any error from MinIO with ListObject APIs #214

Throw any error from MinIO with ListObject APIs #214

Uh oh!

Conversation

mocsharp commented Apr 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Status

Types of changes

Uh oh!

codecov bot commented Apr 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Apr 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mocsharp commented Apr 4, 2023 •

edited

Loading

codecov bot commented Apr 4, 2023 •

edited

Loading