Agent impact on ListObjectsAsync

[tayyebi]

Environment:

Package Reference: Minio Version 6.0.1

In depth details of the issue:

I have an strange problem with a code base. There is a private object storage behind my API, which is in common with other clients, and some devices such as cURL are not able to download files because of a back-end error (Authorization Exception).

Why a client device (maybe headers sent by the device or connection type) has such effects on a server-side request?

Demonstration of the issue:

Screencast.from.2024-08-19.18-12-03.webm

Dependency Injection:

            #region minio
            services.AddMinio(configureClient => configureClient
                .WithEndpoint(configuration.GetSection("AWS:RegionEndpoint").Value)
                .WithCredentials(configuration.GetSection("AWS:Key").Value, configuration.GetSection("AWS:Secret").Value)
                .WithSSL(Convert.ToBoolean(configuration.GetSection("AWS:SSL").Value))
                );
            #endregion

Handler:

        public async Task<List<(string ObjectName, string MimeType)>> List(string directory)
        {
            var bucketName = configuration.GetSection("AWS:BucketName").Value;
            var listObjectsArgs = new ListObjectsArgs()
                .WithBucket(bucketName)
                .WithPrefix(directory)
                .WithRecursive(true);

            var objects = new List<(string ObjectName, string MimeType)>();
            var tcs = new TaskCompletionSource<List<(string ObjectName, string MimeType)>>();

            var observable = minioClient.ListObjectsAsync(listObjectsArgs);
            var subscription = observable.Subscribe(
                onNext: item =>
                {
                    var stat = minioClient.StatObjectAsync(new StatObjectArgs()
                        .WithBucket(bucketName)
                        .WithObject(item.Key));
                    stat.Wait();
                    objects.Add((stat.Result.ObjectName, stat.Result.ContentType));
                },
                onError: ex => { tcs.TrySetException(ex); },
                onCompleted: () => { tcs.TrySetResult(objects); });

            tcs.Task.Wait();
            return tcs.Task.Result;
        }

Exception trace:

Exception has occurred: CLR/System.AggregateException
Exception thrown: 'System.AggregateException' in System.Private.CoreLib.dll: 'One or more errors occurred.'
 Inner exceptions found, see $exception in variables window for more details.
 Innermost exception 	 Minio.Exceptions.AuthorizationException : Exception_WasThrown
   at Minio.MinioClient.ParseErrorFromContent(ResponseResult response)
   at Minio.MinioClient.ParseError(ResponseResult response)
   at Minio.Handlers.DefaultErrorHandler.Handle(ResponseResult response)
   at Minio.RequestExtensions.HandleIfErrorResponse(IMinioClient minioClient, ResponseResult response, IEnumerable`1 handlers, DateTime startTime)
   at Minio.RequestExtensions.<ExecuteTaskCoreAsync>d__3.MoveNext()
   at Minio.RequestExtensions.<>c__DisplayClass2_0.<<ExecuteTaskAsync>b__0>d.MoveNext()
   at Minio.MinioClient.<GetObjectListAsync>d__30.MoveNext()
   at Minio.MinioClient.<>c__DisplayClass6_0.<<ListObjectsAsync>b__0>d.MoveNext()

   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Market.Services.Network.ObjectStorageService.List(String directory) in /home/Projects/x/Services/Network/ObjectStorageService.cs:line 69

The Inner Exception:

   Exception of type 'Minio.Exceptions.AuthorizationException' was thrown.

   at Minio.MinioClient.ParseErrorFromContent(ResponseResult response)
   at Minio.MinioClient.ParseError(ResponseResult response)
   at Minio.Handlers.DefaultErrorHandler.Handle(ResponseResult response)
   at Minio.RequestExtensions.HandleIfErrorResponse(IMinioClient minioClient, ResponseResult response, IEnumerable`1 handlers, DateTime startTime)
   at Minio.RequestExtensions.ExecuteTaskCoreAsync(IMinioClient minioClient, IEnumerable`1 errorHandlers, HttpRequestMessageBuilder requestMessageBuilder, Boolean isSts, CancellationToken cancellationToken)
   at Minio.RequestExtensions.<>c__DisplayClass2_0.<<ExecuteTaskAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Minio.MinioClient.GetObjectListAsync(GetObjectListArgs args, CancellationToken cancellationToken)
   at Minio.MinioClient.<>c__DisplayClass6_0.<<ListObjectsAsync>b__0>d.MoveNext()

Thanks.

[tayyebi]

HAR of the chrome request (which works fine):

{
  "log": {
    "version": "1.2",
    "creator": {
      "name": "WebInspector",
      "version": "537.36"
    },
    "pages": [],
    "entries": [
      {
        "_initiator": {
          "type": "other"
        },
        "_priority": "VeryHigh",
        "_resourceType": "document",
        "cache": {},
        "connection": "61843",
        "request": {
          "method": "GET",
          "url": "https://localhost:5001/User/Picture/10014",
          "httpVersion": "http/2.0",
          "headers": [
            {
              "name": ":authority",
              "value": "localhost:5001"
            },
            {
              "name": ":method",
              "value": "GET"
            },
            {
              "name": ":path",
              "value": "/User/Picture/10014"
            },
            {
              "name": ":scheme",
              "value": "https"
            },
            {
              "name": "accept",
              "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
            },
            {
              "name": "accept-encoding",
              "value": "gzip, deflate, br, zstd"
            },
            {
              "name": "accept-language",
              "value": "en-US,en;q=0.9,de-DE;q=0.8,de;q=0.7"
            },
            {
              "name": "cache-control",
              "value": "no-cache"
            },
            {
              "name": "cookie",
              "value": "userId=10014"
            },
            {
              "name": "pragma",
              "value": "no-cache"
            },
            {
              "name": "priority",
              "value": "u=0, i"
            },
            {
              "name": "sec-ch-ua",
              "value": "\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\""
            },
            {
              "name": "sec-ch-ua-mobile",
              "value": "?0"
            },
            {
              "name": "sec-ch-ua-platform",
              "value": "\"Linux\""
            },
            {
              "name": "sec-fetch-dest",
              "value": "document"
            },
            {
              "name": "sec-fetch-mode",
              "value": "navigate"
            },
            {
              "name": "sec-fetch-site",
              "value": "none"
            },
            {
              "name": "sec-fetch-user",
              "value": "?1"
            },
            {
              "name": "upgrade-insecure-requests",
              "value": "1"
            },
            {
              "name": "user-agent",
              "value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
            }
          ],
          "queryString": [],
          "cookies": [
            {
              "name": "userId",
              "value": "10014",
              "path": "/",
              "domain": "localhost",
              "expires": "2024-11-16T16:50:12.648Z",
              "httpOnly": false,
              "secure": false
            }
          ],
          "headersSize": -1,
          "bodySize": 0
        },
        "response": {
          "status": 200,
          "statusText": "",
          "httpVersion": "http/2.0",
          "headers": [
            {
              "name": "content-disposition",
              "value": "attachment; filename=\"Profile_10014/34ff287483408ad6078427605e6d0503.webp\"; filename*=UTF-8''Profile_10014%2F34ff287483408ad6078427605e6d0503.webp"
            },
            {
              "name": "content-length",
              "value": "3052"
            },
            {
              "name": "content-type",
              "value": "image/webp"
            },
            {
              "name": "date",
              "value": "Mon, 19 Aug 2024 15:08:14 GMT"
            },
            {
              "name": "server",
              "value": "Kestrel"
            }
          ],
          "cookies": [],
          "content": {
            "size": 0,
            "mimeType": "image/webp"
          },
          "redirectURL": "",
          "headersSize": -1,
          "bodySize": -1,
          "_transferSize": 3289,
          "_error": "net::ERR_ABORTED",
          "_fetchedViaServiceWorker": false
        },
        "serverIPAddress": "[::1]",
        "startedDateTime": "2024-08-19T15:08:14.361Z",
        "time": 436.0949999972945,
        "timings": {
          "blocked": 5.410999998416053,
          "dns": -1,
          "ssl": -1,
          "connect": -1,
          "send": 0.18699999999999983,
          "wait": 426.69900000096413,
          "receive": 3.797999997914303,
          "_blocked_queueing": 1.6179999984160531,
          "_workerStart": -1,
          "_workerReady": -1,
          "_workerFetchStart": -1,
          "_workerRespondWithSettled": -1
        }
      }
    ]
  }
}

[tayyebi]

I just updated to 6.0.3 from 6.0.1 and I have the error below:

Minio.Exceptions.ErrorResponseException : MinIO API responded with message=HTTP status-code 403: Forbidden. Status code=Forbidden, response=Exception of type 'Minio.Exceptions.AuthorizationException' was thrown., content=<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code>...

There is still a question left: My API is launching the HttpClient and is directly communicating with the Object Storage; why should my end-user impact this communication?

[tayyebi]

I DO NOT face this problem using Amazon.S3 library. You can compare the video below, with the video mentioned in previous messages.

Screencast.from.2024-08-20.20-32-59.webm

MinIO Method:

    public class ObjectStorageService : IObjectStorageService
    {
        private IConfiguration configuration;
        private readonly IMinioClient minioClient;
        public ObjectStorageService(
                IConfiguration configuration,
                IMinioClient minioClient
        )
        {
            this.minioClient = minioClient;
            this.configuration = configuration;
        }

        public async Task<List<string>> ListDirectories()
        {
            var bucketName = configuration.GetSection("AWS:BucketName").Value;
            var listObjectsArgs = new ListObjectsArgs()
                .WithBucket(bucketName)
                .WithRecursive(true);

            var objectNames = new List<string>();

            var observable = minioClient.ListObjectsAsync(listObjectsArgs);
            foreach (var item in observable)
            {
                objectNames.Add(item.Key);
            }

            return objectNames;
        }


        public async Task<List<(string ObjectName, string MimeType)>> List(string directory)
        {
            var bucketName = configuration.GetSection("AWS:BucketName").Value;
            var listObjectsArgs = new ListObjectsArgs()
                .WithBucket(bucketName)
                .WithPrefix(directory)
                .WithRecursive(true);

            var objects = new List<(string ObjectName, string MimeType)>();
            var tcs = new TaskCompletionSource<List<(string ObjectName, string MimeType)>>();

            var observable = minioClient.ListObjectsAsync(listObjectsArgs);
            var subscription = observable.Subscribe(
                onNext: item =>
                {
                    var stat = minioClient.StatObjectAsync(new StatObjectArgs()
                        .WithBucket(bucketName)
                        .WithObject(item.Key));
                    stat.Wait();
                    objects.Add((stat.Result.ObjectName, stat.Result.ContentType));
                },
                onError: ex => { tcs.TrySetException(ex); },
                onCompleted: () => { tcs.TrySetResult(objects); });

            tcs.Task.Wait();
            return tcs.Task.Result;
        }

        public async Task<MemoryStream> Get(string objectName)
        {
            var bucket = configuration.GetSection("AWS:BucketName").Value;
            using (var ms = new MemoryStream())
            {
                await minioClient.GetObjectAsync(
                    new GetObjectArgs { }
                        .WithBucket(bucket)
                        .WithObject(objectName)
                        .WithCallbackStream(s => s.CopyTo(ms))
                );
                ms.Position = 0;
                return ms;
            }
        }

        public async Task Put(MemoryStream ms, string objectName, string contentType = "application/octet-stream")
        {
            var bucket = configuration.GetSection("AWS:BucketName").Value;
            await minioClient.PutObjectAsync(
                new PutObjectArgs { }
                    .WithBucket(bucket)
                    .WithObject(objectName)
                    .WithContentType(contentType)
                    .WithStreamData(ms)
                    .WithObjectSize(ms.Length)
            );

        }
    }

S3 Method:

    public class ObjectStorageService : IObjectStorageService
    {
        private IConfiguration configuration;
        private readonly IAmazonS3 client;
        private readonly string bucketName;
        public ObjectStorageService(
                IConfiguration configuration,
                IAmazonS3 objectStorageClient
        )
        {
            this.configuration = configuration;
            this.client = objectStorageClient;
            this.bucketName = configuration.GetSection("AWS:BucketName").Value;
        }

        public async Task<List<string>> ListDirectories()
        {
            ListObjectsRequest request = new ListObjectsRequest
            {
                BucketName = bucketName
            };

            var output = new List<string>();

            do
            {
                // Build your call out to S3 and store the response
                ListObjectsResponse response = await client.ListObjectsAsync(request);

                // Filter through the response to find keys that:
                // - end with the delimiter character '/' 
                // - are empty. 
                var folders = response.S3Objects.Where(x =>
                    x.Key.EndsWith(@"/") && x.Size == 0);

                // Add response to the output
                output.AddRange(folders.Select(x => x.Key));

                // If the response is truncated, we'll make another request 
                // and pull the next batch of keys
                request = null;
                if (response.IsTruncated)
                    request.Marker = response.NextMarker;
            } while (request != null);

            return output;
        }
        public async Task<List<(string ObjectName, string MimeType)>> List(string directory)
        {
            var request = new ListObjectsRequest
            {
                BucketName = bucketName,
                Prefix = directory
            };

            var response = await client.ListObjectsAsync(request);

            var objectList = response.S3Objects
                .Select(obj => (obj.Key, "application/octet-stream"))
                .ToList();

            return objectList;
        }


        public async Task Put(MemoryStream ms, string objectName, string ContentType)
        {
            var request = new PutObjectRequest
            {
                BucketName = bucketName,
                Key = objectName,
                InputStream = ms,
                ContentType = ContentType
            };

            await client.PutObjectAsync(request);
        }

        public async Task<MemoryStream> Get(string objectName)
        {
            var request = new GetObjectRequest
            {
                BucketName = bucketName,
                Key = objectName
            };

            using var response = await client.GetObjectAsync(request);
            using var memoryStream = new MemoryStream();

            await response.ResponseStream.CopyToAsync(memoryStream);

            return memoryStream;
        }
    }