Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mc ls fails on DigitalOcean Spaces with "Unable to list folder. Truncated response should have continuation token set" #3067

Closed
genevera opened this issue Feb 4, 2020 · 17 comments
Closed

mc ls fails on DigitalOcean Spaces with "Unable to list folder. Truncated response should have continuation token set" #3067

genevera opened this issue Feb 4, 2020 · 17 comments
Labels
community out of scope

Comments

@genevera
Copy link

genevera commented Feb 4, 2020

Expected behavior

mc ls lists a prefix containing many files and exits 0.

Actual behavior

Failure with the error: "Unable to list folder. Truncated response should have continuation token set"

Steps to reproduce the behavior

mc ls digitaloceanspacesbucket/username/prefix-with-lots-of-files

mc --version

  • (paste output of mc --version)
    mc version RELEASE.2020-01-25T03-02-19Z

System information

darwin/amd64

example debug output:

mc: <DEBUG> GET /genevera/?location= HTTP/1.1
Host: nyc3.digitaloceanspaces.com
User-Agent: MinIO (darwin; amd64) minio-go/v6.0.45 mc/2020-01-25T03:02:19Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20200204/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20200204T022623Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 131
Content-Type: application/xml
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload

mc: <DEBUG> Response Time:  1.078912981s

mc: <DEBUG> GET /genevera/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=PREFIX HTTP/1.1
Host: nyc3.digitaloceanspaces.com
User-Agent: MinIO (darwin; amd64) minio-go/v6.0.45 mc/2020-01-25T03:02:19Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20200204/nyc3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200204T022624Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 04 Feb 2020 02:26:24 GMT
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Amz-Request-Id: tx000000000000018433970-005e38d650-33d0876-nyc3a

mc: <DEBUG> Response Time:  340.814825ms

mc: <DEBUG> GET /genevera/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=PREFIX HTTP/1.1
Host: nyc3.digitaloceanspaces.com
User-Agent: MinIO (darwin; amd64) minio-go/v6.0.45 mc/2020-01-25T03:02:19Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20200204/nyc3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200204T022624Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 04 Feb 2020 02:26:25 GMT
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Amz-Request-Id: tx000000000000018f97f13-005e38d651-33d083a-nyc3a

mc: <DEBUG> Response Time:  569.465558ms

mc: <DEBUG> GET /genevera/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=PREFIX%2F HTTP/1.1
Host: nyc3.digitaloceanspaces.com
User-Agent: MinIO (darwin; amd64) minio-go/v6.0.45 mc/2020-01-25T03:02:19Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20200204/nyc3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200204T022625Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 04 Feb 2020 02:26:25 GMT
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Amz-Request-Id: tx0000000000000187d7e3a-005e38d651-33c84fd-nyc3a

mc: <DEBUG> Response Time:  540.980968ms

mc: <ERROR> Unable to list folder. Truncated response should have continuation token set
 (1) ls.go:132 cmd.doList(..) Tags: [https://nyc3.digitaloceanspaces.com/genevera/PREFIX/]
 (0) client-s3.go:1906 cmd.(*s3Client).listInRoutine(..)
 Release-Tag:RELEASE.2020-01-25T03-02-19Z | Commit:b70a79e39bbd | Host:hostname.SOMEDOMA.IN | OS:darwin | Arch:amd64 | Lang:go1.13.6 | Mem:2.7 MB/72 MB | Heap:2.7 MB/66 MB
@harshavardhana
Copy link
Member

Do you have a sample list of files where this issue happens? @genevera

@vadmeste
Copy link
Member

vadmeste commented Feb 4, 2020

@genevera this looks like more a digital ocean bug, because mc really needs the continuation token to continue listing

@harshavardhana
Copy link
Member

Yes it looks like a server implementation bug @genevera

@genevera
Copy link
Author

genevera commented Feb 5, 2020 •
edited
Loading

Older versions of mc list with no problem. The last version of mc to be able to list this bucket in its entirety was tags/RELEASE.2019-10-02T19-41-02Z. I used the following command inside of $GOPATH/src/github.com/minio/mc to do the test:

for v in $(git tag|rg '2019|2020'|tac); do
  echo "${v}:\n"; 
  sleep 1; 
  git checkout "${v}" --force; # to avoid git refusing to check out a branch due to new files
  make clean;
  GOOS=darwin GOOARCH=amd64 make build;
  ./mc ls ds/genevera/big-bucket &> /dev/null;
  if [[ "${?}" == 0 ]]; then
    echo "${v} works.";
    break;
  else
    continue;
  fi;
done;

My spaces bucket contains around 130000 items.

@genevera
Copy link
Author

genevera commented Feb 5, 2020

Do you have a sample list of files where this issue happens? @genevera

I do!

18:31:39 in mc on  tags/RELEASE.2019-10-02T19-41-02Z took 6s
➜ ./mc ls ds/genevera/record
[2019-01-04 22:46:52 EST]      0B /genevera/record%2F
[2019-03-15 06:35:04 EDT]  7.4MiB /genevera/record%2F20180816-234631.mp3
[2019-03-15 06:35:04 EDT]  7.4MiB /genevera/record%2F20180817-204901.mp3
[2019-03-15 06:35:03 EDT]  7.4MiB /genevera/record%2F20180817-211903.mp3
[2019-03-15 06:35:03 EDT]  7.4MiB /genevera/record%2F20180817-214907.mp3
[2019-03-15 06:35:06 EDT]  7.4MiB /genevera/record%2F20180817-221910.mp3
[2019-03-15 06:35:06 EDT]  7.4MiB /genevera/record%2F20180818-024937.mp3
[2019-03-15 06:35:07 EDT]  7.4MiB /genevera/record%2F20180818-065010.mp3
[2019-03-15 06:35:07 EDT]  7.4MiB /genevera/record%2F20180819-000157.mp3
[2019-03-15 06:35:08 EDT]  7.4MiB /genevera/record%2F20180819-001158.mp3
[2019-03-15 06:35:09 EDT]  7.4MiB /genevera/record%2F20180819-002159.mp3
[2019-03-15 06:35:10 EDT]  7.4MiB /genevera/record%2F20180819-003200.mp3
[2019-03-15 06:35:10 EDT]  7.4MiB /genevera/record%2F20180819-004200.mp3
[2019-03-15 06:35:11 EDT]  7.4MiB /genevera/record%2F20180819-005200.mp3
[2019-03-15 06:35:12 EDT]  7.3MiB /genevera/record%2F20180819-010201.mp3
[2019-03-15 06:35:14 EDT]  7.3MiB /genevera/record%2F20180819-011201.mp3
[2019-03-15 06:35:13 EDT]  7.3MiB /genevera/record%2F20180819-012202.mp3
[2019-03-15 06:35:14 EDT]  7.3MiB /genevera/record%2F20180819-013205.mp3
[2019-03-15 06:35:15 EDT]  7.3MiB /genevera/record%2F20180819-014205.mp3
[2019-03-15 06:35:16 EDT]  7.3MiB /genevera/record%2F20180819-015206.mp3
[2019-03-15 06:35:17 EDT]  7.3MiB /genevera/record%2F20180819-020208.mp3
[2019-03-15 06:35:17 EDT]  7.3MiB /genevera/record%2F20180819-021208.mp3
[2019-03-15 06:35:18 EDT]  7.3MiB /genevera/record%2F20180819-022209.mp3
[2019-03-15 06:35:18 EDT]  7.3MiB /genevera/record%2F20180819-023211.mp3
[2019-03-15 06:35:19 EDT]  7.3MiB /genevera/record%2F20180819-024215.mp3
[2019-03-15 06:35:20 EDT]  7.3MiB /genevera/record%2F20180819-025216.mp3
[2019-03-15 06:35:21 EDT]  7.3MiB /genevera/record%2F20180819-030217.mp3
[2019-03-15 06:35:21 EDT]  7.3MiB /genevera/record%2F20180819-031217.mp3
[2019-03-15 06:35:22 EDT]  7.3MiB /genevera/record%2F20180819-032217.mp3
[2019-03-15 06:35:22 EDT]  7.3MiB /genevera/record%2F20180819-033217.mp3
[2019-03-15 06:35:24 EDT]  7.3MiB /genevera/record%2F20180819-034220.mp3
[2019-03-15 06:35:24 EDT]  7.3MiB /genevera/record%2F20180819-035221.mp3
[2019-03-15 06:35:25 EDT]  7.3MiB /genevera/record%2F20180819-040221.mp3
[2019-03-15 06:35:26 EDT]  7.3MiB /genevera/record%2F20180819-041222.mp3
[2019-03-15 06:35:28 EDT]  7.3MiB /genevera/record%2F20180819-042227.mp3
[2019-03-15 06:35:29 EDT]  7.3MiB /genevera/record%2F20180819-043227.mp3
[2019-03-15 06:35:30 EDT]  7.3MiB /genevera/record%2F20180819-044228.mp3
[2019-03-15 06:35:30 EDT]  7.3MiB /genevera/record%2F20180819-045228.mp3
[2019-03-15 06:35:31 EDT]  7.3MiB /genevera/record%2F20180819-050230.mp3
[2019-03-15 06:35:31 EDT]  7.3MiB /genevera/record%2F20180819-051231.mp3
[2019-03-15 06:35:33 EDT]  7.3MiB /genevera/record%2F20180819-052233.mp3
[2019-03-15 06:35:33 EDT]  7.3MiB /genevera/record%2F20180819-053233.mp3
[2019-03-15 06:35:34 EDT]  7.3MiB /genevera/record%2F20180819-054233.mp3
[2019-03-15 06:35:34 EDT]  7.3MiB /genevera/record%2F20180819-055235.mp3
[2019-03-15 06:35:35 EDT]  7.3MiB /genevera/record%2F20180819-060236.mp3
[2019-03-15 06:35:35 EDT]  7.3MiB /genevera/record%2F20180819-061237.mp3
[2019-03-15 06:35:38 EDT]  7.3MiB /genevera/record%2F20180819-062238.mp3
[2019-03-15 06:35:37 EDT]  7.3MiB /genevera/record%2F20180819-063241.mp3
[2019-03-15 06:35:38 EDT]  7.3MiB /genevera/record%2F20180819-064241.mp3
[2019-03-15 06:35:38 EDT]  7.3MiB /genevera/record%2F20180819-065241.mp3
[2019-03-15 06:35:40 EDT]  7.3MiB /genevera/record%2F20180819-070244.mp3
[2019-03-15 06:35:40 EDT]  7.3MiB /genevera/record%2F20180819-071245.mp3
[2019-03-15 06:35:41 EDT]  7.4MiB /genevera/record%2F20180819-072246.mp3
[2019-03-15 06:36:07 EDT]  7.3MiB /genevera/record%2F20180819-073246.mp3
[2019-03-15 06:35:42 EDT]  7.3MiB /genevera/record%2F20180819-074247.mp3
[2019-03-15 06:35:43 EDT]  7.3MiB /genevera/record%2F20180819-075248.mp3
[2019-03-15 06:35:45 EDT]  7.4MiB /genevera/record%2F20180819-080249.mp3
[2019-03-15 06:35:45 EDT]  7.4MiB /genevera/record%2F20180819-081249.mp3
[2019-03-15 06:35:45 EDT]  7.4MiB /genevera/record%2F20180819-082252.mp3
[2019-03-15 06:35:48 EDT]  7.4MiB /genevera/record%2F20180819-083254.mp3
[2019-03-15 06:35:49 EDT]  7.4MiB /genevera/record%2F20180819-084255.mp3
[2019-03-15 06:35:48 EDT]  7.4MiB /genevera/record%2F20180819-085255.mp3
[2019-03-15 06:35:50 EDT]  7.4MiB /genevera/record%2F20180819-090256.mp3
[2019-03-15 06:35:51 EDT]  7.4MiB /genevera/record%2F20180819-091257.mp3
[2019-03-15 06:36:07 EDT]  7.4MiB /genevera/record%2F20180819-092258.mp3
[2019-03-15 06:35:53 EDT]  7.4MiB /genevera/record%2F20180819-093258.mp3
[2019-03-15 06:35:53 EDT]  7.4MiB /genevera/record%2F20180819-094259.mp3
[2019-03-15 06:35:56 EDT]  7.3MiB /genevera/record%2F20180819-095302.mp3
[2019-03-15 06:35:56 EDT]  7.4MiB /genevera/record%2F20180819-100303.mp3
[2019-03-15 06:35:59 EDT]  7.3MiB /genevera/record%2F20180819-101304.mp3
[2019-03-15 06:36:02 EDT]  7.3MiB /genevera/record%2F20180819-102304.mp3
[2019-03-15 06:36:04 EDT]  7.3MiB /genevera/record%2F20180819-103305.mp3
[2019-03-15 06:36:05 EDT]  7.4MiB /genevera/record%2F20180819-104307.mp3
[2019-03-15 06:36:07 EDT]  7.4MiB /genevera/record%2F20180819-105309.mp3
[2019-03-15 06:36:08 EDT]  7.4MiB /genevera/record%2F20180819-110311.mp3
[2019-03-15 06:36:10 EDT]  7.4MiB /genevera/record%2F20180819-111315.mp3
[2019-03-15 06:36:10 EDT]  7.3MiB /genevera/record%2F20180819-112315.mp3
[2019-03-15 06:36:10 EDT]  7.3MiB /genevera/record%2F20180819-113316.mp3
[2019-03-15 06:36:12 EDT]  7.4MiB /genevera/record%2F20180819-114317.mp3
[2019-03-15 06:36:13 EDT]  7.3MiB /genevera/record%2F20180819-115317.mp3
[2019-03-15 06:36:13 EDT]  7.3MiB /genevera/record%2F20180819-120317.mp3
[2019-03-15 06:36:14 EDT]  7.3MiB /genevera/record%2F20180819-121319.mp3
[2019-03-15 06:36:14 EDT]  7.3MiB /genevera/record%2F20180819-122320.mp3
[2019-03-15 06:36:15 EDT]  7.3MiB /genevera/record%2F20180819-123320.mp3

(this is a snippet of output from the last working version; it lists all files)

@genevera
Copy link
Author

ping @harshavardhana @vadmeste

@harshavardhana
Copy link
Member

The issue here is very clear, there is a bug in the server side implementation with encoding-type=url requested during listing.

Server clearly is setting wrong things with ListObjectsV2 implementation. ListObjectsV2 implementation mandates that if the response has IsTruncated set then NextContinuationToken should be set as well.

So minio-go is failing for the right reason, server is sending an incorrect response.

@genevera
Copy link
Author

Could there not be implemented a fallback to V1 upon receiving a response that doesn't match up with expectations for a V2 call?

@harshavardhana
Copy link
Member

Could there not be implemented a fallback to V1 upon receiving a response that doesn't match up with expectations for a V2 call?

Such types of hacks would be complicated @genevera there are added performance implications to such implementation and other technical complications such as - Listing is a batched operation so we cannot suddenly make listObjectsV1 calls in the middle of an ongoing listObjectsV2 by using the failed state from listObjectsV2 call.

There will be unexpected results as V1 and V2 vary. The best approach is to fix the server here rather than burdening our software

@harshavardhana
Copy link
Member

You can simply use the older version of mc @genevera as a work-around.

@genevera
Copy link
Author

genevera commented Feb 16, 2020 •
edited
Loading

Okey dokey - thanks for taking the time to respond on this ticket!

@LeoQuote LeoQuote mentioned this issue Feb 24, 2020
Closed
@kannappanr kannappanr mentioned this issue Feb 25, 2020
Closed
@leshik leshik mentioned this issue Apr 16, 2020
Closed
@ureykane
Copy link

ureykane commented Jan 5, 2021 •
edited
Loading

@genevera
You can add --versions parameter to avoid this problem
For example: mc ls --versions s3/mybucket
Can also pass
mc du --versions s3/mybucket
Solve the problem of statistical error

ceph ver:
ceph-12.2.5-0.el7.x86_64
mc ver:
mc version RELEASE.2020-12-18T10-53-53Z

@harshavardhana
Copy link
Member

You can add --versions parameter to avoid this problem

This is not an mc bug @ureykane ListObjectsV2 is not implemented correctly in Ceph - they are returning Truncated set to true while ContiuationToken is not set. minio-go adds strict validation to reject such implementations.

We can list millions of objects regularly on AWS S3 without any issues.

@ureykane
Copy link

ureykane commented Jan 5, 2021

@harshavardhana
Thx for your reply
Mainly I am curious why adding this parameter can avoid errors,There is no time to look at the source code yet, can anyone tell me the answer directly?

@harshavardhana
Copy link
Member

Mainly I am curious why adding this parameter can avoid errors,There is no time to look at the source code yet, can anyone tell me the answer directly?

That is because --versions calls a different API called ListObjectVersions() that is perhaps fortunately correctly implemented.

@dudz94
Copy link

dudz94 commented Jun 2, 2021

@genevera
You can add --versions parameter to avoid this problem
For example: mc ls --versions s3/mybucket
Can also pass
mc du --versions s3/mybucket
Solve the problem of statistical error

ceph ver:
ceph-12.2.5-0.el7.x86_64
mc ver:
mc version RELEASE.2020-12-18T10-53-53Z

It works perfectly. Thanks !

@genevera
Copy link
Author

genevera commented Jun 7, 2021

@ureykane thank you!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
community out of scope
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@vadmeste @harshavardhana @dudz94 @genevera @ureykane