Iterating more than 1k objects failed with ListObjectsV2 but works with ListObjects

[gigatexal]

Given this simple (possibly not very go-like) function where I iterate through all files, looking for a BACKUP file, and then return the s3://bucket/prefix for a given one among other things.

    doneCh := make(chan struct{})
    defer close(doneCh)
    backups := make([]Backup, 0)
    // iterate through all objects looking for .../BACKUP which indicates a finished backup
    objectCh := s.client.ListObjectsV2(s.config.Bucket, "", true, doneCh)
    for object := range objectCh {
        if object.Err != nil {
            return nil, fmt.Errorf("unable to get backups: %s", object.Err)
        }
        if path.Base(object.Key) == "BACKUP" {
            var backupType types.BackupType
            if strings.Contains(object.Key, types.Incremental.String()) {
                backupType = types.Incremental
            } else if strings.Contains(object.Key, types.Full.String()) {
                backupType = types.Full
            }
            u, _ := url.Parse("")
            u.Scheme = "s3"
            u.Host = s.config.Bucket
            u.Path = path.Dir(object.Key)
            b := Backup{
                ID:       u.Path,
                Location: s.addAuthorization(u.String()),
                Typ:      backupType,
            }
            backups = append(backups, b)
        }
    }
    sort.SliceStable(backups, func(i int, j int) bool {
        return backups[i].Location < backups[j].Location
    })

When using ListObjectsV2 in this manner I kept getting the following error:

Truncated response should have continuation token set

But I get no such error and my application works using ListObjects.

[nitisht]

Thanks @gigatexal we'll take a look

[sinhaashish]

@gigatexal
I tried the following with 1200 objects agains play.minio and its works fine for me.

Can you come up with the complete sample code which can help in replicating.

func main() {
	endpoint := "play.min.io:9000"
	accessKeyID := "Q3AM3UQ867SPQQA43P2F"
	secretAccessKey := "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
	useSSL := true

	s3Client, err := minio.New(endpoint, accessKeyID, secretAccessKey, useSSL)
	if err != nil {
		fmt.Println(err)
		return
	}
	doneCh := make(chan struct{})
	defer close(doneCh)
	// List all objects from a bucket-name with a matching prefix.
	for object := range s3Client.ListObjectsV2("sinha1", "", true, doneCh) {
		if object.Err != nil {
			fmt.Println(object.Err)
			return
		}
		fmt.Println(object.Key)
	}
	return
}

[zarbis]

I've experienced same problem and ListObjects also worked for me. My code is straight from example:

	client, err := minio.New(endpoint, accessKey, secKey, ssl)
	if err != nil {
		log.Fatal(err)
	}

	doneCh := make(chan struct{})
	defer close(doneCh)
	recursive := true
	objectCh := client.ListObjectsV2(bucket, "", recursive, doneCh)

	for object := range objectCh { 
	  // delete stuff
}

Changing it to V1 or narrowing scope with second argument does the trick.

After taking look at code, I think the culprit should be S3 vendor (DigitalOcean Spaces in my case) not setting token:

minio-go/api-list.go

Lines 247 to 249 in 753e5f7

	if listBucketResult.IsTruncated && listBucketResult.NextContinuationToken == "" {
	return listBucketResult, errors.New("Truncated response should have continuation token set")
	}

[harshavardhana]

So in your case @gigatexal is the vendor Digital Ocean?

[gigatexal]

No, just an on premises deployment of Mini on top of Rook on K8s.

[harshavardhana]

We can't reproduce this with latest MinIO @gigatexal which version are you running ?

[christianhuening]

@harshavardhana This runs on an on-prem CephObjectStore using Ceph Mimic 13.2.5.

[harshavardhana]

Don't understand the context here @christianhuening ? Are you saying there is an issue or not?

[christianhuening]

Ah i just wanted to provide context against which kind of S3 implementation @gigatexal is using this.
We’re working for the same company.

[harshavardhana]

Are you saying you are using Ceph not MinIO? @christianhuening - then it looks like a Ceph bug which they need to fix. Because ListObjectsV2 works with MinIO server and AWS S3.

[harshavardhana]

It is also possible that they don't implement V2 Listing style yet? @christianhuening @gigatexal @zarbis

[kannappanr]

https://tracker.ceph.com/issues/23908

[zarbis]

https://www.digitalocean.com/community/questions/can-t-list-all-keys-on-spaces-no-continuation-token?answer=43470

[kannappanr]

Can this be closed then?

[kannappanr]

Closing this issue based on the issues in Ceph and digital ocean spaces. Please feel free to re-open the issue if you feel otherwise.

[LeoQuote]

How can I use S3v1 in mc client ? I got this error when diff a huge folder.

[harshavardhana]

We cannot @LeoQuote check why we closed the issue.

[LeoQuote]

@harshavardhana thanks for your reply , I just found a work-around for me , I can use the mc tags/RELEASE.2019-10-02T19-41-02Z according to this issue minio/mc#3067

You can use this version if you encounter this same error.

[kotarusv]

any command is giving the same error. it has anything to do with size of the bucket?
mc du prd-rtp/rtp-registry/
mc: Failed to find disk usage of prd-rtp/rtp-registry/ recursively. Truncated response should have continuation token set