HDDS-4468. Fix Goofys listBucket large than 1000 objects will stuck forever

[maobaolong]

Description

The following is the ListObjectsOutput, ListObjectsV2Output and ListObjectsInput, ListObjectsV2Input from github.com/aws/aws-sdk-go/service/s3/api.go,

For goofys talk to s3g, it use the ListObjectsOutput and ListObjectsInput, so I think s3g should return NextMarker in the ListBucketResult and use mark to list key from.

For this PR, I purpose to make BucketEndpoint of s3g can both serve to V1 and V2

type ListObjectsOutput struct {
	_ struct{} `type:"structure"`
	CommonPrefixes []*CommonPrefix `type:"list" flattened:"true"`
	Contents []*Object `type:"list" flattened:"true"`
	Delimiter *string `type:"string"`
	EncodingType *string `type:"string" enum:"EncodingType"`
	IsTruncated *bool `type:"boolean"`
	Marker *string `type:"string"`
	MaxKeys *int64 `type:"integer"`
	Name *string `type:"string"`
	NextMarker *string `type:"string"`
	Prefix *string `type:"string"`
}

type ListObjectsV2Output struct {
	_ struct{} `type:"structure"`
	CommonPrefixes []*CommonPrefix `type:"list" flattened:"true"`
	Contents []*Object `type:"list" flattened:"true"`
	ContinuationToken *string `type:"string"`
	Delimiter *string `type:"string"`
	EncodingType *string `type:"string" enum:"EncodingType"`
	IsTruncated *bool `type:"boolean"`
	KeyCount *int64 `type:"integer"`
	MaxKeys *int64 `type:"integer"`
	Name *string `type:"string"`
	NextContinuationToken *string `type:"string"`
	Prefix *string `type:"string"`
	StartAfter *string `type:"string"`
}

type ListObjectsInput struct {
	_ struct{} `type:"structure"`
	Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"`
	Delimiter *string `location:"querystring" locationName:"delimiter" type:"string"`
	EncodingType *string `location:"querystring" locationName:"encoding-type" type:"string" enum:"EncodingType"`
	Marker *string `location:"querystring" locationName:"marker" type:"string"`
	MaxKeys *int64 `location:"querystring" locationName:"max-keys" type:"integer"`
	Prefix *string `location:"querystring" locationName:"prefix" type:"string"`
	RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"`
}

type ListObjectsV2Input struct {
	_ struct{} `type:"structure"`

	Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"`
	ContinuationToken *string `location:"querystring" locationName:"continuation-token" type:"string"`
	Delimiter *string `location:"querystring" locationName:"delimiter" type:"string"`
	EncodingType *string `location:"querystring" locationName:"encoding-type" type:"string" enum:"EncodingType"`
	FetchOwner *bool `location:"querystring" locationName:"fetch-owner" type:"boolean"`
	MaxKeys *int64 `location:"querystring" locationName:"max-keys" type:"integer"`
	Prefix *string `location:"querystring" locationName:"prefix" type:"string"`
	RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"`
	StartAfter *string `location:"querystring" locationName:"start-after" type:"string"`
}

What changes were proposed in this pull request?

Fill startAfter with marker parameter when startAfter is null

Fill nextMark to the response, so that Goofys can use the nextMark to list follow-up objects.

What is the link to the Apache JIRA

HDDS-4468

How was this patch tested?

• Create an ozone cluster with s3g.
• Start a goofys with this s3g.
• Generate more than 1000 file in the test bucket
• list the goofys mounted path.

I've tested this PR on my local env.

[runzhiwang]

LGTM.

[maobaolong]

@adoroszlai Would you please take a look at this PR, thank you.

[maobaolong]

@runzhiwang Thanks for your quick review, I push a new commit to add some necessary comments, PTAL.

[maobaolong]

@GlenGeng Thanks for your review, I push a new commit to update the comment, PTAL.

[GlenGeng-awx]

+1 LGTM

[timmylicheng]

LGTM. +1

[captainzmc]

LGTM. +1

[bharatviswa504]

I have one comment.

[maobaolong]

@bharatviswa504 Thanks for your review and suggestion, I've set the marker to response also when marker is not null.

PTAL.

[elek]

+1 thanks the fix @maobaolong

[bharatviswa504]

+1 LGTM

[bharatviswa504]

Thank You everyone for the review and @maobaolong for the contribution.