Refactoring API client Waiters for better refactored API Client

[jasdel]

This issue details how the v2 AWS SDK for Go's API client waiters need to be updated based on the API client refactor proposed in #438.

Proposal #438 would remove the, WaitUntil methods from the API clients, and added as function's to the API client's package. The waiters are higher level functionality provided by the SDK that build on the API to add novel functionality. Such as waiting for a Amazon S3 bucket to exist.

Proposed API Client Waiter Refactor

The proposed refactor of the SDK's clients is split into four parts, API Client, Paginators, Waiters, and Request Presigner helpers. This proposal covers the API client waiter changes. Similar to paginators, waiters are higher-level functionality that build on top of the service's API.

The following example demonstrates an application creating a waiter to wait for a specific S3 object to exist.

client := s3.NewClient(cfg)

err := s3.WaitUntilObjectExists(ctx, client, &s3.HeadObjectInput{
	Bucket: aws.String("myBucket"),
	Key: aws.String("myKey"),
})

The WaitUntilObjectExists function is called with an API client for the HeadObject operation. The waiter blocks until either the object exists or timeout. The HeadObjectInput type provides the input parameters to describe the object to wait for. Additional waiter functional options can be provided as variadic arguments. These options allow you to modify the SDK's wait attempts and duration.

An error is returned when the resource has not reached the desired state within a predetermined expiration period. The expiration period is modeled by the API service team, but can be overridden via the waiter's functional options.

The WaitUntil waiter functions are not methods on the client because they are not a part of the API, but higher level concept that add novel functionality. If the waiter functions were methods on the API client, applications mocking the API client would need to mock the waiter as well or experience unexpected behavior.

The following is an example of the waiter function's signature.

func WaitUntilObjectExists(ctx context.Context, client HeadObjectsClient, input *s3.HeadObjectInput, opts ...func(*aws.Waiter)) error {
	// Implementation
}

Mocking the waiter

While there are several ways to mock out the waiter behavior, the most straight forward would be via an unexported package variable. The behavior of the waiter would only need to be replaced if the your mock API clients doesn't include a successful response for the for the operation used by the waiter. Or you want to exercise your code's handling of a waiter failing.

The following demonstrates how a package variable can be used to replace the behavior of the WaitUntilObjectExists waiter.

var waitUntilObjectExists = s3.WaitUntilObjectExists

// Alternatively the s3api.Client could be used instead of creating a custom
// interface.
type PutFileClient interface {
	s3api.PutObjectClient
	s3api.HeadObjectClient
}

func PutFile(ctx context.Context, client PutFileClient, file io.ReadSeeker, key string) error {
	putRes, err := objectPutter.PutObject(ctx, &s3.PutObjectInput{
		Bucket: aws.String("mybucket"),
		Key: &key,
	})
	if err != nil {
		return fmt.Errorf("failed to put object %v", err)
	}

	err = waitUntilObjectExists(ctx, client, &s3.HeadObjectInput{
		Bucket: aws.String("myBucket"),
		Key: &key,
	})
	if err != nil {
		return fmt.Errorf("object failed to exist, %v", err)
	}

	return nil
}

The following demonstrates how a test unit would replace the behavior of the waiter.

func TestPutFile(t *testing.T) {
	origWaiter := waitUntilObjectExists
	waitUntilObjectExists = mockWaitUntilObjectExists
	defer func() {
		waitUntilObjectExists = origWaiter
	}()

	client := mockPutObjectClient()

	mockFile := mockUploadFile()
	err := PutFile(context.Background, client, mockFile, "key")

	// assert behavior of putFile created by the mock client and waiter.
}

[jasdel]

Should waiter options expose a way for the application to modify or change the modeled waiter's wait state? This option would be in addition to the other wait options like max attempts, delay, etc.

In this proposal, WaitCheck is a func member on the HeadBucketWaiterOptions struct that is auto populated by the waiter method with the modeled wait state. When calling the individual waiter the application could provide alternate wait check behavior, and optionally fallback to the modeled behavior if desired. This function would then be used as a parameter when constructing the waiter retry middleware.

• Questions: Should the waiter's constructor take the functional options as input or just the client for the API operation? If it takes the functional options, the WaitCheck being shared would be awkward since waiters like WaitBucketExists and WaitBucketNotExists are opposites. Alternatives would be to defined waiters with single wait method, (e.g. BucketExistsWaiter), not per operation with multiple waiters methods, (e.g. HeadBucketWaiter).

client := s3.NewFromConfig(cfg)

waiter := s3.NewHeadBucketWaiter(client)
err := waiter.WaitForBucketExists(context.TODO(), &s3.HeadBucketInput{
         Bucket: aws.String("example"),
    }, func(o *s3.HeadBucketWaiterOptions) {
         baseCheck := o.WaitCheck
         o.WaitCheck = func(ctx context.Context, result *s3.HeadBucketOutput) (bool, error) {
                // Some custom wait check, that delegates to the SDK's check if not succeeded
                if resp := awshttp.GetRawResponse(result.ResultMetadata); resp != nil && resp.StatusCode == 404 {
                      return true, nil // continue waiting
                }
                return baseCheck(ctx, result)
         }
    })

[jasdel]

Fixed in aws/smithy-go#237, but API models wait waiters have not yet been added to the v2 SDK.

[jasdel]

Closing this and can track model update in #989

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.