E2E test bootstrapping fails S3 object copy

[RedbackThomson]

Describe the bug
When attempting to use the s3.duplicate_bucket_contents method in the acktest package, the e2e test container fails with the following traceback:

Traceback (most recent call last):
  File "service_bootstrap.py", line 126, in <module>
    config = service_bootstrap()
  File "service_bootstrap.py", line 120, in service_bootstrap
    create_data_bucket(),
  File "service_bootstrap.py", line 97, in create_data_bucket
    subprocess.call(
  File "/usr/local/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'aws'

It appears that, under the hood, the boto3 library is using system calls to the aws CLI. (yeah, I know, why?)

The e2e test container does not contain the aws CLI - see here. This may be the cause of the error.

See this prow job log for full context on the error.

Steps to reproduce

1. Include the s3.duplicate_bucket_contents as part of the service_bootstrap for any given service
2. Run the e2e tests on the Prow cluster

Expected outcome
The bucket contents is duplicate from the source bucket, without any error message or exit code.

Environment

• Kubernetes version
• Using EKS (yes/no), if so version?
• AWS service targeted (S3, RDS, etc.)

[surajkota]

awscli is being used here - https://github.com/aws-controllers-k8s/sagemaker-controller/blob/main/test/e2e/service_bootstrap.py#L94-L109

but while trying to install it in requirements.txt it creates conflicts with pyyaml 5.4

[RedbackThomson]

Ahh good catch. I misunderstood this error as coming from the common test package. In that case I'm going to close this issue as your team is already aware of the bug.