AP1011 - Custom archive bucket/folder

[llehtinen]

Problem

The archive_load_files feature added in #178 only allows archival in the same S3 bucket as is used to the Snowflake imports.

This means that S3 bucket versioning will either be enabled for both archived files and the temporary load files, or neither.

Versioning makes sense for archived files, where accidental deletes could be problematic. It does not, however, make sense for the temporary load files, and having it enabled for them will incur unnecessary cost.

Proposed changes

This PR allows configuring a separate S3 bucket to be used as the archive destination, as well as customizing the S3 prefix used.

Types of changes

What types of changes does your code introduce to PipelineWise?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)

Checklist

• Description above provides context of the change
• I have added tests that prove my fix is effective or that my feature works
• Unit tests for changes (not needed for documentation changes)
• CI checks pass with my changes
• Bumping version in setup.py is an individual PR and not mixed with feature or bugfix PRs
• Commit message/PR title starts with [AP-NNNN] (if applicable. AP-NNNN = JIRA ID)
• Branch name starts with AP-NNN (if applicable. AP-NNN = JIRA ID)
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions

[koszti]

how does the authentication work if we copy between two buckets? Can we use the same s3 upload_client for both buckets that initialised at

pipelinewise-target-snowflake/target_snowflake/upload_clients/s3_upload_client.py

Line 21 in 4de4250

def _create_s3_client(self, config=None):

The aws_profile , aws_access_key_id, aws_secret_access_key , aws_session_token , s3_region_name, and s3_endpoint_url optional parameters should be valid for both buckets. Is that fine?

[llehtinen]

From Boto3 docs:

All copy requests must be authenticated. Additionally, you must have read access to the source object and write access to the destination bucket. For more information, see REST Authentication . Both the Region that you want to copy the object from and the Region that you want to copy the object to must be enabled for your account.

So, if we want to use the copy_object method (rather than uploading object to two different buckets and using two sets of credentials), the credentials used for Snowflake imports must be valid for writing to the archive bucket as well. I tested this in staging and it works. Not sure how different region or endpoint url would play out.