Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Give option to keep CSV files locally and skip upload to S3 #30

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Give option to keep CSV files locally and skip upload to S3 #30

wants to merge 2 commits into from

Commits on Oct 1, 2020

  1. Give option to keep CSV files locally and skip upload to S3 

    When testing pipelines on a local machine, you don't always need
to actually send the files to S3. Sometimes you just want to
verify the data was written correctly. With this change,
specifying the s3_bucket value as 'localhost' will skip the
upload to S3 and not remove the files from temp_dir.

It also skips creation of AWS session, avoiding need to have
AWS credentials for local testing.
    @mpcarter
    mpcarter committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    97ca2a0 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2021

  1. Merge branch 'master' into s3-bucket-local

    @Samira-El
    Samira-El authored Sep 14, 2021
    Configuration menu
    Copy the full SHA
    1bf4dca View commit details
    Browse the repository at this point in the history