Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Provide option to not flatten json data #29

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Provide option to not flatten json data #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a0eaff2
Provide option to not flatten json data
mpcarter Sep 30, 2020
9bf4961
Provide option to not flatten json data
mpcarter Sep 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ Full list of options in `config.json`:
| compression | String | No | The type of compression to apply before uploading. Supported options are `none` (default) and `gzip`. For gzipped files, the file extension will automatically be changed to `.csv.gz` for all files. |
| naming_convention | String | No | (Default: None) Custom naming convention of the s3 key. Replaces tokens `date`, `stream`, and `timestamp` with the appropriate values. <br><br>Supports "folders" in s3 keys e.g. `folder/folder2/{stream}/export_date={date}/{timestamp}.csv`. <br><br>Honors the `s3_key_prefix`, if set, by prepending the "filename". E.g. naming_convention = `folder1/my_file.csv` and s3_key_prefix = `prefix_` results in `folder1/prefix_my_file.csv` |
| temp_dir | String | | (Default: platform-dependent) Directory of temporary CSV files with RECORD messages. |
| flatten | Boolean | No | (Default: True) Flatten json data to columns. |

### To run tests:

Expand Down
6 changes: 4 additions & 2 deletions target_s3_csv/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ def persist_messages(messages, config, s3_client):

delimiter = config.get('delimiter', ',')
quotechar = config.get('quotechar', '"')
flatten = config.get('flatten', True)

# Use the system specific temp directory if no custom temp_dir provided
temp_dir = os.path.expanduser(config.get('temp_dir', tempfile.gettempdir()))
Expand Down Expand Up @@ -89,7 +90,7 @@ def persist_messages(messages, config, s3_client):

file_is_empty = (not os.path.isfile(filename)) or os.stat(filename).st_size == 0

flattened_record = utils.flatten_record(record_to_load)
flattened_record = utils.flatten_record(record_to_load) if flatten else record_to_load

if o['stream'] not in headers and not file_is_empty:
with open(filename, 'r') as csvfile:
Expand Down Expand Up @@ -190,4 +191,5 @@ def main():


if __name__ == '__main__':
main()
main()