Skip to content

Conversation

@gmanolache
Copy link
Contributor

@gmanolache gmanolache commented Nov 16, 2020

Description of changes:

Generating a configuration failed for Parquet as it was expecting all uppercase, changed it everywhere to match the rest of the data formats as well.

Fixed an issue when copying the manifest that contained parquet files because Redshift was expecting the meta property content_length

Here are the cloudwatch logs I was getting:

  | 2020-11-15T17:38:14.824+01:00 | COPY my_test_table from 's3://lambda-redshift-loader-test/files/manifest-2020-11-15 16:38:14-6082' with credentials as 'aws_access_key_id=meyaccesskey;aws_secret_access_key=mykey' manifest format as PARQUET;
-- | -- | --
  | 2020-11-15T17:38:14.824+01:00 | commit;
  | 2020-11-15T17:38:16.200+01:00 | error:
  | 2020-11-15T17:38:16.200+01:00 | -----------------------------------------------
  | 2020-11-15T17:38:16.200+01:00 | error: File entry does not have 'meta' key.
  | 2020-11-15T17:38:16.200+01:00 | code: 15003
  | 2020-11-15T17:38:16.200+01:00 | context:
  | 2020-11-15T17:38:16.200+01:00 | query: 90820823
  | 2020-11-15T17:38:16.200+01:00 | location: spectrum_manifest.cpp:204
  | 2020-11-15T17:38:16.200+01:00 | process: padbmaster [pid=17049]

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Fixed an issue when generating the manifest for Parquet that didnt contain meta information about content length.

(cherry picked from commit d107798)
@IanMeyers IanMeyers merged commit a4a1a77 into awslabs:master Nov 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants