Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix warnings seen when ingesting cloud or storage files #1592

Merged
merged 11 commits into from
Dec 22, 2021

Conversation

eiffel777
Copy link
Contributor

@eiffel777 eiffel777 commented Dec 16, 2021

After merging PR #1586, MySQL warnings that were not displayed are now displayed. Those warnings come from the shredding of storage and cloud data.

For storage data, the dt column truncates the date string specified because it contains timezone information. Date time strings with timezones are not compatible with datetime columns. Changing this column to a varchar and then using STR_TO_DATE to get the correct date format fixes this warning. Since the timezone information does not appear to be used in the storage realm, this warning does not cause a bug.

Example of storage data warning:

2021-12-16 16:31:30 [warning] Warning 1265 Data truncated for column 'dt' at row 1

The date warning seen in the storage data exists for the cloud resource specifications as well. The warning is prevented by converting to a varchar column and using STR_TO_DATE when loading the date into a datetime column. The other warning is for the size column, defined as int(11). The size field for image.* events are measured in bytes, and the Jetstream test data has images sizes that are too big for that column definition. The column can be changed to a bigint to fix this. The image.* events are not used and discarded during ingestion, so this does not cause a bug or require a re-ingest of data.

Example of warning for cloud resource specifications:

2021-12-16 16:27:50 [warning] Warning 1265 Data truncated for column 'fact_date' at row 1

Example of warning for cloud events:

2021-12-16 16:27:49 [warning] Warning 1264 Out of range value for column 'size' at row 1

Tests performed

Tested in docker

Checklist:

  • The pull request description is suitable for a Changelog entry
  • The milestone is set correctly on the pull request
  • The appropriate labels have been added to the pull request

@eiffel777 eiffel777 added enhancement Enhancement of the functionality of an existing feature Category:ETL Extract Transform Load labels Dec 16, 2021
@eiffel777 eiffel777 added this to the 10.0.0 milestone Dec 16, 2021
@eiffel777 eiffel777 self-assigned this Dec 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category:ETL Extract Transform Load enhancement Enhancement of the functionality of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants