Autodetector for ISO date strings #767
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for dlt-hub-docs canceled.
|
rudolfix
requested changes
Nov 17, 2023
dlt/common/schema/utils.py
Outdated
| @@ -698,4 +698,4 @@ def standard_hints() -> Dict[TColumnHint, List[TSimpleRegex]]: | |||
|
|
|||
|
|
|||
| def standard_type_detections() -> List[TTypeDetections]: | |||
| return ["iso_timestamp"] | |||
| return ["iso_timestamp", "iso_date"] | |||
There was a problem hiding this comment.
this is the only thing I'd change. I'm not sure that generating dates should be the default. date-like strings are simple and common - IMO automatic conversion to date database type is an unexpected behavior.
instead you can always use add_type_detection - from within a dlt.source like here https://dlthub.com/docs/general-usage/schema#schema-is-modified-in-the-source-function-body
or just before loading: source().schema.add_type_detection("iso_date").
we'll be releasing dlt soon so if you are OK with it we can do this change here
There was a problem hiding this comment.
I'm fine with that :)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
ISO 8601 defines date strings as
YYYY-MM-DD. We already have autodetection for ISO timestamp strings and detect ISO-compliant date strings like2023-11-15as date objects (see here). However, neither does theis_iso_timestampautodetection function pick date strings up as dates, nor is there a dedicatedis_iso_dateauto-detection function.As a result, sources that yield "native" date objects get loaded with data type
date, whereas sources that yield ISO-compliant date strings get loaded with data typetextThis PR adds an
iso_dateautodetection function to the list of default type detectors. This change is fully backward compatible with existing schemas, so that we don't need to bump the schema engine version.Related Issues