Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timestamps don't round-trip through CSV writer/reader #778

Closed
bjchambers opened this issue Sep 14, 2021 · 2 comments · Fixed by #849
Closed

Timestamps don't round-trip through CSV writer/reader #778

bjchambers opened this issue Sep 14, 2021 · 2 comments · Fixed by #849
Labels

Comments

@bjchambers
Copy link
Contributor

Describe the bug

Create a TimestampNanosecondArray and write it to CSV.
Read the resulting CSV back and it will be parsed as UTF-8 not TimestampNanosecondType.
Cast the parsed string to TimestampNanosecondType::DATA_TYPE and the value will be off by the local timezone offset.

I think the solution may be as simple as having the CSV writer append the Z suffix to indicate the timestamp is in GMT time, but I'm not sure.

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

@bjchambers bjchambers added the bug label Sep 14, 2021
@novemberkilo
Copy link
Contributor

novemberkilo commented Oct 22, 2021

@alamb I suspect that #832 addresses this bug. I will check // @bjchambers

@novemberkilo
Copy link
Contributor

Here's what I think is going on.

#832 addresses the issue raised here apache/datafusion#958

In essence, it allows for strings indicating local time to be read as timestamps (and not UTF-8)

The round-trip test suggested in this bug confirmed that the value read in is off by the local timezone offset. The same test confirms that round tripping is sound once the writer appends a Z - i.e. writes out the timestamp in UTC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants