Allow handling timezone-aware datetime values when inserting or updating #557

amotl · 2023-06-26T08:22:31Z

Problem

As reported at #626, when inserting timezone-aware datetime values, the CrateJsonEncoder fails with

TypeError: can't subtract offset-naive and offset-aware datetimes

This patch intends to fix it.

Reference

[CrateDB] Add support for data acquisition and data export daq-tools/kotori#148 (comment)

matriv

LGTM, Left a comment for the changelog

CHANGES.txt

src/crate/client/http.py

mfussenegger

I think it would be worth also extending the data type section in the docs and explain the behavior of datetime with tagged timezone.

One could assume that it persists the timezone information as well, but it's instead converting it to a timestamp with 1970-01-01 as epoch. That's an important distinction

docs/by-example/client.rst

amotl · 2023-06-26T12:16:16Z

docs/data-types.rst

+.. NOTE::
+
+    Values of ``TIMESTAMP`` columns will always be stored using a ``LONG`` type,
+    representing the `Unix time`_ (epoch) timestamp, i.e. number of seconds which
+    have passed since 00:00:00 UTC on Thursday, 1 January 1970.
+
+    This means, when inserting or updating records using timezone-aware Python
+    ``datetime`` objects, timezone information will not be preserved. If you
+    need to store it, you will need to use a separate column.


@mfussenegger: This paragraph intends to reflect your suggestion. Thank you again, and let me know about any improvements, if you think it could be conveyed better.

-- https://crate-python--557.org.readthedocs.build/en/557/data-types.html

docs/by-example/client.rst

matriv

Thx again, but please wait for @mfussenegger to also check.

CHANGES.txt

mfussenegger · 2023-06-28T13:22:30Z

docs/by-example/client.rst

+Both when inserting or updating data, values for ``TIMESTAMP`` columns can be obtained
+in different formats. Both literal strings and datetime objects are supported.


Suggested change

Both when inserting or updating data, values for ``TIMESTAMP`` columns can be obtained

in different formats. Both literal strings and datetime objects are supported.

Values for ``TIMESTAMP`` columns can be set as either string literal or datetime object::

And then maybe:

"If the datetime object contains timezone information, it is converted to UTC and the timezone information is lost:"

(e.g. do a update returning)

If it [all three variants] contains timezone information, it is converted to UTC, and the timezone information is discarded.

Text has been changed with 677bef3, thanks.

(e.g. do a update returning)

We may elaborate this on another occasion? I will be all ears to adjust the documentation, in order to improve guidance, but I would need a bit of education first.

mfussenegger · 2023-06-28T13:24:26Z

docs/by-example/client.rst

+    >>> import datetime as dt
+    >>> timestamp_full = "2023-06-26T09:24:00.123+02:00"
+    >>> timestamp_date = "2023-06-26"
+    >>> datetime_aware = dt.datetime.fromisoformat("2023-06-26T09:24:00.123+02:00")
+    >>> datetime_naive = dt.datetime.fromisoformat("2023-06-26T09:24:00.123")
+    >>> datetime_date = dt.date.fromisoformat("2023-06-26")
+    >>> cursor.execute("UPDATE locations SET date=? WHERE name='Cloverleaf'", (timestamp_full, ))
+    >>> cursor.execute("UPDATE locations SET date=? WHERE name='Cloverleaf'", (timestamp_date, ))
+    >>> cursor.execute("UPDATE locations SET date=? WHERE name='Cloverleaf'", (datetime_aware, ))
+    >>> cursor.execute("UPDATE locations SET date=? WHERE name='Cloverleaf'", (datetime_naive, ))
+    >>> cursor.execute("UPDATE locations SET date=? WHERE name='Cloverleaf'", (datetime_date, ))


If this includes date as well I guess the sentence above should mention that too.

Thanks, added with 677bef3.

docs/data-types.rst

amotl · 2023-06-28T13:51:33Z

@mfussenegger: Thanks for your valuable suggestions. 677bef3 bundles all of them into a corresponding fixup commit.

Co-authored-by: Mathias Fußenegger <mfussenegger@users.noreply.github.com>

amotl requested review from mfussenegger and matriv June 26, 2023 08:36

amotl marked this pull request as ready for review June 26, 2023 08:36

amotl force-pushed the amo/aware-datetimes branch from 947b470 to 48fe6a8 Compare June 26, 2023 08:38

matriv approved these changes Jun 26, 2023

View reviewed changes

CHANGES.txt Outdated Show resolved Hide resolved

amotl force-pushed the amo/aware-datetimes branch from 48fe6a8 to 55192cf Compare June 26, 2023 11:21

mfussenegger reviewed Jun 26, 2023

View reviewed changes

src/crate/client/http.py Outdated Show resolved Hide resolved

amotl force-pushed the amo/aware-datetimes branch 2 times, most recently from 911765c to e7ad5b5 Compare June 26, 2023 11:52

amotl requested review from mfussenegger and matriv June 26, 2023 11:54

mfussenegger reviewed Jun 26, 2023

View reviewed changes

matriv reviewed Jun 26, 2023

View reviewed changes

docs/by-example/client.rst Outdated Show resolved Hide resolved

amotl force-pushed the amo/aware-datetimes branch from e7ad5b5 to 6c14e44 Compare June 26, 2023 12:13

amotl commented Jun 26, 2023

View reviewed changes

amotl requested a review from mfussenegger June 26, 2023 12:20

amotl commented Jun 26, 2023

View reviewed changes

docs/by-example/client.rst Show resolved Hide resolved

amotl changed the title ~~Allow handling "aware" datetime values when inserting or updating~~ Allow handling timezone-aware datetime values when inserting or updating Jun 26, 2023

amotl force-pushed the amo/aware-datetimes branch from 6c14e44 to f88d10c Compare June 26, 2023 14:15

matriv approved these changes Jun 27, 2023

View reviewed changes

mfussenegger approved these changes Jun 28, 2023

View reviewed changes

amotl force-pushed the amo/aware-datetimes branch 2 times, most recently from 3ebf25d to 677bef3 Compare June 28, 2023 13:50

amotl and others added 3 commits June 28, 2023 16:04

Fix CHANGES.txt

87ef216

Allow handling timezone-aware datetime values when inserting or updating

1fdef25

Update documentation about obtaining timezone-aware timestamps

a494bce

Co-authored-by: Mathias Fußenegger <mfussenegger@users.noreply.github.com>

amotl force-pushed the amo/aware-datetimes branch from 677bef3 to a494bce Compare June 28, 2023 14:04

amotl merged commit 4e20913 into master Jul 4, 2023

amotl deleted the amo/aware-datetimes branch July 4, 2023 08:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow handling timezone-aware datetime values when inserting or updating #557

Allow handling timezone-aware datetime values when inserting or updating #557

amotl commented Jun 26, 2023 •

edited

Loading

matriv left a comment

mfussenegger left a comment

amotl Jun 26, 2023 •

edited

Loading

matriv left a comment

mfussenegger Jun 28, 2023

amotl Jun 28, 2023 •

edited

Loading

mfussenegger Jun 28, 2023

amotl Jun 28, 2023

amotl commented Jun 28, 2023

		Both when inserting or updating data, values for ``TIMESTAMP`` columns can be obtained
		in different formats. Both literal strings and datetime objects are supported.

	Both when inserting or updating data, values for ``TIMESTAMP`` columns can be obtained
	in different formats. Both literal strings and datetime objects are supported.
	Values for ``TIMESTAMP`` columns can be set as either string literal or datetime object::

Allow handling timezone-aware datetime values when inserting or updating #557

Allow handling timezone-aware datetime values when inserting or updating #557

Conversation

amotl commented Jun 26, 2023 • edited Loading

Problem

Reference

matriv left a comment

Choose a reason for hiding this comment

mfussenegger left a comment

Choose a reason for hiding this comment

amotl Jun 26, 2023 • edited Loading

Choose a reason for hiding this comment

matriv left a comment

Choose a reason for hiding this comment

mfussenegger Jun 28, 2023

Choose a reason for hiding this comment

amotl Jun 28, 2023 • edited Loading

Choose a reason for hiding this comment

mfussenegger Jun 28, 2023

Choose a reason for hiding this comment

amotl Jun 28, 2023

Choose a reason for hiding this comment

amotl commented Jun 28, 2023

amotl commented Jun 26, 2023 •

edited

Loading

amotl Jun 26, 2023 •

edited

Loading

amotl Jun 28, 2023 •

edited

Loading