[DRAFT] DAQ: Assume naive human-readable timestamps w/o timezone information to be in UTC

[amotl]

About

We would like to get rid of the current default of assuming CET for all naive timestamps at data acquisition time, and use UTC instead.

Date and time objects may be categorized as “aware” or “naive” depending on whether or not they include timezone information.

With sufficient knowledge of applicable algorithmic and political time adjustments, such as time zone and daylight saving time information, an aware object can locate itself relative to other aware objects. An aware object represents a specific moment in time that is not open to interpretation. ¹

A naive object does not contain enough information to unambiguously locate itself relative to other date/time objects. Whether a naive object represents Coordinated Universal Time (UTC), local time, or time in some other timezone is purely up to the program, just like it is up to the program whether a particular number represents metres, miles, or mass. Naive objects are easy to understand and to work with, at the cost of ignoring some aspects of reality.

-- Python documentation » Basic date and time types » Aware and Naive Objects

Details

The current implementation is:

kotori/kotori/io/protocol/util.py

Lines 180 to 199 in de224af

	def parse_timestamp(timestamp):
	if isinstance(timestamp, text_type):
	# FIXME: Maybe use system timezone here by default.
	# TODO: Make configurable via channel settings.
	# HACK: Assume CET (Europe/Berlin) for human readable timestamp w/o timezone offset
	qualified = any([token in timestamp for token in ['Z', '+', ' CET', ' CEST']])
	if not qualified:
	timestamp += ' CET'
	# Parse datetime string
	# Remark: Maybe use pandas.tseries.tools.parse_time_string?
	# TODO: Cache results of call to gettz to improve performance
	berlin = gettz('Europe/Berlin')
	tzinfos = {'CET': berlin, 'CEST': berlin}
	timestamp = parse(timestamp, tzinfos=tzinfos)
	return timestamp

Status

The patch is currently still a very early draft.

Backlog

• FIXME: Maybe use system timezone by default.
• TODO: Make timezone configurable per channel.

SIG

/cc @Tonkenfo, @einsiedlerkrebs, @msweef, @ClemensGruber, @wetterfrosch, @rohlan, @elektra42

Footnotes

1. If, that is, we ignore the effects of Relativity. ↩

[ClemensGruber]

I see the dilemma! Some of my current nodes (e.g. openhive) sending timestamps as CET some other (weatherstation) sending the timestamp as UTC, so a configuraton per channel would be preferable.

Would it be an easier provisional solution to have a flag that says respect the incoming (with data sent) timestamp and use this as timestamp in the DB vs. an ignore flag, so discard the sent timestamp and use the (server / UTC) time the data packages arrives at the server? In the most cases data is gathered and sent immediately to the server. In less situations data is stored in between. With this ignore flag we could serve both UTC and CET / other timestamped data in case there is no storage in between.

[amotl]

Introduction

I see the dilemma!

Well, true. Operating systems ¹² vs. browsers ³⁴, only picked by example, are unfortunately still not in agreement.

There are two time standards: localtime and Coordinated Universal Time (UTC). The localtime standard is dependent on the current time zone, while UTC is the global time standard and is independent of time zone values. Though conceptually different, UTC is also known as GMT (Greenwich Mean Time).

The standard used by the hardware clock (CMOS clock, the BIOS time) is set by the operating system. By default, Windows uses localtime, macOS uses UTC, other UNIX and UNIX-like systems vary. An OS that uses the UTC standard will generally consider the hardware clock as UTC and make an adjustment to it to set the OS time at boot according to the time zone.

vs.

Modern browsers which support ECMAScript 6 treat time strings like "2018-01-01T12:00:00" without a trailing "Z" as local time rather than Coordinated Universal Time (UTC). For example, recent versions of Chrome and Firefox are ES6-compliant, while Safari 11 is not. If you are using a non-ES6 browser, this means that times displayed in Altair charts may be rendered with a timezone offset, unless you explicitly use UTC time (see Using UTC Time).

Outlook

Some of my current nodes (e.g. openhive) sending timestamps as CET some other (weatherstation) sending the timestamp as UTC, so a configuraton per channel would be preferable.

So we probably won't be able to avoid introducing a dedicated "treat_local_time_as_utc" flag.

Footnotes

1. https://wiki.archlinux.org/title/System_time#Time_standard ↩

2. https://forums.linuxmint.com/viewtopic.php?t=265008 ↩

3. https://stackoverflow.com/questions/51657243/treat-utc-date-as-local-in-javascript-momentjs ↩

4. https://altair-viz.github.io/user_guide/times_and_dates.html ↩

[wetterfrosch]

And where should the information about "which local time it is" come from?

In my opinion: Either send a correct (UTC) timestamp or send no time stamp (and let the server do the stamping upon arrival) sounds just right. @ClemensGruber Can you simply suppress the timestamp of the data which "has to be" acquired in local time?

[ClemensGruber]

Suppressing should be possible by modifying some PHP code of OpenHive. And the solution for buffered data, e.g. all dato of x hours on the node would be mandatory UTC in case it is a "blank" timestamp or a full qualified when you are sending local time?