datetime module has no support for nanoseconds

[goshawk]

BPO	15443
Nosy	@malemburg, @tim-one, @mdickinson, @abalkin, @giampaolo, @bitdancer, @andyclegg, @gareth-rees, @eli-b, @serhiy-storchaka, @pganssle, @shlomoa
PRs	bpo-15443: Nanoseconds support for datetime objects (work in progress) #21987
Files	datetime.nanosecond.patch datetime.nanosecond.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = None
created_at = <Date 2012-07-24.19:36:50.070>
labels = ['type-feature', 'library', '3.10']
title = 'datetime module has no support for nanoseconds'
updated_at = <Date 2021-12-18.16:09:10.349>
user = 'https://bugs.python.org/goshawk'

bugs.python.org fields:

activity = <Date 2021-12-18.16:09:10.349>
actor = 'gdr@garethrees.org'
assignee = 'belopolsky'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2012-07-24.19:36:50.070>
creator = 'goshawk'
dependencies = []
files = ['37509', '37512']
hgrepos = []
issue_num = 15443
keywords = ['patch']
message_count = 61.0
messages = ['166326', '166331', '166333', '166335', '166336', '166338', '166340', '166345', '166361', '166364', '166383', '166385', '166386', '166387', '166414', '180125', '223039', '223042', '223066', '223068', '223071', '223073', '223074', '223075', '223077', '223078', '223080', '223082', '223083', '223106', '224360', '232952', '232962', '237338', '237807', '237809', '237819', '240243', '240244', '240290', '240291', '240292', '240294', '240299', '240398', '270266', '270535', '270885', '270886', '270887', '270888', '276748', '276749', '390474', '390479', '390483', '390486', '390491', '392382', '392418', '408859']
nosy_count = 21.0
nosy_names = ['lemburg', 'tim.peters', 'mark.dickinson', 'belopolsky', 'giampaolo.rodola', 'pythonhacker', 'Arfrever', 'r.david.murray', 'andrewclegg', 'python-dev', 'gdr@garethrees.org', 'Ramchandra Apte', 'Eli_B', 'serhiy.storchaka', 'goshawk', 'Niklas.Claesson', 'mdcb808@gmail.com', 'scoobydoo', 'tomikyos', 'p-ganssle', 'anglister']
pr_nums = ['21987']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue15443'
versions = ['Python 3.10']

Linked PRs

• gh-59648: Nanosecond support for datetime #92078

[goshawk]

As long as computers evolve time management becomes more precise and more granular.
Unfortunately the standard datetime module is not able to deal with nanoseconds even if OSes are able to. For example if i do:

print "%.9f" % time.time()
1343158163.471209049

I've actual timestamp from the epoch with nanosecond granularity.

Thus support for nanoseconds in datetime would really be appreciated

[malemburg]

Vincenzo Ampolo wrote:

As long as computers evolve time management becomes more precise and more granular.
Unfortunately the standard datetime module is not able to deal with nanoseconds even if OSes are able to. For example if i do:

print "%.9f" % time.time()
1343158163.471209049

I've actual timestamp from the epoch with nanosecond granularity.

Thus support for nanoseconds in datetime would really be appreciated

I would be interested in an actual use case for this.

[goshawk]

On 07/24/2012 01:28 PM, Marc-Andre Lemburg wrote:

I would be interested in an actual use case for this.

Alice has a dataset with nanosecond granularity. He wants to make a
python library to let Bob access the dataset. Nowadays Alice has to
implement her own time class losing all the flexibility of the datetime
module. With this enhancement she can provide a library that just uses
the standard python datetime module. Her library will get the needed
time format, including nanoseconds.

Many python sql libraries, like the one in django e the one in web2py,
relay on datetime objects for time representation. Bob has a web2py
website that has some data with nanosecond granularity. Nowadays Bob has
to store this data as a string or a long number without the ability to
use the powerful datetime module. With this enhancement Bob doesn't need
to build or learn another interface, he can just use the datetime module
using microseconds or nanoseconds as needed.

Google search for "python datetime nanoseconds" shows more than 141k
results:
https://www.google.com/search?sourceid=chrome&ie=UTF-8&q=python+time#hl=en&biw=1615&bih=938&sclient=psy-ab&q=python+datetime+nanoseconds&oq=python+datetime+nanoseconds

So this is definitively a requested feature. And as soon as technology
evolves more people will ask for it.

I imagine something like:

import datetime
nano_time = datetime.datetime(year=2012, month=07, day=24, hour=14,
minute=35, second=3, microsecond=53, nanosecond=27)

in case you need nanosecond granularity. if you don't need it just skip
the nanosecond part and the module works like it's now. Of course
strftime format should be updated to support nanoseconds.

I can write a patch if some dev can maybe review it.

Before someone takes the datetime source code and starts a third part
module that supports nanoseconds, I think this enhancement has almost
null impact in existing code and makes the datetime module even more
powerful. It's up to the Cpython admins to decide between maintaining
datetime module up to date with new needs or let third part modules take
care of those lacks.

Best Regards,

Vincenzo Ampolo
http://vincenzo-ampolo.net
http://goshawknest.wordpress.com

[bitdancer]

I believe Marc-Andre was looking for an actual real-world use case rather than a hypothetical one. We discussed this briefly on the irc channel and we think Guido vetoed it on a YAGNI basis (we haven't checked the archives though...) so a real world use case is probably required.

[goshawk]

This is a real use case I'm working with that needs nanosecond precision
and lead me in submitting this request:

most OSes let users capture network packets (using tools like tcpdump or
wireshark) and store them using file formats like pcap or pcap-ng. These
formats include a timestamp for each of the captured packets, and this
timestamp usually has nanosecond precision. The reason is that on
gigabit and 10 gigabit networks the frame rate is so high that
microsecond precision is not enough to tell two frames apart.
pcap (and now pcap-ng) are extremely popular file formats, with millions
of files stored around the world. Support for nanoseconds in datetime
would make it possible to properly parse these files inside python to
compute precise statistics, for example network delays or round trip times.

Other case is in stock markets. In that field information is timed in
nanoseconds and have the ability to easily deal with this kind of
representation natively with datetime can make the standard module even
more powerful.

The company I work for is in the data networking field, and we use
python extensively. Currently we rely on custom code to process
timestamps, a nanosecond datetime would let us avoit that and use
standard python datetime module.

Best Regards,

---
Vincenzo Ampolo
http://vincenzo-ampolo.net
http://goshawknest.wordpress.com

[bitdancer]

Are the nanosecond timestamps timestamps or strings? If they are timestamps it's not immediately obvious why you want to convert them to datetime objects, so motivating that would probably help. On the other hand the fact that you have an application that does so is certain an argument for real world applicability.

[bitdancer]

Even if accepted this can't get fixed in 2.7, so removing that from versions.

[goshawk]

On 07/24/2012 04:20 PM, R. David Murray wrote:

R. David Murray <rdmurray@bitdance.com> added the comment:

Are the nanosecond timestamps timestamps or strings? If they are timestamps it's not immediately obvious why you want to convert them to datetime objects, so motivating that would probably help. On the other hand the fact that you have an application that does so is certain an argument for real world applicability.

It depends. When they are exported for example as csv (this can be the
case of market stock) or json (which is close to my case) that's a
string so having a datetime object may be very helpful in doing datetime
adds, subs, <, deltas and in changing representation to human readable
format thanks to strftime() without loosing precison and maintaining
readability.

Think about a web application. User selects year, month, day, hour,
minute, millisecond, nanosecond of an event and the javascript does a
ajax call with time of this format (variant of iso8601):
YYYY-MM-DDTHH:MM:SS.mmmmmmnnn (where nnn is the nanosecond representation).
The python server takes that string, converts to a datetime, does all
the math with its data and gives the output back using labeling data
with int(nano_datetime.strftime('MMSSmmmmmmnnn')) so I've a sequence
number that javascript can sort and handle easily.

It's basically the same you already do nowadays at microseconds level,
but this time you have to deal with nanosecond data.

I agree with the YAGNI principle and I think that we have a clear
evidence of a real use case here indeed.

Best Regards

[vstinner]

See the PEP-410.

[malemburg]

Vincenzo Ampolo wrote:

Vincenzo Ampolo <vincenzo.ampolo@gmail.com> added the comment:

This is a real use case I'm working with that needs nanosecond precision
and lead me in submitting this request:

most OSes let users capture network packets (using tools like tcpdump or
wireshark) and store them using file formats like pcap or pcap-ng. These
formats include a timestamp for each of the captured packets, and this
timestamp usually has nanosecond precision. The reason is that on
gigabit and 10 gigabit networks the frame rate is so high that
microsecond precision is not enough to tell two frames apart.
pcap (and now pcap-ng) are extremely popular file formats, with millions
of files stored around the world. Support for nanoseconds in datetime
would make it possible to properly parse these files inside python to
compute precise statistics, for example network delays or round trip times.

Other case is in stock markets. In that field information is timed in
nanoseconds and have the ability to easily deal with this kind of
representation natively with datetime can make the standard module even
more powerful.

The company I work for is in the data networking field, and we use
python extensively. Currently we rely on custom code to process
timestamps, a nanosecond datetime would let us avoit that and use
standard python datetime module.

Thanks for the two use cases.

You might want to look at mxDateTime and use that for your timestamps.
It does provide full C double precision for the time part of a timestamp,
which covers nanoseconds just fine.

[abalkin]

On Wed, Jul 25, 2012 at 4:17 AM, Marc-Andre Lemburg <report@bugs.python.org> wrote:

... full C double precision for the time part of a timestamp,
which covers nanoseconds just fine.

No, it does not:

>>> import time
>>> t = time.time()
>>> t + 5e-9 == t
True

In fact, C double precision is barely enough to cover microseconds:

>>> t + 1e-6 == t
False

>>> t + 1e-7 == t
True