Datetime NoneType after calling Py_Finalize and Py_Initialize

[DennyWeinberg]

BPO	27400
Nosy	@ncoghlan, @abalkin, @tiran, @Fak3, @JimJJewett, @zooba, @MojoVampire, @ndjensen, @yan12125, @ammaraskar, @cschramm, @pganssle
Files	issue27400.patch 27400.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = None
created_at = <Date 2016-06-27.13:49:18.523>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'Datetime NoneType after calling Py_Finalize and Py_Initialize'
updated_at = <Date 2018-07-05.17:14:18.266>
user = 'https://bugs.python.org/DennyWeinberg'

bugs.python.org fields:

activity = <Date 2018-07-05.17:14:18.266>
actor = 'pablogsal'
assignee = 'belopolsky'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2016-06-27.13:49:18.523>
creator = 'Denny Weinberg'
dependencies = []
files = ['44678', '46478']
hgrepos = []
issue_num = 27400
keywords = ['patch']
message_count = 15.0
messages = ['269379', '269381', '269751', '269914', '269916', '276598', '276604', '276622', '276626', '276630', '276632', '277184', '285560', '286618', '286636']
nosy_count = 15.0
nosy_names = ['ncoghlan', 'belopolsky', 'christian.heimes', 'palm.kevin', 'Roman.Evstifeev', 'Jim.Jewett', 'steve.dower', 'josh.r', 'ndjensen', 'yan12125', 'Denny Weinberg', 'ammar2', 'cschramm', 'p-ganssle', 'FrankBlabu']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27400'
versions = ['Python 3.6', 'Python 3.7']

Linked PRs

• gh-71587: Clear cached strptime module #101783
• gh-71587: Isolate _datetime #102995
• gh-71587: Establish global state in _datetime #110475
• gh-71587: Drop local reference cache to _strptime module in _datetime #120224
• [3.13] gh-71587: Drop local reference cache to _strptime module in _datetime (gh-120224) #120424
• [3.12] gh-71587: Drop local reference cache to _strptime module in _datetime (GH-120224) #120431

[DennyWeinberg]

After calling Py_Finalize and Py_Initialize I get the message "attribute of type 'NoneType' is not callable" on the datetime.strptime method.

Example:

from datetime import datetime
s = '20160505 160000'
refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')

The first call works fine but it crashes after the re initialization.

Workaround:

from datetime import datetime
s = '20160505 160000'
try:
    refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')
except TypeError:
    import time
    refdatim = datetime.fromtimestamp(time.mktime(time.strptime(s, '%Y%m%d %H%M%S')))

Related Issue: bpo-17408 ("second python execution fails when embedding")

[DennyWeinberg]

Just to be clear:

The error happens after these steps:

1. Call strptime
2. Call cpython function "Py_Finalize" and "Py_Initialize"
3. Call strptime again

Now we get the error "attribute of type 'NoneType' is not callable"

[ncoghlan]

Thanks for the report Denny. Looking at https://hg.python.org/cpython/file/30099abdb3a4/Modules/datetimemodule.c#l3929, there's a problematic caching of the "_strptime" module that is almost certainly the cause of the problem - it will attempt to call _strptime._strptime from the already finalized interpreter rather than the new one.

It should be possible to adjust that logic to permit a check for _strptime._strptime being set to None, and reimporting _strptime in that case.

[ammaraskar]

Is there any particular reason that datetime.strptime caches the imported module like that?

From a quick search, these two other examples don't bother with any caching:

cpython/Modules/timemodule.c

Line 709 in 2d26423

PyObject *strptime_module = PyImport_ImportModuleNoBlock("_strptime");

cpython/Python/traceback.c

Line 277 in 64fe35c

io = PyImport_ImportModuleNoBlock("io");

[ncoghlan]

Aye, skipping the caching entirely would be an even simpler solution - the only thing it is saving in the typical case is a dictionary lookup in the modules cache.

[MojoVampire]

Nick: Looks like it's quite a bit more work than just a dict lookup. That PyImport_ImportModuleNoBlock call (which seems odd; the implementation of NoBlock is just to wrap the blocking function; guess we don't allow non-blocking imports anymore and this is just to avoid changing all the names elsewhere?) involves a *lot* more work than just a dict lookup (it devolves to a PyImport_Import call https://hg.python.org/cpython/file/3.5/Python/import.c#l1743 , which basically does everything involved in the import process aside from actually reading/parsing the file unconditionally, because of how weird __import__ overrides can be, I guess).

While it's not a perfect comparison, compare:

>>> import _strptime  # It's now cached

# Cache globals dict for fair comparison without globals() call overhead
>>> g = globals()     

# Reimport (this might be *more* expensive at C layer, see notes below)
>>> %timeit -r5 import _strptime
1000000 loops, best of 5: 351 ns per loop

# Dict lookup (should be at least a bit cheaper at C layer if done equivalently, using GetAttrId to avoid temporary str)
>>> %timeit -r5 g['_strptime']
10000000 loops, best of 5: 33.1 ns per loop

# Cached reference (should be *much* cheaper at C layer)
>>> %timeit -r5 _strptime
100000000 loops, best of 5: 19.1 ns per loop

Note: I'm a little unclear on whether a Python function implemented in C has its own globals, or whether it's simulated as part of the C module initialization); if it lacks globals, then the work done for PyImport_Import looks like it roughly doubles (it has to do all sorts of work to simulate globals and the like), so that 351 ns per re-import might actually be costlier in C.

Either way, it's a >10x increase in cost to reimport compared to a dict lookup, and ~18x speedup over using a cached reference (and like I said, I think the real cost of the cheaper options would be much less in C, so the multiplier is higher). Admittedly, in tests, empty string calls to _strptime._strptime take around 7.4 microseconds (with realistic calls taking 8.5-13.5 microseconds), so caching is saving maybe a third of a microsecond overhead, maybe 2.5%-4.5% of the work involved in the strptime call.

[MojoVampire]

Hmm... On checking down some of the code paths and realizing there were some issues in 3.5 (redundant code, and what looked like two memory leaks), I checked tip (to avoid opening bugs on stale code), and discovered that bpo-22557 rewrote the import code, reducing the cost of top level reimport by ~60%, so my microbenchmarks (run on Python 3.5.0) are already out of date for 3.6's faster re-import. Even so, caching wasn't a wholly unreasonable optimization before now, and undoing it now still has a cost, if a smaller one.

[abalkin]

I am not sure this is possible to fix without refactoring the datetime module according to PEP-3121. See bpo-15390.

[tiran]

PEP-3121 is a big change. Can we use PyModuleDef->m_clear() for a clever hack?

[abalkin]

Yes, I think something like the attached patch may do the trick.

[tiran]

Wouldn't it clear strptime_module when a subinterpreter shuts down, too? It's not a big deal because it can't cause a crash.

[jimjjewett]

Having to (re-)fill the cache once per interpreter seems like a reasonable price to pay.

Why is 3.5 not included? Did this not cause problems before the import change, or is it just that this bug is small enough that maybe it isn't worth backporting?

[DennyWeinberg]

Any news here? 3.6.0 is also affected by this bug.

[cschramm]

This issue should have a much higher priority as it basically breaks Python embedding unless the user either does not re-initialize the interpreter or avoid the use of _datetime.strptime.

We're currently testing with a patch based on Christian and Alexander's idea but using m_free as using m_clear and Py_CLEAR neither makes sense to me nor did it work in conjunction with Py_Finalize when testing it. A version matching the current tip is attached. We did not run into any issues so far (and the only thing I can think of is clearing the static variable too often and thus causing some extra imports).

[ncoghlan]

I've added Steve Dower to the nosy list, as he's done some work recently on making the Windows builds more embedding-friendly, and I believe at least some of that time may have been funded work.

Unfortunately, we don't currently have anyone I'm aware of that's being specifically paid to improve or maintain CPython's embedding support in general, so getting attention for these kinds of interpreter reinitialization bugs can be a bit hit-or-miss.

[MarkusEh]

Hi,

This issue results in bugs in KODI, see xbmc/xbmc#17311 .
Please advice:

• Should we ask KODI developers to implement workarounds, as we cannot expect this bug to be fixed in the near future?
• Or is it worth while to wait for a bug fix in Python?

[brettcannon]

FYI if anyone wants to fix this, you need to make the following line not static:

cpython/Modules/_datetimemodule.c

Line 5196 in 5b946d3

static PyObject *module = NULL;

Now performance might not suck in this instance if you just change that line, but who knows without testing (last time this was looked at was the 3.5 days and things have obviously changed since then). Otherwise datetime should get updated for multi-phase init or do something such that caching _strptime happens as part of calling PyInit__datetime() and not datetime_strptime().

[abalkin]

@brettcannon - there are two patches attached to this issue from 6 years ago. One is mine and another by Christopher Schramm (@cschramm). Let me convert Christopher's patch to a PR so that we can test and review it.

[ericsnowcurrently]

This has been fixed by gh-120224. (Backports to 3.13 and 3.12 are in progress.)