gh-109653: Improve import time of importlib.metadata / email.utils

[hauntsaninja]

My criterion for delayed imports is that they're only worth it if the majority of users of the module would benefit from it, otherwise you're just moving latency around unpredictably.

mktime_tz is not used anywhere in the standard library and grep.app indicates it's not got much use in the ecosystem either.

Distribution.files is not nearly as widely used as other importlib.metadata APIs, so we defer the csv import.

And it's pretty easy for Python programs to not ever need calendar and csv.

Before:

λ hyperfine -w 8 './python -c "import importlib.metadata"'
Benchmark 1: ./python -c "import importlib.metadata"
  Time (mean ± σ):      65.1 ms ±   0.5 ms    [User: 55.3 ms, System: 9.8 ms]
  Range (min … max):    64.4 ms …  66.4 ms    44 runs

After:

λ hyperfine -w 8 './python -c "import importlib.metadata"'
Benchmark 1: ./python -c "import importlib.metadata"
  Time (mean ± σ):      62.0 ms ±   0.3 ms    [User: 52.5 ms, System: 9.6 ms]
  Range (min … max):    61.3 ms …  62.8 ms    46 runs

for about a 3ms saving with warm disk cache, maybe 7-11ms with cold disk cache.

• Issue: Improve import time of various stdlib modules #109653

[jaraco]

Thanks for working on this change. I appreciate the effort.

I'm a little concerned for two reasons.

First, this change implies that best practice is not to put imports at the top of a module, conflicting with my understanding that imports should generally be placed at the top of the module. If there's a systemic problem with eager imports adding undue latency, I'd prefer to see that issue dealt with systemically, rather than piecemeal moving select imports to be late-imported. The approach implied by this change suggests that any Python developer (and especially CPython developers) should be alert to the cost of their imports and aggressively inline imports to be used on demand. I've seen PEP 690, but it appears to have been rejected (and without any reason that I could find in the PEP). It seems to me than something like PEP 690 would be better than selectively combing through codebases and analyzing their imports and tweaking them.

Second, the motivation for this change is lost in the commit - there's nothing in the code or tests that would indicate to a future contributor not to reverse this change for the sake of consistency. At the very least, I'd expect a comment protecting the motivation.

Third, the importlib.metadata part of this change will need to also be contributed to importlib_metadata. That test suite does have performance tests that measure specific performance targets. Perhaps that could be a good place to ensure that the desired behavior is captured.

Given that PEP 690 was rejected, perhaps this kind of change should be enacted through a linter that would automatically scan for imports that are not used until called and automatically move them inline (and thus could also be included in test suites to protect the intention).

I don't have any real objections on this specific change, just a bit of concern at the underlying implications.

[jaraco]

My criterion for delayed imports

Perhaps it would be better if there were a shared, published criteria for delayed imports, either in CPython or for Python libs in general.

[jaraco]

Perhaps that could be a good place to ensure that the desired behavior is captured.

In python/importlib_metadata@1a209e7e41, I started exploring adding a test that would capture the import time expectation (and at least indicate a regression when if the optimization was removed), but when I added the lazy import for csv, it detected no meaningful performance difference:

exercises.py:import time: 0:00:00.000001 (+0:00:00, 0%)

That indicates to me that the import time is 1 µs before and after the lazy import.

Attempting to understand why the timeit results weren't providing the expected performance gains, I added an assertion and found that the "setup" expression has a different outcome depending on what "test" expression is present:

 importlib_metadata feature/test-import-time @ .tox/py/bin/python -s -m timeit --setup 'import sys; assert "csv" not in sys.modules'
100000000 loops, best of 5: 3.9 nsec per loop
 importlib_metadata feature/test-import-time @ .tox/py/bin/python -s -m timeit --setup 'import sys; assert "csv" not in sys.modules' 'import importlib_metadata'
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/timeit.py", line 330, in main
    number, _ = t.autorange(callback)
                ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/timeit.py", line 226, in autorange
    time_taken = self.timeit(number)
                 ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/timeit.py", line 180, in timeit
    timing = self.inner(it, self.timer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<timeit-src>", line 3, in inner
    import sys; assert "csv" not in sys.modules
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

That's weird, right? I would expect the "setup" to be run exactly once before any test expression was run, so it shouldn't be affected by the contents of the test expression, yet it is.

[hauntsaninja]

@jaraco Thank you for taking the time to review. What timeit is doing is something like:

for _ in range(r):
    import sys; assert "csv" not in sys.modules
    # start recording time
    for _ in range(n):
        import importlib.metadata
    # end recording time

r defaults to 5.
n is dynamically guessed if not specified, this involves running your benchmark code, which can again be problematic

This explains why you see:

# this works
λ python -m timeit -n 1 -r 1 --setup 'import sys; assert "csv" not in sys.modules' 'import importlib.metadata'
1 loop, best of 1: 47.5 msec per loop

# but this fails
λ python -m timeit -n 1 -r 2 --setup 'import sys; assert "csv" not in sys.modules' 'import importlib.metadata'
Traceback (most recent call last):
[...]
AssertionError

In general, Python's caching behaviour for imports makes it very easy to benchmark incorrectly when using timeit, so I prefer using subprocesses or general purpose benchmarking tools (like hyperfine above):

λ cat slow.py
import time
time.sleep(1)

# note the UserWarning is a little misleading, even the 11.7 usec "worst time" is obviously not what we're trying to measure
λ python -m timeit 'import slow'
1 loop, best of 5: 542 nsec per loop
:0: UserWarning: The test results are likely unreliable. The worst time (11.7 usec) was more than four times slower than the best time (542 nsec).
λ python -m timeit -r 1 'import slow'
1 loop, best of 1: 9.33 usec per loop
λ python -m timeit -n 1 -r 1 'import slow'
1 loop, best of 1: 1 sec per loop

timeit -n 1 -r 1 will measure the right thing, but noisily — you still need to repeat, since import time is often greatly affected by the state of the disk cache.

First, this change implies that best practice is not to put imports at the top of a module

I think this is a reasonable reason to reject this change. Another understandable reason to reject this change is simply that the benefit is relatively small and as the primary maintainer of this module you'd like a consistent style.

I think of my personal view as pragmatic: importlib.metadata is the slowest import in a command line tool I work on. It's unfortunate that my CLI is slower because importlib.metadata ends up importing email.utils and email.utils imports calendar only for mktime_tz, a function that almost no one uses.

I agree it could be nice to have a more systemic solution. PEP 690 was a global flag to the interpreter, which complicates things, e.g. I think the SC wasn't keen on having libraries double the size of their test matrix. It's also a good point that it could be nice for there to be something in the dev guide about how to think about this.

Second, the motivation for this change is lost in the commit

Thanks, I'll add a comment :-)

Third, the importlib.metadata part of this change will need to also be contributed to importlib_metadata

I can make the same patch to importlib_metadata. Note that of the two changes, deferral of import calendar in email._parseaddr accounts for most of the time saved — this wouldn't apply to importlib_metadata.

[AlexWaygood]

This LGTM. I agree that these are good candidates for lazy imports, and I agree that it would be nice to improve the import time of importlib.metadata, which is a very useful module that unfortunately does have one of the slower import times in the standard library currently. @jaraco should obviously have the final say, though, as the primary maintainer for this module and the importlib_metadata backport.

First, this change implies that best practice is not to put imports at the top of a module, conflicting with my understanding that imports should generally be placed at the top of the module.

While this is a general style recommendation by PEP-8, I think it's worth noting that we already break this in many places across the stdlib, for reasons of pragmatism:

cpython/Lib/functools.py

Line 19 in a768e12

# import types, weakref # Deferred to single_dispatch()

cpython/Lib/email/__init__.py

Lines 28 to 37 in a768e12

	# Some convenience routines. Don't import Parser and Message as side-effects
	# of importing email since those cascadingly import most of the rest of the
	# email package.
	def message_from_string(s, *args, **kws):
	"""Parse a string into a Message object model.
	Optional _class and strict are passed to the Parser constructor.
	"""
	from email.parser import Parser
	return Parser(*args, **kws).parsestr(s)

cpython/Lib/gettext.py

Lines 351 to 353 in a768e12

	# Delay struct import for speeding up gettext import when .mo files
	# are not used.
	from struct import unpack

cpython/Lib/argparse.py

Lines 141 to 147 in a768e12

	# The copy module is used only in the 'append' and 'append_const'
	# actions, and it is needed only when the default value isn't a list.
	# Delay its import for speeding up the common case.
	if type(items) is list:
	return items[:]
	import copy
	return copy.copy(items)

Also, I'll note that PEP-8 also contains this statement ;)

A Foolish Consistency is the Hobgoblin of Little Minds

FWIW, I agree that:

• We definitely shouldn't run around the whole standard library making all imports lazy; we should look at each one on a case-by-case basis and check that a specific change is impactful before making it
• We should always add comments to the source code when making changes like this
• It would also be nice if we could speedup Python's import statement in general (but I don't think that should block small improvements like this, where they make sense)

[jaraco]

I can make the same patch to importlib_metadata. Note that of the two changes, deferral of import calendar in email._parseaddr accounts for most of the time saved — this wouldn't apply to importlib_metadata.

I'll take this on, since I've got the muscle memory for doing it. I just wanted to flag it for context.

I prefer using subprocesses or general purpose benchmarking tools

Alex provided a similar suggestion in the linked issue I filed about timeit. I would like for pytest-perf to be able to facilitate the testing of import latency. I'd like for pytest-perf to be one of those general purpose benchmarking tools. I've filed jaraco/pytest-perf#12 to track that possibility.

[danielhollas]

I prefer using subprocesses or general purpose benchmarking tools

Note that there's also a command line option -Ximporttime specifically for disentangling the import time cost. Run e.g. like `python -Ximporttime -c "import importlib.metadata > import_time.out",
producing a text file with the import time of all the modules that end up being imported. (you might want to run it couple times to warm up disc cache, then the results are quite stable in my experience)

You can visualize it with tuna to get a nice icicle graph of where the import time goes.

[hugovk]

I've seen PEP 690, but it appears to have been rejected (and without any reason that I could find in the PEP).

To see the SC's rejection notice, click through from "Resolution: Discourse message" in the header; essentially they were concerned over producing a split in the community over how imports work.

You can visualize it with tuna to get a nice icicle graph of where the import time goes.

Yes, tuna is a nice tool. hugovk/pypistats#289 is an example of using it to identify and improve slow imports for a CLI.

As it happens, importlib.metadata accounts for 27% of the final import time in that Python 3.10 example. Measuring again to zoom in, it accounts for 59% (I think because 3.12 is quicker) with 2.3% for calendar and 0.8% for csv so this PR is welcome. Thanks all!

[jaraco]

Backported in python/importlib_metadata@b4ce0ff90e.