Performance regression in pytest-html 4.0.0

[akhilramkee]

• There seems to be significant overhead introduced to execution of pytest by pytest-html 4

pytest-html 4 (Takes 6s)

pytest tests_sample.py --html ~/tmp/results.html
========================================================================================= test session starts =========================================================================================
platform darwin -- Python 3.11.4, pytest-7.0.1, pluggy-1.3.0
plugins: metadata-3.0.0, html-4.0.0
collected 1000 items

tests_sample.py ............................................................................................................................................................................... [ 17%]
............................................................................................................................................................................................... [ 36%]
............................................................................................................................................................................................... [ 55%]
............................................................................................................................................................................................... [ 74%]
............................................................................................................................................................................................... [ 93%]
.............................................................                                                                                                                                   [100%]

------------------------------------------------------------------- Generated html report: file:///Users/user/tmp/results.html -------------------------------------------------------------------
======================================================================================== 1000 passed in 6.28s =========================================================================================

pytest-html 3.2 (Takes 1s)

pytest tests_sample.py --html ~/tmp/results.html
========================================================================================= test session starts =========================================================================================
platform darwin -- Python 3.11.4, pytest-7.0.1, pluggy-1.3.0
plugins: html-3.2.0, metadata-3.0.0
collected 1000 items

tests_sample.py ............................................................................................................................................................................... [ 17%]
............................................................................................................................................................................................... [ 36%]
............................................................................................................................................................................................... [ 55%]
............................................................................................................................................................................................... [ 74%]
............................................................................................................................................................................................... [ 93%]
.............................................................

-------------------------------------------------------------------- generated html file: file:///Users/user/tmp/results.html --------------------------------------------------------------------
======================================================================================== 1000 passed in 1.10s =========================================================================================

To reproduce:

File structure:

pytest_html_reproducer/
├── __init__.py
└── tests/
    └── tests_sample.py

tests_sample.py

import pytest
import random

random_values = [int(random.random() * 1e4) for _ in range(1000)]

@pytest.mark.parametrize("val", random_values)
def test_a(val):
    assert 1

This slowness with pytest-html 4.0.0 is more pronounced in larger test suites (100K tests - tweaking the random_values count also shows the same), as the test execution took a 14+ hours with 90% of the execution time being taken by pytest-html to render the results.

Profile sample: Link

[BeyondEvil]

Any idea where the bottle neck might be?

If not, maybe an option to defer rendering until the suite is finished might fix it.

[akhilramkee]

The bottleneck seems to be coming from _generate_report (mostly from cleanup_unserializable and dumps)

[BeyondEvil]

@akhilramkee Thanks for the profile! 🙏

We can potentially drop cleanup_unserializable. dumps is another matter however.

Regardless, that confirms my initial thought that it's the generate-report-for-every-test that is the bigger issue here.

Maybe that should be an ini-option and we change the default back to what it was in v3. e.g. generate report once entire suite if finished?

WDYT?

[BeyondEvil]

Just realized that this PR #753 should speed things up as well.

Currently we generate the report for each "when" (setup, call, teardown) and also for each rerun. That PR makes it so that the report is only generated once per complete test.

[RonnyPfannschmidt]

creating the reports after the run should also solve #741 by moving the fs access out of the line

[BeyondEvil]

creating the reports after the run should also solve #741 by moving the fs access out of the line

Yeah, we've now got at least three issues that all seem to stem from the generation. So I'll definitely make that optional.

[akhilramkee]

So I'll definitely make that optional.

Yeah this sounds good!

[BeyondEvil]

Mind testing https://pypi.org/project/pytest-html/4.0.3rc0/ @akhilramkee ?

[akhilramkee]

My reproducer does indicate that run time is back to normal with pytest-html==4.0.3rc0.

============================================================================= test session starts ==============================================================================
platform darwin -- Python 3.9.17, pytest-7.0.1, pluggy-0.13.1
rootdir: /Users/akhileshr
plugins: metadata-3.0.0, html-4.0.3rc0, anyio-3.5.0, xdist-2.2.1, easyread-0.1.0, forked-1.3.0
collected 1000 items

tests_sample.py ........................................................................................................................................................ [ 15%]
........................................................................................................................................................................ [ 32%]
........................................................................................................................................................................ [ 48%]
........................................................................................................................................................................ [ 65%]
........................................................................................................................................................................ [ 82%]
........................................................................................................................................................................ [ 99%]
........                                                                                                                                                                 [100%]

------------------------------------------------------- Generated html report: file:///Users/akhileshr/tmp/results.html --------------------------------------------------------
============================================================================= 1000 passed in 0.71s =============================================================================

Thanks for the quick turnaround!

[BeyondEvil]

It even looks to be slightly faster: 0.71s vs 1.10s 🥳

Thanks for the quick turnaround!

Thanks for reporting and helping me test! <3

[BeyondEvil]

I'm going to see if there are any other issues I can fix, and then release a proper 4.0.3

[harmin-parra]

The bottleneck seems to be coming from _generate_report (mostly from cleanup_unserializable and dumps)

Hello @akhilramkee , just for curiosity, which tool or command did you use get this analysis ?

[akhilramkee]

Used pyinstrument for the profiling. You can check it out here - https://pyinstrument.readthedocs.io/en/latest/