[ntuple] Improve RPagePool #16859
[ntuple] Improve RPagePool #16859
Conversation
jblomer
force-pushed
the
ntuple-fix-page-pool
branch
from
0350870
to
ce1da4f
Compare
Test Results 18 files 18 suites 4d 4h 10m 19s ⏱️ Results for commit 15b33c0. ♻️ This comment has been updated with latest results. |
Instead of mapping all synthezised zero pages to the same memory buffer, use real allocated and zeroed out pages. That makes sure no special logic is required when adding and removing pages to and from the page pool.
Allows for O(1) page lookup when a page is returned to the page pool, instead of the O(n) linear search.
Use a hash map to filter the pages in the page pool by column ID and on-disk type on access.
Co-authored-by: Jonas Hahnfeld <hahnjo@hahnjo.de>
jblomer
force-pushed
the
ntuple-fix-page-pool
branch
from
91e1921
to
15b33c0
Compare
|
No changes to the "many pages" test. The "many fields" unit test got significantly faster. The overall complexity is still super-linear but much more benign. |
Ah right, now I remember that I had already profiled this before: The "many pages" tests are actually bound by Edit: Hm, the "many pages" tests will also hit the linear loop over the page set in |
| FileRaii fileGuard("test_ntuple_limits_manyFields.root"); | ||
|
|
||
| static constexpr int NumFields = 40'000; | ||
| static constexpr int NumFields = 100'000; |
There was a problem hiding this comment.
This is fantastic that we can now process models with 100k fields in reasonable time!
Improve the lookup complexity for pages in the page pool from linear to constant in well-behaved cases, i.e. if there is a small number of pages per column and cluster. Some smaller cleanups around the RPage/RPagePool logic.