Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP kvs-watch: do not load all KVS blobs at once #6532

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

chu11
Copy link
Member

@chu11 chu11 commented Dec 21, 2024

Problem: With the FLUX_KVS_WATCH_APPEND and FLUX_KVS_STREAM flags, all content blobs in a valref treeobj will be retrieved from the content store at once. If the valref array is gigantic, this may be a very costly initial transaction. Not to mention, we probably wouldn't want to send an extremely large number (like millions) of content requests all at once.

Solution: Send content requests in more reasonable 32K chunks.

Fixes #6456


So this is what I initially started working on that lead more to #6531 and #6518.

This issue is mostly theoretical, as I don't have a use case where this is a problem at the moment. I want to try and manufacture a use case and test it, which is why I'm posting this as WIP. But I thought I'd post and see if folks think it's atleast a good idea.

I should note, I did run the test suite when having the "max loads" set to 2 instead of 32K, so it generally speaking appears to work safely. Although the question of how big the "maximum number of loads at a time" number could possibly be tweaked up or down some.

Problem: With the FLUX_KVS_WATCH_APPEND and FLUX_KVS_STREAM
flags, all content blobs in a valref treeobj will be retrieved
from the content store at once.  If the valref array is gigantic,
this may be a very costly initial transaction.  Not to mention, we
probably wouldn't want to send an extremely large number (like millions)
of content requests all at once.

Solution: Send content requests in more reasonable 32K chunks.

Fixes flux-framework#6456
Copy link

codecov bot commented Dec 21, 2024

Codecov Report

Attention: Patch coverage is 83.33333% with 8 lines in your changes missing coverage. Please review.

Project coverage is 83.65%. Comparing base (9278fa8) to head (8604a3d).
Report is 11 commits behind head on master.

Files with missing lines Patch % Lines
src/modules/kvs-watch/kvs-watch.c 83.33% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6532      +/-   ##
==========================================
+ Coverage   83.62%   83.65%   +0.02%     
==========================================
  Files         522      522              
  Lines       87809    87829      +20     
==========================================
+ Hits        73433    73473      +40     
+ Misses      14376    14356      -20     
Files with missing lines Coverage Δ
src/modules/kvs-watch/kvs-watch.c 77.93% <83.33%> (-0.38%) ⬇️

... and 8 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

kvs-watch: retrieve content blobs in chunks rather than all at once
1 participant