Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify subscribeRepos datastore queries #30

Open
snarfed opened this issue Aug 3, 2024 · 0 comments
Open

Unify subscribeRepos datastore queries #30

snarfed opened this issue Aug 3, 2024 · 0 comments

Comments

@snarfed
Copy link
Owner

snarfed commented Aug 3, 2024

Right now, in subscribeRepos, we query the datastore separately for each connected subscriber (client). This is fine for historical blocks, but it's duplicative for new blocks. For ongoing subscribers, ideally we should only do a given datastore query once, and then fan out the results to all subscribers.

This would take a bit of rearchitecting. Right now, we do all of this inside the request handler, per client:

arroba/arroba/xrpc_sync.py

Lines 199 to 206 in 351d43f

# serve new events as they happen
logger.info(f'serving new events')
while True:
with new_events:
new_events.wait(NEW_EVENTS_TIMEOUT.total_seconds())
for commit_data in server.storage.read_events_by_seq(start=last_seq + 1):
yield handle(commit_data)

We'd need to start a separate, shared thread for the realtime datastore queries, collect the resulting blocks into events in memory, and have each client's request handler read and emit from there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant