Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send Object to receive task in HTTP payload instead of datastore #1354

Closed
snarfed opened this issue Oct 1, 2024 · 5 comments
Closed

Send Object to receive task in HTTP payload instead of datastore #1354

snarfed opened this issue Oct 1, 2024 · 5 comments

Comments

@snarfed
Copy link
Owner

snarfed commented Oct 1, 2024

Should cut some of our datastore load, especially if we stop storing transient activities like CRUD altogether.

Re #1149

@snarfed snarfed added the now label Oct 1, 2024
snarfed added a commit that referenced this issue Oct 2, 2024
…store

trying to cut down datastore load and costs. this switches receive tasks from storing and fetching Objects from the datastore to passing their properties - notably as2, our_as1, bsky, etc - in the HTTP request body, as serialized JSON inside form-encoded params.

for #1354, #1149
@snarfed
Copy link
Owner Author

snarfed commented Oct 2, 2024

Not a huge difference so far, but some reduction in datastore load.

image

@snarfed
Copy link
Owner Author

snarfed commented Oct 2, 2024

Reopening, this made us lose our task delay logging for AP and Bluesky receive tasks. When we log, the Object is now new in memory and hasn't been stored yet, so it doesn't have created or updated. Argh.

bridgy-fed/protocol.py

Lines 836 to 839 in 7c71feb

if request.headers.get('X-AppEngine-TaskRetryCount') == '0' and obj.created:
delay_s = int((util.now().replace(tzinfo=None) - obj.created).total_seconds())
delay = f'({delay_s} s behind)'
logger.info(f'Receiving {from_cls.LABEL} {obj.type} {id} {delay} AS1: {json_dumps(pruned, indent=2)}')

@snarfed snarfed reopened this Oct 2, 2024
@snarfed
Copy link
Owner Author

snarfed commented Oct 2, 2024

^ also, not caused by that bug, but while I'm in there, I should improve our delay handling for existing Objects. At minimum we should use updated instead of created, but I need to think harder about whether that's actually right.

@snarfed snarfed changed the title Send Object to receive talk in HTTP payload instead of datastore Send Object to receive task in HTTP payload instead of datastore Oct 2, 2024
@snarfed
Copy link
Owner Author

snarfed commented Oct 2, 2024

I've temporarily disabled the tasks delayed alert policy. Once I fix this ^, reminder to myself to re-enable it.

@snarfed
Copy link
Owner Author

snarfed commented Oct 13, 2024

Moved monitoring regression (#1354 (comment)) to new issue #1378.

@snarfed snarfed closed this as completed Oct 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant