Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

worker: restrict size of emitted dataclips #739

Closed
josephjclark opened this issue Jul 30, 2024 · 0 comments · Fixed by #740
Closed

worker: restrict size of emitted dataclips #739

josephjclark opened this issue Jul 30, 2024 · 0 comments · Fixed by #740
Assignees
@josephjclark

Comments

@josephjclark
Copy link
Collaborator

Sending ~40mb json dataclips is blowing up lightning

We always planned to do something about this. It's just started being a problem.

  • 10mb is probbaly the target size env var but and option controlled
  • we're tallking about the size of the serialized JSON, the size on the wire
  • streaming encoder?

What if the limit is blown?

  • kill the run
  • exit reason dataclip too large

Limit will have to apply to log messages too.

So it's a generic payload limit

logs don't need to be a kill, but run:complete does
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@josephjclark josephjclark

Labels
None yet
Projects
v2
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Worker: limit payload size OpenFn/kit
1 participant
@josephjclark