Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running multiple copies of PyAirbyte for the same stream will lead to errors #311

Open
aaronsteers opened this issue Jul 25, 2024 · 1 comment

Comments

@aaronsteers
Copy link
Contributor

aaronsteers commented Jul 25, 2024

PyAirbyte should be able to be run multiple times simultaneously except for these conditions:

  1. Loading the same stream at the same time with two PyAirbyte instances is not supported. We do not block against this happening, but it will lead to race conditions, which could cause failures and/or data corruption.
  2. DuckDB and/or MotherDuck may have their own limitations around concurrent writes.
@Udit107710
Copy link
Contributor

If we created a new cache for all the concurrent runs, then this should not be blocked, right? @aaronsteers
Do you think there can be other issues also?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants