[source-mysql]: support Planetscale MySQL's per-query row limit #40561
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
octavia-squidington-iii
added
area/connectors
theyueli
changed the title
octavia-squidington-iii
added
the
area/documentation
oops, my mistake... these two files are almost identical.. and I got confused by their names... fixed now. Thanks! |
Fixes airbytehq/oncall#5051 Planet MySQL has defined multiple system limits. One of our customers has been hitting the 100K per-query row limit, which causes our connector to emit system errors. This patch defines a chunk size limit for our connector. By default, it is the largest long, and if we find it is a Planetscale MySQL, we will adjust to 100K.
|
@theyueli @akashkulk @evantahler: It doesn't look like this fix made it into the V2 version of the connector that @rodireich @xiaohansong and @clnoll merged in #48369 We at Remi are also running into the 100k per-query row limit when using Airbyte's MySQL connector to replicate PlanetScale to BigQuery (the PlanetScale connector is not working for us either). Can you point us in the direction of what a fix for this would look like? We're highly motivated to fix this and happy to submit a PR if your team is open to it. |
|
I don't know if this is the correct approach but I took stab at fixing this in #51055. |
Fixes https://github.com/airbytehq/oncall/issues/5051
Planet MySQL has defined multiple system limits. One of our customers has been hitting the 100K per-query row limit, making our connector emit system errors. This patch defines a chunk size limit for our connector, by default it is the largest long, and if we find it is a Planetscale MySQL, we will adjust to 100K.