Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySqlBulkCopy should have a BatchSize setting #1457

Open
rferraton opened this issue Mar 1, 2024 · 1 comment
Open

MySqlBulkCopy should have a BatchSize setting #1457

rferraton opened this issue Mar 1, 2024 · 1 comment

Comments

@rferraton
Copy link

Is your feature request related to a problem? Please describe.
I saw that the MySqlBulkCopy (which is great) use a const 1MB to read from IDataReader and write to target. I think it could lead to many IOs and disk stress on high load (parallel). I saw that before chunking with 1MB there was issue with network paquet size

Describe the solution you'd like
I suggest to add the possibility to change this value (1MB) using a class setting BatchSize or BatchLen that would have a 1MB default value
I found the const 1MB here :

const int maxLength = 1048575;

Describe alternatives you've considered

Additional context

@bgrainger
Copy link
Member

This limitation was introduced by feea06b to fix #780.

MySQL Server 8.0 increased the default value of max_allowed_packet to 64MiB (up from 4MiB in 5.7) so this constant may be much too conservative for current servers.

A larger default size with a mechanism (such as a class property) to opt out could be a good solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants