Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow Parameter Usage on MultipleRowsCopy/MultipleRowsCopyAsync #2975

Merged
merged 18 commits into from
Jun 3, 2021
Merged

Allow Parameter Usage on MultipleRowsCopy/MultipleRowsCopyAsync #2975

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
16bd698
Add Option to Force Parameter Usage on MultipleRowsCopy/MultipleRowsC…
to11mtm Apr 29, 2021
9caf4b9
Fixup bad Last string position index logic. Make sure to handle the c…
to11mtm Apr 29, 2021
cbec45a
Add more DB and Param counts for DBs
to11mtm May 3, 2021
74edd8c
Adjust batch size in bulk copy methods
to11mtm May 28, 2021
ff4f35f
Merge branch 'master' of https://github.com/linq2db/linq2db into Allo…
to11mtm May 28, 2021
9b8e172
Merge branch 'master' into Allow-Parameter-Use-On-Bulk-Copy-Part-1
MaceWindu May 29, 2021
fab80eb
Merge branch 'Allow-Parameter-Use-On-Bulk-Copy-Part-1' of https://git…
to11mtm May 29, 2021
f391ad9
Add a little XMLDoc, Allow override of MaxParamters via BulkCopyOptions.
to11mtm May 29, 2021
92ad512
add references for max values.
to11mtm May 29, 2021
7fab586
Quick test to confirm Firebird is angry about null + union
to11mtm May 29, 2021
009331e
Try naieve version of MultipleRowsCopy3 switch where we cast all rows…
to11mtm May 31, 2021
7d91b5f
Use SqlBuilder to build the parameter type info
to11mtm Jun 3, 2021
2240547
Merge branch 'master' into Allow-Parameter-Use-On-Bulk-Copy-Part-1
MaceWindu Jun 3, 2021
12b1748
address firebird issue
MaceWindu Jun 3, 2021
1a94108
fix warning
MaceWindu Jun 3, 2021
848eea8
Merge branch 'master' into Allow-Parameter-Use-On-Bulk-Copy-Part-1
MaceWindu Jun 3, 2021
6b3a921
specify copy type in test
MaceWindu Jun 3, 2021
ed372fc
fix column datatype resolve
MaceWindu Jun 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions Source/LinqToDB/Data/BulkCopyOptions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,5 +88,17 @@ public BulkCopyOptions(BulkCopyOptions options)
/// This callback will not be used if <see cref="NotifyAfter"/> set to 0.
/// </summary>
public Action<BulkCopyRowsCopied>? RowsCopiedCallback { get; set; }

/// <summary>
/// Gets or sets whether to Always use Parameters for MultipleRowsCopy. Default is false.
/// If True, provider's override for <see cref="LinqToDB.DataProvider.BasicBulkCopy.MaxParameters"/> will be used to determine the maximum number of rows per insert,
/// Unless overridden by <see cref="MaxParametersForBatch"/>.
/// </summary>
public bool UseParameters { get; set; }

/// <summary>
/// If set, will override the Maximum parameters per batch statement from <see cref="LinqToDB.DataProvider.BasicBulkCopy.MaxParameters"/>.
/// </summary>
public int? MaxParametersForBatch { get; set; }
}
}
9 changes: 9 additions & 0 deletions Source/LinqToDB/DataProvider/Access/AccessBulkCopy.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,14 @@
{
class AccessBulkCopy : BasicBulkCopy
{
/// <remarks>
/// Settings based on https://www.jooq.org/doc/3.12/manual/sql-building/dsl-context/custom-settings/settings-inline-threshold/
/// We subtract 1 here to be safe since some ADO providers use parameter for command itself.
/// </remarks>
protected override int MaxParameters => 767;
/// <remarks>
/// This max is based on https://support.microsoft.com/en-us/office/access-specifications-0cf3c66f-9cf2-4e32-9568-98c1025bb47c
/// </remarks>
protected override int MaxSqlLength => 64000;
}
}
97 changes: 75 additions & 22 deletions Source/LinqToDB/DataProvider/BasicBulkCopy.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ namespace LinqToDB.DataProvider

public class BasicBulkCopy
{
protected virtual int MaxParameters => 999;
MaceWindu marked this conversation as resolved.
Show resolved Hide resolved
MaceWindu marked this conversation as resolved.
Show resolved Hide resolved
protected virtual int MaxSqlLength => 100000;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting this as a virtual -feels- better, but I still ask whether we should let Max Length be tunable by the user or not (For Network round-trip purposes). I decided not, since MaxBatchSize is good enough to do the same most likely.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COuld be usful. In baselines I see increase in number of roundtrips e.g. for DB2


protected virtual bool CastOnUnionAll => false;
protected virtual bool TypeAllUnionParameters => false;

public virtual BulkCopyRowsCopied BulkCopy<T>(BulkCopyType bulkCopyType, ITable<T> table, BulkCopyOptions options, IEnumerable<T> source)
where T : notnull
{
Expand Down Expand Up @@ -344,20 +350,35 @@ protected static BulkCopyRowsCopied MultipleRowsCopyHelper(
Action<MultipleRowsHelper> prepFunction,
Action<MultipleRowsHelper,object,string?> addFunction,
Action<MultipleRowsHelper> finishFunction,
int maxParameters = 10000,
int maxSqlLength = 100000)
int maxParameters,
int maxSqlLength)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be worth moving these, even with a simple Redirection for now, into a 'Bind' like method on MultipleRowsHelper. Over 15 helper. in a method feels like a smell to me.

{
var adjustedBatchSize = helper.Options.UseParameters ? Math.Min(helper.BatchSize, helper.Options.MaxParametersForBatch.GetValueOrDefault(maxParameters) / helper.Columns.Length): helper.BatchSize;
prepFunction(helper);

foreach (var item in source)
{
helper.LastRowParameterIndex = helper.ParameterIndex;
helper.LastRowStringIndex = helper.StringBuilder.Length;
addFunction(helper, item!, from);

if (helper.CurrentCount >= helper.BatchSize || helper.Parameters.Count > maxParameters || helper.StringBuilder.Length > maxSqlLength)
var needRemove = helper.Parameters.Count > maxParameters ||
helper.StringBuilder.Length > maxSqlLength;
var isSingle = helper.CurrentCount == 1;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this check because there may be an edge case where even a single row was putting us over Max SQL limit.

Since we are now trying to be more correct here, we now 'back off' the last row when we go over maxSqlLength, rather than just continuing and hoping for the best. That said, we should still try to insert in case of Single row, and let provider fail if it really is an issue.

if (helper.CurrentCount >= adjustedBatchSize || needRemove)
{
if (needRemove && !isSingle)
{
helper.Parameters.RemoveRange(helper.LastRowParameterIndex, helper.ParameterIndex-helper.LastRowParameterIndex);
helper.StringBuilder.Length = helper.LastRowStringIndex;
helper.RowsCopied.RowsCopied--;
}
Comment on lines +369 to +374
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do this before finishFunction, in case finishFunction decides to add a parameter in future.

finishFunction(helper);
if (!helper.Execute())
return helper.RowsCopied;
if (needRemove && !isSingle)
{
addFunction(helper, item!, from);
}
}
}

Expand All @@ -378,20 +399,36 @@ protected static async Task<BulkCopyRowsCopied> MultipleRowsCopyHelperAsync(
Action<MultipleRowsHelper, object, string?> addFunction,
Action<MultipleRowsHelper> finishFunction,
CancellationToken cancellationToken,
int maxParameters = 10000,
int maxSqlLength = 100000)
int maxParameters,
int maxSqlLength)
{
var adjustedBatchSize = helper.Options.UseParameters ? Math.Min(helper.BatchSize, helper.Options.MaxParametersForBatch.GetValueOrDefault(maxParameters) / helper.Columns.Length): helper.BatchSize;
prepFunction(helper);

foreach (var item in source)
{
helper.LastRowParameterIndex = helper.ParameterIndex;
helper.LastRowStringIndex = helper.StringBuilder.Length;
addFunction(helper, item!, from);

if (helper.CurrentCount >= helper.BatchSize || helper.Parameters.Count > maxParameters || helper.StringBuilder.Length > maxSqlLength)
var needRemove = helper.Parameters.Count > maxParameters ||
helper.StringBuilder.Length > maxSqlLength;
var isSingle = helper.CurrentCount == 1;
if (helper.CurrentCount >= adjustedBatchSize || needRemove)
{
if (needRemove && !isSingle)
{
helper.Parameters.RemoveRange(helper.LastRowParameterIndex, helper.ParameterIndex-helper.LastRowParameterIndex);
helper.StringBuilder.Length = helper.LastRowStringIndex;
helper.RowsCopied.RowsCopied--;
}
finishFunction(helper);
if (!await helper.ExecuteAsync(cancellationToken).ConfigureAwait(Common.Configuration.ContinueOnCapturedContext))
return helper.RowsCopied;
if (needRemove && !isSingle)
{
addFunction(helper, item!, from);
}
}
}

Expand All @@ -413,20 +450,36 @@ protected static async Task<BulkCopyRowsCopied> MultipleRowsCopyHelperAsync<T>(
Action<MultipleRowsHelper, object, string?> addFunction,
Action<MultipleRowsHelper> finishFunction,
CancellationToken cancellationToken,
int maxParameters = 10000,
int maxSqlLength = 100000)
int maxParameters,
int maxSqlLength)
{
var adjustedBatchSize = helper.Options.UseParameters ? Math.Min(helper.BatchSize, helper.Options.MaxParametersForBatch.GetValueOrDefault(maxParameters) / helper.Columns.Length): helper.BatchSize;
prepFunction(helper);

await foreach (var item in source.ConfigureAwait(Common.Configuration.ContinueOnCapturedContext).WithCancellation(cancellationToken))
{
helper.LastRowParameterIndex = helper.ParameterIndex;
helper.LastRowStringIndex = helper.StringBuilder.Length;
addFunction(helper, item!, from);

if (helper.CurrentCount >= helper.BatchSize || helper.Parameters.Count > maxParameters || helper.StringBuilder.Length > maxSqlLength)
var needRemove = helper.Parameters.Count > maxParameters ||
helper.StringBuilder.Length > maxSqlLength;
var isSingle = helper.CurrentCount == 1;
if (helper.CurrentCount >= adjustedBatchSize || needRemove)
{
if (needRemove && !isSingle)
{
helper.Parameters.RemoveRange(helper.LastRowParameterIndex, helper.ParameterIndex-helper.LastRowParameterIndex);
helper.StringBuilder.Length = helper.LastRowStringIndex;
helper.RowsCopied.RowsCopied--;
}
finishFunction(helper);
if (!await helper.ExecuteAsync(cancellationToken).ConfigureAwait(Common.Configuration.ContinueOnCapturedContext))
return helper.RowsCopied;
if (needRemove && !isSingle)
{
addFunction(helper, item!, from);
}
}
}

Expand All @@ -445,14 +498,14 @@ protected BulkCopyRowsCopied MultipleRowsCopy1<T>(ITable<T> table, BulkCopyOptio
=> MultipleRowsCopy1(new MultipleRowsHelper<T>(table, options), source);

protected BulkCopyRowsCopied MultipleRowsCopy1(MultipleRowsHelper helper, IEnumerable source)
=> MultipleRowsCopyHelper(helper, source, null, MultipleRowsCopy1Prep, MultipleRowsCopy1Add, MultipleRowsCopy1Finish);
=> MultipleRowsCopyHelper(helper, source, null, MultipleRowsCopy1Prep, MultipleRowsCopy1Add, MultipleRowsCopy1Finish,MaxParameters, MaxSqlLength);

protected Task<BulkCopyRowsCopied> MultipleRowsCopy1Async<T>(ITable<T> table, BulkCopyOptions options, IEnumerable<T> source, CancellationToken cancellationToken)
where T : notnull
=> MultipleRowsCopy1Async(new MultipleRowsHelper<T>(table, options), source, cancellationToken);

protected Task<BulkCopyRowsCopied> MultipleRowsCopy1Async(MultipleRowsHelper helper, IEnumerable source, CancellationToken cancellationToken)
=> MultipleRowsCopyHelperAsync(helper, source, null, MultipleRowsCopy1Prep, MultipleRowsCopy1Add, MultipleRowsCopy1Finish, cancellationToken);
=> MultipleRowsCopyHelperAsync(helper, source, null, MultipleRowsCopy1Prep, MultipleRowsCopy1Add, MultipleRowsCopy1Finish, cancellationToken, MaxParameters, MaxSqlLength);

#if NATIVE_ASYNC
protected Task<BulkCopyRowsCopied> MultipleRowsCopy1Async<T>(ITable<T> table, BulkCopyOptions options, IAsyncEnumerable<T> source, CancellationToken cancellationToken)
Expand All @@ -461,7 +514,7 @@ protected Task<BulkCopyRowsCopied> MultipleRowsCopy1Async<T>(ITable<T> table, Bu

protected Task<BulkCopyRowsCopied> MultipleRowsCopy1Async<T>(MultipleRowsHelper helper, IAsyncEnumerable<T> source, CancellationToken cancellationToken)
where T: notnull
=> MultipleRowsCopyHelperAsync(helper, source, null, MultipleRowsCopy1Prep, MultipleRowsCopy1Add, MultipleRowsCopy1Finish, cancellationToken);
=> MultipleRowsCopyHelperAsync(helper, source, null, MultipleRowsCopy1Prep, MultipleRowsCopy1Add, MultipleRowsCopy1Finish, cancellationToken, MaxParameters, MaxSqlLength);
#endif

private void MultipleRowsCopy1Prep(MultipleRowsHelper helper)
Expand Down Expand Up @@ -496,7 +549,7 @@ private void MultipleRowsCopy1Add(MultipleRowsHelper helper, object item, string
helper.StringBuilder
.AppendLine()
.Append('(');
helper.BuildColumns(item!);
helper.BuildColumns(item);
helper.StringBuilder.Append("),");

helper.RowsCopied.RowsCopied++;
Expand All @@ -513,14 +566,14 @@ protected BulkCopyRowsCopied MultipleRowsCopy2<T>(ITable<T> table, BulkCopyOptio
=> MultipleRowsCopy2(new MultipleRowsHelper<T>(table, options), source, from);

protected BulkCopyRowsCopied MultipleRowsCopy2(MultipleRowsHelper helper, IEnumerable source, string from)
=> MultipleRowsCopyHelper(helper, source, from, MultipleRowsCopy2Prep, MultipleRowsCopy2Add, MultipleRowsCopy2Finish);
=> MultipleRowsCopyHelper(helper, source, from, MultipleRowsCopy2Prep, MultipleRowsCopy2Add, MultipleRowsCopy2Finish, MaxParameters, MaxSqlLength);

protected Task<BulkCopyRowsCopied> MultipleRowsCopy2Async<T>(ITable<T> table, BulkCopyOptions options, IEnumerable<T> source, string from, CancellationToken cancellationToken)
where T : notnull
=> MultipleRowsCopy2Async(new MultipleRowsHelper<T>(table, options), source, from, cancellationToken);

protected Task<BulkCopyRowsCopied> MultipleRowsCopy2Async(MultipleRowsHelper helper, IEnumerable source, string from, CancellationToken cancellationToken)
=> MultipleRowsCopyHelperAsync(helper, source, from, MultipleRowsCopy2Prep, MultipleRowsCopy2Add, MultipleRowsCopy2Finish, cancellationToken);
=> MultipleRowsCopyHelperAsync(helper, source, from, MultipleRowsCopy2Prep, MultipleRowsCopy2Add, MultipleRowsCopy2Finish, cancellationToken, MaxParameters, MaxSqlLength);

#if NATIVE_ASYNC
protected Task<BulkCopyRowsCopied> MultipleRowsCopy2Async<T>(ITable<T> table, BulkCopyOptions options, IAsyncEnumerable<T> source, string from, CancellationToken cancellationToken)
Expand All @@ -529,7 +582,7 @@ protected Task<BulkCopyRowsCopied> MultipleRowsCopy2Async<T>(ITable<T> table, Bu

protected Task<BulkCopyRowsCopied> MultipleRowsCopy2Async<T>(MultipleRowsHelper helper, IAsyncEnumerable<T> source, string from, CancellationToken cancellationToken)
where T: notnull
=> MultipleRowsCopyHelperAsync(helper, source, from, MultipleRowsCopy2Prep, MultipleRowsCopy2Add, MultipleRowsCopy2Finish, cancellationToken);
=> MultipleRowsCopyHelperAsync(helper, source, from, MultipleRowsCopy2Prep, MultipleRowsCopy2Add, MultipleRowsCopy2Finish, cancellationToken, MaxParameters, MaxSqlLength);
#endif

private void MultipleRowsCopy2Prep(MultipleRowsHelper helper)
Expand Down Expand Up @@ -560,7 +613,7 @@ private void MultipleRowsCopy2Add(MultipleRowsHelper helper, object item, string
helper.StringBuilder
.AppendLine()
.Append("SELECT ");
helper.BuildColumns(item!);
helper.BuildColumns(item, castParameters: CastOnUnionAll, castAllRows: TypeAllUnionParameters);
helper.StringBuilder.Append(from);
helper.StringBuilder.Append(" UNION ALL");

Expand All @@ -574,15 +627,15 @@ private void MultipleRowsCopy2Finish(MultipleRowsHelper helper)
}

protected BulkCopyRowsCopied MultipleRowsCopy3(MultipleRowsHelper helper, BulkCopyOptions options, IEnumerable source, string from)
=> MultipleRowsCopyHelper(helper, source, from, MultipleRowsCopy3Prep, MultipleRowsCopy3Add, MultipleRowsCopy3Finish);
=> MultipleRowsCopyHelper(helper, source, from, MultipleRowsCopy3Prep, MultipleRowsCopy3Add, MultipleRowsCopy3Finish, MaxParameters, MaxSqlLength);

protected Task<BulkCopyRowsCopied> MultipleRowsCopy3Async(MultipleRowsHelper helper, BulkCopyOptions options, IEnumerable source, string from, CancellationToken cancellationToken)
=> MultipleRowsCopyHelperAsync(helper, source, from, MultipleRowsCopy3Prep, MultipleRowsCopy3Add, MultipleRowsCopy3Finish, cancellationToken);
=> MultipleRowsCopyHelperAsync(helper, source, from, MultipleRowsCopy3Prep, MultipleRowsCopy3Add, MultipleRowsCopy3Finish, cancellationToken, MaxParameters, MaxSqlLength);

#if NATIVE_ASYNC
protected Task<BulkCopyRowsCopied> MultipleRowsCopy3Async<T>(MultipleRowsHelper helper, BulkCopyOptions options, IAsyncEnumerable<T> source, string from, CancellationToken cancellationToken)
where T: notnull
=> MultipleRowsCopyHelperAsync(helper, source, from, MultipleRowsCopy3Prep, MultipleRowsCopy3Add, MultipleRowsCopy3Finish, cancellationToken);
=> MultipleRowsCopyHelperAsync(helper, source, from, MultipleRowsCopy3Prep, MultipleRowsCopy3Add, MultipleRowsCopy3Finish, cancellationToken, MaxParameters, MaxSqlLength);
#endif

private void MultipleRowsCopy3Prep(MultipleRowsHelper helper)
Expand Down Expand Up @@ -615,7 +668,7 @@ private void MultipleRowsCopy3Add(MultipleRowsHelper helper, object item, string
helper.StringBuilder
.AppendLine()
.Append("\tSELECT ");
helper.BuildColumns(item);
helper.BuildColumns(item, castParameters: CastOnUnionAll, castAllRows : TypeAllUnionParameters);
helper.StringBuilder.Append(from);
helper.StringBuilder.Append(" UNION ALL");

Expand Down
12 changes: 11 additions & 1 deletion Source/LinqToDB/DataProvider/DB2/DB2BulkCopy.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,17 @@ namespace LinqToDB.DataProvider.DB2

class DB2BulkCopy : BasicBulkCopy
{
private readonly DB2DataProvider _provider;
/// <remarks>
/// Settings based on https://www.ibm.com/docs/en/i/7.3?topic=reference-sql-limits
/// We subtract 1 here to be safe since some ADO providers use parameter for command itself.
/// </remarks>
protected override int MaxParameters => 1999;
/// <remarks>
/// Setting based on https://www.ibm.com/docs/en/i/7.3?topic=reference-sql-limits
/// Max is actually 2MIB, but we keep a lower number here to avoid the cost of huge statements.
/// </remarks>
protected override int MaxSqlLength => 327670;
private readonly DB2DataProvider _provider;

public DB2BulkCopy(DB2DataProvider provider)
{
Expand Down
16 changes: 16 additions & 0 deletions Source/LinqToDB/DataProvider/Firebird/FirebirdBulkCopy.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,22 @@ namespace LinqToDB.DataProvider.Firebird

class FirebirdBulkCopy : BasicBulkCopy
{

/// <remarks>
/// Number based on http://www.firebirdfaq.org/faq197/
/// TODO: Add Compat Switch. Firebird 2.5 has 64k limit, Firebird 3.0+ 10MB.
/// </remarks>
protected override int MaxSqlLength => 65535;

/// <remarks>
/// Based on https://github.com/FirebirdSQL/firebird/blob/799bca3ca5f9eb604433addc0f2b7cb3b6c07275/src/dsql/DsqlCompilerScratch.cpp#L528
/// Max is 65536/2. We subtract one from that in case ADO provider uses parameter for statemnt.
/// </remarks>
protected override int MaxParameters => 32767;

protected override bool CastOnUnionAll => true;
protected override bool TypeAllUnionParameters => true;

protected override BulkCopyRowsCopied MultipleRowsCopy<T>(
ITable<T> table, BulkCopyOptions options, IEnumerable<T> source)
{
Expand Down
3 changes: 2 additions & 1 deletion Source/LinqToDB/DataProvider/Informix/InformixBulkCopy.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ namespace LinqToDB.DataProvider.Informix

class InformixBulkCopy : BasicBulkCopy
{
private readonly InformixDataProvider _provider;
protected override int MaxSqlLength => 32767;
private readonly InformixDataProvider _provider;

public InformixBulkCopy(InformixDataProvider provider)
{
Expand Down
Loading