Allow Parameter Usage on MultipleRowsCopy/MultipleRowsCopyAsync #2975

to11mtm · 2021-04-29T17:21:38Z

Allow Parameter usage on MultipleRowsCopy/MultipleRowsCopyAsync via BulkCopyOptions. Fix #2144

…opyAsyinc

to11mtm · 2021-04-29T17:28:08Z

Source/LinqToDB/DataProvider/BasicBulkCopy.cs

+				helper.LastRowParameterIndex = helper.ParameterIndex;
+				helper.LastRowStringIndex    = helper.StringBuilder.Length;
 				addFunction(helper, item!, from);
-
-				if (helper.CurrentCount >= helper.BatchSize || helper.Parameters.Count > maxParameters || helper.StringBuilder.Length > maxSqlLength)
+				var needRemove = helper.Parameters.Count > maxParameters ||
+				                 helper.StringBuilder.Length > maxSqlLength;
+				if (helper.CurrentCount >= helper.BatchSize || needRemove)
 				{
+					if (needRemove)
+					{
+						helper.Parameters.RemoveRange(helper.LastRowParameterIndex, helper.ParameterIndex-helper.LastRowParameterIndex);
+						helper.StringBuilder.Length = helper.LastRowStringIndex;	
+					}
 					finishFunction(helper);
 					if (!helper.Execute())
 						return helper.RowsCopied;
+					if (needRemove)
+					{
+						addFunction(helper, item!, from);	
+					}


So, I think this could -possibly- be simplified a little bit. I put LastRowParameterIndex and LastRowStringIndex into MultipleRowsHelper since .Execute() is what is clearing out the other bits we are comparing to. I'm not certain whether that is considered better or worse than just tracking those values locally in these functions.

other option to clean up would be if(CurrentCount > rowcount )->else if (needRemoveCondition).

…ase where a single row takes us over a limit; we should try anyway. Made MaxParameters and MaxSqlLength virtual properties so we can set it per-provider bulk copy. Set for Postgres, oracle, sqlite, sqlserver

to11mtm

Left some more comments since I have questions before I do much more.

Hoping for feedback before I go too far down a rabbit hole:

Should we limit Parameters to 8000 even if provider supports more (for LOH concerns)?

Also, Whether it's OK to move Implementations of MultipleRowsHelper/(Async) methods in BasicBulkCopy into Instance of MultipleRowsHelper, Open to better names than Bind/BindAsync. I tried doing another branch where we cached parameter arrays, and the code got pretty ugly and a bit hard to understand when kept in BasicBulkCopy.

to11mtm · 2021-04-30T14:36:18Z

Source/LinqToDB/DataProvider/Oracle/OracleBulkCopy.cs

-		private readonly OracleDataProvider _provider;
-
+		private const      int                _maxParameters = 32766;
+		private const      int                _maxSqlLength  = 327670;


Really, Oracle allows much longer SQL length; I'm not sure what number really 'makes sense' for batching however, huge strings probably aren't great for a number of reasons. These consts are here because this class has static functions so they need a const or static passed in.

to11mtm · 2021-04-30T14:36:53Z

Source/LinqToDB/DataProvider/SQLite/SQLiteBulkCopy.cs

@@ -8,6 +8,9 @@ namespace LinqToDB.DataProvider.SQLite

 	class SQLiteBulkCopy : BasicBulkCopy
 	{
+		protected override int MaxParameters => 999;
+		protected override int MaxSqlLength => 1000000;


Went higher on this number since SQLite is going to be parsed locally and we won't have a network hop.

Source/LinqToDB/DataProvider/BasicBulkCopy.cs

to11mtm · 2021-04-30T15:39:16Z

Source/LinqToDB/DataProvider/BasicBulkCopy.cs

@@ -15,6 +15,9 @@ namespace LinqToDB.DataProvider

 	public class BasicBulkCopy
 	{
+		protected virtual int MaxParameters => 999;
+		protected virtual int MaxSqlLength => 100000;


Putting this as a virtual -feels- better, but I still ask whether we should let Max Length be tunable by the user or not (For Network round-trip purposes). I decided not, since MaxBatchSize is good enough to do the same most likely.

COuld be usful. In baselines I see increase in number of roundtrips e.g. for DB2

to11mtm · 2021-04-30T16:32:03Z

Source/LinqToDB/DataProvider/BasicBulkCopy.cs

-			int                                       maxParameters = 10000,
-			int                                       maxSqlLength  = 100000)
+			int                                       maxParameters,
+			int                                       maxSqlLength)


I think it might be worth moving these, even with a simple Redirection for now, into a 'Bind' like method on MultipleRowsHelper. Over 15 helper. in a method feels like a smell to me.

to11mtm · 2021-04-30T16:38:29Z

Source/LinqToDB/DataProvider/SqlServer/SqlServerBulkCopy.cs

@@ -12,7 +12,9 @@ namespace LinqToDB.DataProvider.SqlServer

 	class SqlServerBulkCopy : BasicBulkCopy
 	{
-		private readonly SqlServerDataProvider _provider;
+		protected override int                   MaxParameters => 2099;


IDK if this is the value we -really- want to use, because of dotnet/SqlClient#974. Large number of parameters is actually counter-productive for performance.

to11mtm · 2021-04-30T16:43:52Z

Source/LinqToDB/DataProvider/BasicBulkCopy.cs

-				if (helper.CurrentCount >= helper.BatchSize || helper.Parameters.Count > maxParameters || helper.StringBuilder.Length > maxSqlLength)
+				var needRemove = helper.Parameters.Count > maxParameters ||
+				                 helper.StringBuilder.Length > maxSqlLength;
+				var isSingle = helper.CurrentCount == 1;


We need this check because there may be an edge case where even a single row was putting us over Max SQL limit.

Since we are now trying to be more correct here, we now 'back off' the last row when we go over maxSqlLength, rather than just continuing and hoping for the best. That said, we should still try to insert in case of Single row, and let provider fail if it really is an issue.

to11mtm · 2021-04-30T16:45:02Z

Source/LinqToDB/DataProvider/BasicBulkCopy.cs

+					if (needRemove && !isSingle)
+					{
+						helper.Parameters.RemoveRange(helper.LastRowParameterIndex, helper.ParameterIndex-helper.LastRowParameterIndex);
+						helper.StringBuilder.Length = helper.LastRowStringIndex;
+						helper.RowsCopied.RowsCopied--;	
+					}


We do this before finishFunction, in case finishFunction decides to add a parameter in future.

to11mtm · 2021-05-03T21:51:28Z

@viceroypenguin If you get a chance can you beat me up on this one please?

viceroypenguin · 2021-05-04T00:29:42Z

First thing I would suggest is move the maxParameters and maxSqlText from MultipleRowsCopyHelperAsync() method to MultipleRowsHelper constructor. Then you can use maxParameters to reduce MaxBatchSize as necessary (BatchSize = Math.Min(Math.Max(10, Options.MaxBatchSize ?? 1000), maxParameters / Columns.Count); notice the -1). This will remove the need to check for max parameters, since the batch size is implicitly less than max parameters.

The final concern will be the text length. For that, your current solution will work, though I think you can track LastRowXxx as local variables rather than properties on the Helper object? There's no way to move that boiler-plate that's copied three times into a common method or pattern is there? Seems like just enough code to screw up editing later and get the three out of sync w/ each other.

to11mtm · 2021-05-08T13:23:40Z

First thing I would suggest is move the maxParameters and maxSqlText from MultipleRowsCopyHelperAsync() method to MultipleRowsHelper constructor. Then you can use maxParameters to reduce MaxBatchSize as necessary (BatchSize = Math.Min(Math.Max(10, Options.MaxBatchSize ?? 1000), maxParameters / Columns.Count); notice the -1). This will remove the need to check for max parameters, since the batch size is implicitly less than max parameters.

Thank you for the feedback! I'll give that a try. I was trying not to be too aggressive in this PR since next round (i.e. another PR) I'd like to optimize the build stage so that we don't re-build the string/parameter lists. I think to do it well we'll want to re-do the loop.

The final concern will be the text length. For that, your current solution will work, though I think you can track LastRowXxx as local variables rather than properties on the Helper object? There's no way to move that boiler-plate that's copied three times into a common method or pattern is there? Seems like just enough code to screw up editing later and get the three out of sync w/ each other.

We might be able to pull out at least some of the logic, challenge is each one is just different enough where more indirection feels like as much cognitive load as the duped code.

But, I think overall the loop structure can be improved at least; Rather than Adding the item, and then checking length to decide whether to back off, we can check length at the start of the iteration and decide what to do. I think it will make the code a little cleaner, and will probably be cleaner as it will have less jumps.

viceroypenguin · 2021-05-08T13:27:59Z

Definitely agree with re-working to check the length before adding. That would simplify all three back to reasonable levels again and don't need to abstract the three versions of the loop into one.

Personally, I'd say go ahead and do it, rather than trying to piece-meal in two separate PRs, but that's a personal opinion. On the other hand, if you're going to redo it in a second PR, then what you've got works well enough for now I think.

to11mtm · 2021-05-08T13:31:23Z

Definitely agree with re-working to check the length before adding. That would simplify all three back to reasonable levels again and don't need to abstract the three versions of the loop into one.

Personally, I'd say go ahead and do it, rather than trying to piece-meal in two separate PRs, but that's a personal opinion. On the other hand, if you're going to redo it in a second PR, then what you've got works well enough for now I think.

If I wasn't about to deal with a bunch of release stuff at work that's gonna have me busy for a while I'd agree. :) But I think second pr is going to be a bigger refactoring of things, I'm still thinking of doing #2960 and that will mean a decent amount of change to this again most likely.

Question though, based on your other feedback; Should I put the MaxParameters/MaxSqlText into SqlProviderFlags? I think that may be cleaner overall, but it still raises the question of whether we want to have 'Real' MaxParameters or ones that make sense for working in .NET

to11mtm · 2021-05-18T01:33:23Z

I looked into setting MaxBatchSize based on param usage, it unfortunately gets a little hacky to work-around the behavior of Oracle's Array-parameter copy. Might be better to get what is here into 3.4.0 for now, so I am marking ready to see if there is other feedback or if we must make more changes first.

…w-Parameter-Use-On-Bulk-Copy-Part-1

MaceWindu · 2021-05-29T14:05:48Z

/azp run test-all

azure-pipelines · 2021-05-29T14:05:58Z

Azure Pipelines successfully started running 1 pipeline(s).

MaceWindu

Could you provide sources for limits for various providers in comments? I see some of them specified but some not

MaceWindu · 2021-05-29T14:06:30Z

Source/LinqToDB/Data/BulkCopyOptions.cs

@@ -88,5 +88,10 @@ public BulkCopyOptions(BulkCopyOptions options)
 		/// This callback will not be used if <see cref="NotifyAfter"/> set to 0.
 		/// </summary>
 		public Action<BulkCopyRowsCopied>? RowsCopiedCallback { get; set; }
+
+		/// <summary>
+		/// Gets or sets whether to Always use Parameters for MultipleRowsCopy.


makes sense to add default value information and behavior in this mode like splitting operation into batches if parameters limit reached for operation.

Source/LinqToDB/DataProvider/BasicBulkCopy.cs

MaceWindu · 2021-05-29T14:27:22Z

Tests/Linq/Update/BulkCopyTests.cs

@@ -276,5 +276,22 @@ public void ReuseOptionTest([DataSources(false, ProviderName.DB2)] string contex
 				db.Child. BulkCopy(options, new[] { new Child  { ParentID = 111001 } });
 			}
 		}
+
+		[Test]
+		public void UseParametersTest([DataSources(false, ProviderName.DB2)] string context)


What's wrong with DB2?

MaceWindu · 2021-05-29T14:29:33Z

@to11mtm, we plan to release 3.4 next thursday, it would be nice if we can include this one.

…hub.com/to11mtm/linq2db into Allow-Parameter-Use-On-Bulk-Copy-Part-1

MaceWindu · 2021-05-29T15:44:23Z

/azp run test-all

azure-pipelines · 2021-05-29T15:44:34Z

Azure Pipelines successfully started running 1 pipeline(s).

to11mtm · 2021-05-29T15:46:16Z

Could you provide sources for limits for various providers in comments? I see some of them specified but some not

I'll try to find my notes again. Most came from https://www.jooq.org/doc/3.12/manual/sql-building/dsl-context/custom-settings/settings-inline-threshold/ but for the strings I had to look in some other places.

linq2dbot · 2021-05-29T19:00:11Z

Test baselines changed by this PR. Don't forget to merge/close baselines PR after this pr merged/closed.

to11mtm · 2021-05-31T19:24:41Z

Firebird doesn't use type information from parameter, so you need to add it using CAST to first select of UNION

Yeah I found FirebirdSqlOptimizer.WrapParameters and cried for a few minutes.

I'm giving it a try with just casts on all rows. Ironically, after our other optimizations to SQL building (i.e. #2774) It appears that at least for SQL Server and maybe postgres, Parameters are slower than our string literals at this point. my guess/assumption is that the cost of marshaling the parameters is eating away any benefits, and because we aren't treating this as a prepared statement there's no gains to be had there either. My hope is that when myself or someone else does a more thorough refactoring of BulkCopy we can address some of this and some of the other cruft in BulkCopy/MultipleRowsHelper.

to11mtm · 2021-06-01T22:28:48Z

/azp run test-firebird

azure-pipelines · 2021-06-01T22:29:02Z

Azure Pipelines successfully started running 1 pipeline(s).

MaceWindu · 2021-06-02T08:06:45Z

Source/LinqToDB/DataProvider/MultipleRowsHelper.cs

@@ -77,7 +79,27 @@ public virtual void BuildColumns(object item, Func<ColumnDescriptor, bool>? skip
 				{
 					var name = ParameterName == "?" ? ParameterName : ParameterName + ++ParameterIndex;

-					StringBuilder.Append(name);
+					if (castParameters)


CAST needed only for first select in union. For other rows db will use type information from first type

Yeah in retrospect I don't think it would hurt to make this if (castParameters && CurrentCount==0). I was over-thinking the jump cost lol.

MaceWindu · 2021-06-02T08:07:52Z

Source/LinqToDB/DataProvider/MultipleRowsHelper.cs

+						StringBuilder.Append(name);
+						StringBuilder.Append(" AS ");
+						StringBuilder.Append(column.DataType);
+						if (!string.IsNullOrEmpty(column.DbType))


can we use sql builder here as it already have db-specific type builder method that takes into account db-specific stuff?

If you could give some pointers on which method to try to use in SQLBuilder I can give it a shot after work.

BuildDataType https://github.com/linq2db/linq2db/blob/master/Source/LinqToDB/SqlProvider/BasicSqlBuilder.cs#L2716
but you need to make it protected internal

I ran into some challenges here (we are using ISqlBuilder, can't cleanly internal on an interface,) so tried to solve for this by adding to ISqlBuilder Interface, StringBuilder BuildDataType (StringBuilder sb, SqlDataType dataType);. I went with returning the stringbuilder because that was the convention for BuildTableName, Convert, and others.

to11mtm · 2021-06-03T00:03:14Z

/azp run test-all

azure-pipelines · 2021-06-03T00:03:25Z

Azure Pipelines successfully started running 1 pipeline(s).

MaceWindu · 2021-06-03T08:41:17Z

/azp run test-all

MaceWindu · 2021-06-03T08:41:24Z

@to11mtm, if you are interested in working on this area more I have several suggestions for improvements:

issue, similar to one you fought with Firebird, still exists for parameter-less mode when first data row contains nulls (no type information issue for multiple databases)
instead of query generation for each batch, we can reuse generated SQL for batches of same size
it makes sense to investigate new batching feature from .net 6 : New System.Data.Common batching API dotnet/runtime#28633 . Here I would add that some providers already support it. IIRC SAP HANA provider repeat command if command use less parameters than you passed to command, but it is provider with positional parameters, so it is not a common case

azure-pipelines · 2021-06-03T08:41:27Z

Azure Pipelines successfully started running 1 pipeline(s).

MaceWindu · 2021-06-03T09:53:51Z

/azp run test-all

azure-pipelines · 2021-06-03T09:54:03Z

Azure Pipelines successfully started running 1 pipeline(s).

MaceWindu · 2021-06-03T11:03:00Z

/azp run test-all

azure-pipelines · 2021-06-03T11:03:12Z

Azure Pipelines successfully started running 1 pipeline(s).

linq2dbot · 2021-06-03T12:28:35Z

Test baselines changed by this PR. Don't forget to merge/close baselines PR after this pr merged/closed.

to11mtm · 2021-06-03T12:36:09Z

@to11mtm, if you are interested in working on this area more I have several suggestions for improvements:

* issue, similar to one you fought with Firebird, still exists for parameter-less mode when first data row contains nulls (no type information issue for multiple databases)

* instead of query generation for each batch, we can reuse generated SQL for batches of same size

* it makes sense to investigate new batching feature from .net 6 : [dotnet/runtime#28633](https://github.com/dotnet/runtime/issues/28633) . Here I would add that some providers already support it. IIRC SAP HANA provider repeat command if command use less parameters than you passed to command, but it is provider with positional parameters, so it is not a common case

Yeah, I want to do more work on this in future; My big goal is to have version of BulkCopy that will return inserted Identity columns (issue #2960 ), but it is a bigger refactor because every DB that supports returning on MultipleRows has different rules on how to do this right.

I started looking into re-using the generated SQL, I think for it to be of actual benefit we will need to do more refactoring. Our stringbuild is pretty efficent as-is, to get a worthwhile benefit I think we would have to also .Prepare() the statement, and go a little lower level on re-assigning parameter values (at least some data providers, if you add/remove/reassign DbParameter on DbCommand.Parameters the .Prepare() gets thrown out, you have to ONLY touch the actual Value of the existing DbParameter object, if that makes sense.)

MaceWindu · 2021-06-03T14:10:09Z

/azp run test-all

azure-pipelines · 2021-06-03T14:10:22Z

Azure Pipelines successfully started running 1 pipeline(s).

MaceWindu · 2021-06-03T14:15:19Z

/azp run test-all

azure-pipelines · 2021-06-03T14:15:31Z

Azure Pipelines successfully started running 1 pipeline(s).

* [Windows / NET472 / SQLite.MS] baselines * [Windows / NET472 / SQL CE] baselines * [Windows / NET472 / Access ODBC MDB] baselines * [Windows / NET472 / SQL Server 2012 (System.Data.SqlClient)] baselines * [Windows / NET472 / SQL Server 2005 (System.Data.SqlClient)] baselines * [Windows / NET472 / SQL Server 2008 (System.Data.SqlClient)] baselines * [Windows / NET472 / SQL Server 2016 (System.Data.SqlClient)] baselines * [Windows / NET472 / SQL Server 2014 (System.Data.SqlClient)] baselines * [Windows / NET472 / SQL Server 2017 (System.Data.SqlClient)] baselines * [Windows / NET472 / Access Jet] baselines * [Windows / NET472 / SQLite] baselines * [Windows / NETCOREAPP2.1 / SQL CE] baselines * [Windows / NETCOREAPP2.1 / Access ODBC ACE x64] baselines * [Windows / NETCOREAPP2.1 / SQLite.MS] baselines * [Windows / NET472 / SQL Server 2019 (System.Data.SqlClient)] baselines * [Windows / NET472 / SQL Server 2019 (Microsoft.Data.SqlClient)] baselines * [Windows / NETCOREAPP2.1 / SQL Server 2016 (System.Data.SqlClient)] baselines * [Windows / NETCOREAPP2.1 / SQL Server 2005 (System.Data.SqlClient)] baselines * [Linux / NETCOREAPP2.1 / Informix 14.10] baselines * [Windows / NETCOREAPP2.1 / SQL Server 2008 (System.Data.SqlClient)] baselines * [Linux / NETCOREAPP2.1 / DB2 LUW 11.5] baselines * [Windows / NET 5.0 / SQL Server 2019 (Microsoft.Data.SqlClient)] baselines * [Linux / NET5.0 / PostgreSQL 13] baselines * [Linux / NETCOREAPP3.1 / PostgreSQL] baselines * [Linux / NETCOREAPP3.1 / MySQL] baselines * [Windows / NETCOREAPP2.1 / SQL Server 2012 (System.Data.SqlClient)] baselines * [Windows / NETCOREAPP2.1 / SQL Server 2017 (System.Data.SqlClient)] baselines * [Linux / NET5.0 / Sybase ASE 16] baselines * [Linux / NET5.0 / SQLite] baselines * [Linux / NETCOREAPP2.1 / MariaDB] baselines * [Windows / NETCOREAPP2.1 / SQL Server 2014 (System.Data.SqlClient)] baselines * [Linux / NETCOREAPP2.1 / Firebird 3.0] baselines * [Linux / NETCOREAPP2.1 / Firebird 2.5] baselines * [Linux / NETCOREAPP2.1 / MySQL 5.5] baselines * [Linux / NETCOREAPP2.1 / PostgreSQL 10] baselines * [Linux / NETCOREAPP2.1 / PostgreSQL 11] baselines * [Linux / NETCOREAPP2.1 / PostgreSQL 12] baselines * [Linux / NETCOREAPP2.1 / PostgreSQL 9.2] baselines * [Linux / NETCOREAPP2.1 / PostgreSQL 9.3] baselines * [Linux / NETCOREAPP2.1 / PostgreSQL 9.5] baselines * [Linux / NETCOREAPP2.1 / Oracle 11g XE] baselines * [Linux / NETCOREAPP2.1 / Oracle 12c] baselines * [Linux / NETCOREAPP2.1 / SAP HANA 2] baselines * [Linux / NETCOREAPP2.1 / Firebird 4.0 (RC1)] baselines Co-authored-by: Azure Pipelines Bot <azp@linq2db.com>

Add Option to Force Parameter Usage on MultipleRowsCopy/MultipleRowsC…

16bd698

…opyAsyinc

to11mtm commented Apr 29, 2021

View reviewed changes

Fixup bad Last string position index logic. Make sure to handle the c…

9caf4b9

…ase where a single row takes us over a limit; we should try anyway. Made MaxParameters and MaxSqlLength virtual properties so we can set it per-provider bulk copy. Set for Postgres, oracle, sqlite, sqlserver

to11mtm force-pushed the Allow-Parameter-Use-On-Bulk-Copy-Part-1 branch from 49fe3c5 to 9caf4b9 Compare April 29, 2021 19:48

to11mtm commented Apr 30, 2021

View reviewed changes

Add more DB and Param counts for DBs

cbec45a

to11mtm marked this pull request as ready for review May 18, 2021 01:33

MaceWindu added this to the 3.4.0 milestone May 22, 2021

to11mtm and others added 3 commits May 28, 2021 12:29

Adjust batch size in bulk copy methods

74edd8c

Merge branch 'master' of https://github.com/linq2db/linq2db into Allo…

ff4f35f

…w-Parameter-Use-On-Bulk-Copy-Part-1

Merge branch 'master' into Allow-Parameter-Use-On-Bulk-Copy-Part-1

9b8e172

MaceWindu requested changes May 29, 2021

View reviewed changes

MaceWindu requested a review from sdanyliv May 29, 2021 14:29

to11mtm added 2 commits May 29, 2021 11:28

Merge branch 'Allow-Parameter-Use-On-Bulk-Copy-Part-1' of https://git…

fab80eb

…hub.com/to11mtm/linq2db into Allow-Parameter-Use-On-Bulk-Copy-Part-1

Add a little XMLDoc, Allow override of MaxParamters via BulkCopyOptions.

f391ad9

linq2dbot mentioned this pull request May 29, 2021

Baselines for https://github.com/linq2db/linq2db/pull/2975 linq2db/linq2db.baselines#619

Closed

add references for max values.

92ad512

MaceWindu reviewed Jun 2, 2021

View reviewed changes

Use SqlBuilder to build the parameter type info

7d91b5f

MaceWindu added 2 commits June 3, 2021 09:57

Merge branch 'master' into Allow-Parameter-Use-On-Bulk-Copy-Part-1

2240547

address firebird issue

12b1748

fix warning

1a94108

MaceWindu added 2 commits June 3, 2021 12:57

Merge branch 'master' into Allow-Parameter-Use-On-Bulk-Copy-Part-1

848eea8

specify copy type in test

6b3a921

linq2dbot mentioned this pull request Jun 3, 2021

Baselines for https://github.com/linq2db/linq2db/pull/2975 linq2db/linq2db.baselines#629

Merged

fix column datatype resolve

ed372fc

MaceWindu merged commit c815832 into linq2db:master Jun 3, 2021

Allow Parameter Usage on MultipleRowsCopy/MultipleRowsCopyAsync #2975

Allow Parameter Usage on MultipleRowsCopy/MultipleRowsCopyAsync #2975

Conversation

to11mtm commented Apr 29, 2021 • edited by MaceWindu Loading

Choose a reason for hiding this comment

to11mtm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

to11mtm commented May 3, 2021

viceroypenguin commented May 4, 2021

to11mtm commented May 8, 2021

viceroypenguin commented May 8, 2021

to11mtm commented May 8, 2021 • edited Loading

to11mtm commented May 18, 2021

MaceWindu commented May 29, 2021

azure-pipelines bot commented May 29, 2021

MaceWindu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaceWindu commented May 29, 2021

MaceWindu commented May 29, 2021

azure-pipelines bot commented May 29, 2021

to11mtm commented May 29, 2021

linq2dbot commented May 29, 2021

to11mtm commented May 31, 2021

to11mtm commented Jun 1, 2021

azure-pipelines bot commented Jun 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaceWindu Jun 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

to11mtm commented Jun 3, 2021

azure-pipelines bot commented Jun 3, 2021

MaceWindu commented Jun 3, 2021

MaceWindu commented Jun 3, 2021

azure-pipelines bot commented Jun 3, 2021

MaceWindu commented Jun 3, 2021

azure-pipelines bot commented Jun 3, 2021

MaceWindu commented Jun 3, 2021

azure-pipelines bot commented Jun 3, 2021

linq2dbot commented Jun 3, 2021

to11mtm commented Jun 3, 2021

MaceWindu commented Jun 3, 2021

azure-pipelines bot commented Jun 3, 2021

MaceWindu commented Jun 3, 2021

azure-pipelines bot commented Jun 3, 2021

to11mtm commented Apr 29, 2021 •

edited by MaceWindu

Loading

to11mtm commented May 8, 2021 •

edited

Loading

MaceWindu Jun 2, 2021 •

edited

Loading