-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Fix LargeArrayBuilder.CopyTo returning incorrect end-of-copy position #23817
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -198,7 +198,7 @@ public void CopyTo(T[] array, int arrayIndex, int count) | |
public CopyPosition CopyTo(CopyPosition position, T[] array, int arrayIndex, int count) | ||
{ | ||
Debug.Assert(arrayIndex >= 0); | ||
Debug.Assert(count >= 0 && count <= Count); | ||
Debug.Assert(count > 0 && count <= Count); | ||
Debug.Assert(array?.Length - arrayIndex >= count); | ||
|
||
// Go through each buffer, which contains one 'row' of items. | ||
|
@@ -216,25 +216,36 @@ public CopyPosition CopyTo(CopyPosition position, T[] array, int arrayIndex, int | |
int row = position.Row; | ||
int column = position.Column; | ||
|
||
for (; count > 0; row++, column = 0) | ||
T[] buffer = GetBuffer(row); | ||
int copied = CopyToCore(buffer, column); | ||
|
||
while (count > 0) | ||
{ | ||
buffer = GetBuffer(++row); | ||
copied = CopyToCore(buffer, 0); | ||
} | ||
|
||
return copied == buffer.Length | ||
? new CopyPosition(row + 1, 0) | ||
: new CopyPosition(row, copied); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm, I believe the previous setup I had in place which looked like /* first copy */
if (count == 0)
{
// Note it's `column + copied` instead of `copied`
return new CopyPosition(row, column + copied).Normalize(buffer.Length);
}
do
{
/* subsequent copies */
}
while (count > 0);
return new CopyPosition(row, copied).Normalize(buffer.Length); was correct. With the current code, something like new TE<int>(new[] { 1 }).Concat(new TE<int>(new[] { 2 })).Concat(new TE<int>(new[] { 3 })).Concat(new[] { 4 }).ToArray() or new[] { 1, 2, 3, 4 }.SelectMany(i => i == 4 ? new[] { i } : new TE<int>(new[] { i })).ToArray() should result in If it is, you can add a regression test for them named
The first LAB.CopyTo takes (row = 0, column = 0) as input, copies 1 item ( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The previous code is not needed because we now assert that |
||
|
||
int CopyToCore(T[] sourceBuffer, int sourceIndex) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: Is it customary in corefx to put a local func at the very end of the containing method? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We have very few local methods; there's no convention yet. But unless there's an obviously better place, end of the method makes sense. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure what is the custom for this repo. I've not seen a lot of local functions used here yet. But that pattern is common. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this could be simpler without a local function, but this will work too. |
||
{ | ||
T[] buffer = GetBuffer(index: row); | ||
Debug.Assert(sourceBuffer.Length > sourceIndex); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You mentioned previously this assert failed. It's good to see that's no longer the case. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, because of the |
||
|
||
// During this iteration, copy until we satisfy `count` or reach the | ||
// end of the current buffer. | ||
int copyCount = Math.Min(buffer.Length, count); | ||
// Copy until we satisfy `count` or reach the end of the current buffer. | ||
int copyCount = Math.Min(sourceBuffer.Length - sourceIndex, count); | ||
|
||
if (copyCount > 0) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We don't have any tests where copyCount == 0. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @OmarTawfik I don't think this can happen anymore. Add these asserts and remove the conditional: +Debug.Assert(count > 0);
Debug.Assert(sourceBuffer.Length > sourceIndex);
// Copy until we satisfy `count` or reach the end of the current buffer.
int copyCount = Math.Min(sourceBuffer.Length - sourceIndex, count);
+Debug.Assert(copyCount > 0); |
||
{ | ||
Array.Copy(buffer, column, array, arrayIndex, copyCount); | ||
Array.Copy(sourceBuffer, sourceIndex, array, arrayIndex, copyCount); | ||
|
||
arrayIndex += copyCount; | ||
count -= copyCount; | ||
column += copyCount; | ||
} | ||
} | ||
|
||
return new CopyPosition(row: row, column: column); | ||
return copyCount; | ||
} | ||
} | ||
|
||
/// <summary> | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -149,8 +149,11 @@ public void CopyTo(T[] array, int arrayIndex, int count) | |
count -= reservedCount; | ||
} | ||
|
||
// Finish copying after the final marker. | ||
_builder.CopyTo(position, array, arrayIndex, count); | ||
if (count > 0) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is beneficial for scenarios when we insert a marker at the very end, e.g. |
||
{ | ||
// Finish copying after the final marker. | ||
_builder.CopyTo(position, array, arrayIndex, count); | ||
} | ||
} | ||
|
||
/// <summary> | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -421,5 +421,18 @@ public void GetEnumerableOfConcatCollectionChainFollowedByEnumerableNodeShouldBe | |
Assert.Equal(0xf00, en.Current); | ||
} | ||
} | ||
|
||
[Fact] | ||
public void CollectionInterleavedWithLazyEnumerables_ToArray_Regression() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: Do we need the "_Regression" suffix on these tests? In a year it won't matter to someone reading the code whether we added a test proactively or in response to a bug. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @stephentoub I personally added public void CollectionInterleavedWithLazyEnumerables_ToArray()
{
// This is a regression test for https://github.com/dotnet/corefx/issues/23680
...
} There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
That's why test naming is important, to qualify what the test is for. Comments are good, too, of course. But "_Regression" as a suffix is so moment-in-time. If you were to follow true TDD, for example, everything would need to be suffixed with "_Regression" because the tests are all written before any code is and thus all fail initially. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll add a comment with the bug URL. |
||
{ | ||
var results = new TestEnumerable<int>(new[] { 1 }) | ||
.Concat(new[] { 2 }) | ||
.Concat(new TestEnumerable<int>(new[] { 3 })) | ||
// Do not omit this ToArray()! There was a previous bug where the ToArray() implementation | ||
// was incorrect, while iterating with foreach produced the correct results. | ||
.ToArray(); | ||
|
||
Assert.Equal(new[] { 1, 2, 3 }, results); | ||
} | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -477,5 +477,19 @@ public void ThrowOverflowExceptionOnConstituentLargeCounts(int[] counts) | |
IEnumerable<int> iterator = counts.SelectMany(c => Enumerable.Range(1, c)); | ||
Assert.Throws<OverflowException>(() => iterator.Count()); | ||
} | ||
|
||
[Fact] | ||
public void CollectionInterleavedWithLazyEnumerables_ToArray_Regression() | ||
{ | ||
var results = new[] { 4, 5, 6 } | ||
.SelectMany(i => i == 5 ? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems like we should have tests with other operators and with varying sizes, not just SelectMany/Concat and not just with one or a few items. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also, since various operators specialize on the source collection type, seems like we need tests on various source types. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I explained in my last PR's description that since Append/Prepend never use the incorrect returned There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Even if that's true of the current implementation, you're assuming the current implementation will never change. Tests are important both to verify existing behavior and to help prevent future regressions. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The bug is affecting only SelectMany() and Concat(). I've added tests for small and large data sets. |
||
(IEnumerable<int>)new List<int>() { i } : | ||
new TestEnumerable<int>(new int[] { i })) | ||
// Do not omit this ToArray()! There was a previous bug where the ToArray() implementation | ||
// was incorrect, while iterating with foreach produced the correct results. | ||
.ToArray(); | ||
|
||
Assert.Equal(new[] { 4, 5, 6 }, results); | ||
} | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't have any tests that enter this loop.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added tests with larger data sets