Change Feed StartFrom.Now() not respected with multiple physical partitions

[abranaugh]

We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.

Describe the bug
We updated to the newest version of the change feed pull model, when we updated our code to use the new ChangeFeedStartFrom.Time() or ChangeFeedStartFrom.Now(), we received documents that were changed prior to the time we specified. We tested this with several different collections. ChangeFeedStartFrom.Time() and ChangeFeedStartFrom.Now() returned the expected number of documents when we ran against a collection with a single physical partition. We only are seeing this for collections with multiple physical partitions.

To Reproduce
We pulled the latest version of the SDK down, added the following test to the Microsoft.Azure.Cosmos.EmulatorTests.FeedRanges.ChangeFeedIteratorCoreTests file and pointed to our dev environment

[TestMethod]
public async Task ChangeFeedIteratorCore_StartTime_LargeContainer()
{
    int totalCount = 0;
    int batchSize = 25;
    await this.CreateRandomItems(this.LargerContainer, batchSize, randomPartitionKey: true);
    await Task.Delay(1000);
    DateTime now = DateTime.UtcNow;
    await Task.Delay(1000);
    await this.CreateRandomItems(this.LargerContainer, batchSize, randomPartitionKey: true);
    ContainerInternal itemsCore = this.LargerContainer;

    FeedIterator feedIterator = itemsCore.GetChangeFeedStreamIterator(
		ChangeFeedStartFrom.Time(now));
    while (feedIterator.HasMoreResults)
    {
        using (ResponseMessage responseMessage =
	    await feedIterator.ReadNextAsync(this.cancellationToken))
        {
            if (responseMessage.IsSuccessStatusCode)
            {
				Collection<ToDoActivity> response = TestCommon.SerializerCore.FromStream<CosmosFeedResponseUtil<ToDoActivity>>(responseMessage.Content).Data;
                totalCount += response.Count;
            }
        }
    }
	
    Assert.AreEqual(batchSize, totalCount);
}

Expected behavior
We were expecting the second set of 25 items that were inserted to be returned.

Actual behavior
We are seeing 42 items returned.

Environment summary
SDK Version: 3.13.0 - Preview and 3.13.0

[j82w]

@bchong95 can you confirm this is fixed by the PR #1933 ?

[j82w]

This will be fixed in 3.15.0-preview which should be released in the next few days.

[j82w]

@abranaugh please try the latest 3.15.0-preview

[abranaugh]

@j82w I tested out the change yesterday and it looks like it is working as expected now, I did have to make a few changes though to our tests.

In the original test that I posted, I had to modify the while loop to this:

    while (feedIterator.HasMoreResults)
    {
        using (ResponseMessage responseMessage =
	    await feedIterator.ReadNextAsync(this.cancellationToken))
        {
            if (!responseMessage.IsSuccessStatusCode)
            {
                break;
            }
	        Collection<ToDoActivity> response = TestCommon.SerializerCore.FromStream<CosmosFeedResponseUtil<ToDoActivity>>(responseMessage.Content).Data;
                totalCount += response.Count;
            }
        }
    }

It looked like the HasMoreResults never flipped to false, even after I have received all of the documents in the container. Instead, the responseMessage had a 304 and the IsSuccessStatusCode was false. I also ran a test against the same container, and used ChangeFeedStartFrom.Now(). I didn't add anything to the container in this test, so there should have been no changes to retrieve, and in this case the HasMoreResults is still true. Based on the description of the HasMoreResults property on the Feed Iterator, I would expect that to be false when there aren't any more documents to be processed.

[j82w]

@bchong95 please take a look at the above feedback since you made the change.

[ghost]

Closing due to in-activity, pease feel free to re-open.