Change Feed Pull Model Memory Issues

[abranaugh]

We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.

Describe the bug
We are trying to update our SDK version from 3.9.1 - preview to the latest preview version. We use the change feed pull model. We noticed when we updated our version to 3.15.2 - preview and deployed, our machines quickly ran out of RAM. Looking at the logs, we got up to about 23gb of RAM in use and then crashed. I was able to reproduce locally and noticed that the RAM on my machine quickly ballooned.

I put together a console app to verify the behavior wasn't anything in our code, the only dependencies in the console app is to Cosmos SDK. In my test, I read off the change feed for 10 minutes and monitored the diagnostics in VS2019. I tested with various versions and took screenshots of the memory usage in the diagnostic session. These tests were against a collection that is about ~300GB

3.15.2 - Preview - Ended up at around 1.8gb for the 10 minutes

3.15.1 - Preview - 900mb in 10 minutes

3.15.0 - Preview - 600mb in 10 minutes

3.14.0 - Preview - 150mb in 10 minutes

3.13.0 - Preview - 150mb in 10 minutes

3.9.1 - Preview - 150mb in 10 minutes

Here is the code for the methods I ran:

3.13 and up

public async Task RunFeed()
{
   var connection = "";
   var container = new CosmosClient(connection).GetContainer("", "");
   var totalDocs = 0;
   var options = new ChangeFeedRequestOptions
   {
     PageSizeHint = 5000
   };
    
   using (var iterator = container.GetChangeFeedStreamIterator(ChangeFeedStartFrom.Beginning(), options))
   {
      var watch = Stopwatch.StartNew();
      while (iterator.HasMoreResults && watch.Elapsed < TimeSpan.FromMinutes(10))
      {
          using (var response = await iterator.ReadNextAsync())
          {
              if (!response.IsSuccessStatusCode)
              {
                 break;
              }

              using (var changes = await JsonDocument.ParseAsync(response.Content))
              {
                  var count = changes.RootElement.GetProperty("_count").GetInt32();
                  Console.WriteLine($"Retrieved {count} documents");
                  totalDocs += count;
              }
          }
      }
      Console.WriteLine($"Found {totalDocs} total documents in {watch.Elapsed.TotalMinutes} minutes");
    }
}

3.9.1 - Preview

public async Task RunFeed()
{
    var connection = "";
    var container = new CosmosClient(connection).GetContainer("", "");
    var totalDocs = 0;
    
    var iterator = container.GetChangeFeedStreamIterator(changeFeedRequestOptions: new ChangeFeedRequestOptions
    {
        StartTime = DateTime.MinValue,
        MaxItemCount = 5000
    });
    
    var watch = Stopwatch.StartNew();
    while (iterator.HasMoreResults && watch.Elapsed < TimeSpan.FromMinutes(10))
    {
        using (var response = await iterator.ReadNextAsync())
        {
            using (var changes = await JsonDocument.ParseAsync(response.Content))
            {
                var count = changes.RootElement.GetProperty("_count").GetInt32();
                Console.WriteLine($"Retrieved {count} documents");
                totalDocs += count;
            }
        }
    }
    
    Console.WriteLine($"Found {totalDocs} total documents in {watch.Elapsed.TotalMinutes} minutes");
}

Environment summary
SDK Version: 3.15.0 - Preview, 3.15.1 - Preview, 3.15.2 - Preview
.Net Core 3.1

[j82w]

Likely caused by #1933

[j82w]

@abranaugh please contact support if this is impacting your production so a support ticket is created and will ensure the issue is properly prioritized.

[j82w]

@abranaugh any chance you can provide a memory dump or perf trace to help root cause the issue?

[abranaugh]

@j82w I can provide a memory dump, how would you like me to provide that to you? When I dumped out the version running 3.15.2 - preview, it was about 1.8gb. I don't want to post it publicly on github due to potentially exposing our connection strings and whatnot. I can open up a support ticket and attach it to that if that is easiest.

[j82w]

Please open a support ticket and attach the memory dump as it is the easiest way.

[shibayan]

I also encountered this problem using the latest preview version (3.15.2-preview). Based on the findings of this issue, I decided to use 3.14.0-preview, but now I've encountered the issue #1918 and have a difficult version to choose.

[j82w]

Fix in PR #2129