Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change Feed Estimator: Adds support for detailed estimation per lease in PREVIEW #1830

Merged
merged 55 commits into from
Sep 30, 2020

Conversation

ealsur
Copy link
Member

@ealsur ealsur commented Sep 2, 2020

Description

This PR adds a new public API to obtain the detailed estimation per Lease on demand.
This API produces an iterator that returns pages of estimations. The iterator can have a customizable page size that will limit the number of estimations done at any given time. By default, it will estimate all leases on a single page.

In the Container, we have:

     public abstract ChangeFeedEstimator GetChangeFeedEstimator(
            string processorName,
            Container leaseContainer);

And the ChangeFeedEstimator:

public abstract class ChangeFeedEstimator
{
    public abstract FeedIterator<ChangeFeedProcessorState> GetCurrentStateIterator(ChangeFeedEstimatorRequestOptions?);
}

Where the ChangeFeedEstimatorRequestOptions gives an option to set the MaxItemCount:

public class ChangeFeedEstimatorRequestOptions
{
    public int MaxItemCount { get; set; }
}

Each page will contain a list of ChangeFeedProcessorState:

public class ChangeFeedProcessorState
{
    public string LeaseToken { get; }

    public long EstimatedLag { get; }

    public string InstanceName { get; }
}

Where the LeaseToken represents the identifier for that lease, the EstimatedLag is the estimation, and InstanceName is the name of the Change Feed processor instance processing that lease.

How can a user consume it

ChangeFeedEstimator estimator =  container.GetChangeFeedEstimator(processorName: "<processorNameUsedForProcessor>", leaseContainer);
FeedIterator<RemainingLeaseWork> iterator =  estimator.GetRemainingLeaseWorkIterator();

while (HasMoreResults)
{
    FeedResponse<RemainingLeaseWork> leaseWork = await iterator.ReadNextAsync(CancellationToken);
    foreach (RemainingLeaseWork estimation in leaseWork)
    {
        Console.WriteLine($"Lease {estimation.LeaseToken} owned by {estimation.InstanceName} has {estimation.RemainingWork} estimated pending items.");
    }
}

Code refactorings

The PR also refactors part of the CFP code. Originally there was code shared between the Processor, and the Estimator (when the Estimator was only a push model). Code refactoring had to be done to include the Estimator pull model and also maintain the Estimator push model (push now uses pull internally)

Also included some Code Analysis fixes on modified files plus new tests all around (emulator and unit tests).

Relevant files: ChangeFeedEstimatorIterator.cs, ContainerCore.Items.cs, ChangeFeedProcessorState.cs, ChangeFeedEstimatorRequestOptions.cs, ContractTests.cs.

Type of change

  • New feature (non-breaking change which adds functionality)

Closing issues

Closes #1730

@ealsur ealsur changed the title Change Feed Estimator: Adds support for detailed estimation per lease Change Feed Estimator: Adds support for detailed estimation per lease in PREVIEW Sep 4, 2020
@ealsur ealsur merged commit 9e8723e into master Sep 30, 2020
@ealsur ealsur deleted the users/ealsur/estimatorleasespull branch September 30, 2020 16:34
@ghost
Copy link

ghost commented Dec 15, 2021

Closing due to in-activity, pease feel free to re-open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Change Feed Processor: Add support to estimate change feed progress on a per lease basis
4 participants