feat!: introduce result-pattern and automatic finalizer management

[kimpenhaus]

Hey Christoph @buehler

This is a comprehensive PR containing changes from integrating the operator into our cluster environment.

Summary

This PR introduces breaking changes to the KubeOps SDK, implementing a result pattern inspired by the Go operator implementation. Controllers and finalizers now return ReconciliationResult<TEntity> enabling explicit success/failure states, centralized requeuing via RequeueAfter, and automatic finalizer lifecycle management. Additional improvements include extensible requeue mechanisms, const value support in source generators, and configurable leader election types.

Breaking Changes ⚠️

1. Result Pattern

Controller and finalizer interfaces now return Task<ReconciliationResult<TEntity>> instead of Task:

Before:

public interface IEntityController<TEntity>
{
    Task ReconcileAsync(TEntity entity, CancellationToken cancellationToken);
    Task DeletedAsync(TEntity entity, CancellationToken cancellationToken);
}

After:

public interface IEntityController<TEntity>
{
    Task<ReconciliationResult<TEntity>> ReconcileAsync(TEntity entity, CancellationToken cancellationToken);
    Task<ReconciliationResult<TEntity>> DeletedAsync(TEntity entity, CancellationToken cancellationToken);
}

The ReconciliationResult<TEntity> provides:

• Success/failure status with error information
• Optional RequeueAfter timespan for delayed reprocessing
• Access to the updated entity after reconciliation (which allows, for example, changing the entity's state before finalizer detachment, which was not possible before as the entity would have been in a modified state)

Migration Example:

// Old implementation
public async Task ReconcileAsync(V1TestEntity entity, CancellationToken token)
{
    // ... reconciliation logic
}

// New implementation
public async Task<ReconciliationResult<V1TestEntity>> ReconcileAsync(V1TestEntity entity, CancellationToken token)
{
    // ... reconciliation logic

    // Success - requeue after 5 minutes
    return ReconciliationResult<V1TestEntity>.Success(entity, TimeSpan.FromMinutes(5));

    // Or failure with error message
    return ReconciliationResult<V1TestEntity>.Failure(entity, "Failed to process entity");
}

2. Namespace Reorganization

Types moved to new namespaces:

• IEntityController<TEntity>: KubeOps.Abstractions.Controller → KubeOps.Abstractions.Reconciliation.Controller
• IEntityFinalizer<TEntity>: KubeOps.Abstractions.Finalizer → KubeOps.Abstractions.Reconciliation.Finalizer
• EntityRequeue: KubeOps.Abstractions.Queue → KubeOps.Abstractions.Reconciliation.Queue
• IEntityRequeueFactory: KubeOps.Abstractions.Queue → KubeOps.Abstractions.Reconciliation.Queue

Migration: Update using statements in your controllers and finalizers.

3. Queue Interface Changes

The internal queue interface is now public and extensible:

public interface ITimedEntityQueue<TEntity>
{
    Task Enqueue(TEntity entity, RequeueType type, TimeSpan requeueIn, CancellationToken cancellationToken);
    Task Remove(TEntity entity, CancellationToken cancellationToken);
}

This enables implementing durable requeue mechanisms (e.g., backed by Redis, Service Bus, database) by overriding the default in-memory implementation.

New Features

1. Automatic Finalizer Management

Two new settings provide automatic finalizer lifecycle management:

builder.Services
    .AddKubernetesOperator(settings =>
    {
        // Automatically attach finalizers during reconciliation (default: true)
        settings.AutoAttachFinalizers = true;

        // Automatically detach finalizers after successful finalization (default: true)
        settings.AutoDetachFinalizers = true;
    });

Benefits:

• No manual finalizer management required
• Consistent finalizer handling across operators
• Reduces boilerplate code
• Can be disabled for custom finalization workflows

2. Const Value Support in Source Generator

The syntax receiver now supports constant values in Kubernetes entity attributes:

public static class Constants
{
    public const string ApiGroup = "mycompany.com";
    public const string ApiVersion = "v1";
}

[KubernetesEntity(
    Group = Constants.ApiGroup,  // Const values now supported
    ApiVersion = Constants.ApiVersion,
    Kind = "MyResource")]
public class V1MyResource : CustomKubernetesEntity<V1MyResourceSpec>
{
}

Benefits:

• Centralized API group/version management
• Compile-time constant validation
• Better code organization for multi-resource operators

3. Leader Election Type Configuration

Introduction of LeaderElectionType enum for explicit leader election configuration:

public enum LeaderElectionType
{
    None = 0,    // No leader election (default)
    Single = 1,  // Single leader election using Kubernetes leases
    Custom = 2   // Custom user-defined leader election mechanism
}

Configuration:

builder.Services
    .AddKubernetesOperator(settings =>
    {
        settings.LeaderElectionType = LeaderElectionType.Single;
        settings.LeaderElectionLeaseDuration = TimeSpan.FromSeconds(15);
        settings.LeaderElectionRenewDeadline = TimeSpan.FromSeconds(10);
        settings.LeaderElectionRetryPeriod = TimeSpan.FromSeconds(2);
    });

Benefits:

• Explicit configuration of leader election behavior
• Support for custom leader election implementations
• Clear distinction between single-instance and multi-instance deployments

4. Extensible Requeue Mechanism

Introduction of RequeueType enum and ITimedEntityQueue<TEntity> interface:

public enum RequeueType
{
    Added,
    Modified,
    Deleted
}

Use Cases:

• Implement durable requeue using external storage (Redis, Service Bus, database)
• Survive operator restarts
• Implement custom requeue strategies
• Add monitoring and metrics for requeue operations

Example Implementation:

public class DurableEntityQueue<TEntity> : ITimedEntityQueue<TEntity>
{
    public async Task Enqueue(TEntity entity, RequeueType type, TimeSpan requeueIn, CancellationToken cancellationToken)
    {
        // Store in Redis/Database with execution time
        await _storage.SaveAsync(entity, type, DateTime.UtcNow.Add(requeueIn));
    }

    public async Task Remove(TEntity entity, CancellationToken cancellationToken)
    {
        // Remove from external storage
        await _storage.DeleteAsync(entity);
    }
}

5. ReconciliationContext

New context object providing metadata about reconciliation triggers:

public sealed record ReconciliationContext<TEntity>
{
    public TEntity Entity { get; }
    public WatchEventType EventType { get; }
    public ReconciliationTriggerSource ReconciliationTriggerSource { get; }
}

Helps distinguish between API server events and operator-initiated requeues.

Implementation Details

Core Components

1. ReconciliationResult (src/KubeOps.Abstractions/Reconciliation/ReconciliationResult{TEntity}.cs)

   • Immutable record type with success/failure semantics
   • Optional requeue after duration
   • Error message and exception support

2. Reconciler (src/KubeOps.Operator/Reconciliation/Reconciler.cs)

   • Centralized reconciliation orchestration
   • Handles controller and finalizer invocation
   • Manages generation-based caching
   • Automatic finalizer attachment/detachment
   • Better testability

3. ITimedEntityQueue (src/KubeOps.Operator/Queue/ITimedEntityQueue{TEntity}.cs)

   • Public interface for queue implementations
   • Async methods with cancellation token support
   • Extensibility point for custom implementations

Alignment with Go Implementation

This implementation draws inspiration from controller-runtime (Go):

• Result pattern for reconciliation outcomes
• RequeueAfter concept for delayed reprocessing
• Clear separation of success/error states
• Flexible error handling strategies

Testing

• ✅ Comprehensive unit tests for ReconciliationResult<TEntity>
• ✅ Unit tests for ReconciliationContext<TEntity>
• ✅ Integration tests for finalizer auto-attach/detach
• ✅ Tests for const value support in syntax receiver
• ✅ Queue functionality tests with new RequeueType
• ✅ All existing integration tests updated and passing

Documentation

• Updated controller examples with new result pattern
• Added advanced configuration guide
• Updated finalizer documentation with auto-attach/detach settings
• Added caching documentation
• Migration guide included in this PR description

Additional Notes

• This pull request also includes the changes of chore(deps): updated k8s KubernetesClient to v18.0.5 #976

Migration Checklist

For operators upgrading to this version:

• Update controller methods to return ReconciliationResult<TEntity>
• Update finalizer methods to return ReconciliationResult<TEntity>
• Update namespace imports for reconciliation types
• Review automatic finalizer settings (defaults are enabled)
• Review leader election configuration (default: None)
• Consider using const values for entity attributes (optional)
• Test requeue behavior with new result pattern
• Review error handling using result pattern instead of exceptions

[kimpenhaus]

@ralf-cestusio I've now found the time to submit the pull request - you're welcome to review the changes and provide feedback. Thanks.

[buehler]

Hey @kimpenhaus
Wow. Thanks for this big contribution! It will surely take a while to comprehend what you've done :-)

One question to start with: The first change with the returning result pattern. This was implmeneted in a long past version of the sdk (v6 or so) and I changed it because I thought it is more extensible when you inject finalizer and requeue mechanisms instead of relying on return values. The return values are parsed by the SDK core engine and thus, feature implementations and enhancements must also touch the core. With the injection of such extensions (e.g. finalizers, requeue), you can provide those without touching the core. Or: at least, that was my intention.

wdyt?

[kimpenhaus]

Hey Christoph @buehler,

Yeah - take your time. I know it's a lot of work on your side. I appreciate the time you'll be investing - thanks for that. 🙏🏼

Regarding your point on the result pattern:

My intention is as follows:

• We had trouble with changing the entity while finalizing. With automatic detaching and no real option to route back the changed entity, this leads to a 409 conflict as the entity has already changed when trying to remove the finalizer. Returning the entity solves this issue.

• Regarding the flexibility you mentioned: from my point of view, you don't lose it - as retryAfter is optional. The idea was to optimize recurring code within the controller and finalizer, and remove code that can be centralized and might otherwise blur the logic in the controller and finalizer.

• Finalizer attaching and detaching could be configured, which also helps remove redundant, recurring code from the finalizer.

• I tried to orient the design around the Go implementation, which handles it in a similar way.

• In my opinion, this helps to better follow the responsibilities of each component (that's why I also introduced the reconciler).

Looking forward to your feedback! 😊

[ralf-cestusio]

I wanted to give some qualitative feedback (completely from a user perspective)

I have adapted out operator to use this pr and i really like how development feels.
Especially finalizer management has become a lot more expressive and easier to use.
The ability to handle detaching a finalizer with the result pattern (or scheduling a rerun of the finalizer in case it is not done yet) feels a lot more natural.

Normal reconcile code has also become more readable and i managed to now avoid all 409 errors, because we can harness updates more easily.

[buehler]

Cool! Thanks for the insights. Definitely looking forward to the code :)

[buehler]

Thank you for this big contribution! I really like the changes and I'm looking forward to see how people integrate those into their operators. Getting more aligned with the go implementation makes sense apparently.

I do have some minor comments/questions. Feel free to comment on them and let us have a discussion :-)

Thanks again!

[buehler]

Same here. Since the return type of the finalizeasync method has changed, please update the template

[kimpenhaus]

fixed

[buehler]

same :)

[kimpenhaus]

fixed

[buehler]

same :)

[kimpenhaus]

fixed

[buehler]

for consistency: can you make this one "sealed" as well? (you made the others sealed)

[kimpenhaus]

fixed

[buehler]

since you changed the structure and return values, can you update the template return types as well? this will fail when templates are used. also, for consistency, I'd like to see the controller / finalizer "sealed" like you did in the example codes.

[kimpenhaus]

wasn't aware of the templates - sorry - fixed all of them :)

[buehler]

Is "not being able to finalize" a success or rather an error? Depends on the finalizer I guess. wdyt?

[buehler]

If made public (intentionally I guess?), then there should be an XML Documentation for the class to be consistent.

[kimpenhaus]

I made it public to have a chance to overload it, as we do. we wanted requeuing to be durable and are using azure service bus as scheduler for it (added a sample to the docs how one can use it). this also gives the opportunity to trigger reconciliation over the service bus from different scenarios we have. e.g. when having a "infrastructural" deployments with some changes which need reconciliation to be triggered.

added the xml docs

[buehler]

short xml documentation about the class/record

[kimpenhaus]

added

[buehler]

why use "CreateFor" instead of the normal constructor? this seems redundant.

[kimpenhaus]

valid point - I thought a factory method is more expressive in the code but removing it and changing the record to required init properties reduced the code a lot - changed it

[ralf-cestusio]

I don't think this should be a warning.
There are valid cases why there is a finalizer on a resource, but its not handled by the operator.
A good example is a cascading foreground delete where k8s itself sets a finalizer