Release v9.7.0

Summary

Runtime support for GDPR redactions. This adds support for redacting personal data from previously committed events.

Redactions are scoped to a single artifact-type (EventTypeId) and an EventSourceId.

It will recognize events with the correct GUID prefix "de1e7e17-bad5-da7a" that match the event payload and is valid.

    public class Event
    {
        public required string EventId { get; init; }
        public required string EventAlias { get; init; }

        /// <summary>
        /// The properties that will be redacted, and the replacement values.
        /// Can be null, in which case the properties will be redacted with a default value
        /// </summary>
        public required Dictionary<string, object?> RedactedProperties { get; init; }

        public required string RedactedBy { get; init; }
        public required string Reason { get; init; }

        public bool IsValid => !string.IsNullOrWhiteSpace(EventId)
                               && !string.IsNullOrWhiteSpace(EventAlias)
                               && RedactedProperties.Count > 0
                               && !string.IsNullOrWhiteSpace(RedactedBy)
                               && !string.IsNullOrWhiteSpace(Reason);
    }

Any valid redactions will then be performed in the same transaction as it writes the new events. Replays of these events will then return the updated version of the event, with the redactions performed.

Added

• GDPR redaction support

Release v9.6.5

Summary

This release adds offset metadata initialization on startup, and does not wait until streams are being written. This supports the sparse streams use case, allowing the runtime to work correctly with events removed from the log.

Added

• Stream offset metadata init

Release v9.6.4

Summary

External behavior

This release adds support for sparse event logs, which allows users to delete events that are no longer needed. Previously the runtime would not tolerate any events being removed. Now it will just info log the fact and process as normally

Internal

The runtime has been refactored to use central package management, which reduces friction when upgrading dependency versions. It also removes a legacy dependency and replaces it with channels. In addition it will now enforce limits to work in progress by EventHorizon.

Added

• Central package management
• Support for sparse event logs

Removed

• Internal dependency on Nito.AsyncEx, rewritten to use channels

Release v9.6.3

Summary

This is a pure maintenance release, upgrading the runtime internal dependencies to their latest stable version.

No breaking changes or interface additions had been added.

Release v9.6.2

Summary

Hardened the handling of already written stream events. Will now log and skip in filter processors. Previously it was able to get stuck while processing a filter that was in an inconsistent state.

Fixed

• Improved error handling for filter processors, allowing the runtime to self-correct when the stream and processor state does not match

Release v9.6.1

Summary

This fixes the build scripts for the Dolittle CLI and switches the performance benchmarks to be only built for dotnet 8.

Fixed

• CLI builds
• Benchmark runner

Release v9.6.0

Summary

This release adds OpenTelemetry metrics to the runtime. It also changes the behavior of stream processors to be able to provide consumer lag metrics for all event handlers.

Stream processors

Previously if the event handler did not consume an event, it would also not update its current offset in the event log at all. Only processed events would cause an update. This release changes that, so all events are taken into consideration when storing how far each handler is in the stream. This allows the runtime to also provide consumer lag metrics that reflects this offset against the latest committed event.

OpenTelemetry

OpenTelemetry metrics will be enabled by default if there is defined an OpenTelemetry endpoint to publish to.
It can be disabled with Dolittle__Runtime__OpenTelemetry__Tracing=false

Added

• OpenTelemetry metrics support
• New metrics:
  • dolittle_customer_runtime_stream_processor_consumer_lag
  • dolittle_customer_runtime_stream_processors_offset
  • dolittle_customer_runtime_stream_processors_processed_total

Changed

• Stream processor state behavior - now considers unhandled events as well as processed events.

Release v9.5.1

Summary

Fixed stream ordering when using DB's that do not use the natural sort order of _id. This has been proven to be a potential issue on MongoDB Atlas.

Fixed

• Added explicit sorting on _id when querying streams

Release v9.5.0

Summary

Upgrades the production image of the runtime to use .NET 8, with the performance improvements this brings.

The development image is still on 7, keeping the bundled MongoDB as is.

Release v9.4.0

Summary

Projections are now supported within the SDK (from V23), and runs on top of standard event handlers with no need for special support in the runtime. This replaces the need for the projections feature in the runtime.

Updated dependencies for Proto.Actor, Grpc, OTEL etc.

Removed

• Runtime Projections.