Events and interception (aka lifecycle hooks)

[divega]

Done in 2.1

• Add events for entity state changes Add events for entity state changes #10895

Done in 3.1

• Add IDbCommandInterceptor or similar to allow injecting custom query execution service Add IDbCommandInterceptor or similar to allow injecting custom query execution service #15066
• IDbConnectionInterceptor and IDbTransactionInterceptor

Done in 5.0

• Implement events for before and after SaveChanges Implement events for before and after SaveChanges #15910

Done in 6.0

• Interception: raw query vs. LINQ Interception: raw query vs. LINQ #22899

Done in 7.0

• Implement ObjectMaterialized event Interception for object materialization (a.k.a. "ObjectMaterialized") #15911
• ChangeTracker: New event for "entity about to get tracked" ChangeTracker: New event for "entity about to get tracked" #27093
• Don't check for a connection string until after ConnectionOpening has been called Don't check for a connection string until after ConnectionOpening has been called #23085
• Add interception event to be fired when EF has finished consuming a result set Add interception event to be fired when EF has finished consuming a result set #23535
• Utility of DataReaderDisposing logging/interception Utility of DataReaderDisposing logging/interception #24295
• Please consider adding two methods to the DbConnectionInterceptor Creating/Created Add ConnectionCreating/ConnectionCreated to DbConnectionInterceptor #23087
• Consider delaying DbCommandInterceptor.CommandCreated until DbCommand properties are intialized Delay DbCommandInterceptor.CommandCreated until DbCommand properties are initialized #17261
• Interception for DbUpdateConcurrencyException Interception for DbUpdateConcurrencyException #28315
• Interception to modify the LINQ expression tree Interception to modify the LINQ expression tree #28505
• Provide event that can be used to get notifications when all property changes have been detected Provide event that can be used to get notifications when all property changes have been detected #26506

Backlog

• Add interception for transaction enlistment Add interception for transaction enlistment #16261
• Add interception for execution strategies Add interception for execution strategies #16257
• Add interception for batches Add interception for batches #16260
• Implement interceptors for Cosmos Implement interceptors for Cosmos #26478
• Add an interceptor to an existing DbContext instance Add an interceptor to an existing DbContext instance #20273
• Ability to register IInterceptor without an IDbContextOptionsExtension Ability to register IInterceptor without an IDbContextOptionsExtension #21578
• Interceptors for SQL generation Interceptors for SQL generation #19748
• Interception for model building Interception for model building #31206

---

Note: below is a copy of a very old EF specification and reflects thinking from several years ago. A lot of things aren't valid anymore.

We define EF Core lifecycle hooks as the general feature that enables an application or library to sign up to be invoked or notified whenever certain interesting conditions or actions occur as part of the lifecycle of entities, properties, associations, queries, context instances, and other elements in the Entity Framework stack.

For example:

1. An application can provide a method that will be invoked automatically whenever an object is about to be saved, or it can subscribe to an event that fires when an object is created and its properties initialized, etc.
2. A framework extension can register an interceptor that gives it an opportunity to rewrite query expression trees before they get translated by EF. This could be used to validate whether a user has access to specific information or to filter query results based on per DbContext filter (see Policy validation #6440).
3. Execute SQL after a DbConnection is opened (to use features such as SQL Server App Role)

The need for lifecycle hooks

We want to enable customers to write business logic that triggers in the different stages of the lifecycle of these objects, following well factored coding patterns. We also want framework writers to be able to use these hooks to extend EF Core in useful ways.

In previous versions of Entity Framework we already exposed a few lifecycle hooks. For instance, we had the AssociationChanged and ObjectStateManagerChanged events since the first version, and the ObjectMaterialized event was added in EF4. Up until EF6.x many of the existing hooks are not exposed in the DbContext API. In EF6 we also added several low level extensibility points in Interception that can be used too as lifecycle hooks.

There is a continuum of capabilities related and overlapping with lifecycle hooks, e.g.:

• We want to improve everyone’s ability to diagnose functional and performance issues with their code using Entity Framework by recording information about interesting events and conditions in the EF stack, but that is essentially logging (tracked in Logging: Design all up logging approach #218 and other work items). By comparison, lifecycle hooks allows to modify the outcome of these events.
• Also the DI-based architecture of EF Core allows for a very granular capability to wrap and replace individual implementation services, which can potentially be used to extend EF and to execute business logic. By comparison, lifecycle hooks is about adding simple and first class hooks that can easily be used to react to interesting conditions without having to re-implement the complete interface of a service.

Target customers

1. Application developers that need to implement business logic that is triggered at certain points in the lifecycle of an application, for instance, before an added or modified object is saved to the database, just before loading the contents of a navigation property, etc. Someone could add an OnValidate method to an entity that gets executed right before the entity is stored.
2. Framework/tool writers that need to extend the behavior of Entity Framework to support new application scenarios or to integrate it with other products or frameworks can write their own frameworks on top of EF that customize the behavior, collect data about the execution of specific actions, etc. Someone can write a full-fledged profiler taking advantage of the hooks.

Goals & Principles

1. Should work well and be consistent with DbContext design principles
2. Should have a consistent story on when and how it is allowed to re-enter: With a very small set of rules users should be able to predict what is allowed and what is not. The stack should be resilient to reentrancy in some cases and throw good exceptions if reentrancy occurs in other cases. We should prevent doing the same work more than once.
3. Should have minimal performance impact, especially when hooks are not being used the impact should be insignificant.

A brief survey of hooking mechanisms

There are not only different interesting conditions or actions an application may need to listen to, but also different kinds of mechanisms to implement hooks that present distinctive characteristics along the following dimensions:

1. Compile-time vs. run-time binding
2. Performance
3. Cancellability
4. Override/customization of default behavior
5. Single vs. multi-cast
6. Simplicity
7. Discoverability
8. Familiarity
   Note: We are looking for criteria that will help us choose the best type of hook for each extensibility point, as well as for the possibility of defining unified hook mechanisms that we can leverage to support different patterns with the same framework code.

.NET Events

Events are the most common hook pattern that almost every API in .NET uses. Events are messages sent by sender object to one or more receiver objects through a multicast delegate that acts as a dispatcher. Among the characteristics of events, they support runtime subscribe/unsubscribe, multiple listeners, and are relatively easy to use. Events can be slower than other hook mechanisms, but they have the advantage of being very discoverable (they are usually public members on the sender object) and familiar to customers. Events also provide a standard way to model cancellable actions with CancelEventArgs.

Virtual methods

Virtual methods require the application code to declare a derived type and override the method. Virtual methods provide better performance than events but are slower than regular method invocation. Virtual methods are easy to discover and provide a very nice model for overriding/customizing default behavior and chaining with subsequent derivate types. Visual Studio provides a nice Intellisense experience for virtual methods: when you write the overrides keyword in C#, Intellisense provides the list of all the virtual methods available.

Delegates

A more efficient alternative to events, regular (non-multicast) delegates can also be used as a hook. Users can normally provide some implementation of a predefined delegate signature (usually a Func<T…> or Action<T…> that can be implemented as a regular method, and anonymous method or a lambda expression) as a parameter to a framework method, as the return type from a method or as a property. Then the delegate is invoked by the framework at appropriate times. While delegates are often compared to strongly typed function pointers, they in fact are very flexible with regards to the signature (they support variance).

Partial methods

Partial methods were introduced in .NET 3.5 as a means to extend generated code in separate partial classes. Partial methods are void methods that are both defined and invoked in the right places in generated code. Users can choose to provide the implementation of partial methods in a separate partial class, and the compiler will resolve the partial method to the implementation provide by the user. When the implementation of a partial method is not provided, all calls to the method and its definition are removed by the compiler. Since partial methods are either turned into regular methods or removed, the mechanism is extremely efficient. Partial methods are discoverable because Visual Studio provides a nice Intellisense experience for them. Similar to virtual methods, once you write the keyword partial inside the partial class, Visual Studio editor will list all the partial method definitions that haven’t been implemented.

Magic methods

The concept of magic methods is that the user can write a method that follows a particular naming convention and signature, and a framework component will make sure the method will get invoked automatically at runtime. Usually an expression is compiled at runtime to produce a delegate that be used to invoke the method multiple times very efficiently. There is some runtime overhead in compiling the expression, but this is paid only once. Magic methods can be instance or static method. A common practice for magic methods, when code generation is involved, is to provide the declaration of the magic method as a partial method. Although no actual calls are generated, the partial method that gets an implementation will be compiled into the assembly so that it can be invoked at runtime. The only reason for this is the Intellisense experience you get.
Magic methods can be used to override default behaviors but it is necessary to expose a public method that implements the default behavior so that the user has the option to invoke this method from within the magic method if he doesn’t want to completely override it.
LINQ to SQL uses this mechanism pervasively.

Attributed methods

Similar to magic methods, an instance or static method with the right signature can decorated with a special attribute that specifies a runtime role for this method. The runtime examines types for the presence of these attributes and registers the method to be executed on the occurrence of certain conditions.
WCF Data Services uses this pattern pervasively.

Listener interfaces

This approach consists on defining a class that implements a custom interface provided by the framework and at a later point register an instance of this class as a listener for particular events. Usually, the interface defines one or more methods with a very specific purpose, but the listener class can be a composite of multiple interfaces. Discoverability can be improved in this programing model with a base listener interfaces and by placing all the related interfaces under the same namespace.

IObservable<T>

In many situations a mechanism is required to handle a stream of asynchronous events coming from the same source. IObservable<T> is an analog to IEnumerabe<T> that can be used to represent this kind of source.

Context hooks vs. entity and property hooks

DbContext provide a very good place for us to focus when defining extensibility hooks for anything that has to do with the functions that they encompass:

1. Store Connection
2. Launchpad for queries
3. Unit of work
4. Update adapter
   But for hooks that are specific to the lifecycle of entity or complex types, properties, etc., other options exist:
5. Factor business logic into the entity type itself: The typical example of this is the OnValidate method that LINQ to SQL supports: a user can define the method on a particular entity type, and have the framework will automatically invoke the method at appropriate times.
6. Extend the Code-First API to support hooks: The Code-First API provides a nice central configuration point for entity types, complex types and properties. We could extend the API with methods to register listeners, e.g.:

mb.Entity<Product>().Notify(new EntityLoadedListener());
mb.Entity<Customer>().Property(c => c.Orders).Notify(new PropertyLoadedListener());

The one potential issue with this approach is that the hook configuration would become part of the model and therefore it would not be possible to change it once the context object has been instantiated. This might be an acceptable limitation however, since any necessary changes in behavior can be coded into the listener class itself. Making the configuration of hooks immutable also has the advantage of allowing for compiling the invocations to listeners into efficient delegates.

EF requirements for a hooking mechanims

We are trying to find a design that has the following characteristics:

1. Works well for both app developers and framework developers.
2. Makes hooks very discoverable.
3. Invocation is efficient.
4. Requires only very simple coding patterns to use.
5. Should allow get the hooks at runtime and from a separate assembly.
6. Hook handlers can be implemented outside the entity types.
7. Hook handlers can be implemented as part of the entity types.
8. Ideally, the way you implement a hook handler in an entity shouldn’t break POCO.
9. The complexity of supporting multiple hook mechanism should be completely hidden from the sender.
   This requires further analysis, but it seems that it would be reasonable to support a mix of hook mechanisms (but not too many) to optimize for different scenarios.
   The current thinking is that we would use a single generic class to represent a hook. The sender should just need to call a method or instantiate a class and call a method, nothing much more complex than the typical “On[EventName]” pattern, and the hook class would make sure all the hook handlers are invoked.
   We are also considering building a convention system for wiring up hook handlers in the entity types.
   When designing this we should take advantage of any opportunity to reuse some hook mechanism as building blocks for others. For instance:
10. If we decide that the basic mechanism is listener interfaces, then we can have context objects implement those interfaces and re-cast the hooks as events.
11. We can have a single convention system that understands about naming patterns and attributes (similar to how the Code-First convention system understands naming patterns and data annotations as different vocabularies to express the same kind of concepts).
     

Lifecycle hooks list

The following is an incomplete list that presents various hooks we could consider adding. Intentionally the list does not try to be specific on each hook about certain details:

1. Hooking mechanisms
2. Method / argument signatures
3. All names of new hooks are up for discussion

Name	Pri	Location	Cancel or override	Description & sample scenario
QueryExecuting	0	DbContext	Yes	Query interception, custom query caching.
QueryExecuted	3	DbContext	No	When Execute happened, before the reader is read. Tracing?
QueryCompleted	3	DbContext	No	After DbDataReader is closed. Tracing?
EntityStateChanged	0	DbContext	No	Signals all state changes
EntityStateChanging	3	DbContext	?	Undo changes or change proposed values before they are set?
ConnectionProvisioning	2	DbContext	Yes	Execute additional code to make sure the connection is alive, or do logging
ConnectionReleasing	2	DbContext	Yes	Cleanup something done during Ensure / StartUsingConnection
ConnectionOpened	2		No	More likely for tracing. Since SqlClient has fixed invalid connection pools, then this is lower priority
ConnectionOpening	1	DbContext	Yes	Slightly simpler to use than Ensure/Start, would not require user to check current state. Could also be used for tracing.
ConnectionClosed	1		No
ConnectionClosing	1	DbContext	Yes	Slightly simpler to use than Release/Stop, would not require user to check the initial state. Could also be used for tracing.
OnModelCreating	0	DbContext	Yes	Tweak model before it is cached.
OnModelCreated	1	DbContext	Yes	Signal that the model is done and execute some custom code, possibly related to caching logic. . Issue: do we need this for ObjectContext? Issue: if the user is going to implement his own caching, we should have an abstract class or interface for that.
ModelCacheLookup	2	DbContext	Yes	Implement your own caching logic. Tracing?
ModelCacheHit	2	DbContext	Yes	Execute additional code when the model is found in the cache. Tracing?
EntityLoading	1	DbContext, DbEntityEntry	No	After object instance is created but before its properties are initialized. Can be used to reset a flag that will be set in newly created instances but shouldn’t be set during initialization, i.e. for validation.
EntityLoaded	0	DbContext, DbEntityEntry	No	Can be used to setup anything after an object has been materialized, i.e. event handlers, flags, etc.
CollectionLoading	1	DbContext, DbEntityEntry DbCollectiohnEntry	No	Can be used to setup anything on a collection after it is created but before it is populated. Issue: Could be used to provide your own collection?
CollectionLoading	1	Context, Entity or Collection	No	Can be used to setup anything on a collection after it has been created and populated, i.e. listeners for its changed event.
ObjectTypeResolving	1	Context	Yes	Could be used to specify a different type than the original one, i.e. to implement your own proxy mechanism. It should be per type but could return a Func<T> that returns a new instance and the result could be compiled into materialization delegates.
CollectionTypeResolving	1	Context	Yes	Something similar to ObjectTypeResolving but for collections. Could be used to replace the default collection type with a custom proxy collection with additional functionality (i.e. paging, fine grained lazy load).
Virtual OnSavingChanges	Medium	DbContext	No	Can be used to re-implement SaveChanges but still invoke the existing SavingChanges event
SavedChanges	Low	Context	No	Could be used to execute cleanup code after SaveChanges. For instance, to call AcceptChanges on each STE change tracker. It is lower priority because virtual SaveChanges covers most scenarios.
EntityStateChanging	Low	Context, Entity	Yes	For an entity instance or type in particular we could avoid putting in the modified state. So even if the properties are read-write, the context ignores changes to this entity. Could be also used to suspend fixup on an entity that is being detached.
EntityStateChanged	High	Context, Entity	No	Executes logic after an entity has been put in a certain state. Can be used to setup property values, restore state after the changing event.
PropertyChanging	Low	Context, Entity	Yes	Any time a property is about to be changed by the framework or any party, if notification or interception is enabled by the entity type. Should make original and new value available. Should also work for navigation, scalar and complex types properties. Tracing?
PropertyChanged	High	Context, Entity	No	Any time a change in a property value change is detected.
PropertyLoading	High	Context, Entity, Collection	Yes	Intercepts, overrides de loading of a property. Could be used to support loading of properties using stored procedures.
PropertyLoaded	Medium	Context, Entity, Collection	No	Tracing?
Writetable IsLoaded	High	Context, Entity	Yes	Allows cancelling the loading of a property.
CollectionChanging	Medium	Context	Yes	Any time a collection is about to be changed by the framework or any party, if interception is enabled
CollectionChanged	Medium	Context, Entity	No	Any time a change to a collection has been detected.
AssociationChanging	Low	Context, RelatedEnd	Yes	Can be used to prevent an association from being changed, or to execute business logic when the association is about to change.
AssociationChanged	Medium	Context, RelatedEnd	No	Can be used to execute additional logic after an association is changed, i.e. user can explicitly cascade relationships removals into dependent removals, workaround current databinding shortcomings.
RowValidate , RowValidateAdded, RowValidateModified, RowValidateDeleted	Medium	Context	Yes	Storage level version of ObjectValidate. Tracing?
SavingChanges event	0	DbContext	?	Currently only available on ObjectContext. Should make trigger OnSavingChanges method protected.

Existing hooks

Name	Description & sample scenario
virtual Dispose	This can be used to do additional cleanup, i.e. on entity instances.
virtual SaveChanges	Can be used to execute additional logic before, after or instead of saving changes.

Some open issues:

1. Need to prototype some coding patterns and try them.
2. Is logging and tracing part of this API? It seems that ideally we should have the same level of flexibility for hooking mechanisms with logging and tracing as we end up having with this API.
3. Second level cache should probably expose its hooks through the same mechanisms.
4. Should we provide low level query interception points with the same mechanisms, i.e. as command tress and store commands? Should we do the same for CUD store commands? Would need to make sure those work well with caching.
5. Can we get some level of support for async execution of queries with this hook?
6. Should we provide enough lifecycle hooks to implement custom fixup logic?
7. Areas of overlap with other extensibilities: read and write properties in object mapping, proxy type creation (can be imperative vs. event driven), equality and snapshot comparisons for change tracking extensibility.
8. What about customizing identity resolution?
9. Is Logging and Tracing part of the lifecycle hooks
10. Is Query interception part of the lifecycle hooks
11. Even without query interception we should expose when we are about to execute (imagine a profiling tool that measures how much query compilation costs).
12. Is ContinueOnConflict part of the lifecycle hooks
13. How do we improve diagnostics? Can we have OnError
14. Need to do prioritization, costing and scoping
15. Should we have a fine grained version of CollectionAdding / CollectionRemoving with support for magic methods on the entities to enable collection patterns? We would need a pattern for Contains checks also.
16. There is a conversation about splitting AssociationChanged this event into properties and collection changes. However, there should be a way to tell the difference between a scalar property change and a nav prop. Should we make AssociationChanged more accessible and add AssociationChanging? This would provide a way to intercept changes in associations independently of cardinality, constraints, etc.
17. AssociationChanging would need the entity and collection types to collaborate to avoid changes from being made to the graph.

[natemcmaster]

I started prototyping some ideas as a part of my investigation of setting SQLite pragma options when a connection opens. It seems this feature would be generally useful in solving several other feature requests. Can we considering moving this off backlog?

[rowanmiller]

I think we need @divega in the office to design this one, the all up feature is pretty large and wide-reaching. We also have some higher priority things to work on first. If you are specifically looking at connection pre-amble then I think we can treat that as a smaller feature that we work on now.

[janhartmann]

Will this make it into the next release?

[Antaris]

@rowanmiller In terms of an EF Core ObjectMaterialized style hook, can you give me some pointers as to where I can tackle this as of now?

[divega]

Permission validation and filtering are other scenarios that could be attained with lifecycle hooks (see #6440).

[dazinator]

Will this also take care of the following scenario?

I'd like to add an entity to a DbContext (to be inserted or updated etc), but also subscribe to be notified once the entity has been saved, so i can handle taking its newly updated values (i.e the database generated values that get poplated after a SaveChanges() - like it's ID etc) and do something with them.

                var dbContext = GetExistingDbContextInstance();           
                var newItem = new SomeEntity();
                dbContext.SomeEntities.Add(newItem);
               // I am quite far down in an object graph, and SaveChanges() will eventually be called
               // later on by something higher up, co-ordinating this transaction. I'd like to be notified
              // here though once SaveChanges() has been called and this entity has been persisted, so I can grab the
              // the updated entity values.
              //   dbContext.OnceSaved(newItem, ()=>{   // ooh someEntity.Id is now populated! })  

Something along the lines of:

 dbContext.OnceSaved(someEntity, ()=>{   // ooh someEntity.Id is now populated! })

Where the callback would be invoked after dbContext.SaveChanges() is called.

[Antaris]

@dazinator I'm currently working on an extension project for EFCore (https://github.com/Antaris/EntityFrameworkCoreExtensions) which will allow you to do what you want, eventually.

I have a number of working hooks including change tracking, value materialization, querying and storage. It's still a WIP though.

[SidShetye]

@rowanmiller What's the expected release date on this? It's been open for almost 2-1/2 years

[Evengard]

Is there a OnModelCreating override implemented, or smth along this issue: #9330 ?

[ajcvickers]

@Evengard I opened #31206.

[xaviergxf]

is there any interceptor available that allow audit feature when using ExecuteUpdate, ExecuteDelete?

[roji]

@xaviergxf command interceptors should work for ExecuteUpdate/Delete just like for any other EF database operation.