Support for "dynamic" entity models

[divega]

The goal of this issue is to track and rationalize various levels or flavors of "dynamic behavior" we could attain with EF in the future.

The initial list is based on digesting information provided in various bugs such as #1513 and #2141 as well as conversations with various teams and previous research on feature gaps with other persistence frameworks.

Note that none of this is planned for EF Core 1.0.0 and before we start bringing dynamic features into EF we should probably open more issues and prioritize each individual feature independently. But at least for now should any new "dynamic" scenario be brought to our attention by the community or should our understanding of them improve we can just keep this list up to date:

• Shadow objects: (already tracked individually as Fully implement entire entities in shadow state #749) This is about having certain entities or complex types defined in the model that do not get materialized as user objects but are just tracked as "state" in the DbContext instance. This is useful for such scenarios as link tables in many-to-many associations or any other situation in which there is an entire entity that contains information that EF needs to track but that the user has chosen not to include in the object model.
• Property bags: Currently the EF Core stack assumes a distinct CLR type is mapped to each distinct entity type (and eventually for complex types) in the entity model and a CLR property for each entity property (minus any shadow property). This could be changed to enable EF to materialize and track simple property bags (e.g. any object that implements IDictionary<string, object> or contains an indexer property or any other mechanism to get and set values based on a property name). Even the name of the entity instance belongs to could become another property in the bag. This feature would make working with entity models that are created programmatically easier because it would remove the need to emit new CLR types for each entity and complex type at runtime. (Shared-type entities (part of property bag entities) #9914)
• Dynamic aggregates: Allow the entity type to be determined by the set of contained components (Support dynamic aggregates (Entity–component–system) #26123)
• Proper dynamic objects: This refers specifically to supporting entities and complex types in which each specific instance can contain a different set of properties. I anticipate that the most likely path for us to support proper dynamic objects mapped to relational databases could be to serialize the dynamic part of the object into a BLOB or JSON column instead of shredding it into the individual scalar properties. Querying based on standard properties can be enabled and optimized through computed columns and indexing as described in the comments of Native JSON support #2141. Even within this category there are several alternative levels of support which will probably pose different challenges:
  1. Only scalar properties can differ vs. relationships can also differ
  2. Whole dynamic objects vs. "open types" (i.e. dynamic properties are added at runtime to a set of predefined properties).
  3. Property bag representation vs. actual dynamic/ExpandoObject CLR types (the latter can lead to nicer syntax but might be impossible to use in LINQ).

[divega]

Moving this straight to the backlog.

[AcousticGuitar]

Hello, just a thought on this, instead of DB first there could be a JSON or even XML first equivalent.

[natemcmaster]

This would help with materializing Annotatable objects. Currently we can get close by using shadow properties and then doing a post-materialization fixup.

var post = context.Posts.First();
var entry = context.Entry(post);
var isDraft = entry.Property("IsDraft").CurrentValue as bool;
post.AddAnnotation("IsDraft", isDraft);

Of course this has caveats: AFAIK the only way to get a shadow property value after materialization is to use change tracking. This means to AsNoTracking() cannot be used if the shadow property is required outside of query.

[dmitry-a-morozov]

Are there plans to make a progress on this feature request?
This will enable implementation of F# erased type providers for EF Core.

[ajcvickers]

Triage: Consider scenario described here when tackling this issue: #10555

[weitzhandler]

Dynamic functionality is very essential in almost any growing app, especially LoB apps or apps that grow quickly and dynamically.

Let me just demonstrate with a couple of examples using a Contact class.

• A list of values or tuples, for example a list of phones, where each phone has a label (Home, Fax, or whatever), so we should be able to search using EF:

var number = CleanNumber(number); //remove formatting
dbContext.Contacts
  .Where(c => 
    c.Phones.Where(p => p.Label = "Work" && p.Number.Contains(number));

Remember that we don't want to predefine the phone shape in the DB (only in the EF back-end).

• Furthermore, a list of dynamic types. For instance I don't want to define the fields in my Contact entity at all, instead I need them to be all dynamic, for example, I'd sometimes want Contact to have a person name, whereas sometimes a company name, in many other scenarios the user would want to add custom fields.
  If it's supposed to be a dynamic app so that the user is the one that defines which columns should be there, after the data is stored in the DB, we still obviously still want to let the user search all contacts that have a company name x without having any of this defined in the server.
  We might want to store schema data externally, but that's already not up to the DB but to the business layer.
  From EF's point of view, we should be able when configuring the model, to define some fields, collections, or entities, as dynamic data, which feel as natural columns from the front-ends point of view, but will be transformed to a dynamic means when stored to the DB.

Let's take Microsoft SQL Server as our DB provider. It provides out the box support for that using JSON/XML
datatype. We could transform all the data fields that we defined as dynamic to be stored and manipulated as JSON/XML.

[smitpatel]

@weitzhandler - I was reading this thread. While dynamic entity types has a wide scope, I believe the scenario you posted could possibly work in current system already.
Tuple/list may be slightly difficult, but if you are using Dictionary/Array and using JSON on server side to store, you could value converter to store your data. And for querying part use DbFunction to access data on server and get queries translated. It also covers when you don't have LabelNames defined and you want to use dynamic "key".

[weitzhandler]

In the future, this (especially the queryability) has to be implemented in the system with no need for DbFunctions.
We will have to distinguish between 'converted' fields and 'dynamic' fields. The difference is that converted fields is uncontrolled, and queryability is not expected from it in the first place. Whereas 'dynamic' fields, are such that the EF is aware of them being dynamic, hence takes care of its conversion/transformation/serialization (JSON/XML etc.), and have the internal query providers generate a different query if the columns in question are of JSON/XML.

[smitpatel]

@weitzhandler - Of course, functions related to JSON support on query side would be implemented in #4021 XML could achieve the similar. This also have some overlap with property bag entities. Regardless, most of work here would be about changing the metadata for representation. Query side comes into picture depending on how it is stored on server & what we can translate.

[weitzhandler]

Please check out the Impatient repo, which relates here.
Read the issue I posted there.

[ajcvickers]

Note from triage: remaining work here tracked by related issues.