Cosmos DB provider

[grahamehorner]

Note: This issue is for discussion and questions about the EF Core Cosmos provider.

Issue #12086 is tracking the work being done on the implementation.

---

Please consider creating entity framework core providers for Cosmos DB.

[divega]

We should consider investing on this. Keeping it in the backlog for now because we don't have short term plans for it.

[divega]

We are going to start investigating this in 2.1. We have a lot of work to do to figure out what the goals should be for EF Core support for Cosmos DB in the short and long term, e.g. from answering basic questions like what value EF Core support should bring to the table to which exact APIs of Cosmos DB we should support.

[MisinformedDNA]

This issue encompasses #1142, so I think #1142 can be marked as a duplicate.

[grahamehorner]

I would love to see support doe take, skip foe paging as at present Cosmos doesn't server side support paging as I understand it

[divega]

@MisinformedDNA from our initial discussions with Cosmos DB, we are not likely going to target the Table Storage API. We could close #1142 because we are not doing it, but it is not necessarily encompassed.

[DavidBurela]

Now that EF core 2.0 has been released. Hopefully this can get considered for 2.1 now <3
Been looking forward to it since 2014 when the original #1035 was created

[MisinformedDNA]

@divega Cosmos DB is essentially four different APIs to do different things. I would expect either a single Cosmos DB provider to handle all four APIs (seemingly this issue) or four separate Cosmos DB providers, one for each of Cosmos DB's APIs. The latter seems more reusable, composable and maintainable.

[satyajit-behera]

This will be an interesting feature. Just curious, any planned date for 2.1 release of EF? Will be helpful for new projects with more interesting use cases.

[ErikEJ]

@satyajit-behera The EF Core roadmap follows the ASP.NET Core roadmap, which you can see here: https://github.com/aspnet/Home/wiki/Roadmap (or maybe the .NET Core roadmap??? https://github.com/dotnet/core/blob/master/roadmap.md )

[VagueGit]

Neither of those roadmaps acknowledges the existence of Cosmos DB. Does this mean there is no plan for EF to support Cosmos DB?

[JonPSmith]

A question. Will you be building a default ExecutionStrategy for the cosmos provider in a similar way to SQL Server's SqlServerRetryingExecutionStrategy or do we need to write that ourselves?

[ErikEJ]

@JonPSmith I think it is already built into the Cosmos SDK - but of course it would be nice to be able to configure the options https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.client.connectionpolicy.retryoptions?view=azure-dotnet

[JonPSmith]

Thanks @ErikEJ,

I had a look at the actual code and there is a CosmosExecutionStrategy. so the answer to my question, yes there is a ExecutionStrategy for the cosmos provider but there isn't an extension like SQL Server has. Here is how you add the strategy.

var builder = new DbContextOptionsBuilder<NoSqlDbContext>()
    .UseCosmos(
        "urltocosmos",
        "somelongkey",
        "databaseName",
        options => options.ExecutionStrategy(c => new CosmosExecutionStrategy(c)));

[AndriySvyryd]

@JonPSmith Cosmos will use a retrying execution strategy by default, so the user doesn't need to specify one unless they need to change the way it behaves.

[JonPSmith]

@AndriySvyryd. Thanks for letting me know about that as I am writing an article on EF Core and Cosmos DB and was going to put the ExecutionStrategy on my example

PS. It makes sense that retries on with Cosmos as it doesn't support transactions, so there is no down side by having retries on.

[Mortana89]

@AndriySvyryd What about retries for 412? Will we support ETag concurrency in the cosmos DB provider?
I'm trying to see how I can implement it myself, but I'm a bit pestered with the fact that you could have partial saved state in the db (if, e.g. 3 changes are sent to the DB, but the 3th got a 412 due to different ETag).
Without worrying about that case, would it be a good idea to throw a dbconcurrency exception from the cosmosclient, when we received a 412? And then catch this in the callers code by using something like Polly? I would then only need to add a shadow prop for the ETag value, but it would be a begin to implement concurrency mechanisms. What are your thoughts?

[AndriySvyryd]

@Mortana89 We'll likely throw a DbConcurrencyException, but it would be up to the caller to decide how it should be handled - like resending updates for all entity types overriding any changes or querying the updated data, determining which ETags are mismatched and merging the changes.

[Bassist067]

Hi everyone! I'm using the CosmosDB EF Core Provider 2.2.4 and here's the strange issue - EF Core seems to ignore the [JsonProperty(NullValueHandling = NullValueHandling.Ignore)] attribute when I try to write null in Cosmos.

So the next time I try to deserialize it, I've got Unable to cast object of type 'Newtonsoft.Json.Linq.JValue' to type 'Newtonsoft.Json.Linq.JObject'. It seems like when I deserialize the null value to some class model, it's crashing.

So the questions -

• how can I force EF Core to ignore nulls when I write into Cosmos?

• how can I deserialize object with one of the properties set to null?

[AndriySvyryd]

@Bassist067 The Cosmos provider currently doesn't handle any attributes. Properties of nullable type generally accept nulls by default.
You are using an early preview version of the provider, please try 3.0.0-preview5 and if you are still running into this open a new issue.

[AndriySvyryd]

All the feature work planned for 3.0 has been completed.
Open new issues for any further questions.

[githubavi]

Hi, Any plan to support EF Core for Cosmos Db Gremlin Api?

[ajcvickers]

@githubavi Currently no plans.

[richardrobberse]

EDIT: apparently the ETag concurrency token was implemented recently by this PR: #19581
@jviau @AndriySvyryd can you guys shine some light on how to effectively use the ETag concurrency token? Thanks in advance!

Few questions that popped-up after creating a simple CRUD REST API with CosmosDB:

• How to implement concurrency handling with ETags?
  Does the provider take the ETag on the model into account during updates or should I build the ETag comparison myself?

• I've added the _etag to my model and it is returned when querying for entities, so far so good.
  When I, for example, add a new record through the API the _etag property is not set on the object after calling Add() and SaveChanges(). I kind of expect this to work the same way as it works with the Id property.
  Is there a way to get the ETag when adding a new record without retrieving the same record again?

[JohnGalt1717]

@richardrobberse CosmosDB provider isn't even remotely close for being used in a realworld use case. Maybe .net 5.

[richardrobberse]

@richardrobberse CosmosDB provider isn't even remotely close for being used in a realworld use case. Maybe .net 5.

That's quite a statement

[JohnGalt1717]

@richardrobberse Look at all of the basics it doesn't support (try sub arrays of objects the CosmosDb way, I dare you) and tell me that I'm wrong. You can't take any marginally complex CosmosDb database with multiple collections and make it work in EF Core 3.1 as it stands today. The basics (like concurrency) aren't there.

[richardrobberse]

@JohnGalt1717 I'm not in need of anything complex, I just need a simple CRUD API so I think I'll manage once the concurrency gets added (hopefully in 3.1.2)
If you have other suggestions, please let me know!

Have you tried to use the CosmosDB SDK? There's not much documentation on it but I managed to get it to work at some point, I wasn't satisfied with my implementation so I kind of moved away from it.

[JohnGalt1717]

@richardrobberse Just use direct CosmosDb table api with linq until they get it working right and fully support child lists. (i.e. right now you can't even do Contact.Addresses properly with it, so your basic CRUD will probably fail too)

[jviau]

EDIT: apparently the ETag concurrency token was implemented recently by this PR: #19581 @jviau @AndriySvyryd can you guys shine some light on how to effectively use the ETag concurrency token? Thanks in advance!

Few questions that popped-up after creating a simple CRUD REST API with CosmosDB:

• How to implement concurrency handling with ETags?
  Does the provider take the ETag on the model into account during updates or should I build the ETag comparison myself?
• I've added the _etag to my model and it is returned when querying for entities, so far so good.
  When I, for example, add a new record through the API the _etag property is not set on the object after calling Add() and SaveChanges(). I kind of expect this to work the same way as it works with the Id property.
  Is there a way to get the ETag when adding a new record without retrieving the same record again?

When it is released, Cosmos ETag concurrency will be handled automatically when a property, shadow or concrete, is mapped to the JSON property _etag and is marked as a concurrency token. Anything mapped to "_etag" must be either string, or have a string value converter. There are helpers for this:

entityBuilder.UseEtagConcurrency(); // adds a shadow property for "_etag" as a concurrency token
entityBuilder.Property(x => x.MyProperty).SetEtagConcurrency(); // maps to "_etag" json property, sets concurrency token

There is no convention for automatically detecting properties named "etag" or anything similar and making it the concurrency token, it needs to be set explicitly in the fluent API for now.

When an entity has an etag concurrency token, it will automatically be included in the ItemRequestOptions.IfMatch for all applicable requests to cosmos (insert, replace, delete), so you will start seeing DbUpdateConcurrencyException on conflicts during SaveChanges.