Improve documentation on how to write an EF Core provider

[divega]

Currently we have something very brief at https://docs.microsoft.com/ef/core/providers/writing-a-provider pointing to @ajcvickers's blog.

It would be good to update this to cover:

• Support for EF Core 2.1 features
• The mechanics of setting up spec tests (e.g. https://github.com/dotnet/efcore/issues/27107#issuecomment-1004923695)
• The mechanics of adding provider specific extension methods

Whether this should be in a blog post or if it needs to be part of the documentation is something we can discuss, but I would be inclined to have it in the actual documentation.

[omatrot]

I'd like to write a scaffolding only provider around edmx files in order to fill the gap for all of us having existing edmx files that we want to keep around in order to create entities and DbContexts.

Any information that could allow me to start would be very much appreciated. May be this would be an interesting contribution to the docs.

[ajcvickers]

Suggestion from @ErikEJ on dotnet/efcore#20793 (comment)

It would lower the barrier to have a "template/null" provider to use as a starting point. But of course it can quickly become stale.

[roji]

Ideally this would be built as part of CI, to keep it in at least minimal sync with EF...

[ErikEJ]

@roji do you think there would be any value in this?

[roji]

Yeah, I think it would... Our docs repo has a set of samples - which now even get built in CI - so it could live there. On the other hand, if we put it in the normal repo it would naturally evolve as we change EF Core...

[ErikEJ]

I think it is better to fix it to depend on a released version of EF Core, and then just update on new release (if needed at all)

[smitpatel]

Releases to consider

• LTS release
• Current release (stable)
• Current preview

I think current preview would provide slightly lesser value. If I am writing a provider from scratch, I would want to depend on stable release so I can build my provider and release it. It would be easy after 3.1 to move to next major release for providers even with breaking changes. The issue with targeting previews would be constantly moving target. If EF Core added some API in one preview but changed it in another preview (we love API reviews), then provider has unnecessary effort to keep changing things when writing the core provider at the same time.

[ErikEJ]

@smitpatel exactly my experience/pain during EF 7 previews

[roji]

I think we should be much more stable now than we used to be - but I agree that there's not that much value to a provider template that tracks previews. So maybe just tracking the current stable release is enough (an additional one for LTS seems to be a bit overkill)? If so, then just putting it in our doc repo under samples should be good...

On another note, I assume we're discussing about a relational provider template - or do we think that a non-relational sample is valuable as well?

Are you interested in doing this @ErikEJ?

[ErikEJ]

@roji yes, the relational. I could just base it on something similar to the initial commit I mentioned above. A no-name ANSI SQL provider...

[dellams]

Are there any plans to have the docs updated for EFCore5?

The blog it points to is for EFCore1.1! Is the doc still valid? I cannot find DatabaseProviderServices?

I searched in the linked example providers such as SQL Server, npsql, etc but could not find DatabaseProviderServices or RelationalDatabaseProviderServices anywhere in them...

Any ideas? I really need to write a provider but haven't a clue how to or where to start?

Any help would be really appreciated.

Many thanks,
David.

[roji]

@dellams we're indeed lacking good documentation on how to write a provider... Assuming you're looking to develop a provider for a relational database, it's recommended you start with the Npgsql or Pomelo MySQL providers, as these are external projects that aren't part of the EF Core repo. Where relevant, the other providers (Sqlite, SQL Server) can also be consulted.

Start by looking at the different services implemented by these providers, and getting familiar with the general architecture of EF Core. We can then help out with any specific questions.

[dellams]

Yes, thank you, I have already checked them out and was surprised with just how much work and how complex it was to write a provider. I was hoping you could provide a simple bare bones template with the minimum needed to implement a provider?

Do you know when some up to date documentation will be created for this?

What I am trying to do is something a bit more special than just a relational provider, I am attempting to integrate EFCore into the OASIS API I have been building, which integrates all blockchains, holochain, cloud, db's, networks, platforms, APIs, etc. It is like a network of networks, an abstraction layer over the entire internet.

I would love to get some support with this very valuable project, I feel once people understood what I am building a LOT would come and help me build it! :)

https://github.com/NextGenSoftwareUK/Our-World-OASIS-API-HoloNET-HoloUnity-And-.NET-HDK

It would be super cool and powerful to have this abstraction API using EFCore making it super easy to bridge everything to everything through one simple ORM (I love EF and have worked with it many years being a big fan of .NET since its initial release many years ago just after I graduated from University in 2002.) :)

I have worked as a IT consultant/contractor in London building commercial .NET applications of all sizes in all markets, etc so have experience of most of the amazing .NET stack since I have grown as a developer along with .NET... you could say we grew up together! ;-)

BIG FAN! :) Keep up the great work, so happy it is Open Source now and cross platform,... very excited and passionate to help grow the .NET ecosystem with you guys...

I hope to get on the .NET Foundation soon... is that something you guys could help with? When you all understand the wonders I plan to do for the .NET community I think you would want to help me help you... ;-)

Many thanks,
David.

[ErikEJ]

@dellams you may find the series of commits in the develop branch here useful: https://github.com/GibraltarSoftware/VistaDB.EFCore

[dellams]

Great thank you. Is this an example of the steps bones minimum needed? 🙂

[ErikEJ]

It depends...

[ElanHasson]

@dellams sounds like you're describing F# Type Providers

[mguinness]

One of the providers that is stalled out is FileContextCore. The fact that there are so many breaking changes between major versions means that ongoing maintenance of providers is very labor intensive especially when there is no documentation.

[DaveRMaltby]

I have a need to create many EF Core providers and decided to create a template repo for this purpose. It is still a work in progress (i.e. need to get passing unit tests going), but am wondering if this repo would be something that the community would be interested in? And also if it might go a ways toward solving this open issue?

The readme describes the concept of this sample provider.

[roji]

@DaveRMaltby I do think we skeleton/template provider is a good thing to have, as a starting point for people writing providers.

Some specific points come to mind when looking at your repo; in particular, many services are being overridden that would only be needed in relatively advanced scenarios (e.g. SqlExpressionFactory only for custom SQL expressions, QueryTranslationPostprocessorFactory only for certain advanced scenarios, etc.). I'd advise going with a more minimum approach, implementing only services which a typical provider would have to override, rather than everything.

[DaveRMaltby]

@DaveRMaltby I do think we skeleton/template provider is a good thing to have, as a starting point for people writing providers.

Some specific points come to mind when looking at your repo; in particular, many services are being overridden that would only be needed in relatively advanced scenarios (e.g. SqlExpressionFactory only for custom SQL expressions, QueryTranslationPostprocessorFactory only for certain advanced scenarios, etc.). I'd advise going with a more minimum approach, implementing only services which a typical provider would have to override, rather than everything.

@roji, thank you for the feedback. Yes, your advice points to the issue that I'm struggling with and I'm sure others who have no idea about the nature of the services offered here. I just used the Npgsql, Pomelo MySQL and SQLite providers as a basis to copy from for this template repo without understanding their purposes. I was trying to get at least one of the tests in EFCore.Relational.Specification.Tests to pass. So far, I have been unsucessful in that endevour and it has been overwhelming as just simply adding the EFCore.SampleProvider.FunctionalTests.Query tests amounts to 443 tests.

Anyhow, I believe now that I need to take a completely different approach. I've now stepped back and am going to take one of the many EF Core providers that I plan to write and create a few unit tests based off of the examples in Getting Started with EF Core. Getting those CRUD tests to pass will give me the understanding and basis for the bare minimum services that are needed. Later, I will add more advanced features to this provider as needed (eventually, adding the migrations/scaffolding that I also need). Ultimately, I'll try to circle around and update the SampleProvider to provide a skeleton/template for others to make use. (And also I hope to provide descriptions about the services as well)

Can you possibly provide a quick list of services that you know are required just to achieve the first step mentioned above of getting simple CRUD unit tests to pass? Thank you.

[ErikEJ]

@DaveRMaltby Maybe some of the initial commits here could help? GibraltarSoftware/VistaDB.EFCore@main...develop

[DaveRMaltby]

Thank you for the lead. I'll take a look on it soon.

[roji]

@DaveRMaltby it's unfortunate that we don't have better docs for provider writers - it really is something we want to improve, but tends to be de-prioritized in favor of user-facing features. However, we're here to help you out with any questions - please don't hesitate to ask.

[DaveRMaltby]

@DaveRMaltby it's unfortunate that we don't have better docs for provider writers - it really is something we want to improve, but tends to be de-prioritized in favor of user-facing features. However, we're here to help you out with any questions - please don't hesitate to ask.

@ErikEJ and @roji, other priorities took me away from this topic. I'm trying to focus again here on providing a useful template repo for the community. I don't like the direction that I was initially going there. I'm now thinking that such a template repo should just be able to take any full fledged ADO.NET data provider (that supports .NET Core, of course) as a basis without needing much else. Is this a naive statement? I will likely try to go in this direction, as it may allow most of the specific database logic for anyone who needs to create an EF Core provider to be placed in their own ADO.NET data provider and then use this template repo to get most of the way there very quickly.

[mguinness]

You could also take a look at FileBaseContext which is probably one of the simpler providers out there. Maybe @dualbios can provide some tips on creating a provider from scratch.

[roji]

@DaveRMaltby I'm not quite sure what you mean with "may allow most of the specific database logic for anyone who needs to create an EF Core provider to be placed in their own ADO.NET data provider and then use this template repo to get most of the way there very quickly"; there's typically various database-specific logic that has to go into each EF provider, which does not belong in the lower-level ADO.NET provider (e.g. how to produce various SQL constructs). The idea of "general ADO.NET-compatible EF provider" which would somehow just work with any ADO.NET provider simply doesn't work.

I think the main value in a template here would be to have a minimal provider sample which implements the services that must be overridden (or would typically be overridden for most providers). This would only provide a starting point for new providers, or for understanding how a provider looks like.

Note also that a minimal relational (ADO.NET-based) EF provider looks very different from a non-relational provider; for example, a relational EF provider typically needs to have its version of QuerySqlGenerator, which non-relational provider obviously don't need. I agree it's worth concentrating on a relational template; non-relational providers in general can be extremely different based on the database being targeted, and there's not likely to be a lot of commonality there.

[DaveRMaltby]

@roji and @ErikEJ, I didn't forget about this. I have finally completed what I was referring to above. I have taken and built a custom EF Core provider that depends on a SQLite ADO.NET provider. Of course, I knew that there would be some custom code that is unique to support the SQLite provider, but it doesn't seem that much as you can see in MyCustom EF Core provider repo.
See what yáll think. Look at the README for this repo. I've tried to logically break up what is needed into 4 steps. The basic unit tests are passing for me.
Note: It builds against EF Core 6.x, but I only did that because currently my product is using that version. If/when you want to add this into an EF Core repo, of course we'll rev it up to the latest version.
Anyhow, let me know if you want more of my involvement here. For my company's use, I have 4 providers to implement and will be working on those efforts, unless you need something further here. Thanks for the guidance!

[JohnGoldInc]

So is it a correct statement that the critical part is working outwards from the ...ServiceCollectionExtensions classes' "IServiceCollection AddEntityFramework..." section, and create your versions of those classes listed there as needed? And that InMemoryServiceCollectionExtensions.cs or CosmosServiceCollectionExtensions.cs is the version probably close to the minimum needed to implement?

[DaveRMaltby]

So is it a correct statement that the critical part is working outwards from the ...ServiceCollectionExtensions classes' "IServiceCollection AddEntityFramework..." section, and create your versions of those classes listed there as needed? And that InMemoryServiceCollectionExtensions.cs or CosmosServiceCollectionExtensions.cs is the version probably close to the minimum needed to implement?

Honestly, I'm still not an expert and got a lot of guidance from ChatGPT. I asked it to indicate the minimal classes required. Anyhow, the ServiceCollectionExtensions class does register your provider services that the Entity Framework will use via DI for itself. It is only your DbContext derived classes (BloggingContext in my unit tests) that end up kicking off the framework to do things using your provider registered services. Anyhow, this was the minimal classes that I found was need to get the GettingStartedTests running with the SQLite in-memory ADO.NET provider. Note: This hasn't been reviewed by @roji and @ErikEJ as far as know, so this example may be missing things.