Azure Mobile Apps v8.0 #802
Replies: 10 comments 20 replies
-
|
EF Core on client side is very interesting for standard LINQ for SQLite and customizable sync (override SaveChanges/Async) as you said |
Beta Was this translation helpful? Give feedback.
-
|
Yes - I'll be switching over to System.Text.Json. I've just been waiting for the OData library to support System.Text.Json before I did the switch. Since it's a breaking change, it had to be done within a major version change as well, so this is the right time to do the swap. |
Beta Was this translation helpful? Give feedback.
-
|
Using EF Core should also simplify cases such as #804 |
Beta Was this translation helpful? Give feedback.
-
|
I'm excited about this potential direction as it sounds like it could address many of the things we are already trying to do through workarounds to the current limitations. How significant do you expect the changes to be to the current API surface (both client and server)? We are currently dealing with one-to-many and many-to-many relationships between synchronized tables and one of the challenges is synchronizing insertions vs synchronizing deletions in correct orders. Right now, where the master is centralized at the server, and client db's aren't also "masters" the concept of "hard" deletions on the client make it harder (but not impossible) to enforce foreign key constraints across tables during synchronization. Curious how you're thinking of handling this?
This makes me think that the client database will at least have some historical state data, at least as far as deletion states are concerned (i.e. current state only, but no history). Or are you planning on more than just that? Would there be the potential to have the option of having a historical ledger to support multi master merging or version rollbacks? (i.e. https://github.com/vlcn-io/cr-sqlite#how-does-it-work) Also going to EF Core opens options to support nested JSON queries: dotnet/efcore#28816. Would you still support the same client query features locally ( |
Beta Was this translation helpful? Give feedback.
-
|
The changes to the server are light - pretty much the same API surface, but I'm switching over to System.Text.Json and OData v8. The changes to the client will be significant. There won't be an IOfflineTable / IRemoteTable any more. Instead, there will be an OfflineDbContext that you will inherit from to maintain the offline queue state The offline queue provides the capability of rolling back a state change before the push to the server happens. I'm not expecting to support multi-master merging or version rollbacks. If you want that, cr-sqlite is probably a better fit. I'm not interested in duplicating that sort of functionality. The developer will write EF Core queries, so the capabilities are independent between offline and remote. Remote is all about identifying the slice of data that the user wants to make available offline (the slice of data the user is allowed to see if provided at the server by the access control provider). I expect I'll keep the ability to download data from the server (and submit data to the server) with the restricted LINQ capability. You can also use any OData client you want since the server code does expose the CSDL for the OData EDM Model. |
Beta Was this translation helpful? Give feedback.
-
|
I agree with the support of hybrid functionality between Remote and Offline tables. I have MAUI app with two different versions:
It would be neat if we can still have the relationship defined between them for querying purposes and keep the same API in between Remote and Offline version. |
Beta Was this translation helpful? Give feedback.
-
|
This is an interesting case. Unfortunately, the proposed architecture won't fix that. Mobile versions will be able to acquire client-driven subsets of the data, but the remote interface will be completely different since it's DbContext vs. RemoteTable - different setup and different sets of LINQ compatibility. |
Beta Was this translation helpful? Give feedback.
-
|
This is unfortunate because it means we will have to set up additional RPC like endpoints which reduces code reusability. If you have a Blazor client that is based around the IRemoteTable abstraction and mobile apps which are using the IOfflineTable abstraction, currently they can share the majority of code because of the API match. How would we accomplish this sort of design with the proposed architecture? Blazor server may be able to use DbContext access in Entity framework backend which might share similar concepts, but Blazor client (or any SPA client over REST) would require a REST API that now has to be developed independently. |
Beta Was this translation helpful? Give feedback.
-
|
I'm wondering what the impact would be of an EF Core query driver for IndexedDb. The API compatibility is really nice, but it comes with some major gotchas around joins and relationships. It also requires us to re-implement EF Core query drivers to support. I'd rather approach this from a "use the regular stuff" kind of architecture rather than "re-invent everything to support API compatibility" That being said, v6 isn't going anywhere. |
Beta Was this translation helpful? Give feedback.
-
|
One of the main problems I struggle with is handling user access per table. Let's say I have table A with field userId, which I use to filter records designated for given user, i.e. in the backend I have my own AccessProvider set for the Controller of table A where I filter the table based on token data from the request. Let's say table A is linked to table B (1:N). The approach is the same but now the accessor needs to query table A first. Let's say there is table C linked to table table B, so now it's 1:N:N relationship. Naturally it all becomes very ugly. If we can have full fledged offline database with relationships, it would be nice if it can pull data from tables B and C based on the relationship constrains. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure how I could do that - open to suggestions on what code you would write to make that happen. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, well now that I think of it, it doesn't sound doable. :D I guess the only time the backend has any clue of what's supposed to be inside the client's table A is when table A gets synchronized. That means the only way to do something like this would be to synchronize tables B and C always together with A within one request, but that wouldn't work either because the subsets of updated records are not always going to match in incremental syncs. Not to mention that if relationships are more complicated and tables have multiple dependencies it all falls apart. SQL is hell sometimes. |
Beta Was this translation helpful? Give feedback.
-
|
... not to mention that you WANT paging to protect resource utilization on the server and to ensure minimal redos when the network disappears or the IP address changes on the client mid-transfer. The current clients just repeat the call and re-get the data from the last known good spot. However, if you put relationships in there, you would have to re-do the entire transfer. |
Beta Was this translation helpful? Give feedback.
-
|
A quickie update on where we are at with v8 at this time. This week, I'm concentrating on "live repository tests". There is an infra folder that deploys an Azure SQL, PostgreSQL, and Cosmos DB database, and each supported database will get a common set of tests. The tests can easily be re-used for your own repositories as well. You basically create a test class that inherits from Right now, there are issues all over the place:
If you want to do the live tests yourself (and you are working with the v8 branch), then you have to create a .runsettings file or set environment variables (before running dotnet test). Specifically, set these environment variables to the appropriate connection string:
I'll add a My aim is for 100% coverage of the repositories by the end of the week so that I have confidence that the "supported databases" actually work, and to give folks a set of DbContext classes that they can use as models for their own work. My list of supported repositories / databases:
I did get with the Cosmos DB folks about having a native Cosmos/SQL implementation. The only real way to do this is to have /id be the partition key, or to do cross-partition queries (which are expensive), or to do something custom (i.e. don't have a standard library). The problem is that our offline "promise" is that the offline client can generate an ID. If you package the partition key inside the ID, you are enforcing a format on the ID which breaks that offline "promise". So, at best, we have to write code that says "on create, the ID may change". This creates further problems in offline change tracking and data binding. My intent is to have better documentation around repository writing, so I'm going to use a specialist Cosmos repository as the example and let folks write the Cosmos based repository themselves based on the documentation. The official Cosmos support will be via Entity Framework Core. Cosmos does have "other facets" - PostgreSQL, MongoDB, etc. Do you all want me to add tests / instructions for those facets as well? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Adrian, I've been working with Offline Client and Cosmos DB. I think in terms of the id and the partitioning I can see 2 scenarios. The first one is that the data is partitioned on the server and the client is not aware of how the data is partitioned. E.g. datasets partitioned by logged user or tenant which are filtered server side. In the scenario the id won't need to be packed of altered and the partition keys are determined on the server. So the offline promise is maintained and the existing offline framework works as is. The second scenario is that the client is aware on how the data is partitioned and the key needs to be packed to the same specification as the server. With the v6 framework I've managed hide this away in my client services / repositories, so that the id set on the entity is renamed for the offline store and the actual id is generated from a combination of the fields the same as the server. The current drawback of this is that you can't for example query the endpoint using the unpacked id. Is this reasonable though given that you've had to put some thought into how the data is partitioned in the first place? I think the example cosmos repository in v8 handles both these scenarios well. The options can be overridden to handle the server or context based partition keys and the default implementation handles the packed keys. If the intention is to support cosmos through EF Core could the generic repository have extension points to alter the lookup queries. So for cosmos db you could inject WithPartitionKey into the query expression, from the ef cosmos library for example, to limit the cross partition queries? With a shift to EF Core on the client side. I'm wondering if the key generation / packing could be handled by the entity configuration. A little bit like you would have to configure cosmos on the server side via attributes and Entity configurators. Mayne a separate extension library for comso db partitioned data? Should the packed key be stored or is there potential for extension points for modifying the operation queue or the requests? |
Beta Was this translation helpful? Give feedback.
-
|
Cc @desjoerd |
Beta Was this translation helpful? Give feedback.
-
|
Interesting viewpoint. Since we abstract the repository, I believe it's doable, but you would have to transfer the partition key somehow. My belief is that the client should not know about partitioning in the generic sense - it's an implementation detail, much like the choice between SQL and Cosmos in our case. However, there are ways to deal with this that would make the client aware - a query string, for example, or a dynamic lookup as a middleware. |
Beta Was this translation helpful? Give feedback.
-
|
Quickie update - now that .NET8 is out and LTS, the v8 branch will be upgraded to .NET 8 over the next couple of weeks. |
Beta Was this translation helpful? Give feedback.
-
|
Since whe are near to the End of Support of .NET 7, what's timeline of the vnext with .NET 8 support ? |
Beta Was this translation helpful? Give feedback.
-
|
I notice this thread is related to using joins in the offline database. So I am assuming joins do work already using the remote tables. Can you please point me to a sample if any of how we can use a View to fetch data from 2 or more tables? FYI we're using the 6.0.4 version of the library with a MAUI app. |
Beta Was this translation helpful? Give feedback.
-
|
Nothing in the (v6) library can do joins. Your best bet is to define a VIEW in your SQL server and then use EF core to access the view. However, a view is read-only, so you will need some other mechanism to split apart the create/delete/replace operations. If you are doing joins and presenting it as a single table controller, your best bet is to create a new repository to do what you want. The IRepository (in the Abstractions nuget) gives you the interface you need to implement. Each of the implementations (EFCore, InMemory, LiteDb) implement the same interface, so plenty of samples there. |
Beta Was this translation helpful? Give feedback.
-
|
And, to follow on from this, the v8 library will be re-orientated towards providing an EF Core interface on the client, allowing you to do joins between disparate tables on the client using LINQ statements. While it is possible to do joins today, it’s not “easy” and expects you to understand SQLite and to do your own deserialization of stored typed, which isn’t always obvious (for example, DateTimeOffset is stored as a number of ticks so it can be numerically processed since the Date type in SQLite is only accurate to the second). |
Beta Was this translation helpful? Give feedback.
-
|
Ok that is great. We are still evaluating this framework for our huge and complex application. So if I understand correctly, there won't be different interfaces for Offline and Remote from v8 which is great since we need about 30 odd tables locally on the device. When you say you will provide an optional synchronization provider, what exactly does that mean? Currently we have a similar concept implemented in our app. We let the user work on the app and have a sync button for when they want to transfer data to the server. But with Azure Mobile Apps we thought we could leverage the 2 interfaces and let the synchronization happen seamlessly when there is a n/w switch with no indication to the user. We also have about 60 odd tables on the server which are not needed on the client. From v8 will it be possible to use joins and fetch data from a view on the server without having the need for all the server tables to be replicated on the client? |
Beta Was this translation helpful? Give feedback.
-
|
There will be different interfaces for offline and remote in v8. We are making the v8 model more offline centric. Example, you’ll be able to do one of the following: Var items = myDbContext.Items.Where(…).ToListAsync();
// Or Remote
Var items = myDbContext.Remote.Items.Where(…).ToListAsync();The inclusion of the Remote will be what sets it apart. At least, that’s what I’m thinking right now. Joins will be done on the client. If you want to do joins on the server, you’ll still need to write a custom repository to get at them. |
Beta Was this translation helpful? Give feedback.
-
|
Hey folks - another update for my post-Christmas work. The server now supports GeoJSON!!!!! This includes including GeographyPoint and GeometryPoint in the server-side model, and using geo.distance(p0, p1) (where either p0 or p1 can be a field in your model or geography'POINT(a b)' to indicate a static value for the GeographyPoint. Note that GeographyPoint and GeometryPoint are part of the Microsoft.Spatial library which is an ABSTRACTION of a Geo location. You would likely need to transform these before storing in the database. You can do this either by doing GeographyPoint.Create(x y) and storing longitude and latitude OR by transforming to NetTopologySuite (or another library that supports your provider). Almost done with the server side at this point (just a few more tests to write for corner cases). I've also started work on the client side, but not much has been done there yet. |
Beta Was this translation helpful? Give feedback.
-
|
Adrian, On the Controller side, in particular the Repository, I am modifying your class to enable more complex filtering to manage user access permission to the database tables which needs to only allow data back to the client depending on their permission levels across users and tenants etc. I think I saw another similar discussion on this https://github.com/Azure/azure-mobile-apps/issues/873#issuecomment-1890660271 . As an example on our Blazor API controller we use MediatR and pass a query or command in which the dbcontext is accessed and with linq do any permission checking and querying we want and to return entities or results. Its not really the MediatR pattern Im raising but the beauty of the complete control one has by defining query and command classes that can be passed to something like MediatR. A Respository redesign could be a nice improvement although it will more than serve my purposes as is since I can modify myself scott |
Beta Was this translation helpful? Give feedback.
-
|
I'd be interested in what you are thinking. The repository pattern is just a simple implementation of CRUDL (where the L is a queryable). Simplicity was a design goal of this API. How would the API change to support what you want to do? |
Beta Was this translation helpful? Give feedback.
-
|
In our solution we override the query part on the controller, because we do not support Linq in our query implementation. So we directly parse the odata query by only allowing the "updatedAt" gte expression and using the result of that to query our data store. I think a mode where we're not using odata but just "custom" query params would greatly help having non-linq implementations. The only real requirement is the support of the updatedAt filter, which could fit in a simple queryparam which can be easily used with modelbinding. |
Beta Was this translation helpful? Give feedback.
-
|
In https://github.com/Azure/azure-mobile-apps/issues/838, support was asked for encryption of the on-device SQLite database. I saw you already made a solution for that 👍🏼, and I was wondering whether that route will keep working when azure-mobile-apps will switch to EF Core? |
Beta Was this translation helpful? Give feedback.
-
|
It should do - the system won't actually rely on SQLite specifically - just EF Core. So as long as you have a system that works with EF Core, it will work with the datasync client side as well. |
Beta Was this translation helpful? Give feedback.
-
Hey folks,
I've been doing a little bit of experimentation on another repo to try and solve two basic problems with Azure Mobile Apps.
I'm also researching how to support one-to-many relationships in the offline database. These things seem to come up frequently in discussions and issues. Fortunately, I believe I have found a way to do both of these, but it fundamentally changes how we think of Azure Mobile Apps.
In the current version, we instantiate a client, which is a representation of the REMOTE service. The offline bit is an extension of the remote service. You can have communication with the service and no offline database, but you can't have the reverse. In order to re-orientate the capability to be offline first, we have to start thinking that the offline database is first, and remote connectivity is an optional extra.
To support this, I believe that we need to switch the central capability to be an Entity Framework Core database context. We can override the SaveChanges() and SaveChangesAsync() methods to maintain the offline operations queue, then provide an optional synchronization provider that synchronizes the data to the remote service.
This new architecture will, I believe, allow us to support all of the LINQ queries that are supported by SQLite as well as allow us to support additional synchronization mechanisms beyond the OData+REST mechanism we use today. It will also give you all more control over the database capabilities. I've seen requests, for example, for offline only data - this would now be possible by having an offline only table and then joining it with the online table with LINQ.
I'm going to start a new branch (likely to be called v8 or rearch) to do the work - which will be extensive. I should note that while the architecture is solid (I've passed it around a number of folks here at Microsoft to validate it), there is no guarantee that it will be adopted.
This process also depends on YOUR input. I'll explicitly need you to try out the new architecture in your applications so we can come up with guidance around migrating from the v6 version to the v8 version. If you would like to write samples for MAUI, WinUI3, WPF, Avalonia, or Uno, then I'd love to hear from you.
Unfortunately, because of the reliance of this on Entity Framework Core, I'll be limiting the v8 version to .NET 6 and beyond. Ideally, I'd like this to work with the latest .NET 8 when it reaches release in a few weeks, but I also want it to be backwards compatible to .NET 6 while that is still supported by Microsoft.
For those of you still developing .NET Standard, UWP, and Xamarin apps, the 6.0 train will be the last train to support these platforms. It's not going anywhere. I'll just be cutting a 6.0 branch right before I merge in the 8.0 branch to main.
Feel free to put your questions in the comments below. I CRAVE YOUR FEEDBACK ON THIS. Don't be shy - no stupid questions. Let's prod and poke at this new direction to make sure it's a good one.
Beta Was this translation helpful? Give feedback.
All reactions