Support for Apollo Federation and Microservice Architecture #24
Comments
|
Hello @johnkm516. I was planning on setting up a Discord at some point, but currently here is the best place to discuss things. Yes, I was trying to reduce all the most redundant tasks possible by unifying as many systems as possible. I'm glad you see the value in it 😁. In terms of architecture, it is a sort of monolith in that the API was intended to be a single endpoint to give access to Prisma via GraphQL. I have been looking into Apollo Federation lately and have been mulling over how it would best fit into this project. I'm trying to gain a bit more experience with federating multiple graphs to get a handle of the technology first. I am completely open to re-architecting things if it aids in making things more modular. The original aim was to have a single javascript import endpoint in which you could access the API. This ended up as an Nx Angular library named You can see the configuration that is generating PURE magic comments here: Line 12 in a1abedf Again, I am a bit inexperienced with GraphQL federation and not sure what would be a generalized solution that would work well. I'd be happy to work on this issue with you. Could you elaborate a bit more on how you were envisioning the services to be separated? |
|
Hi, @ZenSoftware, To be honest with you, I'm a complete newbie myself. I've spent the past few months trying to figure out all the bits and pieces for a new backend architecture that my organization can adopt (my org currently uses a monolith), and it has been really really difficult trying to understand and fit pieces like Prisma, Nexus, Apollo, together. I want to use GraphQL and the power it brings with Apollo Studio and introspection, but there was so much boilerplate code compared to normal CRUD APIs I had to find a solution that auto-generated much of the boilerplate code. After cloning literally dozens of repos trying to figure all this out your repository was the closest one I came across where it's an schema first approach (where the prisma schema is the single source of truth) and generates all the typedefs and resolvers (using Prisma was important to me). While I'm not going to be much help on the development front (at least right now as I'm still working to find a solution while simultaneously learning), I did learn a lot in the architecture side of things and the best practices on this end. The architecture I'm envisioning looks like this :
The microservice orchestration portion would belong to a different repo for main business processes and is outside the scope of this boilerplate project but just explaining what I'm thinking here. Transitioning from a monolith to new technology like GraphQL and microservices and trying to decide what the "right" architecture is has been extremely difficult. There's so many different opinions and different implementations and so many different frameworks, libraries / modules after learning and researching so many different stacks and services I still haven't gotten to actually coding (It's always like, oh what's this? Is this better than the other thing? Can this fit into the stack I am currently thinking of?). While I've learned a lot the past months and formed my own opinions on what the backend architecture should look like I don't know for certain if the architecture I describe above is "correct". I'd love to discuss this with you further either here or somewhere else like discord or slack if you'd like. Let me know what you think, or my ideas are even feasible in terms of actually implementing things. |
|
Still researching on solutions in terms of implementation, and found this : Looks like we can apply Middleware while leaving the current code generation as is to apply Apollo directives such as @key and make all the types extendable so that other subgraphs may extend the types, after the fact. Edit : Found a gateway solution that comes with a whole range of features including auth and subscription and can unify not just apollo federated graphql but basically every other API as well, and composes during compile time a "virtual graph" : https://docs.wundergraph.com/docs/use-cases/api-composition-and-integration#docker-but-for-ap-is |
|
I am going to rework the dev ops so it will be possible to provide arguments that specify the I'd like to bring up some complexities in how to get the tooling for auto-completion to work with multiple GraphQL schemas within the VSCode. There will need to be multiple apollo.config.js files. I actually submitted a feature request for this a while back. This used to be pretty buggy in the past, but hopefully the team working on their extension has ironed everything out with the issues with monorepos and multiple GraphQL schemas. I'm going to play with some of the tools that you have linked. Thanks for sharing your investigations with me. Give me a little bit to work on this and I will get back to you with a more comprehensive response. All of this should be do-able. I will be working on this at federation branch currently. I plan to merge it into the Here is the latest commit that contains the baseline changes that integrates |
|
@ZenSoftware I'm playing around with the graphql-transform-federation in an effort to update the libraries to the latest versions and adapt the code accordingly.
This very much simplifies code generation, we can simply add the @key directive to every entity object that is code generated from schema. Furthermore, any schemas containing the key only (stubs that are used only to refer to other subgraphs rather than extend any additional data of the entity) can additionally have a code generation rule to add the "resolvable : false" directive :
I'm currently working on a brand new project from scratch that adds pal.js code generation and transforms the graphql schema object using my updated graphql-transform-federation. I'm ironing out all the errors currently occurring from updating all the libraries to latest. I'll let you know if I get it working. |
|
Hi John. Thanks for sharing your progress. I'm glad you are investigating solutions into getting Apollo Federation v2 working for us. I agree with your analysis that we would want to be v2 compatible. I have worked tirelessly to get all of the dependencies within the project to the latest versions possible myself. We are currently blocked on upgrading to Prisma v4 due to PalJS. I was thinking of directly pulling out the Apollo SDL generation of PalJS into something we would have more control over as well. In particular we seem to need the ability to inject the appropriate As I mentioned previously. I am reworking the dev ops currently. I am ripping out Gulp and refactoring things to simply use
Of course other front-end technologies like React will be able to use our web APIs, but because there is so much technological investment in Auth seems to touch everything, and I believe I will need to extract out the Nest modules under zen/apps/api/src/app into their own standalone node libraries that multiple Nest apps can import & configure. Let me first get most of the dev ops reworked before I extract the Nest modules into their own standalone libraries. I will need to workout the details on how to best model authorization in our sort of federated model. I haven't worked out all the details out on this just yet. I'll report back once I have made more progress on this. 🎐 |
|
@johnkm516 The
I managed to reduce the total size of the docker image as well by going over all the packages with a fine tooth comb and removing all unused packages. You can pull the latest changes from the |
|
@ZenSoftware I also created another repo https://github.com/johnkm516/maple off of another template to test if the SDL transform works, and it does (querying the SDL from the Apollo sandbox shows the directive and resolve reference correctly). This portion of the code transforms the already built schema to add the key directive to the User model and resolveReference for findUniqueUser : However I question the usefulness of this as this part also has to be code-generated, for all the different queries and mutations. I think it would be far more useful to code-generate the federation subschema from the get-go from paljs by extending paljs with a I'll clone your federation branch and also look into changing paljs directly. I'll be honest with you, while I've been looking into NestJS for quite a while I never actually got around to coding in it (in fact I never touched typescript before), so as I mentioned before I'm a huge newbie so my code might look weird. I like to think I'm a fast learner so I'll just dive in anyway and hopefully be of some use. I'll again update on any progress I have if I have any. |
|
@johnkm516 I changed the output directory for PalJS generated assets from generate.ts const generator = new ZenGenerator({
palConfig: require('./pal.js'),
apiOutPath: path.join(__dirname, 'apps/api/src/app/graphql'),
caslOutFile: path.join(__dirname, 'apps/api/src/app/auth/casl/generated.ts'),
frontend: {
outPath: path.join(__dirname, 'libs/graphql/src/lib'),
},
});
await generator.run();We can reuse this script for any number of On another note, for any given Prisma schema there is a lot that PalJS is generating for us. It generates both the TypeScript types under resolversTypes.ts and the GraphQL SDL types within ../typeDefs.ts files. There is so much that it is taking care of for us that it feels a bit daunting to extract out the relevant bits of code from a fork. I'm concerned about losing out on the benefits of getting future upgrades as well. Instead, maybe it would be best to keep PalJS within the pipeline and do some sort of post-processing step after PalJS that would modify the Before I start experimenting with Apollo Federation v2 myself, I need to extract out the authorization module for Nest and turn it into a standalone library. Multiple Nest modules will be able to import and configure their own Let me know if you run into any issues or have any questions. Cheers! |
|
I read through the PalJS code generation yesterday and it's actually quite easy to understand. I created my own fork and added the for federation 2 header in typedefs, as well as adding the "federation : boolean" in config of the generator class. While I agree with you that our changes should be some sort of post-processing to change the generated files after they're created to not modify PalJS directly, as Apollo gets updated with new directives and Prisma also gets updated, we will have to change our post-processing accordingly; whereas modifying the PalJS generator directly allows us to use @paljs/schema and all the data models that it uses to generate. While my fork of graphql-transform-federation works after updating the dependencies, the way it's implemented it's not easy to add new directives like "@Shareable" that's currently not in the implementation, while modifying the generation from the get-go in PalJS it's a simple string concatenation so I don't have to worry about inputting a GraphQLSchema object, reading and modelling the schema to modify it, then build the modified output GraphQLSchema. So I am more leaning toward merging my changes with any future PalJS upgrades manually since my changes won't be too extensive. Furthermore, the SDL generation actually only uses a few files and has very little cross-dependencies. It's very much self contained and the code looks extremely simple to extract. We could just extract the SDL generator, upgrade the dependencies to latest including Prisma v4, and modify it for our own use. All the files that are relevant to the SDL generation that we use is contained within :
with some utility libraries like Prettier, fs, and some custom type objects from @paljs/types. Let me know what you think. |
|
I upgraded all the deps to latest including to Prisma v4, and tested SDL generation. Works fine. |
|
I have completed modifications to the PalJS code to generate a federation subschema. I haven't tested through a gateway yet but testing the Edit : Ok never mind I'm having issues when I tried building everything from a clean proj. Need to fix deps and build issues. I'll let you know when I get everything working properly. Edit 2 : I think it works now? |
|
Well done my friend. I have made substantial progress myself. Let me share with you my findings. According to the Apollo Federation subgraph docs the signature for the
Therefore the code should look something like this. paljs/User/resolvers.ts const resolvers: Resolvers = {
// ...
User: {
__resolveReference(reference, { prisma }) {
const [field, value] = Object.entries(reference).find(e => e[0] !== '__typename');
return prisma.user.findUnique({ where: { [field]: value } });
},
},
};There might be a better way to do this, but notice how I extract the Furthermore, the TS type can simply be set to resolversTypes.ts export interface User {
[key: string]: Resolver<any, any, any>;
id?: Resolver<Client.User, {}, number>;
// ...
__resolveReference?: any
}Because there is an index signature type of The @ResolveReference()
resolveReference(@Parent() reference, @Context() ctx: IContext) {
return resolvers.User.__resolveReference(reference, ctx);
}My code gen templates will need to include the above code for us when generating the Nest resolvers. I'll be sure to do that once we figure out our situation with the PalJS fork. As you suggested, I used pal.js module.exports = {
schema: 'prisma/schema.prisma',
backend: {
generator: 'sdl',
output: 'apps/api/src/app/graphql/paljs',
doNotUseFieldUpdateOperationsInput: true,
federation: true,
},
};Furthermore, I thought that you should also know that I added the GraphQL SDL definitions for the By the way, there is a small bug at johnkm516/prisma-tools/packages/generator/src/sdl/index.ts#L123 You can pull the latest changes from the |
|
@ZenSoftware
I'm currently away on holiday (it's a Thanksgiving-like long weekend in my country) so I won't be able to work until the 13th. At the very top level is the Tenant. Separating the databases even further from DB per service to DB per (tenant + service), it would be easier to also have something like a test tenant when the data gets more and more complicated. The second level would be Role. Superadmin would have access to all the tenants and all permissions, and admin with all permissions to a specific tenant, and a variety of user Roles such as Finance, Sales, etc. The third level will be Organizational Group. in a hierarchical tree structure as all organizations would have. For example, if I belong in the X Organization, in the IT department, and in team A of the IT department, I should have all the permissions that belong to X Organization, the IT department, and team A. Team B of the IT department will have all the permissions of X Organization and the IT department, but not necessarily permissions specific to team A, and vice versa. I plan on creating a separate service that syncs the organizational hierarchy with the company's Active Directory later down the road. Let me know what you think about this data model. If you agree with it I'll make a pull request to this repo later when I start work on it. Figuring out auth comes first though. |
|
@johnkm516 I would suggest reading up on the Casl docs and familiarize yourself with the ability builder. From there you can play with casl-ability.factory.ts to learn how to grant/deny access. The way I plan to refactor the Nest auth module would be such that other applications can provide their own As for how this will work in our federated model, I was hoping we would be able to just forward the HTTP I'll respond back with more details on auth once I work more on it myself. Have a great long weekend. Things are in a very good place for us thus far. We have the core of the functionality working! 🥂 Edit
|
|
@johnkm516 There were changes from Prisma v3 to v4 that broke |
|
Hey, @ZenSoftware! I'll update you on progress as I have any. |
|
@ZenSoftware
|
|
@johnkm516 Though there is more work to be done. I've been working on trying to modularize the Specifically how it works is that any Nest app within our project will look for the I exposed project wide assets under the Let me know if you need further clarification. Auth is always a rather complex subject when done right. |
|
@johnkm516 Otherwise, the |
|
@ZenSoftware For Prisma in a monorepo, I'm still trying to figure out a good solution. I have a few ideas but I need to work out the details and see if it's actually doable. I'll get back to you on this soon. |
|
@johnkm516 To address your question about where the user is being stored, the actual persistence of the user will be the responsibility of the subgraph with the A User: {
__resolveReference(reference, { prisma }) {
const [field, value] = Object.entries(reference).find(e => e[0] !== '__typename');
return prisma.user.findUnique({ where: { [field]: value } });
},
},I hope that all made sense. |
|
It would probably help you if you just saw some high level code of how tokens are being issued. Take a look at For the Angular app, you interact with the authorization service via the |
|
@ZenSoftware For Prisma, here's what I have so far :
As you can see in the examples in Prismix's README, posts.prisma extends the Accounts entity of base.prisma by declaring a stub ID. If we were to follow Federation 2's conventions Accounts in posts.prisma would be resolvable on its own with no originating entity. I'm going to look into Prismix and see if I can salvage some of the code to create our own schema merger that fits with our own needs. I don't think it's going to take a huge amount of time since looking into Paljs has shown me Prisma's DMMF and just how easy it is to parse. The devil is in the details though, I'm sure there are a huge amount of cases that need to be considered when merging the schemas. To summarize what we need :
Let me know what you think, or perhaps what problems you foresee. |
|
@johnkm516 There are so many advantages to an Nx style of repo. Guaranteed consistency of package versions and simplicity of project management. Nx style of repo would be the preferred choice for this specific starter kit. This adventure we seem to be on is now deserving of it's own repo. What do you think about opening up a repo where we co-share ownership and setup a fresh Lerna style project. We can begin to migrate over the technology from this starter kit into the federated GraphQL project. I think that would be the correct path forward. |
|
@johnkm516 |
My investigations led to the same conclusion; the Prisma client as it's currently implemented wasn't designed to be used in a monorepo. I looked into Nx vs Lerna further and it seems Nx has taken ownership of Lerna. A Lerna style monorepo is still possible, and integrating Nx's task scheduling with Lerna is possible as well. Just something to keep in mind. I definitely agree with you that for our purposes we should have an Nx style of monorepo. Let's hold off on reorganizing the entire project structure for now until a better solution is found for the Prisma client. I'll ask around and investigate further while working on what I mentioned in the previous post.
I would be honored. To be honest I did not realize the amount of extensive modifications that modularizing and supporting federation would require, so I understand it's now running a bit out of scope of the original repo. The code is all yours, so far I have just contributed a little here and there as my skills allow. So it's entirely up to you what you want to do; I look forward to working with you on this project to realize all that we've discussed here regardless. |
|
For prismix and paljs/generator, here's where I'm at : Prismix is as simple as paljs/generator in terms of modifying it. For our custom implementation, I'm thinking of the following when we eventually restructure the project to a monorepo :
I'm currently working on the schema validation portion of Prismix now. It will validate :
Let me know what you think. |
|
I decided to just support every directive in Federation 2 using a commented out attribute in the prisma schema. I added support for shareable, external, and some others just as a test in code generation for paljs/generator and it works beautifully. Keep in mind there is no validation here at all, all it does is read the schema and add the The validation of all these directives as the subgraphs are merged into the supergraph are meant to be done by a managed federation. Since Prisma only deals with databases, the user can change the directive freely and code-generate again without any issues. |
|
@johnkm516 I'm glad your investigations into Prismix were so fruitful. Very nice work. I agree, it would be ideal to have a processing step whereby we validate the multiple schema files of the project to ensure correctness within the supergraph. Since you worked out the details on this, it now seems we have the majority of the pieces we need for a fully functioning federated supergraph. It's just a matter of getting all the dev ops in place and work out all the conventions for the organization of the project assets. I'm currently investigating the integration between Nx and Lerna to work out some of the more nittier gritty details. Let me play around with things for just a bit longer and I'll create the new repo for us and share access. I'll keep you informed. 🎐 |
|
@ZenSoftware I've finished the Federation 2 directive parsing, and am making huge changes to the logic for both Prismix and Paljs/Generator for code generating stubs and validating them. I spent the last week really struggling to wrap my head around how to implement this, trying to figure out which fields a subgraph should resolve (and therefore store in the DB). While our initial modifications to Paljs/Generator worked out great since we were able to update to Prisma 4, there are many many more cases we have to solve. The most difficult case that I'm currently working on :
A Reviews subgraph contains a Product entity, which is not resolvable in the Reviews subgraph. So in our Prisma schema, we need to be able to declare a stub of Product without Prisma creating a separate table for it, and without us having to declare a relation for it. Here is how I plan on a subschema declaring an non-resolvable entity : My implementation of Prismix will first parse all the schema files in a raw string (Prismix already did this through the Implementation 1A :input reviews.prisma output reviews.prisma Paljs/Generator would read stub and generate @key directives Implementation 1B :input reviews.prisma output reviews.prisma For both 1A and 1B, if Product is required in Review, it would select the first keyField if multiple keyFields are declared. Note how the output subschema generated by Prismix autogenerates fields that will actually be stored in a Prisma database, and therefore must be declared in the schema that Prisma client will actually use for migration for the subgraph. Paljs/Generator will recognize Pros : input schema follows very closely with Prisma model declarations (1A) or Apollo Federation model declarations (1B). Implementation 2A :input reviews.prisma output reviews.prisma Pros : All fields are explicitly declared Therefore I'm leaning towards implementation 2 right now, but I need a second opinion in case I might be missing a case here. The above examples are only in cases of stub models that won't be saving any data to a database other than a reference foreign key resolved in another subgraph, because in one-to-many or many-to-many cases Product would be extended and therefore resolvable in the same subgraph (and therefore a valid Prisma model must be declared to save the relationship accordingly) There is a lot to do. I also want to be able to handle explicitly declared keyfields especially in cases where there are nested object keys like : Since this thread is getting quite bloated, I'm going to start creating issues with the above design specs and more when I get around to it in my repositories for Prismix-Federation and Prisma-Tools-Federation so I can document and organize the information better. I'll link them here when I post them. Also, if you happen to have a discord channel or a slack channel, please send me an invite. I'd love to talk to you in a more live setting; I promise I won't bother you too much 😂. Since you're in Canada there's going to be quite a time difference anyways (I'm in South Korea), but it would be good to share information and talk about design decisions that don't necessarily belong in this thread. I look forward to contributing and working with you further. |
|
@johnkm516 In terms of project structure I imagine we would have an
As an example: model User {
id Int @id
orders Order[]
@@schema("base")
}apps/transactional/schema.prisma model Order {
id Int @id
user User @relation(fields: [id], references: [id])
user_id Int
@@schema("transactional")
}Finally, concatenation would result into this: generator client {
provider = "prisma-client-js"
previewFeatures = ["multiSchema"]
}
datasource db {
provider = "postgresql"
url = env("DATABASE_URL")
schemas = ["base", "transactional"]
}
model User {
id Int @id
orders Order[]
@@schema("base")
}
model Order {
id Int @id
user User @relation(fields: [id], references: [id])
user_id Int
@@schema("transactional")
}Ideally, we would leverage off of Prisma's multi-schema feature to do the heavy lifting for validation. We then architect things in such a manner such that a valid Prisma multi-schema would also produce valid Apollo subgraphs/supergraph. This would be the architecture that we might ultimately want. That is if Prisma multi-schema was actually feature complete. I haven't tried my hand at playing with the preview feature just yet. But depending on how much is usable, we may want to rethink our efforts as there may be a better way of doing things. We may not have to integrate these custom directives you have been working on if we can get things working with this preview feature. Let's investigate this further and see what we discover. Yes, I totally agree that it would be really nice to have a more direct line of communication. Discord is what I typically use. Could you please add me as a friend. I'll delete the comment after I've confirmed you've read it. |
|
@ZenSoftware |
|
About Prisma multiSchemas, I also read up on it and immediately gave up on it after reading this : Another thing is that Prisma also doesn't support custom attributes, models, nor custom validation extensions. It's understandable considering Prisma schemas were meant to be used to model databases only, but since like us, many people want to directly code-generate from Prisma schemas, they are aware of this feature request and are working on it. We should definitely keep track of this feature once it matures into something we can use in a monorepo. However for our immediate purposes I'll keep working on Prismix; Prisma type, model, and custom validation extensions won't arrive for a long while. I'll document in detail the exact cases we must consider to declare and translate a prisma schema into a federation 2 subschema with all its features. Stub models that aren't resolvable in the subschema is one such case that I outlined above, but more test cases will help here. |
|
Ah, ok! Yes, it is very problematic that there is a requirement for unique names. I think there is actually no choice but to continue with your efforts with Prismix as we do need an immediate solution. We will just keep it in the back of our minds and do our best to harmonize a solution with an expectation that this feature is eventually coming. I'll review your suggestions for the style of syntax that is to be used and get back to you with a more thorough response. We would ideally choose the option that causes for the least amount of redundancy. |
|
@ZenSoftware Please read over my issue here : johnkm516/prismix-federation#1 |
|
@johnkm516
Historically, the reason you spin up a new database is for the purposes of isolating that data from other schemas. You generally want most of your data models to exist within a single database. The reason for this is that you can define foreign key constraints between tables, which enables the expressiveness to query data utilizing a query language. When you move a data model into a different schema, the foreign key between relations are simply stored as a primitive value, without defining the relational constraints. There is technically a relation between the tables when you only store the ID and don't define the foreign key constraint, but you lose most of the robustness of the query language because the database doesn't have any information about the relation. This is the problem that we are encountering. When you move a data model into its a different Speaking more specifically about Apollo federation, you can see the limitations we are facing more clearly with the following code: Nest Docs Apollo Federation example // User resolver
@ResolveReference()
resolveReference(reference: { __typename: string; id: string }) {
return this.usersService.findById(reference.id);
}The only information we have is the User I heard the Prisma team describing multi-schema support as being a very difficult problem to solve. I am beginning to see why. We are probably encountering the same problems that they are trying to solve. Despite this, I don't believe our efforts are in vain. There is much more clarity in how to modularize the repo now. We've also managed to integrate Apollo Federation v2 into the tech stack! Which enables many new possibilities. I think it is still a good idea to refactor things such that we can specifying custom locations for generated Prisma clients, and have the all tooling work for multiple Nest apps within the monorepo. The stitching together of Apollo subgraphs (even with the limitations that we are now aware of) can be handled at the level of Nest. Let me know what your thoughts are. Sorry it took so long for the response. It took up until now to really understand the nature of the problem. |
|
@ZenSoftware
With this work now done I'm waiting on you to modularize Auth. When you finish putting auth in its own module I can help you modify the type declarations such that it composes in the supergraph. In terms of future plans for a feature, I am currently contemplating a design for the generator to generate idempotent mutations for the Saga pattern. For example, if I have the following prisma model :
I would want a I haven't seen you on for a while. I hope you're doing ok. Leave me a message on Discord when you can. |
|
@johnkm516 I just wanted to let you know that I finally refactored the project to allow for multiple Thanks for your patience my friend. We'll catch up soon.🎐 |
|
@ZenSoftware You should check my profile for a standalone fork I made of zen, called zen-federation. During your hiatus I implemented an absolutely colossal amount of features. Everything I talked about with you on Discord few months back, from the monorepo with multiple schema files, app generation using the Nx generator, modifying PalJS generator to latest Prisma using the latest Prisma features, etc. Some of the things I did include but are not limited to :
I suggest you clone my repo and take a look. I could easily modify the app-generator to use the modularized auth lib rather than generating auth-related files. Needless to say there are a huge list of changes that I made that I cannot list off the top of my head. I'll be on Discord on Monday KST. Let's discuss combining the code into a new repo. |
This is more of a question than an issue. I couldn't find a discussion board or slack, sorry if this is the wrong place to ask.
This repo is exactly the starter-kit I was looking for as it minimizes writing repetitive code that is defined in the SDL anyways, and makes full use of all generated Prisma Inputs in the resolvers.
However I am wondering how scalable is the current architecture of this project? Right now all the generated schemas are input into one application so it looks like it's a monolith. Would it be possible to separate services into different app folders, and federate them with Apollo Federation using Apollo Router? Looking over the gulpfile and the relevant code-gen I definitely think it's possible, but I was wondering if it is worth the effort, as I might be missing something (the current architecture might not be a monolith at all).
The text was updated successfully, but these errors were encountered: