MongoDB Connector

[marktani]

This feature requests serves as a central place to discuss development and progress for the MongoDB connector. Please contribute in the comments below. I will update the top comment as consensus forms.

[alexandrakl]

I would like to work on this! Where is a good place to start looking? How to wrap head around the task and break it into its constituent parts? Please provide some guidelines if possible, would be much appreciated! Thanks for your work on this project, it is very cool, reminds me of RESTHeart ;)

[cyr-l]

@ookangzheng Have a look at the project README file. It says "Prisma can be used for MySQL Databases out of the box" ;)

[sorenbs]

Hi @alexandrakl it would be great to collaborate on this!

The first step is to map out the kind of queries that are particularly well supported by Mongo as well as queries currently supported by Prisma that does not fit the document model.

Based on this we can design the API for Document-flavoured GraphQL.

Feel free to dump your thoughts in this thread so we can get the conversation going :-)

[GilUpstream]

@alexandrakl @sorenbs I'd love to work with you all on this.

[sorenbs]

Here's an outline of some things we need to consider:

• align with existing Mongo-based GraphQL APIs (vulcanJs)
• active/passive (does schema migration even make sense?)
• mutations (can we support the powerful nested mutations? relations?)
• queries (What are the query limitations?)
• embedded types and relations
• subscriptions

Take a look at https://github.com/graphcool/prisma/issues/1340 for the API spec for the MySQL conenctor

@GilUpstream @alexandrakl and anyone who want to contribute. It would be great if you could start mapping out some of these questions in this thread.

[sorenbs]

Top level fields

For each type with an id field, the following fields are generated:

Types without an id field are called embedded types and are stored as nested documents.

The id field is represented as _id in the mongo document per normal mongo convention. Prisma does not support having both _id and id fields.

The type User is used as an example throughout this document

Query

• user returns a single user
• users returns a list of users
• usersConnection returns a relay style Connection of users

In addition to the generated query fields, the following field is present:

• node returns a single node by id

field name generation

The user field name is generated by taking the type name and lowercasing the first letter

The users and usersConnection field names are generated by taking the type name and lowercasing the first letter and then using the evo-inflector library

Mutation

• createUser create a single user
• updateUser update a single user
• deleteUser delete a single user
• upsertUser update a single user if exists, otherwise create
• updateManyUsers update multiple users
• deleteManyUsers delete multiple users

Note: The API has no explicit mutations for managing relations. Instead relations are managed through nested mutations #1280

MongoDB has no concept of foreign key constraints and cascading deletes. Prisma ensures that relations stays valid and that cascading deletes are enforced.

Examples:

A -> B

• Deleting B will fail if the relation is required and there is no cascading delete
• Deleting B will succeed if the relation is required and there is cascading delete
• Deleting B will succeed if the relation is not required and there is no cascading delete. In this case the relation field will be set to null
• Deleting B will (obviously) succeed if the relation is not required and there is cascading delete

Note: MongoDB has no concept of transactions that span more than one document. As mutations that involve relations or mutate many nodes, involve multiple documents, it is possible that the mutation will be only partially applied if the server is stopped before the mutation completes.

Details

type Query {
  user(where: UserWhereUniqueInput): User

  users(
    where: UserWhereInput
    orderBy: UserOrderByInput
    skip: Int
    before: String
    after: String
    first: Int
    last: Int
  ): [User!]!

  usersConnection(
    where: UserWhereInput
    orderBy: UserOrderByInput
    skip: Int
    before: String
    after: String
    first: Int
    last: Int
  ): UserConnection!
}

type Mutation {
  createUser(data: UserCreateInput!): User!
  updateUser(data: UserUpdateInput!, where: UserWhereUniqueInput!): User
  deleteUser(where: UserWhereUniqueInput!): User
  upsertUser(where: UserWhereUniqueInput!, create: UserCreateInput!, update: UserUpdateInput!): User

  updateManyUsers(data: UserUpdateInput!, where: UserWhereInput!): BatchPayload
  deleteManyUsers(where: UserWhereInput!): BatchPayload
}

query fields

where

Same filter as for the SQL connector, except relational filters only work for embedded types.
See #1890
All predicates map directly to mongo:

scalar

• field: $eq
• field_not: $ne
• substring matching: contains, starts_with, ends_with: $regex
• field_lt: $lt
• field_lte: $lte
• field_gt: $gt
• field_gte: $gte
• field_in: $id
• field_not_in: $nin

relation to embedded type

• field_every: $all
• field_some: $elemMatch
• field_none: $not $elemMatch

orderBy

Same enum as for SQL connector

orderBy: CATEGORY_ASC maps to sort: { category: 1 }

skip

Integer specifying the number of documents to skip

before

id of the document to start from, going backwards: max

after

id of the document to start from: min

first

Integer limiting the number of documents to return: limit

last

Integer limiting the number of documents to return: limit

Nested Mutations

Mongo supports arbitrarily nested documents but has no understanding of relations between documents.

Prisma supports two ways to model related data:

• relations
• embedded types.

relations is a great fit for relational databases and Prisma implements embedded types for SQL databases by automatically managing all the tables and foreign keys required.

In the same way, embedded types is a great fit for mongos document model. Prisma implements relations by performing multiple inserts/updates or queries under the hood.

When is a type embedded?

In Prisma a type is considered to be embedded when it has no id field. Types without an id field can only ever exist when referenced from an "owning" type. The mongo connector takes advantage of this fact by using mongos native capability to nest documents. This can result in improved performance compared to SQL.

Note: MongoDB has a storage limit of 16mb per document. This limits the aggregate size of a single node with embedded types. The SQL connector does not have this limit.

Note: The SQL connector is able to change a type from embedded to normal and back again very easily because they underlying storage mechanism is the same. This is not the case for the MongoDB connector as embedded types are embedded in the parent document.

handling of relations

todo: specify how mutations and queries are handled for embedded types

scalar lists

todo: specify how scalar lists are implemented. They should be embedded. Document how each operation maps to mongos primitives. Document storage limit

[juicycleff]

There needs the be a common api or interface so one can switch to different connectors.

[sorenbs]

That's a great point @juicycleff

We plan to support database specific APIs as well a a Core API. Most of the database specific APIs will have significant overlap with the core API making it easy to switch between connectors. For Mongo and SQL specifically we will be able to have almost 100% overlap :-)

[Yogu]

If you need some inspiration for a GraphQL type system of a document database, have a look at cruddl. It's a TypeScript library that serves a GraphQL API similar the one of prisma, but it uses ArangoDB as database (might also be interesting for #1645). It is built to embrace the possibilities of document databasees like embedding. We invented four kinds of object types to support different use cases of embedding and relating objects - like when ids should be auto-generated and whether omitted fields in update mutations should remove them or leave them unchanged.

Considering the similarities between cruddl and prisma, we would look forward to work towards a common "standard" GraphQL API for document databases.

[DavyBello]

Have a look at graphql-compose it's a toolkit for construction of complex GraphQL schemas. Provides bunch of plugins for type generation. Allows to build your own plugins/helper-functions for type construction and modification and graphql-compose-mongoose a Mongoose model converter to GraphQL types with resolvers. They also have type generator plugins for elasticsearch and aws

[duytai]

@sorenbs in my opinion, we should consider a model without embedded type to remove the complexity of mongodb connector. As far as I know:

1. Current generated Prisma api does not support embedded type
2. We just implement connector to support current Prisma API

[sorenbs]

@duytai i agree.

The term embedded type is used here only to describe a certain way to lay out your data. Prismas SQL connector has no understanding of that term and neither will the Mongo connector. The final api will be almost identical apart from less powerful filters.

[0xDaksh]

any updates on this?

[ojongerius]

Is someone working on this at the moment?

[marktani]

Yes, we are 🙂

We plan to start a preview for the MongoDB connector later this month. Please reach out to me in Slack (@nilan) if you're interested in participating.

[lfades]

An update for this note:

Note: MongoDB has no concept of transactions that span more than one document. As mutations that involve relations or mutate many nodes, involve multiple documents, it is possible that the mutation will be only partially applied if the server is stopped before the mutation completes.

Transactions will be supported in 4.0

[lfades]

todo: specify how mutations and queries are handled for embedded types

In Mongodb joins/relations are not recommended for cases where the related documents don't change often, better said, we have a lot more of queries than mutations, for example, suppose that our User has a books fields which can be a one-to-many relation or just an embedded field with the books instead, A Book will probably never change so we don't need a one-to-many relation here.

I believe embedded types can be handled in the same way Prisma handles relations, but every embedded document should contain an id (and the operations for update/remove/insert will be using different mongodb operators). In most cases embedded documents will contain an id, or something that will allow them to be identified inside the array (like the index), if the embedded document is just an object and not an array, all operations will be for that document.

[nhuesmann]

@sorenbs Have you considered including any mongoose-specific abstractions like virtuals and references/reference population (i.e. using references as an alternative to embedded documents)? I've really liked utilizing mongoose's populate operator to keep document sizes to a predictable limit, avoid arrays growing without bound, and normalize the data as much as possible. Virtuals/getters and setters are more of a nice-to-have but have still proven to be handy in certain situations, acting like Redux selectors.

I know the purpose of the mongodb connector isn't to be a mongoosejs connector, however I anticipate a large number of developers having likely interacted with mongoosejs as much if not more than the mongodb API directly, so having some familiar features might make the adoption process easier.

Also, do you plan to support GeoJSON object types for geospatial queries? They've been indispensable in a few projects and I'd really hope for them to be supported.

[heysailor]

+1 for GeoJSON. My primary use case for mongo is to enable efficient geospatial queries. These complex queries can be done with mongo directly, but the ability to resolve GeoJSON data through Prisma would be very handy.

[sorenbs]

While I appreciate the passion displayed by the community in the comments here, this thread is a place to discuss design decisions for the upcoming MongoDB connector. I have deleted a number of comments that were off-topic. If you are interested in these other topics, feel free to reach out to me in slack or open a conversation in the forum.

[agolomoodysaada]

Take a look at mongodb references. Equivalent to foreign keys in pgsql.

It was also previously mentioned that multi-document transactions is supported by mongo 4.0.

[sorenbs]

Thanks @agolomoodysaada!

We will most likely support multi-document transactions in the future. Keep in mind that this is a feature that currently does not support sharding, and even when implemented in MongoDB 4.2 will incur a performance penalty that might be too high for many applications.

References similarly does not support sharding, and I don't know of any public plans to support it in the future.

[sorenbs]

As we are getting closer to the first beta release of the Mongo connector, we are looking for early adopters to take it for a spin.

If you are planning to build an application using Prisma and MongoDB, now is a good time to get involved to make sure the MongoDB connector covers all your needs.

If you are interested, please ping @do4gr here or in the Prisma slack 🙏

[agolomoodysaada]

@do4gr , I'm interested in testing it out... any docs I can follow?

[do4gr]

The documentation how to set up Prisma to use Mongo and how to use the new features can be found here: https://github.com/prisma/Mongo-Connector-Preview

[houmark]

Also interested in testing out the MongoDB connector.

[amilner42]

To those trying out this new connector, it would be great to leave links if your work is open source.

Hoping to try this myself soon.

[nhuesmann]

@do4gr I'd like to test the MongoDB connector and hope to use it in production on a project I'm currently building.

[do4gr]

The current documentation how to use it can be found here: https://github.com/prisma/Mongo-Connector-Preview

We'll release full documentation in the next days.

[schickling]

https://www.prisma.io/blog/mongodb-preview-ow4wahkekaep/ 🎉

[vibhanshu909]

I was waiting a long time for this, and I must say prisma is doing a great job, I was wondering if there is support for multi-document ACID transactions that were introduced in mongodb v4. I can see that the current documentation is using mongodb v3.6 not v4+. Thanks!

[do4gr]

Hey @vibhanshu909,

we haven't done extensive testing on the compatibility with Mongo 4.0. The core is structured in a way that we should be able to introduce transactionality with 4.0 fairly easily. The challenge is in properly testing this for correctness and performance afterwards. This is why we are still holding off on this. But I would expect us to enable this at some point.

[001123]

Wow amazing, wish have support mongo 4.0 soon 👍

[pantharshit00]

You can track the progress on multi-document ACID transactions in #3785

[epileftro85]

Nothing about GeoJson? this future is mandatory for many MongoDB projects.