DocumentDB: cursur within transaction not supported, rest API 500 error above 101 results

[mvdve]

Link to reproduction

No response

Describe the Bug

When calling a rest endpoint with a limit higher then 101 and more then 101 results, DocumentDB throws the following error:

MongoServerError: Feature not supported: cursor within transaction. Try increasing the batchsize

This seems to be causes by a limitation of Documentdb

To Reproduce

• Add more then 101 items to a collection
• Open de admin interface and copy a collection view rest api call
• Up the limit to 102 and the api call will return an 500 error {"errors":[{"message":"Something went wrong."}]}

Payload Version

2.3.1

Adapters and Plugins

@payloadcms/db-mongodb

[DanRibbens]

Sorry for the delay. You might consider using the new db: mongooseAdapter( { transactionOptions: false } ) option which disables the use of transactions until this is sorted out.

[franciscolourenco]

We are having the same problem. Any possible solution that doesn't require disabling transactions?

[franciscolourenco]

@DanRibbens or, is there a way to disable transactions only in a specific (local or REST) API call? Our current workaround would be to use a custom endpoint which fetches all documents using the db adapter in order to not use a transaction, but this way we still need to deal with authorization.

[tak-amboss]

@DanRibbens Based on our analysis, this issue seems to stem from the interplay between enabled transactions in Payload CMS, the batchSize setting in Mongoose, and the transaction limitations within AWS DocumentDB.

We require transactions to be enabled for certain hooks to function as expected, ensuring that changes can be reliably rolled back in case of errors. Therefore, disabling transactions entirely is not a viable option for us.

By default, Mongoose uses a batchSize of 101. When a query in DocumentDB returns more than 101 items, Mongoose resorts to using a cursor, which is not supported within transactions in DocumentDB.

We see the following potential solutions:

1. Increase the batchSize for individual transactions (though this may introduce performance or memory issues).
2. Disable transactions for specific queries.
3. Implement a limit and use pagination.

These solutions are applicable in cases where we can directly use the DatabaseAdapter instead of Payload's Local/GraphQL/REST APIs. However, it would be ideal if Payload could provide an option to disable cursor usage within transactions and handle such cases internally.

[DanRibbens]

This all makes sense @tak-amboss. Thank you for the analysis on this. I'll get back to you after I do some research into the feasibility for disabling the cursor as a feature flag for the db adapter.

[DanRibbens]

@r1tsuu and I just talked through this issue and possible solutions. Before we go ahead with a new solution to this problem, we're thinking this is already mitigated by the changes made in v3.

In the beta currently, I've made it so that any read operations don't call initTransaction (they'll use a transaction if one already exists on the req). If I'm correct in this, the only time you'd see a use of the cursor is in a find with pagination that exceeds the cursor limit.

It could be that you also get this when trying to update > 100 docs, in which case we may need a batching mechanism or some custom flags around cursors built into payload or the database adapter.

Can anyone confirm that this is not an issue with documentDB on a recent version of v3 beta of Payload?

[franciscolourenco]

@DanRibbens that should fix a lot of the issues, but this will still be an issue in situations where you want to payload.find inside afterChange hooks, where you might want to find changes that are still in the uncommitted transaction.

[r1tsuu]

We could implement some batching logic into all find operations after/to this PR #9594 to findMany https://github.com/payloadcms/payload/blob/0e02f3f7f24ed1a40d099cc508e72241df82049f/packages/db-mongodb/src/utilities/findMany.ts

[franciscolourenco]

@r1tsuu the workaround that we have used for now was to increase the batchsize when querying transactions with mongoose. However this prevents us from using the db adaptor methods. Which batching logic did you have in mind?