"MongoError: Cursor not found" on relational query

[mrmarcsmith]

Issue Description

This iOS query to get users of a certain role always return a "MongoError: Cursor not found" error

PFRole *role = [PFRole objectWithoutDataWithObjectId:@"mYoBjEcTId"]; // this is the _id of the  "Role"
    PFRelation *rel = [role relationForKey:@"users"];
    PFQuery *userQuery = rel.query;
    [userQuery whereKey:@"group" equalTo:"myGroup"];

Steps to reproduce

1. set up server and database
2. create a role
3. add users to the role
4. run the above query

Expected Results

The query returns users who are apart of the given role.

Actual Outcome

An internal server error is thrown and nothing is returned

Environment Setup

• Server

  • parse-server version (Be specific! Don't say 'latest'.) : 2.8.4
  • Operating System: Ubuntu 16.04.4 LTS
  • Hardware: 1Gb Digital Ocean Droplet
  • Localhost or remote server? (AWS, Heroku, Azure, Digital Ocean, etc): Digital Ocean

• Database

  • MongoDB version: 4.0.4
  • Storage engine: WiredTiger
  • Hardware: Kubernetes shared cluster (mongos) on 4x 8GB DO Droplet
  • Localhost or remote server? (AWS, mLab, ObjectRocket, Digital Ocean, etc): DigitalOcean

Logs/Trace

verbose: REQUEST for [GET] /parse/classes/_Role: {
  "where": {
    "districtNumber": 1,
    "$relatedTo": {
      "key": "users",
      "object": {
        "__type": "Pointer",
        "className": "_Role",
        "objectId": "xyz"
      }
    },
    "automaticSearchString": {
      "$regex": "\\Qa\\E"
    }
  },
  "redirectClassNameForKey": "users"
} method=GET, url=/parse/classes/_Role, x-real-ip=1.2.3.4, x-forwarded-for=1.2.3.4, x-nginx-proxy=true, host=my.example.com, connection=close, content-length=228, x-parse-client-version=i1.16.0, accept=*/*, x-parse-session-token=r:123, x-parse-application-id=123, x-parse-client-key=123, x-parse-installation-id=123, x-parse-os-version=12.0 (16A366), accept-language=en-us, accept-encoding=br, gzip, deflate, content-type=application/json; charset=utf-8, user-agent=E%20Busy%20Bees%20Babysitting/11397 CFNetwork/974.2.1 Darwin/18.0.0, x-parse-app-build-version=11397, x-parse-app-display-version=14.1.0, districtNumber=1, key=users, __type=Pointer, className=_Role, objectId=123, $regex=\Qa\E, redirectClassNameForKey=users
error: Error generating response. { MongoError: Cursor not found (namespace: 'parsedata._Join:users:_Role', id: 3314076861576347886).
    at /parse-server/node_modules/mongodb-core/lib/connection/pool.js:595:61
    at authenticateStragglers (/parse-server/node_modules/mongodb-core/lib/connection/pool.js:513:16)
    at Connection.messageHandler (/parse-server/node_modules/mongodb-core/lib/connection/pool.js:549:5)
    at emitMessageHandler (/parse-server/node_modules/mongodb-core/lib/connection/connection.js:309:10)
    at TLSSocket.<anonymous> (/parse-server/node_modules/mongodb-core/lib/connection/connection.js:452:17)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at TLSSocket.Readable.push (_stream_readable.js:134:10)
    at TLSWrap.onread (net.js:559:20)
  name: 'MongoError',
  message: 'Cursor not found (namespace: \'parsedata._Join:users:_Role\', id: 3314076861576347886).',
  ok: 0,
  errmsg: 'Cursor not found (namespace: \'parsedata._Join:users:_Role\', id: 3314076861576347886).',
  code: 43,
  codeName: 'CursorNotFound',
  operationTime: Timestamp { _bsontype: 'Timestamp', low_: 2, high_: 1544077155 },
  '$clusterTime': 
   { clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 2, high_: 1544077155 },
     signature: { hash: [Object], keyId: [Object] } } } name=MongoError, message=Cursor not found (namespace: 'parsedata._Join:users:_Role', id: 3314076861576347886)., ok=0, errmsg=Cursor not found (namespace: 'parsedata._Join:users:_Role', id: 3314076861576347886)., code=43, codeName=CursorNotFound, _bsontype=Timestamp, low_=2, high_=1544077155, _bsontype=Timestamp, low_=2, high_=1544077155, _bsontype=Binary, sub_type=0, position=20, 0=58, 1=138, 2=88, 3=45, 4=70, 5=37, 6=153, 7=43, 8=160, 9=55, 10=29, 11=11, 12=12, 13=220, 14=202, 15=251, 16=186, 17=194, 18=246, 19=177, _bsontype=Long, low_=18, high_=1540979722
error: Uncaught internal server error. { MongoError: Cursor not found (namespace: 'parsedata._Join:users:_Role', id: 3314076861576347886).
    at /parse-server/node_modules/mongodb-core/lib/connection/pool.js:595:61
    at authenticateStragglers (/parse-server/node_modules/mongodb-core/lib/connection/pool.js:513:16)
    at Connection.messageHandler (/parse-server/node_modules/mongodb-core/lib/connection/pool.js:549:5)
    at emitMessageHandler (/parse-server/node_modules/mongodb-core/lib/connection/connection.js:309:10)
    at TLSSocket.<anonymous> (/parse-server/node_modules/mongodb-core/lib/connection/connection.js:452:17)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at TLSSocket.Readable.push (_stream_readable.js:134:10)
    at TLSWrap.onread (net.js:559:20)
  name: 'MongoError',
  message: 'Cursor not found (namespace: \'parsedata._Join:users:_Role\', id: 3314076861576347886).',
  ok: 0,
  errmsg: 'Cursor not found (namespace: \'parsedata._Join:users:_Role\', id: 3314076861576347886).',
  code: 43,
  codeName: 'CursorNotFound',
  operationTime: Timestamp { _bsontype: 'Timestamp', low_: 2, high_: 1544077155 },
  '$clusterTime': 
   { clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 2, high_: 1544077155 },
     signature: { hash: [Object], keyId: [Object] } } } MongoError: Cursor not found (namespace: 'parsedata._Join:users:_Role', id: 3314076861576347886).
    at /parse-server/node_modules/mongodb-core/lib/connection/pool.js:595:61
    at authenticateStragglers (/parse-server/node_modules/mongodb-core/lib/connection/pool.js:513:16)
    at Connection.messageHandler (/parse-server/node_modules/mongodb-core/lib/connection/pool.js:549:5)
    at emitMessageHandler (/parse-server/node_modules/mongodb-core/lib/connection/connection.js:309:10)
    at TLSSocket.<anonymous> (/parse-server/node_modules/mongodb-core/lib/connection/connection.js:452:17)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at TLSSocket.Readable.push (_stream_readable.js:134:10)
    at TLSWrap.onread (net.js:559:20)
MongoError: Cursor not found (namespace: 'parsedata._Join:users:_Role', id: 3314076861576347886).
    at /parse-server/node_modules/mongodb-core/lib/connection/pool.js:595:61
    at authenticateStragglers (/parse-server/node_modules/mongodb-core/lib/connection/pool.js:513:16)
    at Connection.messageHandler (/parse-server/node_modules/mongodb-core/lib/connection/pool.js:549:5)
    at emitMessageHandler (/parse-server/node_modules/mongodb-core/lib/connection/connection.js:309:10)
    at TLSSocket.<anonymous> (/parse-server/node_modules/mongodb-core/lib/connection/connection.js:452:17)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at TLSSocket.Readable.push (_stream_readable.js:134:10)
    at TLSWrap.onread (net.js:559:20)

[flovilmart]

Can you try with the latest version of parse-server and mongodb with version 3.x not 4.x?

[mrmarcsmith]

We have a significant overhaul to be ready for parse-server 3.x and downgrading mongo is pretty high risk for a sharded production cluster.
In the mean time we just hot fixed the way the query was constructed to avoid a relational query and moved to a whole new cluster. I'll do some testing tomorrow and let you know if the new cluster stops the error.

[flovilmart]

So how was this issue discovered?

[flovilmart]

As a side question, what is the necessity for sharding?

[mrmarcsmith]

we have a lesser used administrative feature that executed the query. because of the non-critical nature of this feature it isn't covered by tests.

Sharding allows for a unified database that enforces sending requests to "shards" that reside in the same region as the user. for a New York user, why send a request from California to New York when you could access the New York servers directly.

[flovilmart]

So what’s going on is likely the roles not being discoverable because of your usage of sharding. Not sure how to fix nor provide a proper reproduction step.

[mrmarcsmith]

@flovilmart I discovered the issue here. TL;DR We load balance our mongos servers behind a Kubernetes "service" but apparently mongos instances require session affinity. I'm looking for a solution now, I'll post what we find incase anyone else stumbles across the same issue.

[mrmarcsmith]

For future searchers, Per Mongo documentation each application servers should have it's own "mongos" instance. For us that looked like colocating a Parse Server and a Mongos in the same pod.