Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PG: Index Support and Improvements #4629

Closed
wants to merge 14 commits into from

Conversation

dplewis
Copy link
Member

@dplewis dplewis commented Mar 11, 2018

Closes: #4719

Seeing how far this goes...

  • Getting indexes on startup
  • Text, Polygon, GeoPoint (TODO)
  • Each class will have a _id index in the by schema default
  • Index on pointers and default fields

@codecov
Copy link

codecov bot commented Mar 11, 2018

Codecov Report

Merging #4629 into master will decrease coverage by 82.93%.
The diff coverage is 0.74%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #4629       +/-   ##
==========================================
- Coverage   92.88%   9.95%   -82.94%     
==========================================
  Files         119     119               
  Lines        8836    8934       +98     
==========================================
- Hits         8207     889     -7318     
- Misses        629    8045     +7416
Impacted Files Coverage Δ
...dapters/Storage/Postgres/PostgresStorageAdapter.js 2.92% <0%> (-94.38%) ⬇️
src/Routers/SchemasRouter.js 10% <0%> (-87.92%) ⬇️
src/Adapters/Storage/Mongo/MongoStorageAdapter.js 5.35% <0%> (-85.92%) ⬇️
src/Controllers/DatabaseController.js 4.13% <0%> (-90.57%) ⬇️
src/Adapters/Storage/Mongo/MongoTransform.js 3.98% <11.11%> (-84.19%) ⬇️
src/requiredParameter.js 0% <0%> (-100%) ⬇️
src/cloud-code/HTTPResponse.js 0% <0%> (-100%) ⬇️
src/Adapters/Cache/NullCacheAdapter.js 0% <0%> (-100%) ⬇️
src/Adapters/AdapterLoader.js 0% <0%> (-100%) ⬇️
src/LiveQuery/Subscription.js 4.76% <0%> (-95.24%) ⬇️
... and 102 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a2c4036...4d4a751. Read the comment docs.

@dplewis dplewis requested review from flovilmart and vitaly-t March 11, 2018 20:02
Copy link
Contributor

@flovilmart flovilmart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits with the flow types otherwise LGTM.

@@ -1355,13 +1372,15 @@ export class PostgresStorageAdapter implements StorageAdapter {

const qs = `SELECT ${columns} FROM $1:name ${wherePattern} ${sortPattern} ${limitPattern} ${skipPattern}`;
debug(qs, values);
return this._client.any(qs, values)
.catch(error => {
// @flow-disable-next
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove :) we try to enforce flow in ther.

Copy link
Member Author

@dplewis dplewis Mar 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I keep getting Missing type annotation for U. Don't know how to resolve it.

}
return [];
return Promise.reject(err);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why now throw err?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No Idea I wrote this like a month ago

@@ -1745,28 +1764,132 @@ export class PostgresStorageAdapter implements StorageAdapter {
});
}

createIndex(className: string, index: any, type: string = 'btree', conn: ?any) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make it a Promise<void> as return type please.

@dplewis
Copy link
Member Author

dplewis commented Mar 11, 2018

@vitaly-t Can you look at this when you have a moment?

@dplewis
Copy link
Member Author

dplewis commented Mar 12, 2018

@flovilmart I noticed something when I'm using updateSchemaWithIndexes on startup.

In DatabaseController.js -> performInitialization()

return Promise.all([usernameUniqueness, emailUniqueness, roleUniqueness, adapterInit, indexPromise]);

Shouldn't I get the indexes for _User, _Role?

@flovilmart
Copy link
Contributor

@dplewis which indexes are you expecting?

@dplewis
Copy link
Member Author

dplewis commented Mar 12, 2018

I'll look into it.

_User

{
  _id_: { _id: 1 },
  unique_email: { email: 1 },
  unique_username: { username: 1 }
}

_Role

{
  _id_: { _id: 1 },
  unique_name: { name: 1 }
}

@dplewis
Copy link
Member Author

dplewis commented Mar 12, 2018

@flovilmart I think I got it

Copy link
Contributor

@vitaly-t vitaly-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@dplewis
Copy link
Member Author

dplewis commented Mar 24, 2018

@flovilmart Almost forgot about this one.

@flovilmart
Copy link
Contributor

Looks good, but I have one concern, it seems that the index names are not consistent from one DB to the other and those get exposed up the JSON responses.

@dplewis
Copy link
Member Author

dplewis commented Mar 25, 2018

@flovilmart The indexes will be consistent excluding the _User and _Role unique indexes that were already there.

Do you see any potential issues with exposing the names in the JSON Response?

I use the index names in the test cases dropIndex(name). It will also make handing the response easier in the dashboard.

@dplewis
Copy link
Member Author

dplewis commented May 2, 2018

@flovilmart In the case of same index with different name, mongo and pg have different outputs.

  fit('same index with different name', (done) => {
    const index = {
      name: 1
    };
    const schema = new Parse.Schema('TestObject');
    const obj = new Parse.Object('TestObject');
    obj.set('name', 'parse');
    obj.save().then(() => {
      schema.addIndex('name_index_1', index);
      return schema.update();
    }).then((results) => {
      schema.addIndex('name_index_2', index);
      return schema.update();
    }).then((res) => {
      return config.database.adapter.getIndexes('TestObject');
    }).then((indexes) => {
      expect(indexes._id_).toEqual({ _id: 1 });
      expect(indexes. name_index_1).toEqual(index);
      expect(indexes. name_index_2).toEqual(index);
      done();
    });
  });

Mongo: Creates name_index_1 and ignores name_index_2 (no error)

PG: Creates both name_index_1 and name_index_2 (no error)

To have consistency which one makes sense or should error be thrown?

fields.forEach((field) => {
if (field === 'objectId') {
key = { _id: 1 };
name = '_id_';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this preclude compound indices that include _id as a field?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, I'll write a test case for it. Regarding #4719 (comment) can you show me which line you are talking about.

Copy link
Contributor

@ClaireNeveu ClaireNeveu May 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In your tests the schema you are expecting uses the _id field. While that is what backs objectId in the mongo backend, what I would expect Parse to store and report to the user is _id_: { objectId: 1 } since "objectId" is the name of the field in the parse interface. It makes it easier to write tooling if the name in indexes matches the name in fields.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might introduce breaking change. I'm all for changing it tho.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is definitely a breaking change but it seems unlikely anybody is using index support right now since it basically doesn't work. Parse isn't even consistent in whether it returns the _id_ index. @TylerBrock Thoughts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Index support currently works for Mongo, this PR is more geared towards Postgres so there will be some inconsistencies.

If we are introducing breaking changes, I'd recommend removing the ability for users to add names to their indexes, and have parse autogenerate them. This makes it easier to keep track of duplicate indexes, transforming keys, and provide support for future DB adapters. Thoughts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It technically works but you can't create indices on most of the fields you would want to do so (pointers, updatedAt, etc).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are test for compound text indexes, I'll write a few more for compound that include (objectId, pointers, updatedAt, etc)

@dplewis
Copy link
Member Author

dplewis commented May 25, 2018

@flovilmart Have you seen this? #4629 (comment)

@flovilmart
Copy link
Contributor

@dplewis is there any issue with going forward with the current implementation and keep different behaviors if it’s not problematic?

@flovilmart
Copy link
Contributor

@dplewis build looks broken on the CI, can you double check?

@flovilmart
Copy link
Contributor

@dplewis any chance we can get this one in?

@dplewis
Copy link
Member Author

dplewis commented Sep 22, 2018

@flovilmart Sorry for the late reply. Had a lot of emails from stalebot. I'll try to get this in, its gone stale

@stale
Copy link

stale bot commented Nov 6, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants