Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres: prepend className to unique indexes #6741

Merged
merged 13 commits into from
Oct 12, 2020

Conversation

cbaker6
Copy link
Contributor

@cbaker6 cbaker6 commented Jun 19, 2020

When using the PostgresStorageAdapter.ensureUniqueness, the current index name is always "unique_fieldName". This isn't an issue as long as an index field name isn't contained in another table, but causes issues if you want to create a unique index for "uuid" for two different tables. I deleted the comments that mentioned postgres allows the same index name multiple times. I don't know if that was true for older versions of postgres (it doesn't work on postgres 10, 11, and 12), but even the autogenerated primaryKey unique index prepends the table name. The update in this PR should work for older and newer.

The Uniqueness.spec.js makes sure the indexes are created. Update: I added a test case to make sure multiple unique indexes can be added to different classes that have the same field.

Probably need to add a recommendation: After upgrading to this version of parse-server, you should log into postgres and drop the old constraints. This should be done manually as it can block writes to the database:

ALTER TABLE "_User" DROP CONSTRAINT IF EXISTS unique_username;
ALTER TABLE "_User" DROP CONSTRAINT IF EXISTS unique_email;
ALTER TABLE "_Role" DROP CONSTRAINT IF EXISTS unique_name;

The above way seems to be recommended by Heroku, as this is essentially is reindexing unique_username, unique_email, and unique_name without blocking writes on the table. Below is a blurb from the link:

However be cautious about reindexing big indexes as write locks are obtained on the parent table. One strategy to achieve the same result on a live site is to build an index concurrently on the same table and columns but with a different name, and then dropping the original index and renaming the new one. This procedure, while much longer, won’t require any long running locks on the live tables.

@codecov
Copy link

codecov bot commented Jun 19, 2020

Codecov Report

Merging #6741 into master will decrease coverage by 0.03%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6741      +/-   ##
==========================================
- Coverage   93.78%   93.75%   -0.04%     
==========================================
  Files         169      169              
  Lines       12221    12221              
==========================================
- Hits        11462    11458       -4     
- Misses        759      763       +4     
Impacted Files Coverage Δ
...dapters/Storage/Postgres/PostgresStorageAdapter.js 95.86% <100.00%> (-0.16%) ⬇️
src/RestWrite.js 93.65% <0.00%> (-0.33%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c88d81...815cc7d. Read the comment docs.

@cbaker6
Copy link
Contributor Author

cbaker6 commented Jun 22, 2020

Ready for review... I tested this PR and it's testcase on postgres 10, 11, and 12 and the change passes on all 3. The testcase fails on all 3 versions without the change as only 1 unique index is created instead of 2 unique indexes (1 for each table).

@cbaker6
Copy link
Contributor Author

cbaker6 commented Jun 24, 2020

Added another small fix to get rid of console warnings due to attempting to create an index every time during a restart of Postgres. The fix simply adds, IF NOT EXISTS when attempting to create indexes, just like CREATE TABLE IF NOT EXISTS. The error messages that show up during restart (before the fix is applied) are:

db_1         | 2020-06-24 01:36:32.197 UTC [40] ERROR:  relation "unique_username" already exists
db_1         | 2020-06-24 01:36:32.197 UTC [40] STATEMENT:  ALTER TABLE "_User" ADD CONSTRAINT "unique_username" UNIQUE ("username")
db_1         | 2020-06-24 01:36:32.201 UTC [38] ERROR:  relation "case_insensitive_email" already exists
db_1         | 2020-06-24 01:36:32.201 UTC [38] STATEMENT:  CREATE INDEX "case_insensitive_email" ON "_User" (lower("email") varchar_pattern_ops)
db_1         | 2020-06-24 01:36:32.203 UTC [40] ERROR:  relation "unique_name" already exists
db_1         | 2020-06-24 01:36:32.203 UTC [40] STATEMENT:  ALTER TABLE "_Role" ADD CONSTRAINT "unique_name" UNIQUE ("name")
db_1         | 2020-06-24 01:36:32.209 UTC [41] ERROR:  relation "case_insensitive_username" already exists
db_1         | 2020-06-24 01:36:32.209 UTC [41] STATEMENT:  CREATE INDEX "case_insensitive_username" ON "_User" (lower("username") varchar_pattern_ops)
db_1         | 2020-06-24 01:36:32.219 UTC [42] ERROR:  relation "unique_email" already exists
db_1         | 2020-06-24 01:36:32.219 UTC [42] STATEMENT:  ALTER TABLE "_User" ADD CONSTRAINT "unique_email" UNIQUE ("email")

@cbaker6
Copy link
Contributor Author

cbaker6 commented Jun 24, 2020

Fixed error messages outputting on restarts in the previous comment. There are still a set of errors below that show on initial start, but I believe those are okay and I don't know how to fix them. From what I see online, it seems this is due to the driver (node-postgres) having multiple connections and issuing the same command over multiple threads. It looks like postgres creates what's necessary and then throws the error from the other threads (I'm guessing). Not a big deal IMO

db_1         | 2020-06-24 15:36:34.784 UTC [105] ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
db_1         | 2020-06-24 15:36:34.784 UTC [105] DETAIL:  Key (typname, typnamespace)=(_SCHEMA, 2200) already exists.
db_1         | 2020-06-24 15:36:34.784 UTC [105] STATEMENT:  CREATE TABLE IF NOT EXISTS "_SCHEMA" ( "className" varChar(120), "schema" jsonb, "isParseClass" bool, PRIMARY KEY ("className") )
db_1         | 2020-06-24 15:36:34.784 UTC [106] ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
db_1         | 2020-06-24 15:36:34.784 UTC [106] DETAIL:  Key (typname, typnamespace)=(_SCHEMA, 2200) already exists.
db_1         | 2020-06-24 15:36:34.784 UTC [106] STATEMENT:  CREATE TABLE IF NOT EXISTS "_SCHEMA" ( "className" varChar(120), "schema" jsonb, "isParseClass" bool, PRIMARY KEY ("className") )
db_1         | 2020-06-24 15:36:34.785 UTC [107] ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
db_1         | 2020-06-24 15:36:34.785 UTC [107] DETAIL:  Key (typname, typnamespace)=(_SCHEMA, 2200) already exists.
db_1         | 2020-06-24 15:36:34.785 UTC [107] STATEMENT:  CREATE TABLE IF NOT EXISTS "_SCHEMA" ( "className" varChar(120), "schema" jsonb, "isParseClass" bool, PRIMARY KEY ("className") )
db_1         | 2020-06-24 15:36:34.786 UTC [104] ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
db_1         | 2020-06-24 15:36:34.786 UTC [104] DETAIL:  Key (typname, typnamespace)=(_SCHEMA, 2200) already exists.
db_1         | 2020-06-24 15:36:34.786 UTC [104] STATEMENT:  CREATE TABLE IF NOT EXISTS "_SCHEMA" ( "className" varChar(120), "schema" jsonb, "isParseClass" bool, PRIMARY KEY ("className") )
db_1         | 2020-06-24 15:36:34.786 UTC [108] ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
db_1         | 2020-06-24 15:36:34.786 UTC [108] DETAIL:  Key (typname, typnamespace)=(_SCHEMA, 2200) already exists.
db_1         | 2020-06-24 15:36:34.786 UTC [108] STATEMENT:  CREATE TABLE IF NOT EXISTS "_SCHEMA" ( "className" varChar(120), "schema" jsonb, "isParseClass" bool, PRIMARY KEY ("className") )
db_1         | 2020-06-24 15:36:34.787 UTC [109] ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
db_1         | 2020-06-24 15:36:34.787 UTC [109] DETAIL:  Key (typname, typnamespace)=(_SCHEMA, 2200) already exists.
db_1         | 2020-06-24 15:36:34.787 UTC [109] STATEMENT:  CREATE TABLE IF NOT EXISTS "_SCHEMA" ( "className" varChar(120), "schema" jsonb, "isParseClass" bool, PRIMARY KEY ("className") )
db_1         | 2020-06-24 15:36:34.788 UTC [110] ERROR:  duplicate key value violates unique constraint "pg_type_typname_nsp_index"
db_1         | 2020-06-24 15:36:34.788 UTC [110] DETAIL:  Key (typname, typnamespace)=(_SCHEMA, 2200) already exists.
db_1         | 2020-06-24 15:36:34.788 UTC [110] STATEMENT:  CREATE TABLE IF NOT EXISTS "_SCHEMA" ( "className" varChar(120), "schema" jsonb, "isParseClass" bool, PRIMARY KEY ("className") )

@cbaker6
Copy link
Contributor Author

cbaker6 commented Jul 29, 2020

@dplewis this one is ready for review

@cbaker6
Copy link
Contributor Author

cbaker6 commented Sep 27, 2020

@dplewis and @mtrezza can you review this? I believe it’s a pretty big deal for parse-server users who use Postgres and attempt to create index’s through parse-server because if a field with the same name exists in multiple tables, an index can only be created for 1 of those tables because of a bug in the code.

@@ -2063,13 +2066,11 @@ export class PostgresStorageAdapter implements StorageAdapter {
schema: SchemaType,
fieldNames: string[]
) {
// Use the same name for every ensureUniqueness attempt, because postgres
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here’s the main part of the bug. The assumption that was made in the comments is incorrect

Copy link
Member

@davimacedo davimacedo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@davimacedo davimacedo merged commit de7ec58 into parse-community:master Oct 12, 2020
@cbaker6 cbaker6 deleted the fix_unique_index branch October 12, 2020 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants