Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[YSQL] Backup for colocated databases #4874

Closed
ndeodhar opened this issue Jun 24, 2020 · 1 comment
Closed

[YSQL] Backup for colocated databases #4874

ndeodhar opened this issue Jun 24, 2020 · 1 comment
Assignees
@hulien22
Labels
area/ysql Yugabyte SQL (YSQL)

Comments

@ndeodhar
Copy link
Contributor

ndeodhar commented Jun 24, 2020 •
edited by daniel-yb
Loading

Add support to do a distributed back up for colocated databases.

Since a colocation tablet includes table ID in the row key, we need to preserve the table IDs during restore. This will require the following changes:

  1. ysql_dump to add a WITH TABLEID=xx clause to CREATE TABLE statement.
  2. YSQL should support WITH TABLEID clause to create a table with the specified OID if it does not already exist.
@ndeodhar ndeodhar added the area/ysql Yugabyte SQL (YSQL) label Jun 24, 2020
@hulien22 hulien22 assigned hulien22 and unassigned OlegLoginov Oct 7, 2020
@hulien22 hulien22 mentioned this issue Oct 8, 2020
Closed
hulien22 added a commit that referenced this issue Oct 28, 2020
@hulien22
[YSQL] Add support for CREATE TABLE/INDEX ... WITH (table_oid = x) (#…
8d0ef3f 
…5982)

Summary:
First part of backup for colocated databases (#4874), and also a necessary component of adding 2dc
support for colocated tables (#4516).
Allows for use of `CREATE TABLE ... WITH (table_oid = x);` where the created table will be assigned
the given oid if it is free, and return an error if the oid is already in use.
Similarly, also allows for index creation with (table_oid = x).

Note that the minimum table_oid we allow is FirstNormalObjectId (which is 16384 by default as defined [[ https://github.com/yugabyte/yugabyte-db/blob/master/src/postgres/src/include/access/transam.h#L71-L94 | here ]]).

Adding postgres session variable `yb_enable_create_with_table_oid` to enable and disable this feature (defaults to false).

Test Plan:
```
ybd --java-test org.yb.pgsql.TestPgRegressTable
ybd --java-test org.yb.pgsql.TestPgRegressIndex
ybd --java-test org.yb.pgsql.TestPgWithTableOid
```

Reviewers: dmitry, mihnea, zyu

Reviewed By: zyu

Subscribers: zyu, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D9588
hulien22 added a commit that referenced this issue Dec 4, 2020
@hulien22
[#4874] ysql: Backup for colocated databases
b598f62 
Summary:
Depends on D9588.

Adding support for backups for colocated databases. Main changes made are detecting if a table is
colocated during ysql_dump and then using `WITH (table_oid = x)` to recreate the table with the
correct oid. Also adding checks in ysql_dump to detect that if a database is colocated and to create
those databases with `colocated = true`.

Also making changes to snapshot creation/importing. When creating a snapshot, if we have a colocated
table, then also add an entry for the parent colocated table. When importing the snapshot, we still do
schema validation on the normal colocated tables, but we associate the colocated tablet to the parent
colocated table. Thus, normal colocated tables will be shown in the yb-admin import_snapshot table
without any linked tablets. Sample output from yb-admin import_snapshot now:

```
Importing snapshot 83aa94f5-da5b-4a90-8b9d-f96f3463a3eb (COMPLETE)
Table type: colocated table
Target imported colocated table name: 00004600000030008000000000000000.colocated.parent.tablename
Colocated table being imported: 00004000000030008000000000000000.colocated.parent.tablename
Table type: colocated table
Target imported colocated table name: coloc2.products
Colocated table being imported: coloc.products
Table type: colocated table
Target imported colocated table name: coloc2.users
Colocated table being imported: coloc.users
Table type: colocated table
Target imported colocated table name: coloc2.orders
Colocated table being imported: coloc.orders
Table type: colocated table
Target imported colocated table name: coloc2.reviews
Colocated table being imported: coloc.reviews
Table type: colocated table
Target imported colocated table name: coloc2.test
Colocated table being imported: coloc.test
Table type: table
Target imported table name: coloc2.test2
Table being imported: coloc.test2
Successfully applied snapshot.
Object                  Old ID                                  New ID
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ParentColocatedTable    00004000000030008000000000000000.colocated.parent.uuid  00004600000030008000000000000000.colocated.parent.uuid
Tablet 0                ffd937db100d4826b46ad143bd275d93        143da2ce10f946f09262dda9f2c37554
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ColocatedTable          00004000000030008000000000004003        00004600000030008000000000004003
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ColocatedTable          0000400000003000800000000000400c        0000460000003000800000000000400c
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ColocatedTable          00004000000030008000000000004014        00004600000030008000000000004014
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ColocatedTable          0000400000003000800000000000401c        0000460000003000800000000000401c
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ColocatedTable          00004000000030008000000000004409        00004600000030008000000000004409
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
Table                   0000400000003000800000000000440c        00004600000030008000000000004616
Tablet 0                95dc6f04c5e3417682d7b68d9beea8ee        23b25c18ec8a4bf19ff91f9de60ddba4
Tablet 1                24e0d1034e6e4feb8df9224a1899f72d        093de326b717430e881f7852b4b6fc11
Snapshot                83aa94f5-da5b-4a90-8b9d-f96f3463a3eb    95046a41-96b7-402e-b4a9-74c25e66b649
```
In this output, there are 5 colocated tables `(00004000000030008000000000004003, 0000400000003000800000000000400c, 00004000000030008000000000004014, 0000400000003000800000000000401c, 00004000000030008000000000004409)`
all sharing the new tablet `143da2ce10f946f09262dda9f2c37554` under `00004600000030008000000000000000.colocated.parent.uuid`.
Note that these colocated tables have the same old and new oids (but different database ids).
Also note that there is an additional uncolocated table in the same database that has 2 tablets,
and that this table is treated as normal.

Finally, changes have also been made to `yb_backup.py` so that it can deal with the slightly
different output from import_snapshot. It will skip any tables marked as ColocatedTables and
will process the ParentColocatedTable. This processing can be done the same as other tables
since the colocated tablet and data are all stored under the directory with the parent colocated
uuid.

Test Plan:
```
ybd --java-test "org.yb.pgsql.TestYbBackup"
```

Reviewers: rahuldesirazu, jason, oleg

Reviewed By: oleg

Subscribers: mihnea, nicolas, jenkins-bot, yql, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D9649
OlegLoginov pushed a commit that referenced this issue Jan 28, 2021
@hulien22 @OlegLoginov
[Backport 2.4][#4874] ysql: Backup for colocated databases
a55565c 
Summary:
Depends on D9588.

Adding support for backups for colocated databases. Main changes made are detecting if a table is
colocated during ysql_dump and then using `WITH (table_oid = x)` to recreate the table with the
correct oid. Also adding checks in ysql_dump to detect that if a database is colocated and to create
those databases with `colocated = true`.

Also making changes to snapshot creation/importing. When creating a snapshot, if we have a colocated
table, then also add an entry for the parent colocated table. When importing the snapshot, we still do
schema validation on the normal colocated tables, but we associate the colocated tablet to the parent
colocated table. Thus, normal colocated tables will be shown in the yb-admin import_snapshot table
without any linked tablets. Sample output from yb-admin import_snapshot now:

```
Importing snapshot 83aa94f5-da5b-4a90-8b9d-f96f3463a3eb (COMPLETE)
Table type: colocated table
Target imported colocated table name: 00004600000030008000000000000000.colocated.parent.tablename
Colocated table being imported: 00004000000030008000000000000000.colocated.parent.tablename
Table type: colocated table
Target imported colocated table name: coloc2.products
Colocated table being imported: coloc.products
Table type: colocated table
Target imported colocated table name: coloc2.users
Colocated table being imported: coloc.users
Table type: colocated table
Target imported colocated table name: coloc2.orders
Colocated table being imported: coloc.orders
Table type: colocated table
Target imported colocated table name: coloc2.reviews
Colocated table being imported: coloc.reviews
Table type: colocated table
Target imported colocated table name: coloc2.test
Colocated table being imported: coloc.test
Table type: table
Target imported table name: coloc2.test2
Table being imported: coloc.test2
Successfully applied snapshot.
Object                  Old ID                                  New ID
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ParentColocatedTable    00004000000030008000000000000000.colocated.parent.uuid  00004600000030008000000000000000.colocated.parent.uuid
Tablet 0                ffd937db100d4826b46ad143bd275d93        143da2ce10f946f09262dda9f2c37554
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ColocatedTable          00004000000030008000000000004003        00004600000030008000000000004003
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ColocatedTable          0000400000003000800000000000400c        0000460000003000800000000000400c
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ColocatedTable          00004000000030008000000000004014        00004600000030008000000000004014
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ColocatedTable          0000400000003000800000000000401c        0000460000003000800000000000401c
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
ColocatedTable          00004000000030008000000000004409        00004600000030008000000000004409
Keyspace                00004000000030008000000000000000        00004600000030008000000000000000
Table                   0000400000003000800000000000440c        00004600000030008000000000004616
Tablet 0                95dc6f04c5e3417682d7b68d9beea8ee        23b25c18ec8a4bf19ff91f9de60ddba4
Tablet 1                24e0d1034e6e4feb8df9224a1899f72d        093de326b717430e881f7852b4b6fc11
Snapshot                83aa94f5-da5b-4a90-8b9d-f96f3463a3eb    95046a41-96b7-402e-b4a9-74c25e66b649
```
In this output, there are 5 colocated tables `(00004000000030008000000000004003, 0000400000003000800000000000400c, 00004000000030008000000000004014, 0000400000003000800000000000401c, 00004000000030008000000000004409)`
all sharing the new tablet `143da2ce10f946f09262dda9f2c37554` under `00004600000030008000000000000000.colocated.parent.uuid`.
Note that these colocated tables have the same old and new oids (but different database ids).
Also note that there is an additional uncolocated table in the same database that has 2 tablets,
and that this table is treated as normal.

Finally, changes have also been made to `yb_backup.py` so that it can deal with the slightly
different output from import_snapshot. It will skip any tables marked as ColocatedTables and
will process the ParentColocatedTable. This processing can be done the same as other tables
since the colocated tablet and data are all stored under the directory with the parent colocated
uuid.

Original diff: D9649 / b598f62

Test Plan:
ybd --java-test "org.yb.pgsql.TestYbBackup"
Jenkins: rebase: 2.4, hot

Reviewers: rahuldesirazu, jason, mihnea

Reviewed By: mihnea

Subscribers: bogdan, yql, jenkins-bot, nicolas, mihnea

Differential Revision: https://phabricator.dev.yugabyte.com/D10467
@frozenspider
Copy link
Contributor

This has been implemented, and is currently being reworked to use colocation_id instead of table_oid: #7378

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hulien22 hulien22

Labels
area/ysql Yugabyte SQL (YSQL)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@frozenspider @ndeodhar @OlegLoginov @hulien22