api: support vshard groups #319

DifferentialOrange · 2022-08-16T12:41:23Z

readme: fix args description

changelog: use single format

After discussion with Ecosystem team, we decided not to specify PR in CHANGELOG entries if there is a related issue. This patch make last CHANGELOG entries consistent.

api: deprecate using space id in len

Part of #255

internal: use vshard router object

This patch is the groundwork for vshard groups and custom routers
support.

Test runs have shown that this patch do not affects the performance of
crud requests.

Part of #44

internal: rework replicasets schema reload

This patch is the groundwork for vshard groups and custom routers
support.

After this patch, schema reload works per vshard router object.

Test runs have shown that this patch do not affects the performance of
crud requests.

Part of #44

internal: rework sharding schema reload

This patch is the groundwork for vshard groups and custom routers
support.

After this patch, sharding schema reload works per vshard router object.

Test runs have shown that this patch do not affects the performance of
crud requests.

Part of #44

internal: use single router object

This patch is the groundwork for vshard groups and custom routers
support.

After this patch, vshard router object is retrieved only in the single
point of a request. (Except for name resolving in statistics.)

Test runs have shown that this patch do not affects the performance of
crud requests.

Part of #44

api: support vshard groups

This patch adds Cartridge vshard groups [1] and non-default vshard router [2] support to CRUD operations.

To specify Cartridge vshard group, pass vshard group name with vshard_router option of a crud operation.

crud.select('customers_ddl',
            {{'=', 'age', 41}},
            {vshard_router = 'customers'})

To use non-default vshard router, pass vshard router object with vshard_router option of a crud operation.

local router = vshard.router.new(cfg)

crud.select('customers_ddl',
            {{'=', 'age', 41}},
            {vshard_router = router})

If vshard_router is not specified, default vshard router is used to process the request. If default vshard router is not found (for example, if it is a Cartridge application with vshard groups) and option is not specified, the error is returned.

Closes #44

stats: warn for using id with custom router

CRUD stats are separated by space name. If space id is passed instead of name (only crud.len allows this), space name is resolved with net_box schema. Since, after resolving #44, non-default vshard routers and Cartridge vshard groups are supported, to resolve the name we must know what vshard router should be used. Since using space id for crud.len is deprecated now, we won't support name resolve for operations with custom router statistics. Instead of this, a warning will be logged. crud.len statistics will be signed by space name or space id casted to string depending on is there a default router with space schema or not.

Follows up #44, part of #255

I didn't forget about

Tests
Changelog
Documentation

DifferentialOrange · 2022-08-16T15:07:40Z

Needs to be in sync with #317 since something may change with this patch

DifferentialOrange · 2022-08-18T11:16:35Z

Current implementation may be a bit difficult to review, I'll try to separate the implementation to several commits.

UPD: done.

README.md

crud/borders.lua

crud/common/map_call_cases/base_iter.lua

crud/common/schema.lua

crud/common/sharding/sharding_func.lua

crud/common/utils.lua

oleg-jukovec · 2022-08-22T10:10:46Z

crud/stats/init.lua

+    local vshard_router = vshard.router.static
+
+    if vshard_router == nil then
+        log.warn('Failed to resolve space name for stats: default vshard router not found')
+        return nil
+    end


Could we add a vshard_router option for stats.wrap?

Since this case is related only to crud.len, we can make specific wrappers for it. But this behavior is deprecated and will be removed, so I decided not to make statistics code more complicated since it will become irrelevant after a couple of releases.

Nothing is more permanent than a temporary solution.

The inconsistency of the code is not good, but you know more about the plans for the development of this product. So it's up to you.

test/integration/vshard_custom_test.lua

After discussion with Ecosystem team, we decided not to specify PR in CHANGELOG entries if there is a related issue. This patch make last CHANGELOG entries consistent.

Part of #255

This patch is the groundwork for vshard groups and custom routers support. Test runs have shown that this patch do not affects the performance of crud requests. Part of #44

This patch is the groundwork for vshard groups and custom routers support. After this patch, schema reload works per vshard router object. Test runs have shown that this patch do not affects the performance of crud requests. Part of #44

This patch is the groundwork for vshard groups and custom routers support. After this patch, sharding schema reload works per vshard router object. Test runs have shown that this patch do not affects the performance of crud requests. Part of #44

This patch is the groundwork for vshard groups and custom routers support. After this patch, vshard router object is retrieved only in the single point of a request. (Except for name resolving in statistics.) Test runs have shown that this patch do not affects the performance of crud requests. Part of #44

This patch adds Cartridge vshard groups [1] and non-default vshard router [2] support to CRUD operations. To specify Cartridge vshard group, pass vshard group name with `vshard_router` option of a crud operation. ```lua crud.select('customers_ddl', {{'=', 'age', 41}}, {vshard_router = 'customers'}) ``` To use non-default vshard router, pass vshard router object with `vshard_router` option of a crud operation. ```lua local router = vshard.router.new(cfg) crud.select('customers_ddl', {{'=', 'age', 41}}, {vshard_router = router}) ``` If `vshard_router` is not specified, default vshard router is used to process the request. If default vshard router is not found (for example, if it is a Cartridge application with vshard groups) and option is not specified, the error is returned. 1. https://www.tarantool.io/en/doc/latest/book/cartridge/cartridge_dev/#using-multiple-vshard-storage-groups 2. https://www.tarantool.io/ru/doc/latest/reference/reference_rock/vshard/vshard_router/#router-api-new Closes #44

CRUD stats are separated by space name. If space id is passed instead of name (only `crud.len` allows this), space name is resolved with net_box schema. Since, after resolving #44, non-default vshard routers and Cartridge vshard groups are supported, to resolve the name we must know what vshard router should be used. Since using space id for `crud.len` is deprecated now, we won't support name resolve for operations with custom router statistics. Instead of this, a warning will be logged. `crud.len` statistics will be signed by space name or space id casted to string depending on is there a default router with space schema or not. Follows up #44, part of #255

oleg-jukovec

Thank you for the pull request!

DifferentialOrange force-pushed the DifferentialOrange/gh-44-vshard-groups branch 4 times, most recently from 522aeda to 9de1913 Compare August 16, 2022 14:51

DifferentialOrange changed the title ~~support vshard groups~~ api: support vshard groups Aug 16, 2022

DifferentialOrange force-pushed the DifferentialOrange/gh-44-vshard-groups branch from 9de1913 to ccd69f9 Compare August 16, 2022 15:22

oleg-jukovec self-requested a review August 17, 2022 05:47

DifferentialOrange force-pushed the DifferentialOrange/gh-44-vshard-groups branch 2 times, most recently from 8a9876b to 8677a5e Compare August 17, 2022 13:21

DifferentialOrange force-pushed the DifferentialOrange/gh-44-vshard-groups branch from 8677a5e to 679b582 Compare August 19, 2022 06:53

oleg-jukovec reviewed Aug 22, 2022

View reviewed changes

DifferentialOrange force-pushed the DifferentialOrange/gh-44-vshard-groups branch from 679b582 to 04bd845 Compare August 26, 2022 12:40

DifferentialOrange mentioned this pull request Aug 26, 2022

Dishonest schema reload timeout #320

Open

DifferentialOrange force-pushed the DifferentialOrange/gh-44-vshard-groups branch 6 times, most recently from b332869 to fa69e83 Compare August 29, 2022 15:11

DifferentialOrange added 3 commits August 29, 2022 18:53

readme: fix args description

ad5eb71

changelog: use single format

7988df5

After discussion with Ecosystem team, we decided not to specify PR in CHANGELOG entries if there is a related issue. This patch make last CHANGELOG entries consistent.

api: deprecate using space id in len

e5742cb

Part of #255

DifferentialOrange force-pushed the DifferentialOrange/gh-44-vshard-groups branch from fa69e83 to c3ffcb0 Compare August 29, 2022 15:54

DifferentialOrange added 5 commits August 29, 2022 19:38

internal: use vshard router object

e24d5fa

This patch is the groundwork for vshard groups and custom routers support. Test runs have shown that this patch do not affects the performance of crud requests. Part of #44

internal: rework replicasets schema reload

4c153f2

This patch is the groundwork for vshard groups and custom routers support. After this patch, schema reload works per vshard router object. Test runs have shown that this patch do not affects the performance of crud requests. Part of #44

internal: rework sharding schema reload

2058a06

This patch is the groundwork for vshard groups and custom routers support. After this patch, sharding schema reload works per vshard router object. Test runs have shown that this patch do not affects the performance of crud requests. Part of #44

DifferentialOrange force-pushed the DifferentialOrange/gh-44-vshard-groups branch from c3ffcb0 to b8465f3 Compare August 29, 2022 16:40

DifferentialOrange requested a review from oleg-jukovec August 29, 2022 16:41

oleg-jukovec approved these changes Aug 29, 2022

View reviewed changes

DifferentialOrange requested a review from AnaNek August 30, 2022 07:22

DifferentialOrange merged commit d6f7666 into master Aug 31, 2022

DifferentialOrange deleted the DifferentialOrange/gh-44-vshard-groups branch August 31, 2022 10:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

api: support vshard groups #319

api: support vshard groups #319

DifferentialOrange commented Aug 16, 2022 •

edited

Loading

DifferentialOrange commented Aug 16, 2022

DifferentialOrange commented Aug 18, 2022 •

edited

Loading

oleg-jukovec Aug 22, 2022

DifferentialOrange Aug 26, 2022

oleg-jukovec Aug 29, 2022

oleg-jukovec left a comment

api: support vshard groups #319

api: support vshard groups #319

Conversation

DifferentialOrange commented Aug 16, 2022 • edited Loading

readme: fix args description

changelog: use single format

api: deprecate using space id in len

internal: use vshard router object

internal: rework replicasets schema reload

internal: rework sharding schema reload

internal: use single router object

api: support vshard groups

stats: warn for using id with custom router

DifferentialOrange commented Aug 16, 2022

DifferentialOrange commented Aug 18, 2022 • edited Loading

oleg-jukovec Aug 22, 2022

Choose a reason for hiding this comment

DifferentialOrange Aug 26, 2022

Choose a reason for hiding this comment

oleg-jukovec Aug 29, 2022

Choose a reason for hiding this comment

oleg-jukovec left a comment

Choose a reason for hiding this comment

DifferentialOrange commented Aug 16, 2022 •

edited

Loading

DifferentialOrange commented Aug 18, 2022 •

edited

Loading