Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vshard cluster does not start in tests #432

Open
DifferentialOrange opened this issue Apr 11, 2024 · 0 comments
Open

Vshard cluster does not start in tests #432

DifferentialOrange opened this issue Apr 11, 2024 · 0 comments
Labels
2sp bug Something isn't working flaky test The issue describes the problem with the test being flaky

Comments

@DifferentialOrange
Copy link
Member

DifferentialOrange commented Apr 11, 2024

Despite all the retries and checks, sometimes vshard cluster fails to start from config. For example, see https://github.com/tarantool/crud/actions/runs/8541312165/job/23400336786

Log sample:

router | 2024-03-26 17:43:18.344 [446474] main/140/main/crud.storage_info storage_info.lua:77 E> Error getting storage info for s2-master: Connection closed
router | 2024-03-26 17:43:18.446 [446474] main/140/main/crud.storage_info storage_info.lua:77 E> Error getting storage info for s2-master: Connection closed
router | 2024-03-26 17:43:18.549 [446474] main/140/main/crud.storage_info storage_info.lua:77 E> Error getting storage info for s2-master: Connection closed
router | 2024-03-26 17:43:18.551 [446474] main/122/vshard.master_search._static_router/vshard.router init.lua:1185 E> Error during master search: {"code":32,"base_type":"LuajitError","type":"LuajitError","message":"{\"reason\":\"storage is not configured\",\"code\":33,\"type\":\"ShardingError\",\"message\":\"Storage is disabled: storage is not configured\",\"name\":\"STORAGE_IS_DISABLED\"}","trace":[{"file":"./tarantool/src/lua/utils.c","line":679}]}
router | 2024-03-26 17:43:18.651 [446474] main/140/main/crud.storage_info storage_info.lua:77 E> Error getting storage info for s2-master: Connection closed
router | 2024-03-26 17:43:18.754 [446474] main/140/main/crud.storage_info storage_info.lua:77 E> Error getting storage info for s2-master: Connection closed
router | 2024-03-26 17:43:18.856 [446474] main/140/main/crud.storage_info storage_info.lua:77 E> Error getting storage info for s2-master: Connection closed
router | 2024-03-26 17:43:18.959 [446474] main/140/main/crud.storage_info storage_info.lua:77 E> Error getting storage info for s2-master: Connection closed
router | 2024-03-26 17:43:19.061 [446474] main/140/main/crud.storage_info storage_info.lua:77 E> Error getting storage info for s2-master: Connection closed
router | 2024-03-26 17:43:19.162 [446474] main/140/main/crud.storage_info storage_info.lua:77 E> Error getting storage info for s2-master: Connection closed
router | 2024-03-26 17:43:19.164 [446474] main/140/main/crud.storage_info storage_info.lua:77 E> Error getting storage info for s2-master: Connection closed
    select.backend:"config".engine:"memtx".test_select_safety_too_low_limit ... 
    select.backend:"config".engine:"memtx".test_select_safety_too_low_limit ... (0.000s) error
...ithub/tarantool/crud/test/tarantool3_helpers/cluster.lua:252: crud is not bootstrapped: expected 4 running storages, got the following storage info: {"s1-master":{"status":"running","is_master":true},"s1-replica":{"status":"running","is_master":false},"s2-replica":{"status":"running","is_master":false},"s2-master":{"status":"error","message":"Connection closed","is_master":true}}

luatest_log_capture.zip

DifferentialOrange added a commit that referenced this issue Apr 11, 2024
Sometimes cluster fails to bootstrap in tests. The reasons are yet
unknown and likely unrelated to crud or maybe even crud tests setup.

After this patch, in case cluster preparation had failed for a test,
we retry to create a cluster up t three times.

Part of #432
DifferentialOrange added a commit that referenced this issue Apr 11, 2024
Sometimes cluster fails to bootstrap in tests. The reasons are yet
unknown and likely unrelated to crud or maybe even crud tests setup.

After this patch, in case cluster preparation had failed for a test,
we retry to create a cluster up t three times.

Part of #432
DifferentialOrange added a commit that referenced this issue Apr 11, 2024
Sometimes cluster fails to bootstrap in tests. The reasons are yet
unknown and likely unrelated to crud or maybe even crud tests setup.

After this patch, in case cluster preparation had failed for a test,
we retry to create a cluster up t three times.

Part of #432
DifferentialOrange added a commit that referenced this issue Apr 12, 2024
Sometimes cluster fails to bootstrap in tests. The reasons are yet
unknown and likely unrelated to crud or maybe even crud tests setup.

After this patch, in case cluster preparation had failed for a test,
we retry to create a cluster up t three times.

Part of #432
DifferentialOrange added a commit that referenced this issue Apr 12, 2024
Sometimes cluster fails to bootstrap in tests. The reasons are yet
unknown and likely unrelated to crud or maybe even crud tests setup.

After this patch, in case cluster preparation had failed for a test,
we retry to create a cluster up t three times.

Part of #432
@oleg-jukovec oleg-jukovec added bug Something isn't working 2sp flaky test The issue describes the problem with the test being flaky labels May 6, 2024
DifferentialOrange added a commit that referenced this issue May 20, 2024
Sometimes cluster fails to bootstrap in tests. The reasons are yet
unknown and likely unrelated to crud or maybe even crud tests setup.

After this patch, in case cluster preparation had failed for a test,
we retry to create a cluster up t three times.

Part of #432
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2sp bug Something isn't working flaky test The issue describes the problem with the test being flaky
Projects
None yet
Development

No branches or pull requests

3 participants