Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] UpdateSettingsIT testNoopUpdate failing #70961

Closed
iverase opened this issue Mar 29, 2021 · 1 comment · Fixed by #71401
Closed

[CI] UpdateSettingsIT testNoopUpdate failing #70961

iverase opened this issue Mar 29, 2021 · 1 comment · Fixed by #71401
Assignees
@DaveCTurner
Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@iverase
Copy link
Contributor

iverase commented Mar 29, 2021

Build scan:

https://gradle-enterprise.elastic.co/s/aiwtbwewm6zoe/

Repro line:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.indices.settings.UpdateSettingsIT.testNoopUpdate" -Dtests.seed=D5D8AF01D5F08A9 -Dtests.security.manager=true -Dtests.locale=en-GB -Dtests.timezone=Canada/Mountain -Druntime.java=8

Reproduces locally?:

No

Applicable branches:

This failure correspond to 7. but it seems that the test might fail rarely in any branch

Failure excerpt:

org.elasticsearch.indices.settings.UpdateSettingsIT > testNoopUpdate FAILED
    java.lang.AssertionError: expected same:<cluster uuid: AS95B1tnSdS3kNh4S6APAA [committed: true]
    version: 144
    state uuid: ac5Lt1KvSx64ioJI2AHGng
    from_diff: true
    meta data version: 133
       coordination_metadata:
          term: 1
          last_committed_config: VotingConfiguration{-tGF6-kWS4CSX_iTaGRpdQ,UuiiNCv9TOKpMRPE2NWpYA,9TPYGZvkQwmKAZqYRYkmuA}
          last_accepted_config: VotingConfiguration{-tGF6-kWS4CSX_iTaGRpdQ,UuiiNCv9TOKpMRPE2NWpYA,9TPYGZvkQwmKAZqYRYkmuA}
          voting tombstones: []
       [test/dpcF8CcsSHqHR0VH1qsuAQ]: v[15], mv[1], sv[3], av[1]
          0: p_term [1], isa_ids [thJ8lqXeSI-IsAc3YPi7zw, oZ0Ra-AxSde31reI6DC9cA]
          1: p_term [1], isa_ids [of7P0ROpShqL9_FyZLNAvg, C2RdNomuRCuAMeRTRyVGVA]
          2: p_term [1], isa_ids [81CFauG8SLyd5dh1ZumfJg, ATQMZrEkTiK8uBrB3n4FLg]
          3: p_term [1], isa_ids [IqBYasFXT0SnQuZVK7yf6w, PRkNw_K4QVqnJiU81QaqZQ]
          4: p_term [1], isa_ids [lgByuXdiRY24dnWH7Dsi2Q, 5DvXFMfeQauqG68hH6eSNw]
          5: p_term [1], isa_ids [OCT-3icoRbmDz0rT1mevew, QJER5mjFRB6QO1Y0pbLTXQ]
    metadata customs:
       index-graveyard: IndexGraveyard[[[index=[test/RVzLYgUARv-NhfEgO1E5qA], deleteDate=2021-03-29T06:57:01.208Z], [index=[test/tiYx5RLMTTak3IkYg5jgOg], deleteDate=2021-03-29T06:57:02.268Z], [index=[test/U2WjcTsRQaa2KLUGBOiZ5w], deleteDate=2021-03-29T06:57:02.644Z], [index=[test/Esb_zQrESEuYP9uKbsEmyQ], deleteDate=2021-03-29T06:57:03.055Z], [index=[test/NF8kMhaSTUuuyeYgrGWJpQ], deleteDate=2021-03-29T06:57:03.703Z], [index=[test/Q-ZuF0PZTQyKL1S5TMeuaA], deleteDate=2021-03-29T06:57:04.251Z], [index=[test/4V1rm1a-ScaUJotVJGNU4g], deleteDate=2021-03-29T06:57:04.824Z], [index=[test/RX4LPO2zSPOg_mOUuHq7Wg], deleteDate=2021-03-29T06:57:05.736Z], [index=[test/ep2W0d1cRyWLiRyNAcEk3A], deleteDate=2021-03-29T06:57:06.579Z], [index=[test/aQ4WjibFRtGIHYt5QHJR-g], deleteDate=2021-03-29T06:57:07.127Z]]]
    blocks: 
       test:
          7,index read (api), blocks READ
    nodes: 
       {node_s2}{9TPYGZvkQwmKAZqYRYkmuA}{T-6vKHw1SK2PJYIS50yvjQ}{127.0.0.1}{127.0.0.1:33525}{imr}
       {node_s5}{1ZpiqMBUR6uNheFOUMSqNg}{Y_duQV5iR62vHMzCG6I4Cw}{127.0.0.1}{127.0.0.1:37175}{dir}
       {node_s0}{UuiiNCv9TOKpMRPE2NWpYA}{dMEWlvg0SjSvoFcXOGhCZA}{127.0.0.1}{127.0.0.1:45735}{imr}, local
       {node_s1}{-tGF6-kWS4CSX_iTaGRpdQ}{PnUTbp4jTjC-8uAP78Z9ag}{127.0.0.1}{127.0.0.1:42399}{imr}, master
       {node_s3}{Zp0iC09RQs62rQIs3XXkpw}{0fWMwXBPTpKi_R2xADIa1g}{127.0.0.1}{127.0.0.1:43885}{dir}
       {node_s4}{vcqAAAwjQhGvCVBXa0TNkw}{TTu_2ii3TyedUnVOntkTZw}{127.0.0.1}{127.0.0.1:33529}{dir}
    routing_table (version 112):
    -- index [[test/dpcF8CcsSHqHR0VH1qsuAQ]]
    ----shard_id [test][0]
    --------[test][0], node[1ZpiqMBUR6uNheFOUMSqNg], [P], s[STARTED], a[id=oZ0Ra-AxSde31reI6DC9cA]
    --------[test][0], node[vcqAAAwjQhGvCVBXa0TNkw], [R], s[STARTED], a[id=thJ8lqXeSI-IsAc3YPi7zw]
    ----shard_id [test][1]
    --------[test][1], node[1ZpiqMBUR6uNheFOUMSqNg], [R], s[STARTED], a[id=of7P0ROpShqL9_FyZLNAvg]
    --------[test][1], node[vcqAAAwjQhGvCVBXa0TNkw], [P], s[STARTED], a[id=C2RdNomuRCuAMeRTRyVGVA]
    ----shard_id [test][2]
    --------[test][2], node[1ZpiqMBUR6uNheFOUMSqNg], [R], s[STARTED], a[id=ATQMZrEkTiK8uBrB3n4FLg]
    --------[test][2], node[Zp0iC09RQs62rQIs3XXkpw], [P], s[STARTED], a[id=81CFauG8SLyd5dh1ZumfJg]
    ----shard_id [test][3]
    --------[test][3], node[1ZpiqMBUR6uNheFOUMSqNg], [P], s[STARTED], a[id=PRkNw_K4QVqnJiU81QaqZQ]
    --------[test][3], node[Zp0iC09RQs62rQIs3XXkpw], [R], s[STARTED], a[id=IqBYasFXT0SnQuZVK7yf6w]
    ----shard_id [test][4]
    --------[test][4], node[vcqAAAwjQhGvCVBXa0TNkw], [P], s[STARTED], a[id=5DvXFMfeQauqG68hH6eSNw]
    --------[test][4], node[Zp0iC09RQs62rQIs3XXkpw], [R], s[STARTED], a[id=lgByuXdiRY24dnWH7Dsi2Q]
    ----shard_id [test][5]
    --------[test][5], node[1ZpiqMBUR6uNheFOUMSqNg], relocating [vcqAAAwjQhGvCVBXa0TNkw], [R], s[RELOCATING], a[id=QJER5mjFRB6QO1Y0pbLTXQ, rId=RujUrq1MS7m7YGUaKG0fKA]
    --------[test][5], node[Zp0iC09RQs62rQIs3XXkpw], [P], s[STARTED], a[id=OCT-3icoRbmDz0rT1mevew]

    routing_nodes:
    -----node_id[1ZpiqMBUR6uNheFOUMSqNg][V]
    --------[test][3], node[1ZpiqMBUR6uNheFOUMSqNg], [P], s[STARTED], a[id=PRkNw_K4QVqnJiU81QaqZQ]
    --------[test][5], node[1ZpiqMBUR6uNheFOUMSqNg], relocating [vcqAAAwjQhGvCVBXa0TNkw], [R], s[RELOCATING], a[id=QJER5mjFRB6QO1Y0pbLTXQ, rId=RujUrq1MS7m7YGUaKG0fKA]
    --------[test][1], node[1ZpiqMBUR6uNheFOUMSqNg], [R], s[STARTED], a[id=of7P0ROpShqL9_FyZLNAvg]
    --------[test][2], node[1ZpiqMBUR6uNheFOUMSqNg], [R], s[STARTED], a[id=ATQMZrEkTiK8uBrB3n4FLg]
    --------[test][0], node[1ZpiqMBUR6uNheFOUMSqNg], [P], s[STARTED], a[id=oZ0Ra-AxSde31reI6DC9cA]
    -----node_id[vcqAAAwjQhGvCVBXa0TNkw][V]
    --------[test][4], node[vcqAAAwjQhGvCVBXa0TNkw], [P], s[STARTED], a[id=5DvXFMfeQauqG68hH6eSNw]
    --------[test][5], node[vcqAAAwjQhGvCVBXa0TNkw], relocating [1ZpiqMBUR6uNheFOUMSqNg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=RujUrq1MS7m7YGUaKG0fKA, rId=QJER5mjFRB6QO1Y0pbLTXQ]
    --------[test][1], node[vcqAAAwjQhGvCVBXa0TNkw], [P], s[STARTED], a[id=C2RdNomuRCuAMeRTRyVGVA]
    --------[test][0], node[vcqAAAwjQhGvCVBXa0TNkw], [R], s[STARTED], a[id=thJ8lqXeSI-IsAc3YPi7zw]
    -----node_id[Zp0iC09RQs62rQIs3XXkpw][V]
    --------[test][3], node[Zp0iC09RQs62rQIs3XXkpw], [R], s[STARTED], a[id=IqBYasFXT0SnQuZVK7yf6w]
    --------[test][4], node[Zp0iC09RQs62rQIs3XXkpw], [R], s[STARTED], a[id=lgByuXdiRY24dnWH7Dsi2Q]
    --------[test][5], node[Zp0iC09RQs62rQIs3XXkpw], [P], s[STARTED], a[id=OCT-3icoRbmDz0rT1mevew]
    --------[test][2], node[Zp0iC09RQs62rQIs3XXkpw], [P], s[STARTED], a[id=81CFauG8SLyd5dh1ZumfJg]
    ---- unassigned
    > was not:<cluster uuid: AS95B1tnSdS3kNh4S6APAA [committed: true]
    version: 145
    state uuid: uPyN6XBGRzyJCe_rF0MRGQ
    from_diff: true
    meta data version: 134
       coordination_metadata:
          term: 1
          last_committed_config: VotingConfiguration{-tGF6-kWS4CSX_iTaGRpdQ,UuiiNCv9TOKpMRPE2NWpYA,9TPYGZvkQwmKAZqYRYkmuA}
          last_accepted_config: VotingConfiguration{-tGF6-kWS4CSX_iTaGRpdQ,UuiiNCv9TOKpMRPE2NWpYA,9TPYGZvkQwmKAZqYRYkmuA}
          voting tombstones: []
       [test/dpcF8CcsSHqHR0VH1qsuAQ]: v[16], mv[1], sv[3], av[1]
          0: p_term [1], isa_ids [thJ8lqXeSI-IsAc3YPi7zw, oZ0Ra-AxSde31reI6DC9cA]
          1: p_term [1], isa_ids [of7P0ROpShqL9_FyZLNAvg, C2RdNomuRCuAMeRTRyVGVA]
          2: p_term [1], isa_ids [81CFauG8SLyd5dh1ZumfJg, ATQMZrEkTiK8uBrB3n4FLg]
          3: p_term [1], isa_ids [IqBYasFXT0SnQuZVK7yf6w, PRkNw_K4QVqnJiU81QaqZQ]
          4: p_term [1], isa_ids [lgByuXdiRY24dnWH7Dsi2Q, 5DvXFMfeQauqG68hH6eSNw]
          5: p_term [1], isa_ids [OCT-3icoRbmDz0rT1mevew, RujUrq1MS7m7YGUaKG0fKA]
    metadata customs:
       index-graveyard: IndexGraveyard[[[index=[test/RVzLYgUARv-NhfEgO1E5qA], deleteDate=2021-03-29T06:57:01.208Z], [index=[test/tiYx5RLMTTak3IkYg5jgOg], deleteDate=2021-03-29T06:57:02.268Z], [index=[test/U2WjcTsRQaa2KLUGBOiZ5w], deleteDate=2021-03-29T06:57:02.644Z], [index=[test/Esb_zQrESEuYP9uKbsEmyQ], deleteDate=2021-03-29T06:57:03.055Z], [index=[test/NF8kMhaSTUuuyeYgrGWJpQ], deleteDate=2021-03-29T06:57:03.703Z], [index=[test/Q-ZuF0PZTQyKL1S5TMeuaA], deleteDate=2021-03-29T06:57:04.251Z], [index=[test/4V1rm1a-ScaUJotVJGNU4g], deleteDate=2021-03-29T06:57:04.824Z], [index=[test/RX4LPO2zSPOg_mOUuHq7Wg], deleteDate=2021-03-29T06:57:05.736Z], [index=[test/ep2W0d1cRyWLiRyNAcEk3A], deleteDate=2021-03-29T06:57:06.579Z], [index=[test/aQ4WjibFRtGIHYt5QHJR-g], deleteDate=2021-03-29T06:57:07.127Z]]]
    blocks: 
       test:
          7,index read (api), blocks READ
    nodes: 
       {node_s2}{9TPYGZvkQwmKAZqYRYkmuA}{T-6vKHw1SK2PJYIS50yvjQ}{127.0.0.1}{127.0.0.1:33525}{imr}
       {node_s5}{1ZpiqMBUR6uNheFOUMSqNg}{Y_duQV5iR62vHMzCG6I4Cw}{127.0.0.1}{127.0.0.1:37175}{dir}
       {node_s0}{UuiiNCv9TOKpMRPE2NWpYA}{dMEWlvg0SjSvoFcXOGhCZA}{127.0.0.1}{127.0.0.1:45735}{imr}, local
       {node_s1}{-tGF6-kWS4CSX_iTaGRpdQ}{PnUTbp4jTjC-8uAP78Z9ag}{127.0.0.1}{127.0.0.1:42399}{imr}, master
       {node_s3}{Zp0iC09RQs62rQIs3XXkpw}{0fWMwXBPTpKi_R2xADIa1g}{127.0.0.1}{127.0.0.1:43885}{dir}
       {node_s4}{vcqAAAwjQhGvCVBXa0TNkw}{TTu_2ii3TyedUnVOntkTZw}{127.0.0.1}{127.0.0.1:33529}{dir}
    routing_table (version 113):
    -- index [[test/dpcF8CcsSHqHR0VH1qsuAQ]]
    ----shard_id [test][0]
    --------[test][0], node[1ZpiqMBUR6uNheFOUMSqNg], [P], s[STARTED], a[id=oZ0Ra-AxSde31reI6DC9cA]
    --------[test][0], node[vcqAAAwjQhGvCVBXa0TNkw], [R], s[STARTED], a[id=thJ8lqXeSI-IsAc3YPi7zw]
    ----shard_id [test][1]
    --------[test][1], node[1ZpiqMBUR6uNheFOUMSqNg], [R], s[STARTED], a[id=of7P0ROpShqL9_FyZLNAvg]
    --------[test][1], node[vcqAAAwjQhGvCVBXa0TNkw], [P], s[STARTED], a[id=C2RdNomuRCuAMeRTRyVGVA]
    ----shard_id [test][2]
    --------[test][2], node[1ZpiqMBUR6uNheFOUMSqNg], [R], s[STARTED], a[id=ATQMZrEkTiK8uBrB3n4FLg]
    --------[test][2], node[Zp0iC09RQs62rQIs3XXkpw], [P], s[STARTED], a[id=81CFauG8SLyd5dh1ZumfJg]
    ----shard_id [test][3]
    --------[test][3], node[1ZpiqMBUR6uNheFOUMSqNg], [P], s[STARTED], a[id=PRkNw_K4QVqnJiU81QaqZQ]
    --------[test][3], node[Zp0iC09RQs62rQIs3XXkpw], [R], s[STARTED], a[id=IqBYasFXT0SnQuZVK7yf6w]
    ----shard_id [test][4]
    --------[test][4], node[vcqAAAwjQhGvCVBXa0TNkw], [P], s[STARTED], a[id=5DvXFMfeQauqG68hH6eSNw]
    --------[test][4], node[Zp0iC09RQs62rQIs3XXkpw], [R], s[STARTED], a[id=lgByuXdiRY24dnWH7Dsi2Q]
    ----shard_id [test][5]
    --------[test][5], node[vcqAAAwjQhGvCVBXa0TNkw], [R], s[STARTED], a[id=RujUrq1MS7m7YGUaKG0fKA]
    --------[test][5], node[Zp0iC09RQs62rQIs3XXkpw], [P], s[STARTED], a[id=OCT-3icoRbmDz0rT1mevew]

    routing_nodes:
    -----node_id[1ZpiqMBUR6uNheFOUMSqNg][V]
    --------[test][3], node[1ZpiqMBUR6uNheFOUMSqNg], [P], s[STARTED], a[id=PRkNw_K4QVqnJiU81QaqZQ]
    --------[test][1], node[1ZpiqMBUR6uNheFOUMSqNg], [R], s[STARTED], a[id=of7P0ROpShqL9_FyZLNAvg]
    --------[test][2], node[1ZpiqMBUR6uNheFOUMSqNg], [R], s[STARTED], a[id=ATQMZrEkTiK8uBrB3n4FLg]
    --------[test][0], node[1ZpiqMBUR6uNheFOUMSqNg], [P], s[STARTED], a[id=oZ0Ra-AxSde31reI6DC9cA]
    -----node_id[vcqAAAwjQhGvCVBXa0TNkw][V]
    --------[test][4], node[vcqAAAwjQhGvCVBXa0TNkw], [P], s[STARTED], a[id=5DvXFMfeQauqG68hH6eSNw]
    --------[test][5], node[vcqAAAwjQhGvCVBXa0TNkw], [R], s[STARTED], a[id=RujUrq1MS7m7YGUaKG0fKA]
    --------[test][1], node[vcqAAAwjQhGvCVBXa0TNkw], [P], s[STARTED], a[id=C2RdNomuRCuAMeRTRyVGVA]
    --------[test][0], node[vcqAAAwjQhGvCVBXa0TNkw], [R], s[STARTED], a[id=thJ8lqXeSI-IsAc3YPi7zw]
    -----node_id[Zp0iC09RQs62rQIs3XXkpw][V]
    --------[test][3], node[Zp0iC09RQs62rQIs3XXkpw], [R], s[STARTED], a[id=IqBYasFXT0SnQuZVK7yf6w]
    --------[test][4], node[Zp0iC09RQs62rQIs3XXkpw], [R], s[STARTED], a[id=lgByuXdiRY24dnWH7Dsi2Q]
    --------[test][5], node[Zp0iC09RQs62rQIs3XXkpw], [P], s[STARTED], a[id=OCT-3icoRbmDz0rT1mevew]
    --------[test][2], node[Zp0iC09RQs62rQIs3XXkpw], [P], s[STARTED], a[id=81CFauG8SLyd5dh1ZumfJg]
    ---- unassigned
@iverase iverase added >test-failure Triaged test failures from CI :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. labels Mar 29, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Mar 29, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@DaveCTurner DaveCTurner self-assigned this Mar 31, 2021
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Apr 7, 2021
@DaveCTurner
Wait on relocation in UpdateSettingsIT#testNoopUpdate
e7e788a 
This test works by capturing the applied cluster state and verifying
that it does not change as no-op updates are applied. However the
captured cluster state might contain some ongoing relocations which
subsequently finish, causing an unexpected cluster state update. This
commit fixes that by waiting for any ongoing relocations too.

Relates elastic#61348
Closes elastic#70961
@DaveCTurner DaveCTurner mentioned this issue Apr 7, 2021
Merged
DaveCTurner added a commit that referenced this issue Apr 7, 2021
@DaveCTurner
Wait on relocation in UpdateSettingsIT#testNoopUpdate (#71401)
b24a7e2 
This test works by capturing the applied cluster state and verifying
that it does not change as no-op updates are applied. However the
captured cluster state might contain some ongoing relocations which
subsequently finish, causing an unexpected cluster state update. This
commit fixes that by waiting for any ongoing relocations too.

Relates #61348
Closes #70961
DaveCTurner added a commit that referenced this issue Apr 7, 2021
@DaveCTurner
Wait on relocation in UpdateSettingsIT#testNoopUpdate (#71401)
4518f84 
This test works by capturing the applied cluster state and verifying
that it does not change as no-op updates are applied. However the
captured cluster state might contain some ongoing relocations which
subsequently finish, causing an unexpected cluster state update. This
commit fixes that by waiting for any ongoing relocations too.

Relates #61348
Closes #70961
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DaveCTurner DaveCTurner

Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Wait on relocation in UpdateSettingsIT#testNoopUpdate DaveCTurner/elasticsearch
3 participants
@DaveCTurner @elasticmachine @iverase