Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

πŸ‘©β€πŸŒΎ ReplayStep test regression in Windows #195

Open
chapulina opened this issue Nov 4, 2020 · 5 comments
Open

πŸ‘©β€πŸŒΎ ReplayStep test regression in Windows #195

chapulina opened this issue Nov 4, 2020 · 5 comments
Labels
🏰 citadel Ignition Citadel 🏯 fortress Ignition Fortress 🌱 garden Ignition Garden 🎡 harmonic Gazebo Harmonic πŸ›οΈ ionic Gazebo Ionic tests Broken or missing tests / testing infra Windows Windows support

Comments

@chapulina
Copy link
Contributor

This test used to pass consistently, but started failing since 2020-10-23 for ign-transport8:

https://build.osrfoundation.org/job/ignition_transport-ci-ign-transport8-windows7-amd64/28/testReport/(root)/playback/ReplayStep/history/

It passes on ign-transport9 though:

https://build.osrfoundation.org/job/ignition_transport-ci-ign-transport9-windows7-amd64/1/testReport/(root)/playback/ReplayStep/history/

Failure log:

79: [ RUN      ] playback.ReplayStep
79: Chirping [100] times on [3] topics:
79:  -- /foo
79:  -- /bar
79:  -- /baz
79: Stepping playback...
79: D:\Jenkins\workspace\ignition_transport-ci-ign-transport8-windows7-amd64\ws\ign-transport\log\test\integration\playback.cc(660): error: Value of: MessagesAreEqual(secondMessageData, thirdMessageData)
79:   Actual: true
79: Expected: false
79: Waiting to for playback to finish...
79:  Done waiting...
79: Playback finished!
79: [  FAILED  ] playback.ReplayStep (4688 ms)
@chapulina chapulina added tests Broken or missing tests / testing infra Windows Windows support labels Nov 4, 2020
@chapulina chapulina added the 🏰 citadel Ignition Citadel label Nov 4, 2020
@Crola1702
Copy link
Contributor

I entered the test regression in buildfarmer database and found out that this issue happened less than 25 times. It has happened in the following jobs:

Job Last time seen First time seen (Since 2021)
ign_transport-ign-8-win 2023-01-24 2021-05-01
ign_transport-gz-12-win 2022-12-01 2022-08-31
ign_transport-ign-11-win 2022-09-14 2022-05-25
ign_transport-ci-win 2022-08-02 2022-08-02

This means that this issue is happening on Citadel, Fortress and Garden

@Crola1702 Crola1702 added 🏯 fortress Ignition Fortress 🌱 garden Ignition Garden labels Jan 24, 2023
@Crola1702
Copy link
Contributor

This regression is still happening flaky on transport-11 and transport-13

Flaky ratio in the last 30 days:

job_name last_fail first_fail build_count failure_count failure_percentage
ign_transport-ign-11-win 2023-10-20 2023-10-20 7 1 14.29
ign_transport-gz-13-win 2023-10-17 2023-10-08 10 2 20.0

FYI @azeey

@Crola1702 Crola1702 added the πŸ›οΈ ionic Gazebo Ionic label Jan 10, 2024
@Crola1702
Copy link
Contributor

Flaky ration in the last 30 days:

job_name last_fail first_fail build_count failure_count failure_percentage
gz_transport-ign-transport11-win 2024-01-07 2023-12-18 7 2 28.57
gz_transport-ign-transport8-win 2024-01-03 2024-01-03 7 1 14.29
gz_transport-12-win 2023-12-30 2023-12-17 7 3 42.86
gz_transport-13-win 2023-12-26 2023-12-12 8 2 25.0

@Crola1702
Copy link
Contributor

This issue seems to be happening more often this week.

Flaky ratio in the last 15 days:

job_name last_fail first_fail build_count failure_count failure_percentage
gz_transport-main-win 2024-08-03 2024-07-30 4 2 50.0
gz_transport-ign-transport8-win 2024-08-03 2024-08-03 4 1 25.0
gz_transport-12-win 2024-07-25 2024-07-25 4 1 25.0

@Crola1702
Copy link
Contributor

Log output is almost the same as before:

83: [ RUN      ] playback.ReplayStep
83: Stepping playback...
83: C:\J\workspace\gz_transport-main-win\ws\gz-transport\log\test\integration\playback.cc(652): error: Value of: MessagesAreEqual(firstMessageData, secondMessageData)
83:   Actual: true
83: Expected: false
83: Waiting to for playback to finish...
83:  Done waiting...
83: Playback finished!
83: [  FAILED  ] playback.ReplayStep (11265 ms)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏰 citadel Ignition Citadel 🏯 fortress Ignition Fortress 🌱 garden Ignition Garden 🎡 harmonic Gazebo Harmonic πŸ›οΈ ionic Gazebo Ionic tests Broken or missing tests / testing infra Windows Windows support
Projects
None yet
Development

No branches or pull requests

2 participants