Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky test: port-forwarding for the component: Possible custom port collision when running tests in parallel #6758

Closed
rm3l opened this issue Apr 20, 2023 · 0 comments · Fixed by #6763
Assignees
Labels
area/testing Issues or PRs related to testing, Quality Assurance or Quality Engineering kind/bug Categorizes issue or PR as related to a bug.
Milestone

Comments

@rm3l
Copy link
Member

rm3l commented Apr 20, 2023

/kind bug
/area testing

What versions of software are you using?

Operating System:
Fedora 37

Output of odo version:
odo v3.9.0 (b843ce8)

How did you run odo exactly?

ginkgo -v -nodes=3 \
  -focus 'port-forwarding for the component' --label-filter='podman' \
  --output-interceptor-mode=none \
  tests/integration/

Actual behavior

Tests won't pass, due to conflicting ports, as we can see in the logs below:

Logs on Podman

[...]
  [odo] [starting container d0f4752cdc7a2dc564cf7f873f421e023ce1cecd7412a63dc12f17eb04159df4: rootlessport listen tcp 0.0.0.0:30005: bind: address already in use]                           
f8452b4e27835bb56f40485371d20569e541fcc222f557ae541d9b5b: a dependency of container 8b55f3a6f8452b4e27835bb56f40485371d20569e541fcc222f557ae541d9b5b failed to start: container state improper]
    [odo] Error: failed to start 2 containers                                                                                                                                                  
    [odo]                                                                                                                                                                                      
    Setting current dir to: /home/asoro/work/projects/odo/tests/integration                                                                                                                    
    Deleting dir: /tmp/2993300828                                                                                                                                                              
    Deleting dir: /tmp/3771873096    
[...]

Summarizing 2 Failures:
  [FAIL] odo dev command tests port-forwarding for the component when devfile has single endpoint when running odo dev [BeforeEach] should expose the endpoint on localhost (podman=true, manual=true, customPortForwarding=true) [podman]
  /home/asoro/work/projects/odo/tests/helper/helper_run.go:50
  [FAIL] odo dev command tests port-forwarding for the component when devfile has single endpoint when running odo dev [BeforeEach] when modifying memoryLimit for container in Devfile should react on the Devfile modification (podman=true, manual=true, customPortForwarding=true) [podman]
  /home/asoro/work/projects/odo/tests/helper/helper_run.go:54

When running against a cluster, the tests pass, but looking deeper into the logs, we can see that port-forwarding did not actually start.

Logs on cluster

 ✓  Building your application in container (command: devbuild) [5s]                                                                                                                            
    [odo]  •  Executing the application (command: devrun)  ...                                                                                                                                     [odo]  •  Waiting for the application to be ready  ...                                                                                                                                     
 ✓  Waiting for the application to be ready [1s]                                                                                                                                               
    [odo] Unable to listen on port 30021: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:30021: bind: address already in use una
ble to create listener: Error listen tcp6 [::1]:30021: bind: address already in use]                                                                                                           
    [odo] Failed to setup port-forwarding: unable to listen on any of the requested ports: [{30021 3000}]                                                                                      
    [odo] Unable to listen on port 30021: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:30021: bind: address already in use una
ble to create listener: Error listen tcp6 [::1]:30021: bind: address already in use]
    [odo] Failed to setup port-forwarding: unable to listen on any of the requested ports: [{30021 3000}]
    [odo] Unable to listen on port 30021: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:30021: bind: address already in use una
ble to create listener: Error listen tcp6 [::1]:30021: bind: address already in use]
    [odo] Failed to setup port-forwarding: unable to listen on any of the requested ports: [{30021 3000}]
    [odo] Unable to listen on port 30021: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:30021: bind: address already in use una
ble to create listener: Error listen tcp6 [::1]:30021: bind: address already in use]
    [odo] Failed to setup port-forwarding: unable to listen on any of the requested ports: [{30021 3000}]

Expected behavior

We are currently relying on a global counter to get a free port to use as custom port, as we can see in helper_http.go.
The problem is that Ginkgo actually runs parallel specs in different processes, so they will have their own counter. See https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-runs-parallel-specs

A simple solution could be to rely on the index of the process using GinkgoParallelProcess().

@openshift-ci openshift-ci bot added kind/bug Categorizes issue or PR as related to a bug. area/testing Issues or PRs related to testing, Quality Assurance or Quality Engineering labels Apr 20, 2023
@github-actions github-actions bot added the needs-triage Indicates an issue or PR lacks a `triage/*` and requires one. label Apr 20, 2023
@rm3l rm3l moved this to In Review 👀 in odo Project Apr 21, 2023
@rm3l rm3l self-assigned this Apr 21, 2023
@rm3l rm3l removed the needs-triage Indicates an issue or PR lacks a `triage/*` and requires one. label Apr 21, 2023
@github-project-automation github-project-automation bot moved this from In Review 👀 to Done ✅ in odo Project Apr 25, 2023
@rm3l rm3l added this to the v3.10.0 🚀 milestone Apr 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/testing Issues or PRs related to testing, Quality Assurance or Quality Engineering kind/bug Categorizes issue or PR as related to a bug.
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant