Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

port in use errors in CI #3848

Open
busma13 opened this issue Dec 2, 2024 · 7 comments
Open

port in use errors in CI #3848

busma13 opened this issue Dec 2, 2024 · 7 comments
Assignees

Comments

@busma13
Copy link
Contributor

busma13 commented Dec 2, 2024

We keep getting port in use errors.
I would like to try using ss to list all ports in TIME_WAIT to see if they are not being shut down fast enough.

The other option is to assign random ports.

Example failures:
https://github.com/terascope/teraslice/actions/runs/11981509936/job/33407933870

● Test suite failed to run

    listen EADDRINUSE: address already in use :::55678

      140 |             await new Promise((resolve, reject) => {
      141 |                 // @ts-expect-error
    > 142 |                 this.app.listen(this.port, (err: Error) => {
          |                          ^
      143 |                     if (err) {
      144 |                         reject(err);
      145 |                         return;

      at Function.apply [as listen] (node_modules/express/lib/application.js:635:24)
      at listen (packages/teraslice/src/lib/cluster/services/assets.ts:142:26)
      at AssetsService.initialize (packages/teraslice/src/lib/cluster/services/assets.ts:140:19)
      at Object.<anonymous> (packages/teraslice/test/services/assets-spec.ts:67:9)

https://github.com/terascope/teraslice/actions/runs/11941437795/job/33286764202

docker: Error response from daemon: driver failed programming external connectivity on endpoint ts_test_kafka (fb60d1c2428b6a3ad031693a716c1897da234a8e33577b500cd45bd8242830ae): Error starting userland proxy: listen tcp4 0.0.0.0:49092: bind: address already in use.
@busma13 busma13 self-assigned this Dec 2, 2024
@busma13
Copy link
Contributor Author

busma13 commented Dec 3, 2024

I have found port related errors in 3 different spots:

  • assets service tests
  • teraslice messaging server
  • docker run of service images

These 3 different areas may need to be handled separately.

@busma13
Copy link
Contributor Author

busma13 commented Dec 3, 2024

The teraslice package has a findPort() function that some of the tests use. It returns a random open port. I am adding this to assets-spec.ts, which should resolve the first list item in my previous comment.

@busma13
Copy link
Contributor Author

busma13 commented Dec 3, 2024

https://github.com/terascope/teraslice/actions/runs/11921771709/job/33226709312

This failure uses the findPort() function. I am guessing multiple tests run this function at the same time and there end up being collisions. We use porty to test if the port is open, but it looks like we start at the same port for all these tests and move up until porty says one is open. Maybe 2 tests get assigned the same open port at the same time.

@busma13
Copy link
Contributor Author

busma13 commented Dec 5, 2024

findPort() allows you to set a port range, so I am giving each test a unique range to prevent collisions.

godber pushed a commit that referenced this issue Dec 16, 2024
This PR makes the following changes:
- Adds the `--verbose` flag to the root package.json `build` script and
removes `--silent` from `YARN_SETUP_ARGS`. This will hopefully reveal
why the script fails occasionally. ref: #3849
- Replace unmaintained `porty` library with `get-port` library.
- Within the teraslice package, many tests use `findPort()` to get a
random port to start a server on. The implementation did not actually
return a random port and made conflicts more likely. Update `findPort()`
to use `get-port`, increase the port range, and get a truly random port
in that range. ref: #3848
- When starting services, log all TCP ports just before calling `docker
run`. This should give us more information about why the `docker run`
commands occasionally fail with a `port already in use` error. ref:
#3848
@godber
Copy link
Member

godber commented Dec 18, 2024

Have there been fewer or zero port errors on tests after this was merged?

@godber
Copy link
Member

godber commented Dec 18, 2024

Or has the added netstat command helped narrow trouble down.

@busma13
Copy link
Contributor Author

busma13 commented Dec 18, 2024

There haven't been too many CI runs, but no port errors yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants