Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ephemeral ports are exhausted if stopping the DAS takes a long time #25292

Open
tnagao7 opened this issue Dec 18, 2024 · 2 comments · May be fixed by #25293
Open

Ephemeral ports are exhausted if stopping the DAS takes a long time #25292

tnagao7 opened this issue Dec 18, 2024 · 2 comments · May be fixed by #25293

Comments

@tnagao7
Copy link
Member

tnagao7 commented Dec 18, 2024

Environment Details

  • GlassFish Version (and build number): Eclipse GlassFish 7.0.20
  • JDK version: OpenJDK 21.0.4
  • OS: Windows 10
  • Database: none

Problem Description

If it takes a long time to stop the DAS, many ephemeral ports are opened.
In the worst case, the ephemeral ports provided by the operating system are exhausted.

Steps to reproduce

  1. Compile the following lifecycle listener including a long-running task.
import com.sun.appserv.server.LifecycleEvent;
import com.sun.appserv.server.LifecycleListener;
import com.sun.appserv.server.ServerLifecycleException;

public class TestLifecycleListener implements LifecycleListener {
    @Override
    public void handleEvent(LifecycleEvent event) throws ServerLifecycleException {
        if (event.getEventType() == LifecycleEvent.SHUTDOWN_EVENT) {
            try {
                // do long-running task
                Thread.sleep(15000);
            } catch (InterruptedException e) {
            }
        }
    }
}
> javac -cp C:\glassfish7\glassfish\modules\glassfish-api.jar TestLifecycleListener.java
  1. Enable the lifecycle listener.
> asadmin start-domain
> asadmin create-lifecycle-module --classname TestLifecycleListener --classpath C:/work/lifecycle TestLifecycleListener
> asadmin stop-domain
> asadmin start-domain
  1. Stop the DAS.
> asadmin stop-domain
Waiting for the domain to stop ...........
Waiting finished after 15,668 ms.
Command stop-domain executed successfully.
  1. Just after stoppiong the DAS, by checking netstat, we can see that many ephemeral ports are consumed.
> netstat -ano
-- snip --
  TCP         127.0.0.1:49152        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:49153        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:49154        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:49155        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:49156        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:49157        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:49158        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:49159        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:49160        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:49161        127.0.0.1:4848         TIME_WAIT       0
-- snip --
  TCP         127.0.0.1:65526        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:65527        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:65528        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:65529        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:65530        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:65531        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:65532        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:65533        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:65534        127.0.0.1:4848         TIME_WAIT       0
  TCP         127.0.0.1:65535        127.0.0.1:4848         TIME_WAIT       0

Impact of Issue

If the ephemeral ports are exhausted, new TCP connections requiring ephemeral ports cannot be made.
@OndroMih
Copy link
Contributor

Do you know how the connections are created? Are there quest coming to the server while it’s shutting down?

@tnagao7
Copy link
Member Author

tnagao7 commented Dec 18, 2024

The connections are repeatedly created here while waiting for server shutdown:

glassfish/nucleus/admin/server-mgmt/src/main/java/com/sun/enterprise/admin/servermgmt/cli/StopDomainCommand.java

Line 207 in 6f90816

deathSign = () -> !ProcessUtils.isListening(addr) && !ProcessUtils.isAlive(watchedPid);

I'm preparing a pull request to fix this issue

@tnagao7 tnagao7 linked a pull request Dec 18, 2024 that will close this issue
Open
@dmatej dmatej mentioned this issue Dec 28, 2024
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix #25292 Ephemeral ports are exhausted if stopping the DAS takes a long time tnagao7/glassfish
2 participants
@OndroMih @tnagao7