[CI] TransportServiceHandshakeTests testConnectToNodeLight failing

[pgomulka]

Build scan:
https://gradle-enterprise.elastic.co/s/ry5q53b56iugw/tests/:server:test/org.elasticsearch.transport.TransportServiceHandshakeTests/testConnectToNodeLight

Reproduction line:
./gradlew ':server:test' --tests "org.elasticsearch.transport.TransportServiceHandshakeTests.testConnectToNodeLight" -Dtests.seed=F102EBFC20BCB1FB -Dtests.locale=et -Dtests.timezone=Africa/Mogadishu -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.transport.TransportServiceHandshakeTests&tests.test=testConnectToNodeLight

Failure excerpt:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=820, name=Thread-89, state=RUNNABLE, group=TGRP-TransportServiceHandshakeTests]

  at __randomizedtesting.SeedInfo.seed([F102EBFC20BCB1FB:4BFF542966FA3AD0]:0)

  Caused by: java.lang.AssertionError: cluster:monitor/state

    at __randomizedtesting.SeedInfo.seed([F102EBFC20BCB1FB]:0)
    at org.elasticsearch.transport.InboundAggregator.lambda$new$0(InboundAggregator.java:46)
    at org.elasticsearch.transport.InboundAggregator.initializeRequestState(InboundAggregator.java:197)
    at org.elasticsearch.transport.InboundAggregator.headerReceived(InboundAggregator.java:66)
    at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:138)
    at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
    at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
    at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[DaveCTurner]

Likely duplicates #85589 and I believe the problem is environmental. Relabelling this :Distributed/Network to collect them together.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

This looks to be caused by different test workers using the same port ranges. #85777 will mostly mitigate this, and #85786 tracks work towards a more robust fix. I'm closing this as the mitigation is now in place, and there's nothing particularly special about the tests involved in this failure which needs further attention.