QUICHE_ERR_STREAM_LIMIT with Jetty 12.1.2 on HTTP/3

[Horcrux7]

Jetty version(s)
12.1.2

Jetty Environment
ee11

HTTP version
HTTP 3

Java version/vendor (use: java -version)
21, OpenJDK

OS type/version
Windows

Description
If I try to connect to the Jetty Server I get a QUICHE_ERR_STREAM_LIMIT since version 12.1.x. With version 12.0.x it has worked. You can test it with the jetty client or a curl command. In the debug log from Jetty server I can't see any helpful error for me.

Curl command:

docker run --rm --add-host=my.domain.com:10.10.7.13 ymuski/curl-http3 curl --http3 --verbose https://my.domain.com/

Jetty client code:

    public void run( String server ) throws Exception {
        Immutable httpURI = HttpURI.from( "https://" + server + "/" );
        int port = httpURI.getPort();
        if( port < 0 ) {
            port = 443;
        }

        QuicheClientQuicConfiguration clientQuicConfig = HTTP3ClientQuicConfiguration.configure( new QuicheClientQuicConfiguration() );
        try (HTTP3Client http3Client = new HTTP3Client( clientQuicConfig )) {
            http3Client.start();

            SocketAddress serverAddress = new InetSocketAddress( httpURI.getHost(), port );
            Session.Client session = Blocker.blockWithPromise( 5, TimeUnit.SECONDS, p -> http3Client.connect( new QuicheTransport( clientQuicConfig ), serverAddress, new Session.Client.Listener() {
                @Override
                public void onDisconnect( Session session, long error, String reason ) {
                    System.err.println( "onDisconnect: " + reason );
                }
                public void onFailure( Session session, long error, String reason, Throwable failure ) {
                    failure.printStackTrace();
                    p.failed( failure );  <-- failing
                }
            }, p ) );

            HttpFields requestHeaders = HttpFields.build();
            MetaData.Request request = new MetaData.Request( "GET", httpURI, HttpVersion.HTTP_3, requestHeaders );
            HeadersFrame headersFrame = new HeadersFrame( request, true );

            ThreadAssert.init( 1 );
            session.newRequest( headersFrame, new Stream.Client.Listener() {
                @Override
                public void onResponse( Client stream, HeadersFrame frame ) {
                    MetaData.Response responseMetaData = (MetaData.Response)frame.getMetaData();
                    int statusCode = responseMetaData.getStatus();
                    if( statusCode != 200 ) {
                        ThreadAssert.fail( new IOException( "StatusCode: " + statusCode ) );
                    } else {
                        ThreadAssert.countDown();
                    }
                }

                @Override
                public void onFailure( Client stream, long error, Throwable failure ) {
                    ThreadAssert.fail( failure );
                }
            }, Promise.Invocable.noop() );
            ThreadAssert.await();
        }
    }

The client exception is:

java.io.IOException: failed to write to stream 10; quiche_err=QUICHE_ERR_STREAM_LIMIT
	at org.eclipse.jetty.quic.quiche.jna.JnaQuicheConnection.feedClearBytesForStream(JnaQuicheConnection.java:701)
	at org.eclipse.jetty.quic.quiche.QuicheSession.data(QuicheSession.java:377)
	at org.eclipse.jetty.quic.quiche.QuicheStream.write(QuicheStream.java:304)
	at org.eclipse.jetty.quic.quiche.QuicheStream.data(QuicheStream.java:276)
	at org.eclipse.jetty.quic.common.StreamEndPoint.write(StreamEndPoint.java:447)
	at org.eclipse.jetty.http3.ControlFlusher.process(ControlFlusher.java:106)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:377)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:354)
	at org.eclipse.jetty.http3.client.internal.ClientHTTP3Session.onStart(ClientHTTP3Session.java:173)
	at org.eclipse.jetty.quic.common.ProtocolSession.doStart(ProtocolSession.java:81)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:170)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.addManaged(ContainerLifeCycle.java:483)
	at org.eclipse.jetty.quic.quiche.client.QuicheTransport$ProtocolSessionListener.onOpen(QuicheTransport.java:117)
	at org.eclipse.jetty.quic.common.AbstractSession.notifyOpen(AbstractSession.java:112)
	at org.eclipse.jetty.quic.common.AbstractSession.notifyOpen(AbstractSession.java:105)
	at org.eclipse.jetty.quic.common.AbstractSession.emitOpen(AbstractSession.java:69)
	at org.eclipse.jetty.quic.quiche.QuicheSession.open(QuicheSession.java:451)
	at org.eclipse.jetty.quic.quiche.client.internal.ClientQuicheConnection.onFillable(ClientQuicheConnection.java:217)
	at org.eclipse.jetty.quic.quiche.QuicheConnection$FillableCallback.succeeded(QuicheConnection.java:81)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:54)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:492)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.epcRunTask(AdaptiveExecutionStrategy.java:428)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:401)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:255)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:1009)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1239)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1194)
	at java.base/java.lang.Thread.run(Thread.java:840)

The Jetty server code look like:

import org.eclipse.jetty.http3.server.HTTP3ServerConnectionFactory;
import org.eclipse.jetty.quic.quiche.Quiche;
import org.eclipse.jetty.quic.quiche.QuicheConnection;
import org.eclipse.jetty.quic.quiche.server.QuicheServerConnector;
import org.eclipse.jetty.quic.quiche.server.QuicheServerQuicConfiguration;
import org.eclipse.jetty.server.AbstractConnector;
import org.eclipse.jetty.server.HttpConfiguration;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.util.ssl.SslContextFactory;

    public AbstractConnector addHttp3Connector( @Nonnull Server server, @Nonnull ListenerConnector listenerConnector, @Nonnull HttpConfiguration httpConfig, @Nonnull SslContextFactory.Server sslContextFactory ) throws IOException {
        ClassLoader classLoader = getClass().getClassLoader();
        Thread.currentThread().setContextClassLoader( classLoader ); // for service loader in QuicheConnection
        try {
            Class.forName( QuicheConnection.class.getName(), true, classLoader );
            Class.forName( Quiche.class.getName(), true, classLoader ); // force init the ServiceLoader
        } catch( Throwable ex ) {
            ServerPluginManager.getInstance().setPluginLoadError( Http3ServerPlugin.PLUGIN_ID, ex );
            LOGGER.error( ex );
            return null;
        }

        Path pemDir = Path.of( System.getProperty( "user.home" ), ".ssh", "inet" );
        Files.createDirectories( pemDir );
        QuicheServerQuicConfiguration quicConfiguration = new QuicheServerQuicConfiguration( pemDir );

        HTTP3ServerConnectionFactory http3 = new HTTP3ServerConnectionFactory( httpConfig );
        QuicheServerConnector http3Connector = new QuicheServerConnector( server, sslContextFactory, quicConfiguration, http3 );
        http3Connector.setPort( listenerConnector.getPort() );
        String bindAddress = listenerConnector.getBindAddress();
        if( bindAddress != null && bindAddress.length() > 0 ) {
            http3Connector.setHost( bindAddress );
        }

        server.addConnector( http3Connector );
        return http3Connector;
    }

How to reproduce?

Here is reproducible app for 7 days. In the zip file is a readme with some configuration steps to start.

[sbordet]

@Horcrux7 you are missing the HTTP/3 configuration on the server, like you have done in the client.

On the client:

QuicheClientQuicConfiguration clientQuicConfig = HTTP3ClientQuicConfiguration.configure( new QuicheClientQuicConfiguration() );

On the server:

QuicheServerQuicConfiguration quicConfiguration = new QuicheServerQuicConfiguration( pemDir );

You should do this instead:

QuicheServerQuicConfiguration serverQuicConfiguration = HTTP3ServerQuicConfiguration.configure(new QuicheServerQuicConfiguration( pemDir ));

Let us know if it was just that.

The reason for the extra step if that it is possible to configure a QuicheServerConnector to speak just QUIC, but not HTTP/3.
When speaking just QUIC, using QuicheServerQuicConfiguration is enough.
But for HTTP/3, the QUIC configuration needs to be changed (for example to allow unidirectional streams that are not used by QUIC, but they are used by HTTP/3).

We have tests that speak HTTP/1 over QUIC, and do not need the HTTP/3 configuration.

[sbordet]

And thank you for this issue, because I just discovered that the documentation is wrong, and also few examples and tests.

Doh! 😃

PR incoming.

[Horcrux7]

Yes, this was the trick. My short test are working. Thank you very much. You can close the ticket.