Async servlet failed to send large data frames completely when HTTP/2 trailers are present

[dapengzhang0]

I tried to let a async servlet send out a sequence of large data (40k each) followed by trailers. The client does receive the trailer, but only receives partial data.

• This issue does not happen if I use Undertow 2.0.12 instead of Glassfish 5.0.
• The issue also does not happen if I Thread.sleep for 100ms (the shorter the time is, the more flaky the result will be) before asyncContext.complete()
• This issue also does not happen if only sending out a few bytes.
• The servlet will send out data frames completely if I do not send trailers.

Steps to reproduce:

1. Deploy the following servlet with glassfish 5.0

import java.io.IOException;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Supplier;
import javax.servlet.AsyncContext;
import javax.servlet.ServletOutputStream;
import javax.servlet.WriteListener;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@WebServlet(urlPatterns = {"/large_output_with_trailer"}, asyncSupported = true)
public class LargeOutputWithTrailer extends HttpServlet {
  @Override
  protected void doGet(HttpServletRequest req, HttpServletResponse resp)
      throws IOException {
    AsyncContext asyncCtx = req.startAsync();

    resp.setStatus(HttpServletResponse.SC_OK);
    resp.setContentType("application/binary");
    resp.setHeader("trailers", "hello");

    final TrailerSupplier trailerSupplier = new TrailerSupplier();
    resp.setTrailerFields(trailerSupplier);
    resp.flushBuffer();

    final ServletOutputStream outputStream = resp.getOutputStream();
    outputStream.setWriteListener(new WriteListener() {
      int i;

      @Override
      public void onWritePossible() throws IOException {

        System.out.println("onWritePossible");
        while (outputStream.isReady()) {
          if (i < 3) {
            byte[] bytes = new byte[40000];
            outputStream.write(bytes);
            i++;
          } else {
            // Allow more time to let buffer clear out can workaround the issue
            // try {
            //   Thread.sleep(100);
            // } catch (InterruptedException e) {
            // }

            System.out.println("set trailers");
            trailerSupplier.get().put("hello", "world");
            System.out.println("complete");
            asyncCtx.complete();
            return;
          }
        }

        System.out.println("not ready");
      }

      @Override
      public void onError(Throwable t) {
        t.printStackTrace();
      }
    });
  }

  private static final class TrailerSupplier implements Supplier<Map<String, String>> {
    final Map<String, String> trailers = new ConcurrentHashMap<>();

    @Override
    public Map<String, String> get() {
      return trailers;
    }
  }
}

2. Run HTTP/2 client to send a POST request

nghttp "http://localhost:8080/large_output_with_trailer" -v

Console output:

[  2.445] recv DATA frame <length=16383, flags=0x00, stream_id=13>
[  2.445] recv DATA frame <length=16383, flags=0x00, stream_id=13>
[  2.445] recv DATA frame <length=7234, flags=0x00, stream_id=13>
[  2.445] recv DATA frame <length=16383, flags=0x00, stream_id=13>
[  2.445] recv DATA frame <length=9152, flags=0x00, stream_id=13>
[  2.445] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
          (window_size_increment=40000)
[  2.445] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=13>
          (window_size_increment=40000)
[  2.446] recv DATA frame <length=14465, flags=0x00, stream_id=13>
[  2.455] recv DATA frame <length=16383, flags=0x00, stream_id=13>
[  2.456] recv DATA frame <length=9152, flags=0x00, stream_id=13>
[  2.456] recv (stream_id=13) hello: world
[  2.456] recv HEADERS frame <length=11, flags=0x05, stream_id=13>
          ; END_STREAM | END_HEADERS
          (padlen=0)
[  2.456] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])

3. Sum up the lengths of DATA frames the client received:
   Only 105535 bytes were received, expecting 120000 bytes to be received.
   If you try multiple times, occasionally the client can receive all the 120000 bytes, but it's rare.

[MattGill98]

Hi @dapengzhang0,

After an initial investigation, I just wanted to clarify my understanding of the problem and make sure we're on the same wavelength!

The total data bytes received before a closed connection seem to mostly be one of three values: 65535, 105535, or 120000. If it's 120000, it means all data was received. 65535 is the default window size, and 105535 as per the log you posted means the window size was incremented once (by the client). From what I can tell on the server end, the data seems to be getting sent before the client has a chance to increase the window size, so the client is disregarding this data. I'm looking into why this data is not being held until a window size update. Is that similar to your own diagnosis?

Kind regards,

Matt

[dapengzhang0]

@MattGill98 Thanks you very much for the investigation. I actually don't know what is really happening under the hood. I'm just experiencing data corruption when the Servlet on Glassfish sends large data followed by trailers. This is observed with Netty client as well as nghttp client. If there are no trailers, or the trailers are sent a bit later after the payload like with a 100 ms delay, then the data that the client receives will not be corrupted.

[MattGill98]

Hi @dapengzhang0,

No worries at all, that's fine! I'm fairly sure I know where it's coming from. I'll let you know when I get any further!

Kind regards,

Matt

[MattGill98]

Hi @dapengzhang0,

A pull request for a potential fix has been made here: payara#13.

Once it's been verified and merged to Payara, we can work on making an upstream PR.

Kind regards,

Matt

[benmccann]

The fix has been merged to Payara. Can it be ported back upstream here now?