JDK 21 - Compliance Client's method handlers not receiving the entire request body

[lqiu96]

We are trying to include testing for JDK 21 and our tests are flaky for the Compliance client. We believe the change is behavior is because of some internal JDK changes updated how the HttpRequests are sent over the network.

In JDK 17 and prior, the internal code had a byte buffer of ~8k bytes but the new changes start the capacity at 512 bytes. We think that this change made it so that it now takes multiple "writes" over the stream to send data over (i.e. it used to be able to be sent in one "write", but now takes 2+ "writes")

We believe that the golang io.Reader's Read() call does not block until all the "writes" have finished. Specifically this line:

gapic-showcase/server/genrest/compliance.go

Line 57 in f0e32bc

bodyReader := io.TeeReader(r.Body, &jsonReader)

would Read and it is possible to only Read the first "write" (given that multiple "writes" could be made for a large request). We have logged the numBytes read (https://pkg.go.dev/io#Reader) and see that some messages see numBytes less than the expected value.

Environment details

• Programming language: Java
• OS: Mac + Linux
• Language runtime version: JDK 21 (I believe this is also an issue from JDK 19+)

Logs from Gapic Showcase Server

Error: 2024/07/01 17:05:20 error reading body params '*': proto: syntax error (line 1:513): invalid value

2024/07/01 17:08:59   urlRequestHeaders:
    User-Agent: "Google-HTTP-Java-Client/1.44.2 (gzip)"
    X-Goog-Api-Client: "gl-java/21.0.1__Eclipse-Adoptium__Temurin-21.0.1-12 gapic/ gax/2.50.1-SNAPSHOT rest/"
    Accept: "*/*"
    Content-Length: "514"
    Accept-Encoding: "gzip"
    Content-Type: "application/json; charset=utf-8"
    Connection: "keep-alive"
2024/07/01 17:08:59 Num Bytes Read: 512

The Content-Length header says there are 514 bytes, but the reader only reports that 512 bytes are read.

The logs above are from this updated snippet in th showcase server:

	var jsonReader bytes.Buffer
	bodyReader := io.TeeReader(r.Body, &jsonReader)
	rBytes := make([]byte, r.ContentLength)
	numBytesRead, err := bodyReader.Read(rBytes)
	if err != nil && err != io.EOF {
		backend.Error(w, http.StatusBadRequest, "error reading body content: %s", err)
		return
	}

	backend.StdLog.Printf("Num Bytes Read: %d", numBytesRead)

Reproducer

Compliance ITs: https://github.com/googleapis/sdk-platform-java/blob/308aeafc9f04795d2e1df8206c84689b11c4323a/showcase/gapic-showcase/src/test/java/com/google/showcase/v1beta1/it/ITHttpAnnotation.java#L123-L142

Run with JDK 21

Possible Solution

Update the clients to use io.ReadAll(r.body) when reading the request body.

We have made this change locally and ran the Java Compliance ITs 1000x. We do not see this flaky test anymore and have a strong suspicion that this should fix it.

[lqiu96]

Resolved in #1520