HttpConnection#prepareByteData() : should close bodyStream to reuse socket connection

[ghost]

It is significant when crawling maven repo index,

• reuse socket: 0.3s/page
• open a new socket: 1s/page

---

Will reuse the socket connection

Connection conn = HttpConnection.connect("http://localhost");
conn.execute().parse();
conn.url("http://localhost/index.html");
conn.execute().parse();

Will close and open a new socket connection

Connection conn = HttpConnection.connect("http://localhost");
conn.execute().body();
conn.url("http://localhost/index.html");
conn.execute().body();

• parse() invoke DataUtil.parseInputStream, which close the bodyStream;
• body() invoke DataUtil.readToByteBuffer, which not close the bodyStream.

If the InputStream closed & the host not changed, java.net.HttpURLConnection will reuse the socket, though invoked disconnect().

---

I do the following change:

private void prepareByteData() {
  Validate.isTrue(executed,
      "Request must be executed (with .execute(), .get(), or .post() before getting response body");
  if (byteData == null) {
    Validate.isFalse(inputStreamRead, "Request has already been read (with .parse())");
    try {
      byteData = DataUtil.readToByteBuffer(bodyStream, req.maxBodySize());
    } catch (IOException e) {
      throw new UncheckedIOException(e);
    } finally {
      inputStreamRead = true;
      ////////////////////////////// start
      try {
        bodyStream.close();
      } catch (IOException e) {
        // no-op
      } finally {
        bodyStream = null;
      }
      ////////////////////////////// end
      safeClose();
    }
  }
}

[ghost]

a test case

import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;

public class URLConnectionDemo {

  public static void main(String[] args) throws IOException {
    fetch_url("http://localhost/");
    fetch_url("http://localhost/index.html");
  }

  public static void fetch_url(String link) throws IOException {
    URL url = new URL(link);
    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
    InputStream in = conn.getInputStream();
    Files.copy(in, Paths.get("temp"), StandardCopyOption.REPLACE_EXISTING);
    in.close();
    conn.disconnect();
  }
}

File Server: My Java Server https://github.com/h4675/web

run: java wjs.sample.Debug

[ghost]

a better way: close bodyStream first in safeClose()

private void safeClose() {
  if (bodyStream != null) {
    try {
      bodyStream.close();
    } catch (IOException e) {
      // no-op
    } finally {
      bodyStream = null;
    }
  }
  if (conn != null) {
    conn.disconnect();
    conn = null;
  }
}

[jhy]

Thanks!