Unpredictable errors when OkHttpClient.Builder instance is reused #445
Comments
|
Have you tried to disable gzip compression ? |
|
@majst01 my guess (and this is a loooooooong shot) is: a race condition that could happen when you reuse a custom Checking the OkHttpClient javadoc documentation, you see that |
|
so should we create the okhttpClient instance inside a synchronized block in a helper function and pass that Retrofit.Builder instead? @cantide5ga Can you reproduce this Stackoverflow ? And try a version from a branch ? |
|
@majst01 absolutely - let me know the branch. And as mentioned, the Stackoverflow comes and goes, but I think I can lock it down with a heavy load. re: gzip, I've already disabled and the result is the same. One thing a colleague and I were talking about is how |
|
You should share and use the same instance of InfluxDB, if you are creating many of them, this might be the root cause. Would you please try with a single instance of influxDB ? |
|
If this is the case we should at least document this fact, and bonus try to prevent the creation of multiple influxdb instances to the same endpoint. |
|
I thought this might be the case. This is an addition to some older code, so I'll need to determine how heavy an impact this refactor is. Standby folks, thanks for the help. |
|
Hi @cantide5ga any news here ? |
|
Apologies for being late here. IMO there is no way to fix the reported issue without removing the Errors will happen if we share a single A quick-and-dirty PoC showing the collateral effects: import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import org.influxdb.InfluxDB;
import org.influxdb.InfluxDBFactory;
import okhttp3.OkHttpClient;
public class Main {
public static void main(String[] args) throws InterruptedException, ExecutionException {
ExecutorService executor = Executors.newFixedThreadPool(100);
final int maxCallables = 10_000;
List<Callable<String>> callableList = new ArrayList<>(maxCallables);
for (int i = 0; i < maxCallables; i++) {
callableList.add(new Callable<String>() {
@Override
public String call() throws Exception {
MyInfluxDBBean myBean = new MyInfluxDBBean();
return myBean.connectAndDoNothing1(); //pick one
return myBean.connectAndDoNothing2(); //pick one
}
});
}
System.out.println("Invoking all callableList (size()=" + callableList.size() + ")");
executor.invokeAll(callableList);
System.out.println("Shutting down...");
executor.shutdown();
System.out.println("Shutdown requested and waiting for termination...");
if (!executor.awaitTermination(20_000, TimeUnit.SECONDS)) {
executor.shutdownNow();
}
}
public static class MyInfluxDBBean {
static final AtomicInteger COUNT = new AtomicInteger(0);
static final OkHttpClient.Builder OKHTTP_BUILDER = new OkHttpClient.Builder();
InfluxDB influxClient;
/**
* DO NOT USE THIS CODE! This method will throw a <pre>java.lang.IllegalStateException: Null interceptor</pre>.
*/
String connectAndDoNothing1() {
try {
influxClient = InfluxDBFactory.connect("http://127.0.0.1:8086", "admin", "supersecretpassword", OKHTTP_BUILDER); //LINE56
} catch (Exception e) {
e.printStackTrace();
System.exit(1); // OPS!
}
return null;
}
/**
* DO NOT USE THIS CODE!
*/
String connectAndDoNothing2() {
try {
synchronized (OKHTTP_BUILDER) {
influxClient = InfluxDBFactory.connect("http://127.0.0.1:8086", "admin", "supersecretpassword", OKHTTP_BUILDER);
}
if (COUNT.incrementAndGet() % 1000 == 0) {
System.out.println("connectAndDoNothing2: check how many interceptors the builder will use: " + OKHTTP_BUILDER.interceptors().size());
}
} catch (Exception e) {
e.printStackTrace();
System.exit(1); // OPS!
}
return null;
}
}
}Output when executing Output when executing What do you think, guys? |
|
I agree, seems multiple loggingInterceptor and gzipRequestInterceptor is likely to result in a super deep call stack (and StackOverflowError consequently). I am not sure if it's also the root cause of the SocketTimeotException, however when the system goes unstable (such as running out of memory), many kind of weird errors can happen. So we first should address that redundancy of loggingInterceptor and gzipRequestInterceptor to isolate the problems So how about these approaches
|
This will create a
This may work. Personally I don't like this approach because:
IMO, this "temporary fix" should be clearly stated in the source code (maybe a comment with link to this ticket). We don't want to scary anyone checking the source for the first time and finding this "workaround" without a good explanation. :) For the next major version, we should also remove the |
|
Hi @fmachado,
This is to keep the builder untouched so even this builder is going to be reused multiple times, the InfluxDBImpl constructors will not pile up its interceptor list Actually I prefer this approach most because it's still unclear these constructors can modify the builder (add more interceptors) or not, so try to keep it immutable and we can prevent bad side effects for user's code So I believe cloning the builder is not a workaround but a real fix. I am not sure by now if for next major release we should remove the Builder parameter or not because it gives users some level of customization for their very own network config (such as proxy, key store ...). |
|
@lxhoan OK. Let's move forward with (1). |
|
@majst01 et al, sorry for the month long MIA. We migrated anything impacted here to use a single instance and I haven't seen the problem since. Single instance of Willing to help in this aspect. Regarding the |
|
@cantide5ga , |
lxhoan
changed the title
|
I change the title to reflect the real discussion and the final fix |
influxdb-java version 2.10
There are two similar stack traces throwing different exceptions:
and
Common parts of the two is:
Sometimes I get one, sometimes the other. Sometimes not at all. Maybe a race depending on if the stack is exhausted before the timeout?
Here are the details of my code:
httpClientis a singleton, instantiated like so:I've also
.setLogLevel(InfluxDB.LogLevel.FULL)to gather more information, but immediately preceding the exception, I only get logs on the HTTP request and nothing abnormal.I found this related issue that didn't ever seem to get closure: #379
Again, this is often, but not always. I'm trying to find ways that I can consistently reproduce in order to be more helpful. The
StackOverflowErrorand recursive trace only started happening when I had to explicitly configure the influx timeout.The text was updated successfully, but these errors were encountered: