OutOfMemoryException because PerpetualCache.cache is too big

[rsiggi]

MyBatis version

3.4.5

Database vendor and version

Oracle 12.1.0.2 but not vendor specific

Test case or example project

We use MyBatis for big data migrations. The master select is in a separate thread and in the ResultHandler the data will be get out (pipe) to process the data. So MyBatis can read the next ResultSet when we process the current data. The master select will be always open. We also use nested selects, but all selects use a unique identification key. Actual the master select will be get 200.000 complex results.

The problem

Every result will be filled in BaseExecutor.queryFromDatabase in the localCache . This localCache will be cleared when the select is finished. To late, the program throws a OutOfMemoryException, because the HashMap in PerpetualCache is too big. MyBatis will never use this cache, because every call for selects (also nested) is unique!

Steps to reproduce

Expected result

It would be nice if it is possible to deactivate the localCache for all or specific statements. Or if I can set my one localCache over setter.

Actual result

I wrote a plugin which set (per refection) BaseExecutor.localCache with my one cache implementation

public static class NoCache extends PerpetualCache {

    public NoCache(String id) {
        super(id);
    }

    @Override
    public void putObject(Object key, Object value) {
    }
}

Not nice but it works for me.

[harawata]

Have you tried localCacheScope="STATEMENT"?
http://www.mybatis.org/mybatis-3/configuration.html#settings

[rsiggi]

Yes, remember I call

session.select("Statementname", args, new ResultHander(){
  public void handleResult(ResultContext context) {


  //here we processing every result.

   /* this method will be called 200.000 times and 
  for each result mybatis must call two or more nested selects.
  and so for each result the cache increases by at least 3 entries (arguments and results)
  in the end 600.000 (or more) cache entries for nothing or a OoM.
*/
  }
});
// all work is done.

The setting localCacheScope=STATEMENT had no effect, because that only works after the statement done, but this is too late.

[harawata]

Thanks for the follow-up, @rsiggi !
Check out #437 and see if you can use selectCursor() instead.

[rsiggi]

You are right. I should use selectCursor() instead.
Thanks for the fast solution.