RetryService threads seem to be leaked

[ClaireAB]

We're using Astyanax 2.0.1 to connect to a Cassandra database with multiple keyspaces. We use a separate AstyanaxContext to connect to each keyspace.

I have recently run some testing where I block the traffic to the server running Cassandra periodically and then recover it while making requests to the database continuously. (This testing was prompted by a problem where a server hung when it could not talk to the Cassandra server).

After running this for 4 days I did a Java thread dump and there were 185 RetryService threads. I reconnected the Cassanadra server and stopped my stress test running. A few hours later these threads were still there. So nothing has cleaned them up and they seem to be leaking.

Is this an Astyanax bug or is there something we should be doing to clean them up?

We also have code that shuts the context down when we notice that the seeds for the connection have changed. We then recreate the context from scratch. Is this the correct way to do this or can we change the seeds without shutting down the context? Is it possible that the RetryService threads are related to the contexts that have been shutdown and if so how do I clean them up?

Here is the code we use to build the context:

ConnectionPoolConfigurationImpl poolConfiguration =
new ConnectionPoolConfigurationImpl(MVS_OBS_POOL)
.setPort(DB_PORT)
.setMaxConnsPerHost(1)
.setSeeds(sSeeds);

Builder builder = new AstyanaxContext.Builder()
  .forKeyspace(keyspace)
  .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
    .setDiscoveryType(NodeDiscoveryType.RING_DESCRIBE)
    .setDefaultWriteConsistencyLevel(ConsistencyLevel.CL_LOCAL_ONE)
    .setDefaultReadConsistencyLevel(ConsistencyLevel.CL_LOCAL_ONE)
    .setConnectionPoolType(ConnectionPoolType.TOKEN_AWARE))
  .withConnectionPoolConfiguration(poolConfiguration)
  .withConnectionPoolMonitor(new CountingConnectionPoolMonitor());

AstyanaxContext<Keyspace> context = builder.buildKeyspace(ThriftFamilyFactory.getInstance());