prepared query errors on multi-node Cassandra cluster

[ryane]

I am trying to deploy Kong (0.5.4) and am using a 3-node cassandra cluster that is registered in consul. Here is my database-related configuration.

databases_available:
  cassandra:
    properties:
      contact_points:
        - "cassandra-kong-node.service.consul:9042"
      timeout: 1000
      keyspace: kong
      keepalive: 60000 # in milliseconds

And here is how the host resolves:

host cassandra-kong-node.service.consul
cassandra-kong-node.service.consul has address 10.0.0.6
cassandra-kong-node.service.consul has address 10.0.0.9
cassandra-kong-node.service.consul has address 10.0.0.5

I can confirm that cassandra is listening on 9042 on all of those nodes.

With this configuration, Kong intermittently is returning errors. For example, here is the output when making requests against the Admin API:

$ curl gce-worker-004:4833
{"message":"An unexpected error occurred"}
$ curl gce-worker-004:4833
{"version":"0.5.4","lua_version":"LuaJIT 2.1.0-alpha","tagline":"Welcome to Kong","hostname":"b91b2bb85f1b","plugins":{"enabled_in_cluster":{},"available_on_server":["ssl","jwt","acl","cors","oauth2","tcp-log","udp-log","file-log","http-log","key-auth","hmac-auth","basic-auth","ip-restriction","mashape-analytics","request-transformer","response-transformer","request-size-limiting","rate-limiting","response-ratelimiting"]}}
$ curl gce-worker-004:4833
{"message":"An unexpected error occurred"}

In /usr/local/kong/logs/error.log, I see lots of error like:

2016/01/18 20:22:15 [error] 60#0: *1 [lua] responses.lua:61: Cassandra returned error (Unprepared): Prepared query with ID c5bd483fc757a66975a3c297d62407e7 not found (either the query was not prepared on this host (maybe the host has been restarted?) or you have prepared too many queries and it has been evicted from the internal cache), client: 10.0.0.2, server: , request: "GET / HTTP/1.1", host: "gce-worker-001.node.consul:25038"

This looks somewhat similar to #660 but this is a little different of a scenario, I think.

When I use the exact same configuration for a single-node Cassandra instance, Kong runs fine. I'm pretty new to both Kong and Cassandra - does anyone know what is wrong with this setup?

[thibaultcha]

This is a known issue and has been resolved in 0.6.

[thibaultcha]

(It is not entirely related to #660 but the new driver provides much better support for prepared queries which also has the nice side effect of fixing those occasional errors)

[ryane]

Thanks, I just had a chance to test with 0.6.0rc3 and it is working much better.