Client Node-Discovery and Auto-configuration (aka "autoconfig")

[seancribbs]

Several other clustered datastores (including Couchbase/Membase,
Amazon DynamoDB, MongoDB with the "config server") have ways of
discovering new nodes in the cluster. Since any Riak node can handle
any request, we should be able to advertise the existence of other
nodes to clients so that they can reconfigure automatically, spreading
load more evenly among the Riak nodes. This can reduce operational
burden, because applications will not need to be restarted to take
advantage of a grown cluster. This will also pave the way for the
future work of preflist-hinting or client-side routing.

Client-Server Interaction

The goal of this proposal is to provide as much backward- and forward-
compatibility as possible. As such, existing request/response cycles
are overloaded with new information; the "/" URL on HTTP and the
RpbGetServerInfoResp will be modified for this purpose.

In addition to the one-roundtrip cycle, clients should be able to
stream updates by providing an additional request parameter or
limiting the acceptable media-types. This allows them to receive
cluster changes as they occur and update accordingly, perhaps using a
background thread or reactor loop. It also removes the need to send
reconfiguration messages in-band with other requests.

Implementation Details

PB

The RpbGetServerInfoResp would be modified to add information about
other nodes in the cluster, and RpbGetServerInfoReq would be made a
full message:

// Added message
message RpbGetServerInfoReq {
    optional bool stream = 1 [default=false];
}

message RpbGetServerInfoResp {
    optional bytes node = 1;
    optional bytes server_version = 2;
    // Added field
    repeated RpbNode nodes = 3;
}

message RpbNode {
    // The node name, same as what would be in the original RpbGetServerInfoResp.
    required bytes name = 1;
    repeated RpbClientInterface interfaces = 2;
}

message RpbClientInterface {
    enum RpbProtocol {
        PB = 1;
        HTTP = 2;
        HTTPS = 3;
    }
    required RpbProtocol protocol = 1;
    // This can be a FQDN or a text representation of an IP address.
    required bytes host = 2;
    required uint32 port = 3;    
}

HTTP

The "root" resource riak_core_wm_urlmap would be modified to return
additional information:

1. A link (in the HTML format) with relation "self" that identifies
   the URL of the current node and its name:

   <a href="http://10.0.1.150:8098/" rel="self">riak@riak-dc.bos1.basho.com</a>

   This could also be represented in a similar fashion in the Link
   response header. If the node also has PB configured, it should
   include similar links using the "alternate" relation.

2. Links for other known nodes in the cluster, listed in the
   same fashion as the current node but using the "alternate"
   relation.

3. If given the query string parameter stream=true, then updates to
   the configuration will be streamed back to the client.

The JSON format of riak_core_wm_urlmap would change like so:

{
  "riak_kv_wm_buckets": "/riak",
  "riak_kv_wm_counter": "/buckets",
  "riak_kv_wm_index": "/buckets",
  "riak_kv_wm_keylist": "/buckets",
  "riak_kv_wm_link_walker": "/riak",
  "riak_kv_wm_mapred": "/mapred",
  "riak_kv_wm_object": "/riak",
  "riak_kv_wm_ping": "/ping",
  "riak_kv_wm_props": "/buckets",
  "riak_kv_wm_stats": "/stats",
  "riak_solr_indexer_wm": "/solr",
  "riak_solr_searcher_wm": "/solr",
  // Added below here
  "riak@riak-dc.bos1.basho.com":{
      "interfaces":[
          "http://10.0.1.150:8098/",
          "pbc://10.0.1.150:8087/"
      ]
  }, 
  "riak@riak-dc2.bos1.basho.com":{
      "interfaces":[
          "http://10.0.1.151:8098",
          "pbc://10.0.1.151:8087"
      ]
  }
  // u.s.w.
}    

Server Implementation

A new dedicated process on each node, probably called
riak_api_autoconfig, will be added to broadcast, cache, and serve
listener information among the cluster members. When receiving an
update from another node, the process will update its internal cache
and notify client-socket processes if the state has changed. Client
processes will filter the information according to the addresses that
are reachable from the peer (CIDR).

Existing code from riak_repl and riak_core will be repurposed or
refactored to support detecting and filtering interfaces.
Additionally, base HTTP support will likely be moved into riak_api
so that client-interface configuration can live in a single place.

TBD: Autoconfig information might also be affected by authorization, see #355.

Risks/Problems

1. User/sysadmin can configure the client to connect to Riak through a
   proxy-load-balancer (e.g. HAProxy), in a sense making this feature
   unneeded. The client should have an option to turn off or ignore
   autoconfig, as well as disabling it server-side.
2. Hostname/IP mappings should be identified at startup, or
   inet_gethost calls could needlessly dominate, assuming someone
   bound an interface to a hostname rather than IP address. This could
   get even stickier if administrators use round-robin DNS or VIP,
   in which case autoconfig should likely be disabled.
3. We should be careful about sending autoconfig information over the
   wire when a node is not starting up or cleanly shutting down.
   Network partitions between nodes could cause flappy and undesirable
   behavior in clients.

[slfritchie]

Thinking of @Vagabond's recent RFC & work, would this be information that would be exposed before or after authentication?

[seancribbs]

@slfritchie I think at least it should be filterable by what hosts patterns the user can connect from. This needs more thought!

[bakins]

I think having something akin to what couchbase has with moxi would be helpful. A lightweight "proxy" that understands the autoconfig -- and possibly authentication, as well. Could be used in place of haproxy in most of my use cases and I can continue to use "dumb" clients.

[seancribbs]

@bakins What client(s) are you currently using?

[bakins]

@seancribbs ruby, go, and Lua

[seancribbs]

@bakins Not sure we can manage writing our own proxy, but with the way this is designed, it should be very straightforward to write a daemon that reconfigures existing proxy software. Alternatively, a Chef/Puppet/etc run could be used to rewrite the configuration, which is what most people do nowadays. It is my intention that admins will be able to turn off the feature, and client participation is totally optional, even if the feature is enabled.

[kuenishi]

A small nitpick:

message RpbClientInterface {
    enum RpbProtocol {
        PB = 1;
        HTTP = 2;
        HTTPS = 3;
    }
    required RpbProtocol protocol = 1;
    // This can be a FQDN or a text representation of an IP address.
    required bytes host = 2;
    required uint32 port = 3;    
}

To prevent from wrong value, isn't it good to set the type of port as uint16 ?

After all I totally like this idea because several users are suffering network bottleneck at the load balancer or automatic client-side failover of connection.

[seancribbs]

I'd love to, but there's no uint16 type in protobuff.

On Tue, Aug 27, 2013 at 3:26 AM, UENISHI Kota notifications@github.comwrote:

A small nitpick:

message RpbClientInterface {
enum RpbProtocol {
PB = 1;
HTTP = 2;
HTTPS = 3;
}
required RpbProtocol protocol = 1;
// This can be a FQDN or a text representation of an IP address.
required bytes host = 2;
required uint32 port = 3; }

To prevent from wrong value, isn't it good to set the type of port as
uint16 ?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/356#issuecomment-23320266
.

Sean Cribbs sean@basho.com
Software Engineer
Basho Technologies, Inc.
http://basho.com/

[kuenishi]

A short Google search made me misunderstand that it has uint16. I should have refered here: https://developers.google.com/protocol-buffers/docs/proto#scalar
Anyway, thanks!

[xpe]

+1 I haven't dug into the technical detail above, but this part resonates with my infrastructure needs:

Since any Riak node can handle any request, we should be able to advertise the existence of other nodes to clients so that they can reconfigure automatically, spreading load more evenly among the Riak nodes. This can reduce operational burden, because applications will not need to be restarted to take advantage of a grown cluster.

[seancribbs]

Unfortunately due to time constraints and other big features, this will not be in 2.0.

[jaredmorrow]

Bumped milestone to 2.1

[martincox]

We've talked about this at 365. Definitely need this.