Perform load balancing when communicating with a cluster of Alternators

[julienrf]

Customize the emr-dynamodb-connector to configure the DynamoDB client to use the load balancing request handler from https://github.com/scylladb/alternator-load-balancing.

Unfortunately, the project emr-dynamodb-connector has not been designed to support customization of the underlying DynamoDB client. Therefore, I had to copy-paste code from the project and change it to customize the underlying DynamoDB client. This led to creating a parallel class hierarchy co-existing with the existing one, but with names all prefixed by LoadBalanced.

The PR should be reviewed commit-by-commit and ignoring the content of 424141b and 2b7f076 which copy-paste verbatim content from emr-dynamodb-connector.

Blocked by scylladb/alternator-load-balancing#18
Fixes #117

[tarzanek]

what other classes implemented this?
Aren't we loosing support for other key types than Text ?

[julienrf]

The only implementation in emr-dynamodb-connector was DefaultDynamoDBRecordReader, which also fixes the key type to Text. Maybe there are other implementations in the wild…

[tarzanek]

we need to focus just on what is in emr-dynamodb-connector

[tarzanek]

I understand you are simplifying, but I don't get how only Text can be the key
(I didn't check DynamoDBItemWritable but I assume it's an interface to most data types value can have in Dynamo)

[julienrf]

Hadoop requires a key, but here in emr-dynamodb-connector they always use an empty Text instance as a key. So, it is not really used. DynamoDBItemWritable is a class that only contains a Map<String, AttributeValue> (where AttributeValue comes from the AWS Java SDK and models a column in a DynamoDB row).

[julienrf]

Unfortunately, the PR contains a lot of changes, but in my opinion this is still the best approach to go for now.

I’ve also considered using reflection-based approaches to access the private DynamoDB client of the Hadoop job to configure it to supply our custom request handler, but this is not simple to achieve because we don’t have access to the instance that holds that private DynamoDB client. This means that we would have to instrument the bytecode to impact the places that create and configure the DynamoDB client. This could be achieved with tools like AspectJ or Java Instrumentation, but in my opinion using those tools would not make our maintenance work simpler because they require some specific expertise.

On the other hand, with simple changes in emr-dynamodb-connector, we could easily achieve our goal. For instance, if awslabs/emr-dynamodb-connector#196 is merged, we will be able to set up our custom request handler like so:

// in DynamoUtils.setDynamoDBJobConf
jobConf.set(DynamoDBConstants.CUSTOM_CLIENT_BUILDER_TRANSFORMER, "WithLoadBalancing")
jobConf.set("scylladb.alternator.endpoint", "https://127.0.0.1:8043")

// Where the class WithLoadBalancing would be defined as follows
class WithLoadBalancing extends DynamoDbClientBuilderTransformer with Configurable {
  private var conf: Configuration = null
  def apply(builder: DynamoDbClientBuilder): DynamoDbClientBuilder =
    builder.endpointProvider(
      new AlternatorEndpointProvider(URI.create(conf.get("scylladb.alternator.endpoint")))
    )
  def setConf(conf: Configuration): Unit = {
    this.conf = conf
  }
  def getConf: Configuration = this.conf
}

(note that we would also have to upgrade to v2 of AWS SDK to use the latest version of the connector)

[tarzanek]

we have no power over AWS code, so if they will accept it or not is a dice roll ;-)
so we can worst case go temporarily with this big fork and then cleanup once AWS merges and releases your PR ?

[julienrf]

Superseded by #173