Change certain replica failures not to fail the replica shard #22874

dakrone · 2017-01-30T23:18:09Z

This changes the way that replica failures are handled such that not all
failures will cause the replica shard to be failed or marked as stale.

In some cases such as refresh operations, or global checkpoint syncs, it is
"okay" for the operation to fail without the shard being failed (because no data
is out of sync). In these cases, instead of failing the shard we should simply
fail the operation, and, in the event it is a user-facing operation, return a
5xx response code including the shard-specific failures.

This was accomplished by having two forms of the Replicas proxy, one that is
for non-write operations that does not fail the shard, and one that is for write
operations that will fail the shard when an operation fails.

Relates to #10708

bleskes

Thx @dakrone. I left a bunch of initial comments. Maybe I miss something obvious but where is the logic to respond with an error if one of the shard copies fail (except in the TransportWriteAction)?

bleskes · 2017-02-01T15:14:50Z

core/src/main/java/org/elasticsearch/action/support/replication/ReplicationOperation.java

-                        ReplicationOperation.this::onPrimaryDemoted,
-                        throwable -> decPendingAndFinishIfNeeded()
-                    );
+                    final boolean shardWasFailed = replicasProxy.failShardIfNeeded(shard, replicaRequest.primaryTerm(), message,


I'd rather not introduce more code paths. I would vote to always have the callback pattern. If people have nothing to do they can call onSuccess on the same thread.

Okay, I pushed 3589d9c39d7030b9d7817764626520cb976ba9b2 that switches this to use the callback

bleskes · 2017-02-01T15:17:46Z

core/src/main/java/org/elasticsearch/action/support/replication/ReplicationOperation.java

@@ -209,14 +209,20 @@ public void onFailure(Exception replicaException) {
                    shardReplicaFailures.add(new ReplicationResponse.ShardInfo.Failure(
                        shard.shardId(), shard.currentNodeId(), replicaException, restStatus, false));
                    String message = String.format(Locale.ROOT, "failed to perform %s on replica %s", opType, shard);
+                    // TODO: Boaz wanted to get rid of this warning, should we?
                    logger.warn(


we plan to use this for background operations like the global check point sync, where it's not a big deal if this happen. I don't want the logs to have warning in them. Instead, implementations (i.e., TransportWriteAction) of failShardIfNeeded can log a warning if they are going to fail the shard

Okay, I pushed ccda991ee569490ce2ced3f5bff7763e867e20ac

bleskes · 2017-02-01T15:27:59Z

core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java

+     * interface that performs the actual {@code ReplicaRequest} on the replica
+     * shards. It also encapsulates the logic required for failing the replica
+     * if deemed necessary as well as marking it as stale when needed.
+     */
    final class ReplicasProxy implements ReplicationOperation.Replicas<ReplicaRequest> {


why make this final? WriteReplicasProxy can share a lot of it's logic with this one?

I originally had them sharing knowledge (subclassing), however, ReplicasProxy used parts of TransportReplicationAction and made WriteActionReplicasProxy not able to subclass it well.

This seems to work fine for me?

diff --git a/core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java b/core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java index db037ad..2cd1d8e 100644 --- a/core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java +++ b/core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java @@ -1040,7 +1040,7 @@ public abstract class TransportReplicationAction< * shards. It also encapsulates the logic required for failing the replica * if deemed necessary as well as marking it as stale when needed. */ - final class ReplicasProxy implements ReplicationOperation.Replicas<ReplicaRequest> { + class ReplicasProxy implements ReplicationOperation.Replicas<ReplicaRequest> { @Override public void performOn(ShardRouting replica, ReplicaRequest request, ActionListener<ReplicationOperation.ReplicaResponse> listener) { diff --git a/core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java b/core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java index ac79a8c..8a168ca 100644 --- a/core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java +++ b/core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java @@ -74,7 +74,7 @@ public abstract class TransportWriteAction< @Override protected ReplicationOperation.Replicas newReplicasProxy() { - return new WriteActionReplicasProxy(shardStateAction); + return new WriteActionReplicasProxy(); } /** @@ -335,26 +335,7 @@ public abstract class TransportWriteAction< * replicas, where a failure to execute the operation should fail * the replica shard and/or mark the replica as stale. */ - final class WriteActionReplicasProxy implements ReplicationOperation.Replicas<ReplicaRequest> { - - private final ShardStateAction shardStateAction; - - WriteActionReplicasProxy(ShardStateAction shardStateAction) { - this.shardStateAction = shardStateAction; - } - - @Override - public void performOn(ShardRouting replica, ReplicaRequest request, ActionListener<ReplicationOperation.ReplicaResponse> listener) { - String nodeId = replica.currentNodeId(); - final DiscoveryNode node = clusterService.state().nodes().get(nodeId); - if (node == null) { - listener.onFailure(new NoNodeAvailableException("unknown node [" + nodeId + "]")); - return; - } - final ConcreteShardRequest<ReplicaRequest> concreteShardRequest = - new ConcreteShardRequest<>(request, replica.allocationId().getId()); - sendReplicaRequest(concreteShardRequest, node, listener); - } + class WriteActionReplicasProxy extends ReplicasProxy { @Override public void failShardIfNeeded(ShardRouting replica, long primaryTerm, String message, Exception exception,

@bleskes that's actually a scary thing, that "works" for Intellij's compiler (it reports no compilation errors), but if you actually run gradle compileJava:

/home/hinmanm/es/elasticsearch/core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java:347: error: name clash: failShardIfNeeded(ShardRouting,long,String,Exception,Runnable,Consumer<Exception>,Consumer<Exception>) in TransportWriteAction.WriteActionReplicasProxy and failShardIfNeeded(ShardRouting,long,String,Exception,Runnable,Consumer<Exception>,Consumer<Exception>) in TransportReplicationAction.ReplicasProxy have the same erasure, yet neither overrides the other public void failShardIfNeeded(ShardRouting replica, long primaryTerm, String message, Exception exception, ^ /home/hinmanm/es/elasticsearch/core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java:346: error: method does not override or implement a method from a supertype @Override ^

It fails because of type erasure (I think due to the crazy generics that are in use), I tried a lot of different ways to get around it but I wasn't able to find a way.

bleskes · 2017-02-01T15:28:51Z

core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java

+        private final ClusterService clusterService;
+        private final TransportReplicationAction replicationAction;
+
+        ReplicasProxy(ClusterService clusterService, TransportReplicationAction replicationAction) {


this is an inner class anyway, why pass these along? If you want to make it a static inner class, I'm good with it, although I'm not sure if it's worth the extra clutter.

I originally tried to factor this out into a non-inner class, but turned out that the generics prevented it, so I will removed these.

I pushed b5e91badc401a5aab1f20209ce0096881b07992f

bleskes · 2017-02-01T15:31:02Z

core/src/main/java/org/elasticsearch/action/support/replication/ReplicationOperation.java

         */
-        void failShard(ShardRouting replica, long primaryTerm, String message, Exception exception, Runnable onSuccess,
-                       Consumer<Exception> onPrimaryDemoted, Consumer<Exception> onIgnoredFailure);
+        void markShardCopyAsStaleIfNeeded(ShardId shardId, String allocationId, long primaryTerm, Runnable onSuccess,


why do we need two of these? can't we have just markShardCopyAsStaleIfNeeded? we can rename markUnavailableShardsAsStale to markUnavailableShardsAsStaleIfNeeded

We need two because there are situations when a replica shard needs to be marked as stale without a write action, I originally had this and then the tests that restart ES exposed that they still need to be able to mark it as stale without any actions.

bleskes · 2017-02-01T15:32:39Z

core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java

+                                                 Consumer<Exception> onPrimaryDemoted, Consumer<Exception> onIgnoredFailure) {
+            // Just like with failing the shard, it should only be marked as
+            // stale when write operations fail, so for regular operations
+            // don't mark it as stale when not necessary.
        }

        @Override
        public void markShardCopyAsStale(ShardId shardId, String allocationId, long primaryTerm, Runnable onSuccess,
                                         Consumer<Exception> onPrimaryDemoted, Consumer<Exception> onIgnoredFailure) {
            shardStateAction.remoteShardFailed(shardId, allocationId, primaryTerm, "mark copy as stale", null,


this logic belongs in transportWriteAction

As I mentioned in my previous comment, we still need a way to mark the replica as stale when the shard is unavailable.

What we don't need is the "if needed" version, because we only do this when the shard is not available, not when an operation fails.

I've removed the "if needed" version of this in 1f58744362b756bc9ef6f6bfb412c2836890d8f7

We need two because there are situations when a replica shard needs to be marked as stale without a write action

Which situations do you mean?

bleskes · 2017-02-01T15:37:00Z

core/src/main/java/org/elasticsearch/index/seqno/GlobalCheckpointSyncAction.java

-public class GlobalCheckpointSyncAction extends TransportReplicationAction<GlobalCheckpointSyncAction.PrimaryRequest,
-    GlobalCheckpointSyncAction.ReplicaRequest, ReplicationResponse> {
+public class GlobalCheckpointSyncAction extends TransportReplicationAction<GlobalCheckpointSyncAction.GlobalCheckpointPrimaryRequest,
+    GlobalCheckpointSyncAction.GlobalCheckpointReplicaRequest, ReplicationResponse> {


why rename this , it's already scoped under GlobalCheckpointSyncAction. it gets so long...

I pushed 68d0ddd456d5e42db40c06261af8e8a19c201533 for this.

dakrone · 2017-02-01T17:23:12Z

Maybe I miss something obvious but where is the logic to respond with an error if one of the shard copies fail (except in the TransportWriteAction)?

That logic is in the BroadcastShardResponse now, (getStatus()), which is used on the REST side. The error reporting was already there, this just changes the response to be a 500 instead of 200 in the event there is an error.

dakrone · 2017-02-02T00:00:01Z

Thanks @bleskes, I think I pushed commits and added comments for all of your feedback so far.

bleskes

Thx @dakrone . I responded.

bleskes · 2017-02-02T10:37:54Z

core/src/main/java/org/elasticsearch/action/support/broadcast/BroadcastResponse.java

@@ -71,6 +74,17 @@ public int getFailedShards() {
    }

    /**
+     * The REST status that should be used for the response
+     */
+    public RestStatus getStatus() {


Since this is effects all response that inherit from BroadcastResponse - can you check that all of them do the right thing and filter shard not available failures? might be good to add an assertion in the constructors, looking at all the exceptions this get.

Certainly, I added an assert in f79c0f724819ff428c7c2c1d9bc2701521d96a81

bleskes · 2017-02-02T10:38:54Z

core/src/main/java/org/elasticsearch/action/support/broadcast/BroadcastResponse.java

+     */
+    public RestStatus getStatus() {
+        if (failedShards > 0) {
+            return RestStatus.INTERNAL_SERVER_ERROR;


shall we make an heroic effort to take the worst of all underlying rest status? maybe just the first ?

shall we make an heroic effort to take the worst of all underlying rest status? maybe just the first ?

I don't know, how do you define "worst"? Is a 403 worse than a 500 or is a 500 worse? We can totally go with first though, if that's desired.

I think 5xx (server error) is worst than 4xx, but let's just go with the first.

bleskes · 2017-02-02T10:39:33Z

core/src/main/java/org/elasticsearch/action/support/replication/ReplicationOperation.java

-                Request extends ReplicationRequest<Request>,
-                ReplicaRequest extends ReplicationRequest<ReplicaRequest>,
-                PrimaryResultT extends PrimaryResult<ReplicaRequest>
+                R extends ReplicationRequest<R>,


nit: Why do we call this R and use an T suffix on the rest?

I have seen two standard "formats", one is the single char format, which the JDK uses a lot like Collection<E>, so that's why the R. For the others, PrimaryResult as PR I could do, but I figured since it wasn't a single char I'd suffix it with T for type (another format I've seen in codebases).

I'm happy to change it to R, and PR if you think that would be clearer

I would go with a T suffix - i.e., RequestT

bleskes · 2017-02-02T10:41:32Z

core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java

-            return new ReplicationOperation<>(request, primaryShardReference, listener,
-                executeOnReplicas, replicasProxy, clusterService::state, logger, actionName
-            );
+            return new ReplicationOperation<Request, ReplicaRequest, PrimaryResult<ReplicaRequest, Response>>(request,


out of curiosity - why does <> not work?

This was added unnecessarily, I've removed it in 5eafb1866e85b994fafd178d72c873bba9838e71

bleskes · 2017-02-02T10:50:00Z

core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java

+     * interface that performs the actual {@code ReplicaRequest} on the replica
+     * shards. It also encapsulates the logic required for failing the replica
+     * if deemed necessary as well as marking it as stale when needed.
+     */
    final class ReplicasProxy implements ReplicationOperation.Replicas<ReplicaRequest> {


This seems to work fine for me?

diff --git a/core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java b/core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java index db037ad..2cd1d8e 100644 --- a/core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java +++ b/core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java @@ -1040,7 +1040,7 @@ public abstract class TransportReplicationAction< * shards. It also encapsulates the logic required for failing the replica * if deemed necessary as well as marking it as stale when needed. */ - final class ReplicasProxy implements ReplicationOperation.Replicas<ReplicaRequest> { + class ReplicasProxy implements ReplicationOperation.Replicas<ReplicaRequest> { @Override public void performOn(ShardRouting replica, ReplicaRequest request, ActionListener<ReplicationOperation.ReplicaResponse> listener) { diff --git a/core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java b/core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java index ac79a8c..8a168ca 100644 --- a/core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java +++ b/core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java @@ -74,7 +74,7 @@ public abstract class TransportWriteAction< @Override protected ReplicationOperation.Replicas newReplicasProxy() { - return new WriteActionReplicasProxy(shardStateAction); + return new WriteActionReplicasProxy(); } /** @@ -335,26 +335,7 @@ public abstract class TransportWriteAction< * replicas, where a failure to execute the operation should fail * the replica shard and/or mark the replica as stale. */ - final class WriteActionReplicasProxy implements ReplicationOperation.Replicas<ReplicaRequest> { - - private final ShardStateAction shardStateAction; - - WriteActionReplicasProxy(ShardStateAction shardStateAction) { - this.shardStateAction = shardStateAction; - } - - @Override - public void performOn(ShardRouting replica, ReplicaRequest request, ActionListener<ReplicationOperation.ReplicaResponse> listener) { - String nodeId = replica.currentNodeId(); - final DiscoveryNode node = clusterService.state().nodes().get(nodeId); - if (node == null) { - listener.onFailure(new NoNodeAvailableException("unknown node [" + nodeId + "]")); - return; - } - final ConcreteShardRequest<ReplicaRequest> concreteShardRequest = - new ConcreteShardRequest<>(request, replica.allocationId().getId()); - sendReplicaRequest(concreteShardRequest, node, listener); - } + class WriteActionReplicasProxy extends ReplicasProxy { @Override public void failShardIfNeeded(ShardRouting replica, long primaryTerm, String message, Exception exception,

bleskes · 2017-02-02T10:51:38Z

core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java

+                                                 Consumer<Exception> onPrimaryDemoted, Consumer<Exception> onIgnoredFailure) {
+            // Just like with failing the shard, it should only be marked as
+            // stale when write operations fail, so for regular operations
+            // don't mark it as stale when not necessary.
        }

        @Override
        public void markShardCopyAsStale(ShardId shardId, String allocationId, long primaryTerm, Runnable onSuccess,
                                         Consumer<Exception> onPrimaryDemoted, Consumer<Exception> onIgnoredFailure) {
            shardStateAction.remoteShardFailed(shardId, allocationId, primaryTerm, "mark copy as stale", null,


We need two because there are situations when a replica shard needs to be marked as stale without a write action

Which situations do you mean?

bleskes · 2017-02-02T10:52:35Z

core/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java

@@ -124,7 +138,11 @@ public synchronized void respond(ActionListener<Response> listener) {
        protected void respondIfPossible(Exception ex) {
            if (finishedAsyncActions && listener != null) {
                if (ex == null) {
-                    super.respond(listener);
+                    if (finalResponseIfSuccessful != null) {


why this change?

This was a leftover from me trying something out, I've removed it in 56cc63f612e5281555d5da1202dc9a44748eceea

bleskes · 2017-02-02T10:53:00Z

core/src/main/java/org/elasticsearch/index/seqno/GlobalCheckpointSyncAction.java

@@ -129,7 +129,7 @@ public String toString() {
        private ReplicaRequest() {
        }

-        public ReplicaRequest(PrimaryRequest primaryRequest, long checkpoint) {
+        public ReplicaRequest(ReplicationRequest primaryRequest, long checkpoint) {


Why this change?

I manually reverted the changes to GlobalCheckpointSyncAction and I missed one :)

I undid it in f7b29d78395c2e13c715f6c167d6e8cbffe28e6e

bleskes · 2017-02-02T10:54:02Z

core/src/main/java/org/elasticsearch/rest/action/RestFieldStatsAction.java

@@ -98,7 +98,7 @@ public RestResponse buildResponse(FieldStatsResponse response, XContentBuilder b
                    builder.endObject();
                }
                builder.endObject();
-                return new BytesRestResponse(RestStatus.OK, builder);
+                return new BytesRestResponse(response.getStatus(), builder);


these are not related changes - can we maybe move them to a separate PR?

Sure, though I think it makes it more consistent to change these to be the same, I'll open a separate PR for it after this one to change them to be 500 if a shard fails and remove all of these except for the refresh action

I pushed d16441b19d606d87166d82345c1197f7fa6d8386 for this

dakrone · 2017-02-03T00:00:54Z

@bleskes I think I addressed all the feedback for this round, thanks again for taking a look!

bleskes

LGTM. Left some comments that you can adopt or not. No need for another review. Thx @dakrone

bleskes · 2017-02-03T20:08:14Z

core/src/main/java/org/elasticsearch/action/support/broadcast/BroadcastResponse.java

+     */
+    public RestStatus getStatus() {
+        if (failedShards > 0) {
+            return RestStatus.INTERNAL_SERVER_ERROR;


I think 5xx (server error) is worst than 4xx, but let's just go with the first.

bleskes · 2017-02-03T20:10:09Z

core/src/main/java/org/elasticsearch/action/support/replication/ReplicationOperation.java

-                Request extends ReplicationRequest<Request>,
-                ReplicaRequest extends ReplicationRequest<ReplicaRequest>,
-                PrimaryResultT extends PrimaryResult<ReplicaRequest>
+                R extends ReplicationRequest<R>,


I would go with a T suffix - i.e., RequestT

bleskes · 2017-02-03T20:12:01Z

core/src/main/java/org/elasticsearch/action/support/replication/TransportReplicationAction.java

    private final TransportRequestOptions transportOptions;
    private final String executor;

    // package private for testing
    private final String transportReplicaAction;
    private final String transportPrimaryAction;
-    private final ReplicasProxy replicasProxy;
+    private final ReplicationOperation.Replicas replicasProxy;

    protected TransportReplicationAction(Settings settings, String actionName, TransportService transportService,
                                         ClusterService clusterService, IndicesService indicesService,


we can remove the shard state action here, no?

bleskes · 2017-02-03T20:24:14Z

core/src/test/java/org/elasticsearch/action/support/replication/TransportWriteActionTests.java

+        return indexService;
+    }
+
+    final IndicesService mockIndicesService(ClusterService clusterService) {


can we share this and mockIndexShard with TransportReplicationAction? is there any point in that?

dakrone · 2017-02-03T21:36:54Z

Thank you very much for reviewing this @bleskes!

This changes the way that replica failures are handled such that not all failures will cause the replica shard to be failed or marked as stale. In some cases such as refresh operations, or global checkpoint syncs, it is "okay" for the operation to fail without the shard being failed (because no data is out of sync). In these cases, instead of failing the shard we should simply fail the operation, and, in the event it is a user-facing operation, return a 5xx response code including the shard-specific failures. This was accomplished by having two forms of the `Replicas` proxy, one that is for non-write operations that does not fail the shard, and one that is for write operations that will fail the shard when an operation fails. Relates to elastic#10708

dakrone added :Sequence IDs >breaking v6.0.0-alpha1 labels Jan 30, 2017

dakrone requested a review from bleskes January 30, 2017 23:18

dakrone force-pushed the replica-failures-dont-always-fail-shard branch from d64fac6 to b686146 Compare January 31, 2017 19:35

bleskes suggested changes Feb 1, 2017

View reviewed changes

bleskes suggested changes Feb 2, 2017

View reviewed changes

dakrone force-pushed the replica-failures-dont-always-fail-shard branch from d16441b to d7b277d Compare February 3, 2017 17:11

bleskes approved these changes Feb 3, 2017

View reviewed changes

dakrone force-pushed the replica-failures-dont-always-fail-shard branch 2 times, most recently from 24ddcb0 to 8661750 Compare February 3, 2017 21:03

dakrone force-pushed the replica-failures-dont-always-fail-shard branch from 8661750 to 7af97f5 Compare February 3, 2017 21:38

dakrone force-pushed the replica-failures-dont-always-fail-shard branch from 7af97f5 to 39e7c30 Compare February 3, 2017 21:39

dakrone merged commit 39e7c30 into elastic:master Feb 3, 2017

bleskes mentioned this pull request Feb 3, 2017

Add Sequence Numbers to write operations #10708

Closed

64 tasks

dakrone deleted the replica-failures-dont-always-fail-shard branch December 13, 2017 20:37

clintongormley added :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. and removed :Sequence IDs labels Feb 14, 2018

Change certain replica failures not to fail the replica shard #22874

Change certain replica failures not to fail the replica shard #22874

Conversation

dakrone commented Jan 30, 2017 • edited Loading

bleskes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakrone commented Feb 1, 2017

dakrone commented Feb 2, 2017

bleskes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakrone commented Feb 3, 2017

bleskes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakrone commented Feb 3, 2017

dakrone commented Jan 30, 2017 •

edited

Loading