Get ConsumerStats using Binary Protocol #216

jai1 · 2017-02-16T18:29:05Z

Motivation

Sometimes we need topic statics before consuming messages (Common use cases listed below), currently such information is obtained by making admin calls which use HTTP. Admin calls are expensive because each call spawns a new connection and may cause a redirect if the current broker doesn’t own the topic, we need a way to get this information without using HTTP.

Common use cases

When Pulsar is used as a write-ahead log we decide whether to allow updates to a topic based on the amount of backlog the topic has.
Consumer may need the number of pending acknowledgements.

Modifications

Created a new command to get consumer stats - Changed Protobuf and regenerated files.
Changed the C++ Client to get consumer stats.
Made changed to Broker to respond to consumer stats request
Added a caching mechanism to cache the consumer stats for 30 seconds.
Changed ClientConnection to handle operation timeout

Result

We will be able to get consumer stats without having to use HTTP

merlimat · 2017-02-17T02:22:59Z

protobuf/README.md

@@ -28,6 +29,7 @@ make

 # Re-generate PulsarApi
 cd pulsar-path/pulsar-common
-PROTOC=~/protobuf/src/protoc ./generate_protobuf.sh
+PROTOC=~/protobuf/src/protoc 


If you break it in 2 lines, then you need to use export PROTOC=... otherwise the variable will not picked up by the next script

merlimat · 2017-02-17T02:24:01Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

+        }
+    }
+
+    private Map<String, String> createConsumerStatsMap(Consumer consumer, PersistentSubscription subscription) {


Why not reuse the already existing ConsumerStats class?

We need some subscriber level details as well - backlog, subscription type etc
But I have added a toMap method to convert the Class to a map

ca16032

@rdhabalia, @saandrews and @merlimat Would it be better if I add a class in the protobuf and client side instead of using a map?

merlimat · 2017-02-17T02:25:46Z

pulsar-client-cpp/include/pulsar/Result.h

-
-    ResultMessageTooBig /// Trying to send a messages exceeding the max size
+    ResultMessageTooBig, /// Trying to send a messages exceeding the max size
+    ResultConsumerStatsError /// Broker returned an error while trying to fetch consumer stats


Cannot we use the existing ResultError instead of ResultConsumerStatsError. If I am making a call to get consumer stats, there's no need to re-state the type of the request.

rdhabalia

all the formatting changes are with /pulsar-client-cpp/eclipse-formatter.xml ??

rdhabalia · 2017-02-21T19:47:36Z

pom.xml

@@ -484,6 +484,7 @@ flexible messaging model and an intuitive client API.</description>
            <exclude>pulsar-client-cpp/lib/checksum/int_types.h</exclude>
            <exclude>pulsar-client-cpp/lib/checksum/crc32c*</exclude>
            <exclude>pulsar-client-cpp/lib/lz4/lz4.*</exclude>
+            <exclude>pulsar-client-cpp/.idea/*</exclude>


i think cpp/.gitignore already contains: .idea so, we may not need this.

Git ignore will prevent the idea folder to be checked in and this change will prevent the local build to fail

rdhabalia · 2017-02-21T19:49:24Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

+        Object msg = null;
+        try {
+            PersistentTopic topic;
+            topic = (PersistentTopic) getBrokerService().getTopicReference(topicName);


no need of multiple lines :
PersistentTopic topic = (PersistentTopic) getBrokerService().getTopicReference(topicName);

rdhabalia · 2017-02-21T19:51:56Z

protobuf/README.md

@@ -5,7 +5,7 @@ Pulsar uses protocol buffer messages for the client/broker wire protocol.

 The protocol definition is located at `pulsar-common/src/main/proto/PulsarApi.proto` and the pre-generated Java code is at `pulsar-common/src/main/java/com/yahoo/pulsar/common/api/proto/PulsarApi.java`. 

-When making a change to the `PulsarApi.proto` definition, we have regenerate the `PulsarApi.java` and include that in the same commit.
+When making a change to the `PulsarApi.proto` definition, we have to regenerate the `PulsarApi.java` and include that in the same commit.


now, cpp library is added, we can also mention : PulsarApi.pb.cc

rdhabalia · 2017-02-21T19:52:51Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

+            log.debug("CommandConsumerStats[requestId = {}, topicName = {}, subscriptionName = {}, consumerId = {}]", requestId, topicName, subscriptionName, consumerId);
+        }
+
+        Object msg = null;


ByteBuf msg

rdhabalia · 2017-02-21T19:53:28Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

+                        requestId, topicName, subscriptionName, consumerId);
+            }
+        } catch (Exception e) {
+            log.error("Exception: {}", e);


should write appropriate message such as topic not found

Have written appropriate message in line 244, 240 and 236 (same file)

Log remaining info as well in all the above error logs? topic name, requested, surname & consumerid?

I mean error log such as: Failed to get consumer-stats response so, it can help while reading log file where exactly it occurred.

rdhabalia · 2017-02-21T19:56:55Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

+            }
+        } catch (Exception e) {
+            log.error("Exception: {}", e);
+            msg = Commands.newConsumerStatsResponse("Exception: " + e, requestId, topicName, subscriptionName, consumerId);


same here: error-msg

Have written appropriate message in line 244, 240 and 236 (same file)
This case is for handling some exception

If this exception is sent to the client, shouldn't it be the one client knows about?

rdhabalia · 2017-02-21T19:57:38Z

pulsar-client-cpp/include/pulsar/Consumer.h


 #pragma GCC visibility push(default)

 class PulsarFriend;

 namespace pulsar {

+typedef std::map<std::string, std::string> StatsMap;
+
+class BrokerSideConsumerStats {


should we rename it as BrokerConsumerStats or ConsumerStats?

There is already a ConsumerStats class in Java client which captures the message rate, acks etc as sent by client - created by Sangming and Sid.

I would prefer BrokerConsumerStats or ConsumerStatsInfo.

rdhabalia · 2017-02-21T19:59:41Z

pulsar-client-cpp/lib/BatchMessageContainer.cc

@@ -117,7 +117,8 @@ SharedBuffer BatchMessageContainer::getBatchedPayload() {
 void BatchMessageContainer::clear() {
    LOG_DEBUG(*this << " BatchMessageContainer::clear() called");
    timer_->cancel();
-    averageBatchSize_ = (messagesContainerListPtr_->size() + (averageBatchSize_ * numberOfBatchesSent_))/++numberOfBatchesSent_;
+    averageBatchSize_ = (messagesContainerListPtr_->size() + (averageBatchSize_ * numberOfBatchesSent_))/(numberOfBatchesSent_ + 1);


is there any difference here?

yes, the compiler doesn't give ordering guarantees for ++ if used in complex calculations
Try compiling in your mac m/c and you will see the warning.

rdhabalia · 2017-02-21T20:10:04Z

pulsar-common/src/main/java/com/yahoo/pulsar/common/api/PulsarDecoder.java

+            case CONSUMER_STATS:
+            	checkArgument(cmd.hasConsumerStats());
+            	handleConsumerStats(cmd.getConsumerStats());
+                cmd.getConsumerStats().recycle();


rdhabalia · 2017-02-21T20:10:15Z

pulsar-common/src/main/java/com/yahoo/pulsar/common/api/PulsarDecoder.java

+            case CONSUMER_STATS_RESPONSE:
+            	checkArgument(cmd.hasConsumerStatsResponse());
+            	handleConsumerStatsResponse(cmd.getConsumerStatsResponse());
+                cmd.getConsumerStatsResponse().recycle();


rdhabalia · 2017-02-21T20:19:01Z

just a thought:

if client requires subscription level stats then : if consumerId is not provided then should broker return subscription level stats in the map?

rdhabalia · 2017-02-22T21:52:06Z

protobuf/README.md


-# Apply patch
-patch -p1 < ../pulsar-path/protobuf/protobuf.patch
+## For C++


for C++ -> git checkout v2.6.0

rdhabalia · 2017-02-22T21:55:51Z

pulsar-common/src/main/proto/PulsarApi.proto

+
+message CommandConsumerStatsResponse {
+        required uint64 request_id              = 1;        
+        required string topic_name              = 2;


Do we need topic_name, subscription_name, consumer_id in response? can't we just return request_id as client knows information attached with it?

Yes we can but a little more info might help in debug if we run into some trouble and since this call can be made at most once in 30 seconds - it won't harm the Network BW much.

Still if you feel strongly about it let me know - i will remove the fields

yes, we should not send extra field in response except request_id

rdhabalia · 2017-02-22T21:58:56Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

+                        requestId, topicName, subscriptionName, consumerId);
+            }
+        } catch (Exception e) {
+            log.error("Exception: {}", e);


I mean error log such as: Failed to get consumer-stats response so, it can help while reading log file where exactly it occurred.

rdhabalia · 2017-02-22T22:37:45Z

pulsar-client-cpp/lib/ClientConnection.h

@@ -266,7 +274,12 @@ class ClientConnection : public boost::enable_shared_from_this<ClientConnection>
    // Signals whether we're waiting for a response from broker
    bool havePendingPingRequest_;
    DeadlineTimerPtr keepAliveTimer_;
+    DeadlineTimerPtr consumerStatsRequestTimer_;
+
+    void handleConsumerStatsTimeout(const boost::system::error_code &ec,


do we really need timer to expiry the request. we already have keep alive time which will break the connection if broker is not responding and it should clear-out pending requests.

rdhabalia · 2017-02-22T22:42:43Z

pulsar-client-cpp/lib/PartitionedConsumerImpl.h

@@ -60,6 +60,7 @@ namespace pulsar {
        virtual void redeliverUnacknowledgedMessages();
        virtual const std::string& getName() const;
        virtual int getNumOfPrefetchedMessages() const ;
+        virtual Result getConsumerStats(BrokerSideConsumerStats& brokerSideConsumerStats, int partitionIndex);


should we also provide getConsumerStats(BrokerSideConsumerStats& brokerSideConsumerStats) ?

As discussed yesterday - We can the code is not difficult but the interface will become quite ugly since now we need to return a list of stats always.

rdhabalia · 2017-02-22T22:54:09Z

pulsar-client-cpp/lib/Consumer.cc

@@ -23,6 +23,17 @@ namespace pulsar {

 const std::string EMPTY_STRING;

+BrokerSideConsumerStats::BrokerSideConsumerStats():validTill_(now()) {};


should we initialize validTill_+X so it can be valid for X time.??

we do that in ClientConnection

boost::posix_time::ptime currentTime = now() + milliseconds(consumerStatsTTLMs_); consumerStatsPromise.setValue(BrokerSideConsumerStats(consumerStats, currentTime));

rdhabalia · 2017-02-22T23:12:42Z

pulsar-client-cpp/lib/ConsumerImpl.cc

+                    " Sending getConsumerStats command for Consumer - " << getConsumerId() << ", requestId - "<<requestId);
+
+            BrokerSideConsumerStats consumerStats;
+            Result res = cnx->newConsumerStats(topic_, subscription_, consumerId_, requestId).get(consumerStats);


is it a blocking call on future?

then should we make it non blocking call by attaching the listener?

It will block the user thread not out io thread, similar logic is used to producer.send and consumer.receive

Result Producer::send(const Message& msg) { Promise<Result, Message> promise; sendAsync(msg, WaitForCallbackValue<Message>(promise)); Message m; Result result = promise.getFuture().get(m); return result; }

Do you think Is it worth it to provide callback option also in case of client doesn't want to block such as: sendAsync(msg, callBack)

rdhabalia · 2017-02-22T23:13:09Z

pulsar-client-cpp/lib/ConsumerImpl.cc

+            }
+            return res;
+        } else {
+            LOG_ERROR(getName() << " In getConsumerStats call - Operation not supported since server protobuf version is older than proto::v6");


saandrews · 2017-02-22T00:16:45Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

+            }
+        } catch (Exception e) {
+            log.error("Exception: {}", e);
+            msg = Commands.newConsumerStatsResponse("Exception: " + e, requestId, topicName, subscriptionName, consumerId);


If this exception is sent to the client, shouldn't it be the one client knows about?

saandrews · 2017-02-22T00:17:38Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

+                        requestId, topicName, subscriptionName, consumerId);
+            }
+        } catch (Exception e) {
+            log.error("Exception: {}", e);


Log remaining info as well in all the above error logs? topic name, requested, surname & consumerid?

saandrews · 2017-02-22T00:43:20Z

pulsar-client-cpp/include/pulsar/Consumer.h


 #pragma GCC visibility push(default)

 class PulsarFriend;

 namespace pulsar {

+typedef std::map<std::string, std::string> StatsMap;
+
+class BrokerSideConsumerStats {


I would prefer BrokerConsumerStats or ConsumerStatsInfo.

saandrews · 2017-02-22T01:20:15Z

pulsar-client-cpp/lib/ClientConnection.cc

+                                consumerStats[kv.key()] = kv.value();
+                                LOG_DEBUG(kv.key() << " = " << kv.value());
+                            }
+                            boost::posix_time::ptime currentTime = now() + milliseconds(consumerStatsTTLMs_);


Its not really currentTime

saandrews · 2017-02-23T07:52:43Z

pulsar-client-cpp/lib/ClientConnection.cc

+
+    for (PendingConsumerStatsMap::iterator it = pendingConsumerStatsMap_.begin();
+         it != pendingConsumerStatsMap_.end(); ++it) {
+        LOG_ERROR(cnxString_ << " In getConsumerStats call - Closing Client Connection, please try again later");


Copy pasted message? Also we shouldn't have to explicitly say "In getConsumerStats call"

saandrews · 2017-02-23T07:53:15Z

pulsar-client-cpp/lib/ConsumerImpl.cc

@@ -686,4 +687,44 @@ int ConsumerImpl::getNumOfPrefetchedMessages() const {
    return incomingMessages_.size();
 }

+Result ConsumerImpl::getConsumerStats(BrokerSideConsumerStats& brokerSideConsumerStats, int partitionIndex) {
+    if (partitionIndex != -1) {
+        LOG_WARN(getName() << " In getConsumerStats call - Ignoring the partitionIndex since the topic is not partitioned")


Remove "In getConsumerStats call -"

saandrews · 2017-02-23T07:55:37Z

pulsar-client-cpp/lib/ConsumerImpl.cc

+    }
+
+    if (brokerSideConsumerStats_.isValid()) {
+        LOG_INFO(getName() << " In getConsumerStats call - Serving data from cache");


Why INFO log?

jai1 · 2017-02-23T22:06:28Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

@@ -300,7 +392,7 @@ protected void handleSubscribe(final CommandSubscribe subscribe) {
                        }) //
                        .exceptionally(exception -> {
                            log.warn("[{}][{}][{}] Failed to create consumer: {}", remoteAddress, topicName,
-                                    subscriptionName, exception.getCause().getMessage());


getCause().getMessage() is not very informative

jai1 · 2017-02-23T22:08:56Z

pulsar-client-cpp/lib/Client.cc

@@ -45,7 +45,9 @@ struct ClientConfiguration::Impl {
             operationTimeoutSeconds(30),
             messageListenerThreads(1),
             concurrentLookupRequest(5000),
-             logConfFilePath() {}
+             logConfFilePath(),
+             useTls(false),


In a few test runs I found that tls was automatically enabled - hence initialized the params

jai1 · 2017-02-23T22:10:03Z

pulsar-client-cpp/lib/ConsumerImpl.cc

@@ -113,7 +114,7 @@ void ConsumerImpl::connectionOpened(const ClientConnectionPtr& cnx) {
    lock.unlock();

    ClientImplPtr client = client_.lock();
-    int requestId = client->newRequestId();


datatype compatible with what we send over the wire

jai1 · 2017-02-23T22:12:23Z

pulsar-client-cpp/lib/ConsumerImpl.cc

@@ -483,13 +484,13 @@ inline proto::CommandSubscribe_SubType ConsumerImpl::getSubType() {
    ConsumerType type = config_.getConsumerType();
    switch (type) {
        case ConsumerExclusive:
-            return proto::CommandSubscribe_SubType_Exclusive;


both mean the same - changed since underscores were ugly

static const SubType Exclusive = CommandSubscribe_SubType_Exclusive;
static const SubType Shared = CommandSubscribe_SubType_Shared;
static const SubType Failover = CommandSubscribe_SubType_Failover;

rdhabalia · 2017-02-23T23:34:35Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

@@ -300,7 +392,7 @@ protected void handleSubscribe(final CommandSubscribe subscribe) {
                        }) //
                        .exceptionally(exception -> {
                            log.warn("[{}][{}][{}] Failed to create consumer: {}", remoteAddress, topicName,
-                                    subscriptionName, exception.getCause().getMessage());
+                                    subscriptionName, exception);


this will be CompletionException and it will not be helpful, we have to log exception.getCause()

saandrews · 2017-02-24T00:26:19Z

pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/ServerCnx.java

+                    }
+                    if (!consumerFound) {
+                        log.error(
+                                "Failed to get consumer-stats response - Consumer {} not found for CommandConsumerStats request id {}",


Log topic and subscription id as well?

saandrews · 2017-02-24T00:56:01Z

pulsar-client-cpp/lib/ClientConnection.cc

+        LOG_ERROR(cnxString_ << " In getConsumerStats call - Closing Client Connection, please try again later");
+        it->second.setFailed(ResultConnectError);
+    }
+


Unlock here.

saandrews · 2017-02-24T01:12:35Z

pulsar-client-cpp/lib/ClientConnection.cc

+    // Complex logic since promises need to be fulfilled outside the lock
+    for (int i = 0; i < consumerStatsPromises.size(); i++) {
+        consumerStatsPromises[i].setFailed(ResultTimeout);
+        LOG_ERROR(cnxString_ << " In getConsumerStats call - Operation timedout, didn't get response from broker");


Fix log msg.

saandrews · 2017-02-24T01:23:51Z

pulsar-client-cpp/lib/ClientConnection.cc

+
+                        if (consumerStatsResponse.has_error_code()) {
+                            if (consumerStatsResponse.has_error_message()) {
+                                LOG_ERROR(cnxString_ << " In getConsumerStats call - "


saandrews · 2017-02-25T01:46:28Z

pulsar-client-cpp/lib/ClientConnection.cc

                    }

                    break;
                }

                case BaseCommand::SEND_ERROR: {
-                    const CommandSendError& error = incomingCmd_.send_error();
+                    const CommandSendError &error = incomingCmd_.send_error();


Fix the formatting caused by the editor w.r.t. space and &.

jai1 · 2017-02-26T04:07:44Z

@merlimat - if you find time, can you review this PR.

merlimat · 2017-02-27T17:54:29Z

Ok, I was about to get to it ;)

…

On Mon, Feb 27, 2017 at 9:52 AM jai1 ***@***.***> wrote: Merged #216 <#216>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#216 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD0JFYQ-dU-Mz5EZMqX8kadwm8fhT5bks5rgw1PgaJpZM4MDYHI> .

jai1 · 2017-02-28T23:15:16Z

Sorry - was in a hurry to get this out, will plan better next time onwards On Mon, Feb 27, 2017 at 9:54 AM, Matteo Merli <notifications@github.com> wrote:

…

Ok, I was about to get to it ;) On Mon, Feb 27, 2017 at 9:52 AM jai1 ***@***.***> wrote: > Merged #216 <#216>. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#216 (comment)>, or mute the > thread > <https://github.com/notifications/unsubscribe-auth/AAD0JFYQ-dU- Mz5EZMqX8kadwm8fhT5bks5rgw1PgaJpZM4MDYHI> > . > — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#216 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIQh_F8HALVIdU1obGNbSU6712P0g9_Bks5rgw3XgaJpZM4MDYHI> .

* Rename FunctionConfig names to make them more consistent * Added generated python file

speed up entriesToRecords for consuming multi entries

rdhabalia assigned jai1 Feb 16, 2017

rdhabalia added type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages type/feature The PR added a new feature or issue requested a new feature labels Feb 16, 2017

rdhabalia added this to the 1.17 milestone Feb 16, 2017

jai1 changed the title ~~Cms 2214~~ Get ConsumerStats using Binary Protocol Feb 16, 2017

merlimat reviewed Feb 17, 2017

View reviewed changes

rdhabalia reviewed Feb 21, 2017

View reviewed changes

rdhabalia reviewed Feb 22, 2017

View reviewed changes

jai1 added type/feature The PR added a new feature or issue requested a new feature and removed type/feature The PR added a new feature or issue requested a new feature labels Feb 23, 2017

saandrews reviewed Feb 23, 2017

View reviewed changes

jai1 commented Feb 23, 2017

View reviewed changes

rdhabalia reviewed Feb 23, 2017

View reviewed changes

saandrews reviewed Feb 24, 2017

View reviewed changes

jai1 force-pushed the CMS-2214 branch 2 times, most recently from 83c4319 to 20675ca Compare February 25, 2017 01:30

saandrews reviewed Feb 25, 2017

View reviewed changes

jai1 force-pushed the CMS-2214 branch 4 times, most recently from 97827ef to 12cf4d1 Compare February 25, 2017 06:24

Get ConsumerStats using Binary Protocol

d6671e5

jai1 force-pushed the CMS-2214 branch from 12cf4d1 to d6671e5 Compare February 25, 2017 07:56

saandrews approved these changes Feb 25, 2017

View reviewed changes

jai1 removed the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Feb 25, 2017

jai1 added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Feb 25, 2017

jai1 merged commit ae381b4 into apache:master Feb 27, 2017

jai1 mentioned this pull request Mar 1, 2017

C++ Client - Async call for getConsumerStat #255

Closed

jai1 deleted the CMS-2214 branch March 6, 2017 17:43

sijie pushed a commit to sijie/pulsar that referenced this pull request Mar 4, 2018

Rename FunctionConfig names to make them more consistent (apache#216)

7fd924f

* Rename FunctionConfig names to make them more consistent * Added generated python file

hrsakai pushed a commit to hrsakai/pulsar that referenced this pull request Dec 10, 2020

Use .asf.yaml to configure github repo (apache#216)

4e52758

hangc0276 pushed a commit to hangc0276/pulsar that referenced this pull request May 26, 2021

speed up entriesToRecords for consuming multi entries (apache#216)

91ea416

speed up entriesToRecords for consuming multi entries

xiaotongwang1 mentioned this pull request Aug 4, 2021

Pulsar 2.7.0+ KOP 2.7.2.x getPartitionedTopicMetadata timeout #11532

Closed

		@@ -23,6 +23,17 @@ namespace pulsar {

		const std::string EMPTY_STRING;

		BrokerSideConsumerStats::BrokerSideConsumerStats():validTill_(now()) {};

Get ConsumerStats using Binary Protocol #216

Get ConsumerStats using Binary Protocol #216

Conversation

jai1 commented Feb 16, 2017

Motivation

Modifications

Result

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jai1 Feb 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdhabalia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdhabalia commented Feb 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jai1 Feb 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jai1 commented Feb 26, 2017

merlimat commented Feb 27, 2017 via email

jai1 commented Feb 28, 2017 via email

jai1 Feb 21, 2017 •

edited

Loading

jai1 Feb 23, 2017 •

edited

Loading