[doc]Add document for debug connector #5041

tuteng · 2019-08-26T09:41:29Z

Master Issue: #3624

Motivation

If the current user encounters problems when using the connector, there is not a reasonable debugging process. This document attempts to provide a debugging process for the connector.

Modifications

Add debugging document for the connector.

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

Jennifer88huang-zz · 2019-08-26T11:17:28Z

@Anonymitaet Could you please help review this PR? Thank you.

Anonymitaet · 2019-08-26T11:21:17Z

@jennifer88huang glad to help

sijie · 2019-08-26T19:18:20Z

run java8 tests
run integration tests

Anonymitaet · 2019-08-28T12:33:44Z

The issue of Updating Connector Guide: #5015 (comment)

I've updated the guide based on the following structure:

Anonymitaet · 2019-08-28T12:34:18Z

Making plenty of modifications in a PR is not convenient, so I paste my writings here.
If you don't have comments, you can copy the following contents.

id: io-debug
title: How to debug Pulsar connectors
sidebar_label: Debug

This guide explains how to debug connectors in localrun or cluster mode and gives a debugging checklist.

To better demonstrate how to debug Pulsar connectors, here takes a Mongo sink connector as an example.

Deploy a Mongo sink environment

Start a Mongo service.

docker pull mongo:4
docker run -d -p 27017:27017 --name pulsar-mongo -v $PWD/data:/data/db mongo:4

Create a DB and a collection.

docker exec -it pulsar-mongo /bin/bash
mongo
> use pulsar
> db.createCollection('messages')
> exit

Start Pulsar standalone.

docker pull apachepulsar/pulsar:2.4.0
docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --link pulsar-mongo --name pulsar-mongo-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone

Configure the Mongo sink with the mongo-sink-config.yaml file.

configs:
mongoUri: "mongodb://pulsar-mongo:27017"
database: "pulsar"
collection: "messages"
batchSize: 2
batchTimeMs: 500

docker cp mongo-sink-config.yaml pulsar-mongo-standalone:/pulsar/

Download the Mongo sink nar package.

docker exec -it pulsar-mongo-standalone /bin/bash
curl -O http://apache.01link.hk/pulsar/pulsar-2.4.0/connectors/pulsar-io-mongo-2.4.0.nar

Debug in localrun mode

Start the Mongo sink in localrun mode using the localrun command.

Tip

For more information about the localrun command, see localrun.

./bin/pulsar-admin sinks localrun \
--archive pulsar-io-mongo-2.4.0.nar \ 
--tenant public --namespace default \
--inputs test-mongo \
--name pulsar-mongo-sink \
--sink-config-file mongo-sink-config.yaml \
--parallelism 1

Use connector log

Use one of the following methods to get a connector log in localrun mode:

After executing the localrun command, the log is automatically printed on the console.

The log is located at:

logs/functions/tenant/namespace/function-name/function-name-instance-id.log

Example

The path of the Mongo sink connector is:

logs/functions/public/default/pulsar-mongo-sink/pulsar-mongo-sink-0.log

To clearly explain the log information, here breaks down the large block of information into small blocks and add descriptions for each block.

This piece of log information shows the storage path of the nar package after decompression.
```
08:21:54.132 [main] INFO  org.apache.pulsar.common.nar.NarClassLoader - Created class loader with paths: [file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/, file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/META-INF/bundled-dependencies/,
```
Tip

If class cannot be found exception is thrown, check whether the nar file is decompressed in the folder file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/META-INF/bundled-dependencies/ or not.

This piece of log information illustrates the basic information about the Mongo sink connector, such as tenant, namespace, name, parallelism, resources, and so on, which can be used to check whether the Mongo sink connector is configured correctly or not.

08:21:55.390 [main] INFO  org.apache.pulsar.functions.runtime.ThreadRuntime - ThreadContainer starting function with instance config InstanceConfig(instanceId=0, functionId=853d60a1-0c48-44d5-9a5c-6917386476b2, functionVersion=c2ce1458-b69e-4175-88c0-a0a856a2be8c, functionDetails=tenant: "public"
namespace: "default"
name: "pulsar-mongo-sink"
className: "org.apache.pulsar.functions.api.utils.IdentityFunction"
autoAck: true
parallelism: 1
source {
typeClassName: "[B"
inputSpecs {
    key: "test-mongo"
    value {
    }
}
cleanupSubscription: true
}
sink {
className: "org.apache.pulsar.io.mongodb.MongoSink"
configs: "{\"mongoUri\":\"mongodb://pulsar-mongo:27017\",\"database\":\"pulsar\",\"collection\":\"messages\",\"batchSize\":2,\"batchTimeMs\":500}"
typeClassName: "[B"
}
resources {
cpu: 1.0
ram: 1073741824
disk: 10737418240
}
componentType: SINK
, maxBufferedTuples=1024, functionAuthenticationSpec=null, port=38459, clusterName=local)

This piece of log information demonstrates the status of the connections to Mongo and configuration information.

08:21:56.231 [cluster-ClusterId{value='5d6396a3c9e77c0569ff00eb', description='null'}-pulsar-mongo:27017] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:8}] to pulsar-mongo:27017
08:21:56.326 [cluster-ClusterId{value='5d6396a3c9e77c0569ff00eb', description='null'}-pulsar-mongo:27017] INFO  org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=pulsar-mongo:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 0]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=89058800}

This piece of log information explains the configuration of consumers and clients, including the topic name, subscription name, subscription type, and so on.

08:21:56.719 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer perf with config: {
"topicNames" : [ "test-mongo" ],
"topicsPattern" : null,
"subscriptionName" : "public/default/pulsar-mongo-sink",
"subscriptionType" : "Shared",
"receiverQueueSize" : 1000,
"acknowledgementsGroupTimeMicros" : 100000,
"negativeAckRedeliveryDelayMicros" : 60000000,
"maxTotalReceiverQueueSizeAcrossPartitions" : 50000,
"consumerName" : null,
"ackTimeoutMillis" : 0,
"tickDurationMillis" : 1000,
"priorityLevel" : 0,
"cryptoFailureAction" : "CONSUME",
"properties" : {
    "application" : "pulsar-sink",
    "id" : "public/default/pulsar-mongo-sink",
    "instance_id" : "0"
},
"readCompacted" : false,
"subscriptionInitialPosition" : "Latest",
"patternAutoDiscoveryPeriod" : 1,
"regexSubscriptionMode" : "PersistentOnly",
"deadLetterPolicy" : null,
"autoUpdatePartitions" : true,
"replicateSubscriptionState" : false,
"resetIncludeHead" : false
}
08:21:56.726 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {
"serviceUrl" : "pulsar://localhost:6650",
"authPluginClassName" : null,
"authParams" : null,
"operationTimeoutMs" : 30000,
"statsIntervalSeconds" : 60,
"numIoThreads" : 1,
"numListenerThreads" : 1,
"connectionsPerBroker" : 1,
"useTcpNoDelay" : true,
"useTls" : false,
"tlsTrustCertsFilePath" : null,
"tlsAllowInsecureConnection" : false,
"tlsHostnameVerificationEnable" : false,
"concurrentLookupRequest" : 5000,
"maxLookupRequest" : 50000,
"maxNumberOfRejectedRequestPerConnection" : 50,
"keepAliveIntervalSeconds" : 30,
"connectionTimeoutMs" : 10000,
"requestTimeoutMs" : 60000,
"defaultBackoffIntervalNanos" : 100000000,
"maxBackoffIntervalNanos" : 30000000000
}

Debug in cluster mode

You can use the following methods to debug a connector in cluster mode:

Use connector log
Use admin CLI

Use connector log

In cluster mode, multiple connectors can run on a worker. To find the log path of a specified connector, use the workerId to locate the connector log.

Use admin CLI

Pulsar admin CLI helps you debug Pulsar connectors with the following subcommands:

get
status
topics stats

Create a Mongo sink

./bin/pulsar-admin sinks create \
--archive pulsar-io-mongo-2.4.0.nar \
--tenant public \
--namespace default \
--inputs test-mongo \
--name pulsar-mongo-sink \
--sink-config-file mongo-sink-config.yaml \
--parallelism 1

`get`

Use the get command to get the basic information about the Mongo sink connector, such as tenant, namespace, name, parallelism, and so on.

./bin/pulsar-admin sinks get --tenant public --namespace default  --name pulsar-mongo-sink
{
  "tenant": "public",
  "namespace": "default",
  "name": "pulsar-mongo-sink",
  "className": "org.apache.pulsar.io.mongodb.MongoSink",
  "inputSpecs": {
    "test-mongo": {
      "isRegexPattern": false
    }
  },
  "configs": {
    "mongoUri": "mongodb://pulsar-mongo:27017",
    "database": "pulsar",
    "collection": "messages",
    "batchSize": 2.0,
    "batchTimeMs": 500.0
  },
  "parallelism": 1,
  "processingGuarantees": "ATLEAST_ONCE",
  "retainOrdering": false,
  "autoAck": true
}

Tip

For more information about the get command, see get.

`status`

Use the status command to get the current status about the Mongo sink connector, such as the number of instance, the number of running instance, instanceId, workerId and so on.

./bin/pulsar-admin sinks status 
--tenant public \
--namespace default  \
--name pulsar-mongo-sink

{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
    "instanceId" : 0,
    "status" : {
    "running" : true,
    "error" : "",
    "numRestarts" : 0,
    "numReadFromPulsar" : 0,
    "numSystemExceptions" : 0,
    "latestSystemExceptions" : [ ],
    "numSinkExceptions" : 0,
    "latestSinkExceptions" : [ ],
    "numWrittenToSink" : 0,
    "lastReceivedTime" : 0,
    "workerId" : "c-standalone-fw-5d202832fd18-8080"
    }
} ]
}

Tip

For more information about the status command, see status.

If there are multiple connectors running on a worker, workerId can locate the worker on which the specified connector is running.

`topics stats`

Use the topics stats command to get the stats for a topic and its connected producer and consumer, such as whether the topic has received messages or not, whether there is a backlog of messages or not, the available permits and other key information. All rates are computed over a 1-minute window and are relative to the last completed 1-minute period.

./bin/pulsar-admin topics stats test-mongo
{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "averageMsgSize" : 0.0,
  "storageSize" : 1,
  "publishers" : [ ],
  "subscriptions" : {
    "public/default/pulsar-mongo-sink" : {
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "msgRateRedeliver" : 0.0,
      "msgBacklog" : 0,
      "blockedSubscriptionOnUnackedMsgs" : false,
      "msgDelayed" : 0,
      "unackedMessages" : 0,
      "type" : "Shared",
      "msgRateExpired" : 0.0,
      "consumers" : [ {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "msgRateRedeliver" : 0.0,
        "consumerName" : "dffdd",
        "availablePermits" : 999,
        "unackedMessages" : 0,
        "blockedConsumerOnUnackedMsgs" : false,
        "metadata" : {
          "instance_id" : "0",
          "application" : "pulsar-sink",
          "id" : "public/default/pulsar-mongo-sink"
        },
        "connectedSince" : "2019-08-26T08:48:07.582Z",
        "clientVersion" : "2.4.0",
        "address" : "/172.17.0.3:57790"
      } ],
      "isReplicated" : false
    }
  },
  "replication" : { },
  "deduplicationStatus" : "Disabled"
}

Tip

For more information about the topic status command, see topic status.

Checklist

This checklist indicates the major areas to check when you debug connectors. It is a reminder of what to look for to ensure a thorough review and an evaluation tool to get the status of connectors.

Does Pulsar start successfully?
Does the external service run normally?
Is the nar package complete?
Is the connector configuration file correct?
In localrun mode, run a connector and check the printed information (connector log) on the console.
In cluster mode：
- Use the get command to get the basic information.
- Use the status command to get the current status.
- Use the topics stats command to get the stats for a specified topic and its connected producers and consumers.
- Check the connector log.
Enter into the external system and verify the result.

Anonymitaet · 2019-08-28T12:37:56Z

site2/docs/io-debug.md

+sidebar_label: Debug
+---
+
+You can use the following methods to debug Pulsar Source or Sink:


Please see my comments in conversation.

Anonymitaet · 2019-09-02T03:20:27Z

run cpp tests

Anonymitaet · 2019-09-03T01:53:47Z

run integration tests

tuteng added 6 commits August 22, 2019 09:47

Add io-debug file

36e6f7b

Merge branch 'master' into document/add-document-for-debug-connector

9123c22

Add debug sink for connectors

0739b88

Modify case

cbfa963

Move subject to Topic

3dbdb26

Move command topic stats

9bea758

tuteng changed the title ~~[docs]Add document for debug connector~~ [doc]Add document for debug connector Aug 26, 2019

Jennifer88huang-zz assigned tuteng Aug 26, 2019

Jennifer88huang-zz added doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. area/connector labels Aug 26, 2019

Jennifer88huang-zz added this to the 2.5.0 milestone Aug 26, 2019

Anonymitaet requested changes Aug 28, 2019

View reviewed changes

Anonymitaet mentioned this pull request Aug 29, 2019

[Doc] Update Pulsar Connector Guide #5015

Closed

tuteng added 2 commits August 29, 2019 20:39

Update debug pulsar connectors

a5cb16d

Format content

d4ca7ae

Anonymitaet approved these changes Aug 29, 2019

View reviewed changes

Update topics stats

7ca0886

Merge branch 'master' into document/add-document-for-debug-connector

cc271c9

sijie approved these changes Sep 2, 2019

View reviewed changes

Anonymitaet mentioned this pull request Sep 3, 2019

No documentation on how to get connector logs #3624

Closed

sijie merged commit 2ed7d6e into apache:master Sep 3, 2019

sijie deleted the document/add-document-for-debug-connector branch September 3, 2019 03:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[doc]Add document for debug connector #5041

[doc]Add document for debug connector #5041

tuteng commented Aug 26, 2019

Jennifer88huang-zz commented Aug 26, 2019

Anonymitaet commented Aug 26, 2019

sijie commented Aug 26, 2019

Anonymitaet commented Aug 28, 2019 •

edited

Loading

Anonymitaet commented Aug 28, 2019 •

edited

Loading

Tip

Tip

Tip

Tip

Tip

Anonymitaet Aug 28, 2019

Anonymitaet commented Sep 2, 2019

Anonymitaet commented Sep 3, 2019

[doc]Add document for debug connector #5041

[doc]Add document for debug connector #5041

Conversation

tuteng commented Aug 26, 2019

Motivation

Modifications

Documentation

Jennifer88huang-zz commented Aug 26, 2019

Anonymitaet commented Aug 26, 2019

sijie commented Aug 26, 2019

Anonymitaet commented Aug 28, 2019 • edited Loading

Anonymitaet commented Aug 28, 2019 • edited Loading

id: io-debug title: How to debug Pulsar connectors sidebar_label: Debug

Debug in localrun mode

Tip

Use connector log

Tip

Debug in cluster mode

Use connector log

Use admin CLI

get

Tip

status

Tip

topics stats

Tip

Checklist

Anonymitaet Aug 28, 2019

Choose a reason for hiding this comment

Anonymitaet commented Sep 2, 2019

Anonymitaet commented Sep 3, 2019

Anonymitaet commented Aug 28, 2019 •

edited

Loading

Anonymitaet commented Aug 28, 2019 •

edited

Loading

id: io-debug
title: How to debug Pulsar connectors
sidebar_label: Debug

`get`

`status`

`topics stats`