Server exits unexpectedly due to sink error

[overvenus]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

Create a changefeed that replicates data to a closed pulsar downstream.

TiCDC server exits unexpectedly.

Log

[2021/08/03 13:14:43.195 +08:00] [INFO] [server.go:235] ["owner exited"] [capture-id=59af0cc1-e1c9-470f-b854-f34aa25ea782]
[2021/08/03 13:14:43.195 +08:00] [ERROR] [server.go:132] ["run server"] [error="[CDC:ErrPulsarNewProducer]connection error\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:279\ngithub.com/pingcap/ticdc/pkg/errors.WrapError\n\tgithub.com/pingcap/ticdc/pkg/errors/helper.go:28\ngithub.com/pingcap/ticdc/cdc/sink/producer/pulsar.NewProducer\n\tgithub.com/pingcap/ticdc/cdc/sink/producer/pulsar/producer.go:49\ngithub.com/pingcap/ticdc/cdc/sink.newPulsarSink\n\tgithub.com/pingcap/ticdc/cdc/sink/mq.go:504\ngithub.com/pingcap/ticdc/cdc/sink.init.1.func4\n\tgithub.com/pingcap/ticdc/cdc/sink/sink.go:88\ngithub.com/pingcap/ticdc/cdc/sink.NewSink\n\tgithub.com/pingcap/ticdc/cdc/sink/sink.go:113\ngithub.com/pingcap/ticdc/cdc.runProcessor\n\tgithub.com/pingcap/ticdc/cdc/processor.go:1358\ngithub.com/pingcap/ticdc/cdc.(*Capture).assignTask\n\tgithub.com/pingcap/ticdc/cdc/capture.go:312\ngithub.com/pingcap/ticdc/cdc.(*Capture).handleTaskEvent\n\tgithub.com/pingcap/ticdc/cdc/capture.go:277\ngithub.com/pingcap/ticdc/cdc.(*Capture).Run\n\tgithub.com/pingcap/ticdc/cdc/capture.go:213\ngithub.com/pingcap/ticdc/cdc.(*Server).run.func3\n\tgithub.com/pingcap/ticdc/cdc/server.go:322\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1373"]
Error: run server: [CDC:ErrPulsarNewProducer]connection error
Usage:
  cdc server [flags]

Flags:
      --addr string                           Set the listening address (default "127.0.0.1:8300")
      --advertise-addr string                 Set the advertise listening address for client communication
      --ca string                             CA certificate path for TLS connection
      --cert string                           Certificate path for TLS connection
      --cert-allowed-cn string                Verify caller's identity (cert Common Name). Use ',' to separate multiple CN
      --config string                         Path of the configuration file
      --data-dir string                       the path to the directory used to store TiCDC-generated data
      --gc-ttl int                            CDC GC safepoint TTL duration, specified in seconds (default 86400)
  -h, --help                                  help for server
      --key string                            Private key path for TLS connection
      --log-file string                       log file path
      --log-level string                      log level (etc: debug|info|warn|error) (default "info")
      --owner-flush-interval duration         owner flushes changefeed status interval (default 200ms)
      --pd string                             Set the PD endpoints to use. Use ',' to separate multiple PDs (default "http://127.0.0.1:2379")
      --processor-flush-interval duration     processor flushes task status interval (default 100ms)
      --sorter-chunk-size-limit uint          size of heaps for sorting (default 134217728)
      --sorter-max-memory-consumption uint    maximum memory consumption of in-memory sort (default 17179869184)
      --sorter-max-memory-percentage int      system memory usage threshold for forcing in-disk sort (default 30)
      --sorter-num-concurrent-worker int      sorter concurrency level (default 4)
      --sorter-num-workerpool-goroutine int   sorter workerpool size (default 16)
      --tz string                             Specify time zone of TiCDC cluster (default "System")

run server: [CDC:ErrPulsarNewProducer]connection error

Stack backtrace:

[CDC:ErrPulsarNewProducer]connection error
github.com/pingcap/errors.AddStack
	github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174
github.com/pingcap/errors.(*Error).GenWithStackByCause
	github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:279
github.com/pingcap/ticdc/pkg/errors.WrapError
	github.com/pingcap/ticdc/pkg/errors/helper.go:28
github.com/pingcap/ticdc/cdc/sink/producer/pulsar.NewProducer
	github.com/pingcap/ticdc/cdc/sink/producer/pulsar/producer.go:49
github.com/pingcap/ticdc/cdc/sink.newPulsarSink
	github.com/pingcap/ticdc/cdc/sink/mq.go:504
github.com/pingcap/ticdc/cdc/sink.init.1.func4
	github.com/pingcap/ticdc/cdc/sink/sink.go:88
github.com/pingcap/ticdc/cdc/sink.NewSink
	github.com/pingcap/ticdc/cdc/sink/sink.go:113
github.com/pingcap/ticdc/cdc.runProcessor
	github.com/pingcap/ticdc/cdc/processor.go:1358
github.com/pingcap/ticdc/cdc.(*Capture).assignTask
	github.com/pingcap/ticdc/cdc/capture.go:312
github.com/pingcap/ticdc/cdc.(*Capture).handleTaskEvent
	github.com/pingcap/ticdc/cdc/capture.go:277
github.com/pingcap/ticdc/cdc.(*Capture).Run
	github.com/pingcap/ticdc/cdc/capture.go:213
github.com/pingcap/ticdc/cdc.(*Server).run.func3
	github.com/pingcap/ticdc/cdc/server.go:322
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57
runtime.goexit
	runtime/asm_amd64.s:1373

2. What did you expect to see?

Do not exit.

3. What did you see instead?

Exits unexpectedly.

4. Versions of the cluster

v4.0.14

[overvenus]

/assign @ben1009

[L3T]

/assign

[sunxiaoguang]

@L3T Do we have any updates?

[asddongmen]

@zhaoxinyu can we colse this issue?

[zhaoxinyu]

The affected version is relatively old. At present, the newPulsarSink operation will be executed in a processor tick, it would not cause an unexpected cdc server exit.

[zhaoxinyu]

/close

[ti-chi-bot]

@zhaoxinyu: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nongfushanquan]

/remove affect-6.1

[nongfushanquan]

/remove affects-6.1