Brokers fail starting on GKE Autopilot #416

theoribeiro · 2023-12-21T14:27:25Z

Describe the bug

I'm installing the helm on GKE with Autopilot turned on but my brokers fail starting up. If I turn off functions by setting components.functions: false the brokers start up correctly.

This is the exception on my brokers' initialization:

2023-12-19T00:47:53,609+0000 [main] INFO  org.apache.pulsar.functions.utils.functions.FunctionUtils - Searching for functions in /pulsar/./functions
2023-12-19T00:47:53,610+0000 [main] WARN  org.apache.pulsar.functions.utils.functions.FunctionUtils - Functions archive directory not found
2023-12-19T00:47:53,612+0000 [main] ERROR org.apache.pulsar.functions.worker.PulsarWorkerService - Error Starting up in worker
org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:251) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:301) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:457) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1608) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:795) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
Caused by: java.lang.IllegalStateException: Client instance has been closed.
    at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1319) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:619) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$20(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:292) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    ... 6 more
2023-12-19T00:47:53,619+0000 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:585) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1608) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:795) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:251) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:301) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:457) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    ... 4 more
Caused by: java.lang.IllegalStateException: Client instance has been closed.
    at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1319) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:619) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$20(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:292) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:457) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    ... 4 more
2023-12-19T00:47:53,620+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.broker.PulsarService - Closing PulsarService
2023-12-19T00:47:53,621+0000 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:827) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
Caused by: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:585) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1608) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:795) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    ... 2 more
Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
    at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:251) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:301) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:457) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1608) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:795) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    ... 2 more
Caused by: java.lang.IllegalStateException: Client instance has been closed.
    at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1319) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:619) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$20(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:292) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:613) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:457) ~[org.apache.pulsar-pulsar-functions-worker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1608) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:795) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    ... 2 more

Any ideas on what may be causing this?

To Reproduce
Steps to reproduce the behavior:

Create a GKE Cluster on Autopilot
Install helm
See errors in brokers

Expected behavior
I expected the helm chart to install the cluster correctly with no errors.

The text was updated successfully, but these errors were encountered:

lhotari · 2023-12-21T15:45:07Z

@theoribeiro Does this happen also with the most recent Helm chart release 3.1.0 ?

theoribeiro · 2023-12-21T17:10:50Z

@lhotari yes, I just updated the chart to test it after seeing the announcement on Slack

lhotari · 2023-12-21T20:32:04Z

@lhotari yes, I just updated the chart to test it after seeing the announcement on Slack

The stack traces reference Pulsar 2.10.2 version which is the default for chart version 3.0.0 . Are you pinning Pulsar version to 2.10.2 ? In chart version 3.1.0, the default Pulsar version is 3.0.2 .

theoribeiro · 2023-12-21T20:35:09Z

Understood, I copied and posted the logs before updating to chart 3.1.0. After updating the issue persisted, I just didn't copy the log messages after that because they were exactly the same.

lhotari · 2023-12-21T20:37:57Z

There must be some other error messages in the k8s events etc. . My guess is that it will be necessary to set functions.rbac.limit_to_namespace to true in the values file.

lhotari · 2023-12-21T20:42:21Z

I made this PR #418 to fix the issue that the chart uses cluster level role & role bindings by default.
We missed making this change before the 3.0.0 release and the 3.1.0 release. We'll have that for the next 3.2.0 release.
In the meantime, change the value in your values.yaml file.

lhotari · 2023-12-21T20:48:34Z

You could also test with passing --set functions.rbac.limit_to_namespace=true to the helm install command line.

lhotari · 2023-12-21T21:00:33Z

another one to test is --set functions.rbac.limit_to_namespace=true as long as we are blind about the reason.

@theoribeiro
can you check kubectl get events --sort-by=.lastTimestamp -A / kubectl get -n pulsar_namespace events --sort-by=.lastTimestamp output too?

theoribeiro · 2023-12-21T21:55:38Z

@lhotari
Here's the output of kubectl get -n pulsar_namespace events --sort-by=.lastTimestamp after setting functions.rbac.limit_to_namespace=true

7m18s       Normal    NoPods                           poddisruptionbudget/pulsar-bookie                                No matching pods found
7m18s       Normal    ADD                              secret/sh.helm.release.v1.pulsar.v1                              pulsar/sh.helm.release.v1.pulsar.v1
7m17s       Normal    NoPods                           poddisruptionbudget/pulsar-proxy                                 No matching pods found
7m17s       Normal    NoPods                           poddisruptionbudget/pulsar-zookeeper                             No matching pods found
7m11s       Normal    ADD                              service/pulsar-recovery                                          pulsar/pulsar-recovery
7m11s       Normal    ADD                              service/pulsar-bookie                                            pulsar/pulsar-bookie
7m11s       Normal    ADD                              service/pulsar-broker                                            pulsar/pulsar-broker
7m10s       Normal    UPDATE                           service/pulsar-proxy                                             pulsar/pulsar-proxy
7m10s       Normal    EnsuringLoadBalancer             service/pulsar-proxy                                             Ensuring load balancer
7m10s       Normal    ADD                              service/pulsar-proxy                                             pulsar/pulsar-proxy
7m10s       Normal    ADD                              service/pulsar-zookeeper                                         pulsar/pulsar-zookeeper
7m10s       Normal    ADD                              service/pulsar-toolset                                           pulsar/pulsar-toolset
7m9s        Normal    WaitForFirstConsumer             persistentvolumeclaim/pulsar-bookie-journal-pulsar-bookie-0      waiting for first consumer to be created before binding
7m9s        Normal    ExternalProvisioning             persistentvolumeclaim/pulsar-bookie-journal-pulsar-bookie-0      waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
7m9s        Normal    ExternalProvisioning             persistentvolumeclaim/pulsar-bookie-ledgers-pulsar-bookie-0      waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
7m9s        Normal    Provisioning                     persistentvolumeclaim/pulsar-bookie-ledgers-pulsar-bookie-0      External provisioner is provisioning volume for claim "pulsar/pulsar-bookie-ledgers-pulsar-bookie-0"
7m9s        Normal    SuccessfulCreate                 statefulset/pulsar-recovery                                      create Pod pulsar-recovery-0 in StatefulSet pulsar-recovery successful
7m9s        Normal    WaitForFirstConsumer             persistentvolumeclaim/pulsar-bookie-ledgers-pulsar-bookie-0      waiting for first consumer to be created before binding
7m9s        Normal    Scheduled                        pod/pulsar-recovery-0                                            Successfully assigned pulsar/pulsar-recovery-0 to gk3-trinio-pool-3-afc9cad4-t8tf
7m9s        Normal    SuccessfulCreate                 statefulset/pulsar-bookie                                        create Claim pulsar-bookie-journal-pulsar-bookie-0 Pod pulsar-bookie-0 in StatefulSet pulsar-bookie success
7m9s        Normal    Provisioning                     persistentvolumeclaim/pulsar-bookie-journal-pulsar-bookie-0      External provisioner is provisioning volume for claim "pulsar/pulsar-bookie-journal-pulsar-bookie-0"
7m9s        Normal    SuccessfulCreate                 statefulset/pulsar-bookie                                        create Pod pulsar-bookie-0 in StatefulSet pulsar-bookie successful
7m9s        Normal    SuccessfulCreate                 statefulset/pulsar-bookie                                        create Claim pulsar-bookie-ledgers-pulsar-bookie-0 Pod pulsar-bookie-0 in StatefulSet pulsar-bookie success
7m8s        Normal    Scheduled                        pod/pulsar-proxy-0                                               Successfully assigned pulsar/pulsar-proxy-0 to gk3-trinio-pool-3-afc9cad4-t8tf
7m8s        Normal    DNSRecordProvisioningSucceeded   service/pulsar-proxy                                             DNS records updated
7m8s        Normal    Created                          pod/pulsar-recovery-0                                            Created container pulsar-bookkeeper-verify-clusterid
7m8s        Normal    Scheduled                        pod/pulsar-broker-0                                              Successfully assigned pulsar/pulsar-broker-0 to gk3-trinio-pool-3-afc9cad4-t8tf
7m8s        Normal    SuccessfulCreate                 statefulset/pulsar-toolset                                       create Pod pulsar-toolset-0 in StatefulSet pulsar-toolset successful
7m8s        Normal    Scheduled                        pod/pulsar-toolset-0                                             Successfully assigned pulsar/pulsar-toolset-0 to gk3-trinio-pool-3-afc9cad4-t8tf
7m8s        Normal    Pulled                           pod/pulsar-recovery-0                                            Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
7m8s        Normal    Started                          pod/pulsar-recovery-0                                            Started container pulsar-bookkeeper-verify-clusterid
7m8s        Normal    SuccessfulCreate                 statefulset/pulsar-proxy                                         create Pod pulsar-proxy-0 in StatefulSet pulsar-proxy successful
7m7s        Normal    WaitForFirstConsumer             persistentvolumeclaim/pulsar-zookeeper-data-pulsar-zookeeper-0   waiting for first consumer to be created before binding
7m7s        Normal    Created                          pod/pulsar-proxy-0                                               Created container wait-zookeeper-ready
7m7s        Normal    Pulled                           pod/pulsar-proxy-0                                               Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
7m7s        Normal    SuccessfulCreate                 job/pulsar-pulsar-init                                           Created pod: pulsar-pulsar-init-k8twv
7m7s        Normal    SuccessfulCreate                 job/pulsar-bookie-init                                           Created pod: pulsar-bookie-init-cw8qb
7m7s        Normal    SuccessfulCreate                 statefulset/pulsar-zookeeper                                     create Pod pulsar-zookeeper-0 in StatefulSet pulsar-zookeeper successful
7m7s        Normal    Pulled                           pod/pulsar-toolset-0                                             Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
7m7s        Normal    Pulled                           pod/pulsar-broker-0                                              Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
7m7s        Normal    Created                          pod/pulsar-broker-0                                              Created container wait-zookeeper-ready
7m7s        Normal    Started                          pod/pulsar-broker-0                                              Started container wait-zookeeper-ready
7m7s        Normal    SuccessfulCreate                 statefulset/pulsar-zookeeper                                     create Claim pulsar-zookeeper-data-pulsar-zookeeper-0 Pod pulsar-zookeeper-0 in StatefulSet pulsar-zookeeper success
7m7s        Normal    Started                          pod/pulsar-proxy-0                                               Started container wait-zookeeper-ready
7m7s        Normal    Provisioning                     persistentvolumeclaim/pulsar-zookeeper-data-pulsar-zookeeper-0   External provisioner is provisioning volume for claim "pulsar/pulsar-zookeeper-data-pulsar-zookeeper-0"
7m6s        Normal    Started                          pod/pulsar-toolset-0                                             Started container pulsar-toolset
7m6s        Normal    Created                          pod/pulsar-toolset-0                                             Created container pulsar-toolset
7m5s        Normal    ProvisioningSucceeded            persistentvolumeclaim/pulsar-bookie-journal-pulsar-bookie-0      Successfully provisioned volume pvc-5995ec5a-df45-42de-886a-166d2a7f2378
7m5s        Normal    Scheduled                        pod/pulsar-bookie-0                                              Successfully assigned pulsar/pulsar-bookie-0 to gk3-trinio-pool-3-afc9cad4-t8tf
7m5s        Normal    ProvisioningSucceeded            persistentvolumeclaim/pulsar-bookie-ledgers-pulsar-bookie-0      Successfully provisioned volume pvc-5afbf7bd-2fc8-4a03-94da-1f1f7a3f8938
7m5s        Normal    DNSRecordProvisioningSucceeded   service/pulsar-toolset                                           DNS records updated
7m4s        Normal    ExternalProvisioning             persistentvolumeclaim/pulsar-zookeeper-data-pulsar-zookeeper-0   waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
7m3s        Normal    Scheduled                        pod/pulsar-zookeeper-0                                           Successfully assigned pulsar/pulsar-zookeeper-0 to gk3-trinio-pool-3-afc9cad4-t8tf
7m3s        Normal    ProvisioningSucceeded            persistentvolumeclaim/pulsar-zookeeper-data-pulsar-zookeeper-0   Successfully provisioned volume pvc-d1e96ae9-d07e-485e-ba6a-68323c5fa062
7m          Normal    SuccessfulAttachVolume           pod/pulsar-bookie-0                                              AttachVolume.Attach succeeded for volume "pvc-5afbf7bd-2fc8-4a03-94da-1f1f7a3f8938"
6m56s       Normal    SuccessfulAttachVolume           pod/pulsar-zookeeper-0                                           AttachVolume.Attach succeeded for volume "pvc-d1e96ae9-d07e-485e-ba6a-68323c5fa062"
6m55s       Normal    TriggeredScaleUp                 pod/pulsar-bookie-init-cw8qb                                     pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/trinio-dev/zones/us-central1-f/instanceGroups/gk3-trinio-pool-2-02bc9742-grp 0->1 (max: 1000)}]
6m55s       Normal    TriggeredScaleUp                 pod/pulsar-pulsar-init-k8twv                                     pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/trinio-dev/zones/us-central1-f/instanceGroups/gk3-trinio-pool-2-02bc9742-grp 0->1 (max: 1000)}]
6m53s       Normal    Started                          pod/pulsar-zookeeper-0                                           Started container pulsar-zookeeper
6m53s       Normal    Pulled                           pod/pulsar-zookeeper-0                                           Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
6m53s       Normal    Created                          pod/pulsar-zookeeper-0                                           Created container pulsar-zookeeper
6m52s       Normal    SuccessfulAttachVolume           pod/pulsar-bookie-0                                              AttachVolume.Attach succeeded for volume "pvc-5995ec5a-df45-42de-886a-166d2a7f2378"
6m50s       Normal    Created                          pod/pulsar-bookie-0                                              Created container pulsar-bookkeeper-verify-clusterid
6m50s       Normal    Started                          pod/pulsar-bookie-0                                              Started container pulsar-bookkeeper-verify-clusterid
6m50s       Normal    Pulled                           pod/pulsar-bookie-0                                              Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
6m50s       Normal    DNSRecordProvisioningSucceeded   service/pulsar-zookeeper                                         DNS records updated
6m49s       Normal    DNSRecordProvisioningSucceeded   service/pulsar-bookie                                            DNS records updated
6m24s       Warning   FailedScheduling                 pod/pulsar-bookie-init-cw8qb                                     0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
6m24s       Warning   FailedScheduling                 pod/pulsar-pulsar-init-k8twv                                     0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
6m21s       Normal    EnsuredLoadBalancer              service/pulsar-proxy                                             Ensured load balancer
5m46s       Normal    Scheduled                        pod/pulsar-bookie-init-cw8qb                                     Successfully assigned pulsar/pulsar-bookie-init-cw8qb to gk3-trinio-pool-2-02bc9742-5zkb
5m46s       Normal    Scheduled                        pod/pulsar-pulsar-init-k8twv                                     Successfully assigned pulsar/pulsar-pulsar-init-k8twv to gk3-trinio-pool-2-02bc9742-5zkb
5m44s       Normal    TaintManagerEviction             pod/pulsar-bookie-init-cw8qb                                     Cancelling deletion of Pod pulsar/pulsar-bookie-init-cw8qb
5m44s       Normal    TaintManagerEviction             pod/pulsar-pulsar-init-k8twv                                     Cancelling deletion of Pod pulsar/pulsar-pulsar-init-k8twv
5m17s       Normal    Pulling                          pod/pulsar-pulsar-init-k8twv                                     Pulling image "apachepulsar/pulsar-all:3.0.2"
5m13s       Normal    Pulling                          pod/pulsar-bookie-init-cw8qb                                     Pulling image "apachepulsar/pulsar-all:3.0.2"
4m14s       Normal    Pulled                           pod/pulsar-pulsar-init-k8twv                                     Successfully pulled image "apachepulsar/pulsar-all:3.0.2" in 1m3.151860213s (1m3.151872603s including waiting)
4m14s       Normal    Pulled                           pod/pulsar-bookie-init-cw8qb                                     Successfully pulled image "apachepulsar/pulsar-all:3.0.2" in 39.039710652s (59.646624077s including waiting)
4m10s       Normal    Created                          pod/pulsar-pulsar-init-k8twv                                     Created container wait-zookeeper-ready
4m10s       Normal    Created                          pod/pulsar-bookie-init-cw8qb                                     Created container wait-zookeeper-ready
4m9s        Normal    Started                          pod/pulsar-pulsar-init-k8twv                                     Started container wait-zookeeper-ready
4m9s        Normal    Started                          pod/pulsar-bookie-init-cw8qb                                     Started container wait-zookeeper-ready
4m4s        Normal    Pulled                           pod/pulsar-pulsar-init-k8twv                                     Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
4m4s        Normal    Created                          pod/pulsar-pulsar-init-k8twv                                     Created container pulsar-bookkeeper-verify-clusterid
4m4s        Normal    Pulled                           pod/pulsar-bookie-init-cw8qb                                     Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
4m4s        Normal    Created                          pod/pulsar-bookie-init-cw8qb                                     Created container pulsar-bookie-init
4m3s        Normal    Started                          pod/pulsar-pulsar-init-k8twv                                     Started container pulsar-bookkeeper-verify-clusterid
4m3s        Normal    Started                          pod/pulsar-bookie-init-cw8qb                                     Started container pulsar-bookie-init
3m43s       Normal    Created                          pod/pulsar-pulsar-init-k8twv                                     Created container pulsar-pulsar-init
3m43s       Normal    Pulled                           pod/pulsar-pulsar-init-k8twv                                     Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
3m43s       Normal    Started                          pod/pulsar-pulsar-init-k8twv                                     Started container pulsar-pulsar-init
3m42s       Normal    Completed                        job/pulsar-bookie-init                                           Job completed
3m37s       Normal    Pulled                           pod/pulsar-recovery-0                                            Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
3m37s       Normal    Started                          pod/pulsar-recovery-0                                            Started container pulsar-recovery
3m37s       Normal    Pulled                           pod/pulsar-bookie-0                                              Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
3m37s       Normal    Created                          pod/pulsar-bookie-0                                              Created container pulsar-bookie
3m37s       Normal    Started                          pod/pulsar-bookie-0                                              Started container pulsar-bookie
3m37s       Normal    Created                          pod/pulsar-recovery-0                                            Created container pulsar-recovery
3m35s       Normal    DNSRecordProvisioningSucceeded   service/pulsar-recovery                                          DNS records updated
3m28s       Normal    Completed                        job/pulsar-pulsar-init                                           Job completed
3m25s       Normal    Pulled                           pod/pulsar-broker-0                                              Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
3m25s       Normal    Created                          pod/pulsar-broker-0                                              Created container wait-bookkeeper-ready
3m25s       Normal    Started                          pod/pulsar-broker-0                                              Started container wait-bookkeeper-ready
3m24s       Normal    Pulled                           pod/pulsar-proxy-0                                               Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
3m24s       Normal    Created                          pod/pulsar-proxy-0                                               Created container wait-broker-ready
3m23s       Normal    Started                          pod/pulsar-proxy-0                                               Started container wait-broker-ready
3m21s       Warning   Unhealthy                        pod/pulsar-bookie-0                                              Readiness probe failed: Get "http://10.108.1.168:8000/api/v1/bookie/is_ready": dial tcp 10.108.1.168:8000: connect: connection refused
3m21s       Warning   Unhealthy                        pod/pulsar-bookie-0                                              Liveness probe failed: Get "http://10.108.1.168:8000/api/v1/bookie/state": dial tcp 10.108.1.168:8000: connect: connection refused
3m16s       Normal    Pulled                           pod/pulsar-broker-0                                              Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
3m16s       Normal    Created                          pod/pulsar-broker-0                                              Created container pulsar-broker
3m16s       Normal    Started                          pod/pulsar-broker-0                                              Started container pulsar-broker
2m38s       Warning   Unhealthy                        pod/pulsar-broker-0                                              Readiness probe failed: Get "http://10.108.1.164:8080/status.html": dial tcp 10.108.1.164:8080: connect: connection refused
2m38s       Warning   Unhealthy                        pod/pulsar-broker-0                                              Liveness probe failed: Get "http://10.108.1.164:8080/status.html": dial tcp 10.108.1.164:8080: connect: connection refused
2m6s        Normal    UPDATE                           secret/sh.helm.release.v1.pulsar.v1                              pulsar/sh.helm.release.v1.pulsar.v1
118s        Warning   Unhealthy                        pod/pulsar-broker-0                                              Liveness probe failed: HTTP probe failed with statuscode: 404
114s        Warning   Unhealthy                        pod/pulsar-broker-0                                              Readiness probe failed: HTTP probe failed with statuscode: 404
110s        Normal    Killing                          pod/pulsar-broker-0                                              Stopping container pulsar-broker
110s        Warning   Evicted                          pod/pulsar-broker-0                                              Pod ephemeral local storage usage exceeds the total limit of containers 1Gi.
100s        Warning   ExceededGracePeriod              pod/pulsar-broker-0                                              Container runtime did not kill the pod within specified grace period.
97s         Normal    NoPods                           poddisruptionbudget/pulsar-broker                                No matching pods found
97s         Normal    SuccessfulCreate                 statefulset/pulsar-broker                                        create Pod pulsar-broker-0 in StatefulSet pulsar-broker successful
97s         Warning   RecreatingFailedPod              statefulset/pulsar-broker                                        StatefulSet pulsar/pulsar-broker is recreating failed Pod pulsar-broker-0
97s         Normal    SuccessfulDelete                 statefulset/pulsar-broker                                        delete Pod pulsar-broker-0 in StatefulSet pulsar-broker successful
97s         Normal    Scheduled                        pod/pulsar-broker-0                                              Successfully assigned pulsar/pulsar-broker-0 to gk3-trinio-pool-3-afc9cad4-t8tf
96s         Normal    Pulled                           pod/pulsar-broker-0                                              Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
96s         Normal    Created                          pod/pulsar-broker-0                                              Created container wait-zookeeper-ready
96s         Normal    Started                          pod/pulsar-broker-0                                              Started container wait-zookeeper-ready
95s         Normal    DNSRecordProvisioningSucceeded   service/pulsar-broker                                            DNS records updated
89s         Normal    Pulled                           pod/pulsar-broker-0                                              Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
88s         Normal    Created                          pod/pulsar-broker-0                                              Created container wait-bookkeeper-ready
88s         Normal    Started                          pod/pulsar-broker-0                                              Started container wait-bookkeeper-ready
79s         Normal    Pulled                           pod/pulsar-broker-0                                              Container image "apachepulsar/pulsar-all:3.0.2" already present on machine
79s         Normal    Created                          pod/pulsar-broker-0                                              Created container pulsar-broker
79s         Normal    Started                          pod/pulsar-broker-0                                              Started container pulsar-broker
27s         Warning   Unhealthy                        pod/pulsar-broker-0                                              Readiness probe failed: Get "http://10.108.1.171:8080/status.html": dial tcp 10.108.1.171:8080: connect: connection refused
27s         Warning   Unhealthy                        pod/pulsar-broker-0                                              Liveness probe failed: Get "http://10.108.1.171:8080/status.html": dial tcp 10.108.1.171:8080: connect: connection refused
18s         Normal    UpdatedLoadBalancer              service/pulsar-proxy                                             Updated load balancer with new hosts
7s          Warning   Unhealthy                        pod/pulsar-broker-0                                              Liveness probe failed: HTTP probe failed with statuscode: 404
0s          Warning   Unhealthy                        pod/pulsar-broker-0                                              Readiness probe failed: HTTP probe failed with statuscode: 404

theoribeiro · 2023-12-21T22:15:40Z

@lhotari Also happening after setting rbac.limit_to_namespace=true.

Here are the full logs of the broker's initialization:

[conf/broker.conf] Applying config brokerServicePort = 6650
[conf/broker.conf] Applying config clusterName = pulsar
[conf/broker.conf] Applying config configurationStoreServers = pulsar-zookeeper:2181
[conf/broker.conf] Applying config exposeTopicLevelMetricsInPrometheus = true
[conf/broker.conf] Applying config functionsWorkerEnabled = true
[conf/broker.conf] Applying config managedLedgerDefaultAckQuorum = 1
[conf/broker.conf] Applying config managedLedgerDefaultEnsembleSize = 1
[conf/broker.conf] Applying config managedLedgerDefaultWriteQuorum = 1
[conf/broker.conf] Applying config numHttpServerThreads = 8
[conf/broker.conf] Applying config statusFilePath = /pulsar/status
[conf/broker.conf] Applying config webServicePort = 8080
[conf/broker.conf] Applying config zooKeeperSessionTimeoutMillis = 30000
[conf/broker.conf] Applying config zookeeperServers = pulsar-zookeeper:2181
Connecting to pulsar-zookeeper:2181
2023-12-21T22:12:03,353+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.8.3-6ad6d364c7c0bcf0de452d54ebefa3058098ab56, built on 2023-10-05 10:34 UTC
2023-12-21T22:12:03,440+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local
2023-12-21T22:12:03,440+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.9
2023-12-21T22:12:03,441+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Eclipse Adoptium
2023-12-21T22:12:03,441+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/temurin-17-jdk-amd64
2023-12-21T22:12:03,442+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/io.grpc-grpc-all-1.55.3.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/com.auth0-jwks-rsa-0.22.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-filesystem-storage-3.0.2.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-utils-3.0.2.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.16.3.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-proxy-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-local-runner-original-3.0.2.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.11.3.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.9.3.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/pulsar/lib/io.netty-netty-common-4.1.100.Final.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-x86_64.jar:/pulsar/lib/net.java.dev.jna-jna-platform-jpms-5.12.1.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.6.0.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.8.3.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.16.0.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/io.vertx-vertx-grpc-4.3.5.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.4.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.16.3.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/io.netty-netty-handler-4.1.100.Final.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/io.kubernetes-client-java-proto-18.0.0.jar:/pulsar/lib/io.etcd-jetcd-api-0.7.5.jar:/pulsar/lib/com.fasterxml.jackson.datatype-jackson-datatype-jdk8-2.14.2.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.55.3.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/com.google.guava-guava-32.1.1-jre.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.100.Final.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/pulsar/lib/io.grpc-grpc-servlet-1.55.3.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar:/pulsar/lib/com.google.re2j-re2j-1.6.jar:/pulsar/lib/io.etcd-jetcd-core-0.7.5.jar:/pulsar/lib/io.vertx-vertx-auth-common-4.3.8.jar:/pulsar/lib/io.grpc-grpc-rls-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-api-3.0.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.53.v20231009.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-proto-3.0.2.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-original-3.0.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-auth-sasl-3.0.2.jar:/pulsar/lib/com.squareup.okio-okio-jvm-3.4.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.53.v20231009.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.55.3.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/org.roaringbitmap-RoaringBitmap-0.9.44.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.16.3.jar:/pulsar/lib/io.grpc-grpc-xds-1.55.3.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/com.squareup.okio-okio-3.4.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-managed-ledger-3.0.2.jar:/pulsar/lib/net.java.dev.jna-jna-jpms-5.12.1.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/com.carrotsearch-hppc-0.9.1.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/io.grpc-grpc-context-1.55.3.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/io.grpc-grpc-stub-1.55.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.8.3.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.100.Final.jar:/pulsar/lib/io.perfmark-perfmark-api-0.26.0.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.100.Final.jar:/pulsar/lib/org.jline-jline-3.21.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.10.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:/pulsar/lib/io.netty-netty-transport-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-websocket-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.16.3.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:/pulsar/lib/com.github.oshi-oshi-core-java11-6.4.0.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.53.v20231009.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.8.3.jar:/pulsar/lib/io.etcd-jetcd-grpc-0.7.5.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-oidc-3.0.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.6.jar:/pulsar/lib/com.fasterxml.jackson.datatype-jackson-datatype-jsr310-2.14.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.16.3.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.3.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-resolver-dns-classes-macos-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-core-3.0.2.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.100.Final.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/io.netty-netty-codec-4.1.100.Final.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/io.grpc-grpc-alts-1.55.3.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.55.3.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.16.3.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-secrets-3.0.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.53.v20231009.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-slogger-api-4.16.3.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.9.0.jar:/pulsar/lib/org.apache.pulsar-bouncy-castle-bc-3.0.2-pkg.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.100.Final.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-coordinator-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-api-3.0.2.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/pulsar/lib/io.etcd-jetcd-common-0.7.5.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-buffer-4.1.100.Final.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-core-3.0.2.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar:/pulsar/lib/com.auth0-java-jwt-4.3.0.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/org.checkerframework-checker-qual-3.33.0.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.16.3.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.53.v20231009.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.16.3.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/io.swagger-swagger-core-1.6.10.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.16.3.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-3.0.2.jar:/pulsar/lib/io.grpc-grpc-core-1.55.3.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.16.3.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/io.kubernetes-client-java-18.0.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.53.v20231009.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/io.vertx-vertx-bridge-common-4.3.8.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.3.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-sasl-3.0.2.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-config-validation-3.0.2.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/io.grpc-grpc-servlet-jakarta-1.55.3.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-testclient-3.0.2.jar:/pulsar/lib/io.vertx-vertx-web-common-4.3.8.jar:/pulsar/lib/org.apache.pulsar-pulsar-metadata-3.0.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-3.0.2.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-aarch_64.jar:/pulsar/lib/org.apache.bookkeeper-native-io-4.16.3.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-3.0.2.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.10.5.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.16.3.jar:/pulsar/lib/io.kubernetes-client-java-api-18.0.0.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-common-3.0.2.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.16.3.jar:/pulsar/lib/io.netty-netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.53.v20231009.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/io.grpc-grpc-auth-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-runtime-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.16.3.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:/pulsar/lib/io.grpc-grpc-api-1.55.3.jar:/pulsar/lib/net.jodah-failsafe-2.4.4.jar:/pulsar/lib/io.prometheus-simpleclient-0.16.0.jar:/pulsar/lib/io.vertx-vertx-core-4.3.8.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.16.3.jar:/pulsar/lib/org.apache.avro-avro-1.11.3.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.16.3.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.16.3-tests.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.100.Final-linux-x86_64.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-api-3.0.2.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.16.3.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-3.0.2.jar:/pulsar/lib/io.swagger-swagger-models-1.6.10.jar:/pulsar/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/pulsar/lib/org.apache.commons-commons-text-1.10.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-resolver-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.6.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.53.v20231009.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/pulsar/lib/org.yaml-snakeyaml-2.0.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.61.Final.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.16.3.jar:/pulsar/lib/io.vertx-vertx-web-4.3.8.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.16.3.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/pulsar/lib/com.rabbitmq-amqp-client-5.5.3.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/org.apache.pulsar-testmocks-3.0.2.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.100.Final.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-instance-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-common-3.0.2.jar:/pulsar/lib/io.prometheus.jmx-collector-0.16.1.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-parameter-names-2.14.2.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.0.jar:/pulsar/lib/io.grpc-grpc-services-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-common-3.0.2.jar:/pulsar/lib/com.beust-jcommander-1.82.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.100.Final.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.3.jar:/pulsar/lib/io.netty-netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:/pulsar/lib/org.rocksdb-rocksdbjni-7.9.2.jar:/pulsar/lib/io.grpc-grpc-netty-1.55.3.jar:
2023-12-21T22:12:03,444+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2023-12-21T22:12:03,444+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2023-12-21T22:12:03,444+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2023-12-21T22:12:03,445+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2023-12-21T22:12:03,445+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2023-12-21T22:12:03,445+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.15.109+
2023-12-21T22:12:03,446+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=?
2023-12-21T22:12:03,448+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=?
2023-12-21T22:12:03,448+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar
2023-12-21T22:12:03,448+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=67MB
2023-12-21T22:12:03,449+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=256MB
2023-12-21T22:12:03,449+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=130MB
2023-12-21T22:12:03,454+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-zookeeper:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@696f0212
2023-12-21T22:12:03,537+0000 [main] INFO  org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2023-12-21T22:12:03,549+0000 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
2023-12-21T22:12:03,637+0000 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2023-12-21T22:12:03,651+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/10.108.1.167:2181.
2023-12-21T22:12:03,653+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-12-21T22:12:03,742+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.108.1.180:38318, server: pulsar-zookeeper/10.108.1.167:2181
2023-12-21T22:12:03,750+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper/10.108.1.167:2181, session id = 0x10000d9e80d00a4, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
{"webServiceUrl":"http://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080","pulsarServiceUrl":"pulsar://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:6650","persistentTopicsEnabled":true,"nonPersistentTopicsEnabled":true,"cpu":{"usage":48.74491942069563,"limit":50.0},"memory":{"usage":105.8018798828125,"limit":256.0},"directMemory":{"usage":8.0,"limit":256.0},"bandwidthIn":{"usage":8.333183340832958,"limit":1.0E7},"bandwidthOut":{"usage":7.8358748729230205,"limit":1.0E7},"msgThroughputIn":0.0,"msgThroughputOut":0.0,"msgRateIn":0.0,"msgRateOut":0.0,"lastUpdate":1703196691340,"lastStats":{"public/functions/0x40000000_0x80000000":{"msgRateIn":0.0,"msgThroughputIn":0.0,"msgRateOut":0.0,"msgThroughputOut":0.0,"consumerCount":0,"producerCount":0,"topics":3,"cacheSize":0},"public/functions/0x00000000_0x40000000":{"msgRateIn":0.0,"msgThroughputIn":0.0,"msgRateOut":0.0,"msgThroughputOut":0.0,"consumerCount":1,"producerCount":0,"topics":1,"cacheSize":0}},"numTopics":4,"numBundles":2,"numConsumers":1,"numProducers":0,"bundles":["public/functions/0x40000000_0x80000000","public/functions/0x00000000_0x40000000"],"lastBundleGains":["public/functions/0x40000000_0x80000000","public/functions/0x00000000_0x40000000"],"lastBundleLosses":[],"brokerVersionString":"3.0.2","protocols":{},"advertisedListeners":{},"loadManagerClassName":"org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl","startTimestamp":1703196640862,"loadReportType":"LocalBrokerData","maxResourceUsage":0.9748983979225159}
2023-12-21T22:12:03,838+0000 [main] INFO  org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 0
broker pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local znode still exists ... check in 10 seconds ...
Connecting to pulsar-zookeeper:2181
2023-12-21T22:12:20,755+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.8.3-6ad6d364c7c0bcf0de452d54ebefa3058098ab56, built on 2023-10-05 10:34 UTC
2023-12-21T22:12:20,843+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local
2023-12-21T22:12:20,844+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.9
2023-12-21T22:12:20,844+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Eclipse Adoptium
2023-12-21T22:12:20,844+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/temurin-17-jdk-amd64
2023-12-21T22:12:20,845+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/io.grpc-grpc-all-1.55.3.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/com.auth0-jwks-rsa-0.22.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-filesystem-storage-3.0.2.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-utils-3.0.2.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.16.3.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-proxy-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-local-runner-original-3.0.2.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.11.3.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.9.3.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/pulsar/lib/io.netty-netty-common-4.1.100.Final.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-x86_64.jar:/pulsar/lib/net.java.dev.jna-jna-platform-jpms-5.12.1.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.6.0.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.8.3.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.16.0.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/io.vertx-vertx-grpc-4.3.5.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.4.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.16.3.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/io.netty-netty-handler-4.1.100.Final.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/io.kubernetes-client-java-proto-18.0.0.jar:/pulsar/lib/io.etcd-jetcd-api-0.7.5.jar:/pulsar/lib/com.fasterxml.jackson.datatype-jackson-datatype-jdk8-2.14.2.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.55.3.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/com.google.guava-guava-32.1.1-jre.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.100.Final.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/pulsar/lib/io.grpc-grpc-servlet-1.55.3.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar:/pulsar/lib/com.google.re2j-re2j-1.6.jar:/pulsar/lib/io.etcd-jetcd-core-0.7.5.jar:/pulsar/lib/io.vertx-vertx-auth-common-4.3.8.jar:/pulsar/lib/io.grpc-grpc-rls-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-api-3.0.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.53.v20231009.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-proto-3.0.2.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-original-3.0.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-auth-sasl-3.0.2.jar:/pulsar/lib/com.squareup.okio-okio-jvm-3.4.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.53.v20231009.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.55.3.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/org.roaringbitmap-RoaringBitmap-0.9.44.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.16.3.jar:/pulsar/lib/io.grpc-grpc-xds-1.55.3.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/com.squareup.okio-okio-3.4.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-managed-ledger-3.0.2.jar:/pulsar/lib/net.java.dev.jna-jna-jpms-5.12.1.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/com.carrotsearch-hppc-0.9.1.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/io.grpc-grpc-context-1.55.3.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/io.grpc-grpc-stub-1.55.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.8.3.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.100.Final.jar:/pulsar/lib/io.perfmark-perfmark-api-0.26.0.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.100.Final.jar:/pulsar/lib/org.jline-jline-3.21.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.10.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:/pulsar/lib/io.netty-netty-transport-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-websocket-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.16.3.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:/pulsar/lib/com.github.oshi-oshi-core-java11-6.4.0.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.53.v20231009.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.8.3.jar:/pulsar/lib/io.etcd-jetcd-grpc-0.7.5.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-oidc-3.0.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.6.jar:/pulsar/lib/com.fasterxml.jackson.datatype-jackson-datatype-jsr310-2.14.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.16.3.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.3.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-resolver-dns-classes-macos-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-core-3.0.2.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.100.Final.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/io.netty-netty-codec-4.1.100.Final.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/io.grpc-grpc-alts-1.55.3.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.55.3.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.16.3.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-secrets-3.0.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.53.v20231009.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-slogger-api-4.16.3.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.9.0.jar:/pulsar/lib/org.apache.pulsar-bouncy-castle-bc-3.0.2-pkg.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.100.Final.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-coordinator-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-api-3.0.2.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/pulsar/lib/io.etcd-jetcd-common-0.7.5.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-buffer-4.1.100.Final.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-core-3.0.2.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar:/pulsar/lib/com.auth0-java-jwt-4.3.0.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/org.checkerframework-checker-qual-3.33.0.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.16.3.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.53.v20231009.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.16.3.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/io.swagger-swagger-core-1.6.10.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.16.3.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-3.0.2.jar:/pulsar/lib/io.grpc-grpc-core-1.55.3.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.16.3.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/io.kubernetes-client-java-18.0.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.53.v20231009.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/io.vertx-vertx-bridge-common-4.3.8.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.3.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-sasl-3.0.2.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-config-validation-3.0.2.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/io.grpc-grpc-servlet-jakarta-1.55.3.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-testclient-3.0.2.jar:/pulsar/lib/io.vertx-vertx-web-common-4.3.8.jar:/pulsar/lib/org.apache.pulsar-pulsar-metadata-3.0.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-3.0.2.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-aarch_64.jar:/pulsar/lib/org.apache.bookkeeper-native-io-4.16.3.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-3.0.2.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.10.5.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.16.3.jar:/pulsar/lib/io.kubernetes-client-java-api-18.0.0.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-common-3.0.2.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.16.3.jar:/pulsar/lib/io.netty-netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.53.v20231009.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/io.grpc-grpc-auth-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-runtime-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.16.3.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:/pulsar/lib/io.grpc-grpc-api-1.55.3.jar:/pulsar/lib/net.jodah-failsafe-2.4.4.jar:/pulsar/lib/io.prometheus-simpleclient-0.16.0.jar:/pulsar/lib/io.vertx-vertx-core-4.3.8.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.16.3.jar:/pulsar/lib/org.apache.avro-avro-1.11.3.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.16.3.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.16.3-tests.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.100.Final-linux-x86_64.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-api-3.0.2.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.16.3.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-3.0.2.jar:/pulsar/lib/io.swagger-swagger-models-1.6.10.jar:/pulsar/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/pulsar/lib/org.apache.commons-commons-text-1.10.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-resolver-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.6.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.53.v20231009.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/pulsar/lib/org.yaml-snakeyaml-2.0.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.61.Final.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.16.3.jar:/pulsar/lib/io.vertx-vertx-web-4.3.8.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.16.3.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/pulsar/lib/com.rabbitmq-amqp-client-5.5.3.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/org.apache.pulsar-testmocks-3.0.2.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.100.Final.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-instance-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-common-3.0.2.jar:/pulsar/lib/io.prometheus.jmx-collector-0.16.1.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-parameter-names-2.14.2.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.0.jar:/pulsar/lib/io.grpc-grpc-services-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-common-3.0.2.jar:/pulsar/lib/com.beust-jcommander-1.82.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.100.Final.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.3.jar:/pulsar/lib/io.netty-netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:/pulsar/lib/org.rocksdb-rocksdbjni-7.9.2.jar:/pulsar/lib/io.grpc-grpc-netty-1.55.3.jar:
2023-12-21T22:12:20,846+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2023-12-21T22:12:20,846+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2023-12-21T22:12:20,846+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2023-12-21T22:12:20,847+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2023-12-21T22:12:20,847+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2023-12-21T22:12:20,847+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.15.109+
2023-12-21T22:12:20,848+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=?
2023-12-21T22:12:20,849+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=?
2023-12-21T22:12:20,849+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar
2023-12-21T22:12:20,849+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=67MB
2023-12-21T22:12:20,849+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=256MB
2023-12-21T22:12:20,850+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=130MB
2023-12-21T22:12:20,854+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-zookeeper:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@696f0212
2023-12-21T22:12:20,861+0000 [main] INFO  org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2023-12-21T22:12:20,941+0000 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
2023-12-21T22:12:20,950+0000 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2023-12-21T22:12:21,051+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/10.108.1.167:2181.
2023-12-21T22:12:21,052+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-12-21T22:12:21,065+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.108.1.180:34162, server: pulsar-zookeeper/10.108.1.167:2181
2023-12-21T22:12:21,073+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper/10.108.1.167:2181, session id = 0x10000d9e80d00a5, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
Node does not exist: /loadbalance/brokers/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080
2023-12-21T22:12:21,085+0000 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 1
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

# Set JAVA_HOME here to override the environment setting
# JAVA_HOME=

# default settings for starting pulsar broker

# Log4j configuration file
# PULSAR_LOG_CONF=

# Logs location
# PULSAR_LOG_DIR=

# Configuration file of settings used in broker server
# PULSAR_BROKER_CONF=

# Configuration file of settings used in bookie server
# PULSAR_BOOKKEEPER_CONF=

# Configuration file of settings used in zookeeper server
# PULSAR_ZK_CONF=

# Configuration file of settings used in global zookeeper server
# PULSAR_GLOBAL_ZK_CONF=

# Extra options to be passed to the jvm
PULSAR_MEM=${PULSAR_MEM:-"-Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g"}

# Garbage collection options
PULSAR_GC=${PULSAR_GC:-"-XX:+UseZGC -XX:+PerfDisableSharedMem -XX:+AlwaysPreTouch"}

if [ -z "$JAVA_HOME" ]; then
  JAVA_BIN=java
else
  JAVA_BIN="$JAVA_HOME/bin/java"
fi
for token in $("$JAVA_BIN" -version 2>&1 | grep 'version "'); do
    if [[ $token =~ \"([[:digit:]]+)\.([[:digit:]]+)(.*)\" ]]; then
        if [[ ${BASH_REMATCH[1]} == "1" ]]; then
          JAVA_MAJOR_VERSION=${BASH_REMATCH[2]}
        else
          JAVA_MAJOR_VERSION=${BASH_REMATCH[1]}
        fi
        break
    elif [[ $token =~ \"([[:digit:]]+)(.*)\" ]]; then
        # Process the java versions without dots, such as `17-internal`.
        JAVA_MAJOR_VERSION=${BASH_REMATCH[1]}
        break
    fi
done

PULSAR_GC_LOG_DIR=${PULSAR_GC_LOG_DIR:-"${PULSAR_LOG_DIR}"}

if [[ -z "$PULSAR_GC_LOG" ]]; then
  if [[ $JAVA_MAJOR_VERSION -gt 8 ]]; then
    PULSAR_GC_LOG="-Xlog:gc*,safepoint:${PULSAR_GC_LOG_DIR}/pulsar_gc_%p.log:time,uptime,tags:filecount=10,filesize=20M"
    if [[ $JAVA_MAJOR_VERSION -ge 17 ]]; then
      # Use async logging on Java 17+ https://bugs.openjdk.java.net/browse/JDK-8264323
      PULSAR_GC_LOG="-Xlog:async ${PULSAR_GC_LOG}"
    fi
  else
    # Java 8 gc log options
    PULSAR_GC_LOG="-Xloggc:${PULSAR_GC_LOG_DIR}/pulsar_gc_%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=20M"
  fi
fi

# Extra options to be passed to the jvm
PULSAR_EXTRA_OPTS="${PULSAR_EXTRA_OPTS:-" -Dpulsar.allocator.exit_on_oom=true -Dio.netty.recycler.maxCapacityPerThread=4096"}"

# Add extra paths to the bookkeeper classpath
# PULSAR_EXTRA_CLASSPATH=

#Folder where the Bookie server PID file should be stored
#PULSAR_PID_DIR=

#Wait time before forcefully kill the pulsar server instance, if the stop is not successful
#PULSAR_STOP_TIMEOUT=

2023-12-21T22:12:30,358+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Starting Pulsar Broker service; version: '3.0.2'
2023-12-21T22:12:30,444+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Git Revision 12c92fed7847965e3bc3769a91c866b2f0ec2e44(dirty)
2023-12-21T22:12:30,445+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Git Branch branch-3.0
2023-12-21T22:12:30,445+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Built by fengyubiao <yubiao.feng@streamnative.io> on fengyubiaodeMacBook-Pro.local at 2023-11-17T15:50:55+0800
2023-12-21T22:12:30,950+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.8.3-6ad6d364c7c0bcf0de452d54ebefa3058098ab56, built on 2023-10-05 10:34 UTC
2023-12-21T22:12:30,951+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local
2023-12-21T22:12:30,952+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.9
2023-12-21T22:12:30,953+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Eclipse Adoptium
2023-12-21T22:12:30,953+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/temurin-17-jdk-amd64
2023-12-21T22:12:30,953+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/io.grpc-grpc-all-1.55.3.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/com.auth0-jwks-rsa-0.22.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-filesystem-storage-3.0.2.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-utils-3.0.2.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.16.3.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-proxy-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-local-runner-original-3.0.2.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.11.3.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.9.3.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/pulsar/lib/io.netty-netty-common-4.1.100.Final.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-x86_64.jar:/pulsar/lib/net.java.dev.jna-jna-platform-jpms-5.12.1.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.6.0.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.8.3.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.16.0.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/io.vertx-vertx-grpc-4.3.5.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.4.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.16.3.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/io.netty-netty-handler-4.1.100.Final.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/io.kubernetes-client-java-proto-18.0.0.jar:/pulsar/lib/io.etcd-jetcd-api-0.7.5.jar:/pulsar/lib/com.fasterxml.jackson.datatype-jackson-datatype-jdk8-2.14.2.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.55.3.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/com.google.guava-guava-32.1.1-jre.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.100.Final.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/pulsar/lib/io.grpc-grpc-servlet-1.55.3.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar:/pulsar/lib/com.google.re2j-re2j-1.6.jar:/pulsar/lib/io.etcd-jetcd-core-0.7.5.jar:/pulsar/lib/io.vertx-vertx-auth-common-4.3.8.jar:/pulsar/lib/io.grpc-grpc-rls-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-api-3.0.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.53.v20231009.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-proto-3.0.2.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-original-3.0.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-auth-sasl-3.0.2.jar:/pulsar/lib/com.squareup.okio-okio-jvm-3.4.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.53.v20231009.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.55.3.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/org.roaringbitmap-RoaringBitmap-0.9.44.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.16.3.jar:/pulsar/lib/io.grpc-grpc-xds-1.55.3.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/com.squareup.okio-okio-3.4.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-managed-ledger-3.0.2.jar:/pulsar/lib/net.java.dev.jna-jna-jpms-5.12.1.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/com.carrotsearch-hppc-0.9.1.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/io.grpc-grpc-context-1.55.3.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/io.grpc-grpc-stub-1.55.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.8.3.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.100.Final.jar:/pulsar/lib/io.perfmark-perfmark-api-0.26.0.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.100.Final.jar:/pulsar/lib/org.jline-jline-3.21.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.10.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:/pulsar/lib/io.netty-netty-transport-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-websocket-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.16.3.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:/pulsar/lib/com.github.oshi-oshi-core-java11-6.4.0.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.53.v20231009.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.8.3.jar:/pulsar/lib/io.etcd-jetcd-grpc-0.7.5.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-oidc-3.0.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.6.jar:/pulsar/lib/com.fasterxml.jackson.datatype-jackson-datatype-jsr310-2.14.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.16.3.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.3.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-resolver-dns-classes-macos-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-core-3.0.2.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.100.Final.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/io.netty-netty-codec-4.1.100.Final.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/io.grpc-grpc-alts-1.55.3.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.55.3.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.16.3.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-secrets-3.0.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.53.v20231009.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-slogger-api-4.16.3.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.9.0.jar:/pulsar/lib/org.apache.pulsar-bouncy-castle-bc-3.0.2-pkg.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.100.Final.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-coordinator-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-api-3.0.2.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/pulsar/lib/io.etcd-jetcd-common-0.7.5.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-buffer-4.1.100.Final.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-core-3.0.2.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar:/pulsar/lib/com.auth0-java-jwt-4.3.0.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/org.checkerframework-checker-qual-3.33.0.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.16.3.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.53.v20231009.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.16.3.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/io.swagger-swagger-core-1.6.10.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.16.3.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-3.0.2.jar:/pulsar/lib/io.grpc-grpc-core-1.55.3.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.16.3.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/io.kubernetes-client-java-18.0.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.53.v20231009.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/io.vertx-vertx-bridge-common-4.3.8.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.3.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-sasl-3.0.2.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-config-validation-3.0.2.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/io.grpc-grpc-servlet-jakarta-1.55.3.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.100.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-testclient-3.0.2.jar:/pulsar/lib/io.vertx-vertx-web-common-4.3.8.jar:/pulsar/lib/org.apache.pulsar-pulsar-metadata-3.0.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-3.0.2.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-aarch_64.jar:/pulsar/lib/org.apache.bookkeeper-native-io-4.16.3.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.53.v20231009.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-3.0.2.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.10.5.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.16.3.jar:/pulsar/lib/io.kubernetes-client-java-api-18.0.0.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-common-3.0.2.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.16.3.jar:/pulsar/lib/io.netty-netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.53.v20231009.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/io.grpc-grpc-auth-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-runtime-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.16.3.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:/pulsar/lib/io.grpc-grpc-api-1.55.3.jar:/pulsar/lib/net.jodah-failsafe-2.4.4.jar:/pulsar/lib/io.prometheus-simpleclient-0.16.0.jar:/pulsar/lib/io.vertx-vertx-core-4.3.8.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.16.3.jar:/pulsar/lib/org.apache.avro-avro-1.11.3.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.16.3.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.16.3-tests.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.100.Final-linux-x86_64.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-api-3.0.2.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.16.3.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-3.0.2.jar:/pulsar/lib/io.swagger-swagger-models-1.6.10.jar:/pulsar/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/pulsar/lib/org.apache.commons-commons-text-1.10.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.53.v20231009.jar:/pulsar/lib/io.netty-netty-resolver-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.6.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.53.v20231009.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/pulsar/lib/org.yaml-snakeyaml-2.0.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.61.Final.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.16.3.jar:/pulsar/lib/io.vertx-vertx-web-4.3.8.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.16.3.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/pulsar/lib/com.rabbitmq-amqp-client-5.5.3.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/org.apache.pulsar-testmocks-3.0.2.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.100.Final.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-instance-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-common-3.0.2.jar:/pulsar/lib/io.prometheus.jmx-collector-0.16.1.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-parameter-names-2.14.2.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.0.jar:/pulsar/lib/io.grpc-grpc-services-1.55.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-common-3.0.2.jar:/pulsar/lib/com.beust-jcommander-1.82.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.100.Final.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.3.jar:/pulsar/lib/io.netty-netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:/pulsar/lib/org.rocksdb-rocksdbjni-7.9.2.jar:/pulsar/lib/io.grpc-grpc-netty-1.55.3.jar:
2023-12-21T22:12:30,955+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2023-12-21T22:12:30,955+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2023-12-21T22:12:30,956+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2023-12-21T22:12:30,956+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2023-12-21T22:12:30,956+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2023-12-21T22:12:30,956+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.15.109+
2023-12-21T22:12:30,957+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=?
2023-12-21T22:12:30,957+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=?
2023-12-21T22:12:30,958+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar
2023-12-21T22:12:30,958+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=99MB
2023-12-21T22:12:30,958+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=256MB
2023-12-21T22:12:30,958+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=130MB
2023-12-21T22:12:31,044+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-zookeeper:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@58f174d9
2023-12-21T22:12:31,051+0000 [main] INFO  org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2023-12-21T22:12:31,057+0000 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
2023-12-21T22:12:31,144+0000 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2023-12-21T22:12:31,156+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/10.108.1.167:2181.
2023-12-21T22:12:31,156+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-12-21T22:12:31,243+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.108.1.180:50556, server: pulsar-zookeeper/10.108.1.167:2181
2023-12-21T22:12:31,252+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper/10.108.1.167:2181, session id = 0x10000d9e80d00a6, negotiated timeout = 30000
2023-12-21T22:12:31,257+0000 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
2023-12-21T22:12:31,355+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-zookeeper:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@5bb8f9e2
2023-12-21T22:12:31,356+0000 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
2023-12-21T22:12:31,437+0000 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2023-12-21T22:12:31,438+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/10.108.1.167:2181.
2023-12-21T22:12:31,440+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-12-21T22:12:31,442+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.108.1.180:50564, server: pulsar-zookeeper/10.108.1.167:2181
2023-12-21T22:12:31,446+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper/10.108.1.167:2181, session id = 0x10000d9e80d00a7, negotiated timeout = 30000
2023-12-21T22:12:31,446+0000 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
2023-12-21T22:12:32,951+0000 [main] INFO  org.apache.pulsar.broker.protocol.ProtocolHandlerUtils - Searching for protocol handlers in /pulsar/./protocols
2023-12-21T22:12:32,953+0000 [main] WARN  org.apache.pulsar.broker.protocol.ProtocolHandlerUtils - Protocol handler directory not found
2023-12-21T22:12:33,240+0000 [main] INFO  org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
2023-12-21T22:12:33,249+0000 [main] INFO  org.apache.pulsar.broker.BookKeeperClientFactoryImpl - Applying BookKeeper client configuration setting tlsHostnameVerificationEnabled=false
2023-12-21T22:12:33,942+0000 [main] WARN  org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for 10.108.1.180, using default rack for it : /default-rack.
2023-12-21T22:12:33,943+0000 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:10.108.1.180:0> @ /default-rack : org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping.
2023-12-21T22:12:33,943+0000 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
2023-12-21T22:12:33,955+0000 [main] INFO  org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
2023-12-21T22:12:34,152+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-12-21T22:12:34,155+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-12-21T22:12:34,167+0000 [pulsar-registration-client-17-1] WARN  org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, using default rack for it : /default-rack.
2023-12-21T22:12:34,170+0000 [pulsar-registration-client-17-1] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2023-12-21T22:12:34,297+0000 [main] INFO  org.apache.bookkeeper.mledger.impl.cache.RangeEntryCacheManagerImpl - Initialized managed-ledger entry cache of 64.0 Mb
2023-12-21T22:12:34,493+0000 [main] INFO  org.apache.pulsar.broker.authorization.AuthorizationService - org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider has been loaded.
2023-12-21T22:12:34,496+0000 [main] INFO  org.apache.pulsar.broker.service.plugin.EntryFilterProvider - Searching for entry filters in /pulsar
2023-12-21T22:12:34,545+0000 [main] INFO  org.apache.pulsar.broker.authentication.AuthenticationService - Authentication is disabled
2023-12-21T22:12:34,639+0000 [main] INFO  org.apache.pulsar.broker.service.BrokerService - Disabling per broker unack-msg blocking due invalid unAckMsgSubscriptionPercentageLimitOnBrokerBlocked 0.16 
2023-12-21T22:12:34,949+0000 [main] INFO  org.apache.pulsar.broker.loadbalance.LinuxInfoUtils - [LinuxInfo] The system metrics provider is: cgroupv2
2023-12-21T22:12:35,164+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Starting name space service, bootstrap namespaces=[]
2023-12-21T22:12:35,346+0000 [main] INFO  org.apache.pulsar.broker.BookKeeperClientFactoryImpl - Applying BookKeeper client configuration setting tlsHostnameVerificationEnabled=false
2023-12-21T22:12:35,355+0000 [main] WARN  org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for 10.108.1.180, using default rack for it : /default-rack.
2023-12-21T22:12:35,356+0000 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:10.108.1.180:0> @ /default-rack : org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping.
2023-12-21T22:12:35,357+0000 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
2023-12-21T22:12:35,359+0000 [main] INFO  org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
2023-12-21T22:12:35,363+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-12-21T22:12:35,363+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-12-21T22:12:35,365+0000 [pulsar-registration-client-34-1] WARN  org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, using default rack for it : /default-rack.
2023-12-21T22:12:35,366+0000 [pulsar-registration-client-34-1] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2023-12-21T22:12:35,577+0000 [main] INFO  org.apache.pulsar.broker.intercept.BrokerInterceptorUtils - Searching for broker interceptors in /pulsar/./interceptors
2023-12-21T22:12:35,578+0000 [main] WARN  org.apache.pulsar.broker.intercept.BrokerInterceptorUtils - Pulsar broker interceptors directory not found
2023-12-21T22:12:35,647+0000 [main] INFO  org.apache.pulsar.broker.service.DistributedIdGenerator - Broker distributed id generator started with instance id pulsar-10
2023-12-21T22:12:35,769+0000 [main] INFO  org.apache.pulsar.broker.service.BrokerService - Started Pulsar Broker service on /0.0.0.0:6650, TLS: (none), listener: (none)
2023-12-21T22:12:35,887+0000 [main] INFO  org.apache.pulsar.broker.service.BrokerService - Scheduling a thread to check backlog quota after [60] seconds in background
2023-12-21T22:12:35,890+0000 [main] INFO  org.apache.pulsar.broker.service.persistent.DispatchRateLimiter - configured broker message-dispatch rate DispatchRateImpl(dispatchThrottlingRateInMsg=0, dispatchThrottlingRateInByte=0, relativeToPublishRate=false, ratePeriodInSecond=1)
2023-12-21T22:12:35,890+0000 [main] INFO  org.apache.pulsar.broker.service.persistent.DispatchRateLimiter - setting message-dispatch-rate DispatchRateImpl(dispatchThrottlingRateInMsg=0, dispatchThrottlingRateInByte=0, relativeToPublishRate=false, ratePeriodInSecond=1)
2023-12-21T22:12:35,948+0000 [main] INFO  org.eclipse.jetty.util.log - Logging initialized @14005ms to org.eclipse.jetty.util.log.Slf4jLog
2023-12-21T22:12:36,946+0000 [main] INFO  org.eclipse.jetty.server.Server - jetty-9.4.53.v20231009; built: 2023-10-09T12:29:09.265Z; git: 27bde00a0b95a1d5bbee0eae7984f891d2d0f8c9; jvm 17.0.9+9
2023-12-21T22:12:37,051+0000 [main] INFO  org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0
2023-12-21T22:12:37,051+0000 [main] INFO  org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
2023-12-21T22:12:37,053+0000 [main] INFO  org.eclipse.jetty.server.session - node0 Scavenging every 660000ms
Dec 21, 2023 10:12:39 PM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
2023-12-21T22:12:40,347+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@7c011174{/,null,AVAILABLE}
Dec 21, 2023 10:12:41 PM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
2023-12-21T22:12:44,651+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@673c4f6e{/admin,null,AVAILABLE}
Dec 21, 2023 10:12:45 PM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
2023-12-21T22:12:48,646+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@5562c2c9{/admin/v2,null,AVAILABLE}
Dec 21, 2023 10:12:49 PM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
2023-12-21T22:12:50,040+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@3fb9a67f{/admin/v3,null,AVAILABLE}
Dec 21, 2023 10:12:50 PM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
2023-12-21T22:12:50,350+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@15c487a8{/lookup,null,AVAILABLE}
Dec 21, 2023 10:12:50 PM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
2023-12-21T22:12:50,842+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@127705e4{/topics,null,AVAILABLE}
2023-12-21T22:12:50,844+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@3f36e8d1{/metrics,null,AVAILABLE}
2023-12-21T22:12:50,866+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.h.ContextHandler@794366a5{/static,null,AVAILABLE}
2023-12-21T22:12:50,951+0000 [main] INFO  org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@9679750{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2023-12-21T22:12:50,951+0000 [main] INFO  org.eclipse.jetty.server.Server - Started @29010ms
2023-12-21T22:12:50,952+0000 [main] INFO  org.apache.pulsar.broker.web.WebService - HTTP Service started at http://0.0.0.0:8080
2023-12-21T22:12:50,952+0000 [main] INFO  org.apache.pulsar.broker.web.WebService - HTTPS Service disabled
2023-12-21T22:12:51,242+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
2023-12-21T22:12:51,242+0000 [metadata-store-10-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
2023-12-21T22:12:51,249+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Starting load management service ...
2023-12-21T22:12:51,444+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl - Acquired resource lock on /loadbalance/brokers/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080
2023-12-21T22:12:51,579+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Starting load balancer
2023-12-21T22:12:51,758+0000 [main] INFO  org.apache.pulsar.broker.namespace.OwnershipCache - Trying to acquire ownership of pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/0x00000000_0xffffffff
2023-12-21T22:12:51,841+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl - Acquired resource lock on /namespace/pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/0x00000000_0xffffffff
2023-12-21T22:12:51,846+0000 [metadata-store-10-1] INFO  org.apache.pulsar.broker.namespace.OwnershipCache - Successfully acquired ownership of OwnedBundle(bundle=pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/0x00000000_0xffffffff, isActive=1)
2023-12-21T22:12:51,847+0000 [pulsar-2-1] INFO  org.apache.pulsar.broker.PulsarService - Loading all topics on bundle: pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/0x00000000_0xffffffff
2023-12-21T22:12:51,849+0000 [main] INFO  org.apache.pulsar.broker.namespace.NamespaceService - added heartbeat namespace name in local cache: ns=pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080
2023-12-21T22:12:51,856+0000 [main] INFO  org.apache.pulsar.broker.namespace.OwnershipCache - Trying to acquire ownership of pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/0x00000000_0xffffffff
2023-12-21T22:12:51,937+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl - Acquired resource lock on /namespace/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/0x00000000_0xffffffff
2023-12-21T22:12:51,938+0000 [metadata-store-10-1] INFO  org.apache.pulsar.broker.namespace.OwnershipCache - Successfully acquired ownership of OwnedBundle(bundle=pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/0x00000000_0xffffffff, isActive=1)
2023-12-21T22:12:51,939+0000 [main] INFO  org.apache.pulsar.broker.namespace.NamespaceService - added heartbeat namespace name in local cache: ns=pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080
2023-12-21T22:12:51,940+0000 [pulsar-2-1] INFO  org.apache.pulsar.broker.PulsarService - Loading all topics on bundle: pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/0x00000000_0xffffffff
2023-12-21T22:12:51,947+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - SLA Namespace = sla-monitor/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080 doesn't exist.
2023-12-21T22:12:51,948+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Starting function worker service: serviceUrl = pulsar://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:6650, webServiceUrl = http://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080, functionWebServiceUrl = http://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080
2023-12-21T22:12:52,287+0000 [main] INFO  org.apache.pulsar.functions.worker.WorkerUtils - initialize DistributedLog Namespace with ledgersStoreServers: pulsar-zookeeper:2181 ledgersRootPath: /ledgers uri: distributedlog://pulsar-zookeeper:2181/pulsar/functions
2023-12-21T22:12:52,368+0000 [main] INFO  org.apache.distributedlog.namespace.NamespaceDriverManager - DistributedLog NamespaceDriverManager initialized
2023-12-21T22:12:52,512+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-zookeeper:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@5981f2c6
2023-12-21T22:12:52,514+0000 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
2023-12-21T22:12:52,515+0000 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2023-12-21T22:12:52,517+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/10.108.1.167:2181.
2023-12-21T22:12:52,517+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-12-21T22:12:52,518+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.108.1.180:40434, server: pulsar-zookeeper/10.108.1.167:2181
2023-12-21T22:12:52,526+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper/10.108.1.167:2181, session id = 0x10000d9e80d00a8, negotiated timeout = 30000
2023-12-21T22:12:52,532+0000 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
2023-12-21T22:12:52,542+0000 [main] INFO  org.apache.distributedlog.ZooKeeperClient - Close zookeeper client default.
2023-12-21T22:12:52,543+0000 [main] INFO  org.apache.distributedlog.ZooKeeperClient - Closing zookeeper client default.
2023-12-21T22:12:52,651+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x10000d9e80d00a8 closed
2023-12-21T22:12:52,651+0000 [main] INFO  org.apache.distributedlog.ZooKeeperClient - Closed zookeeper client default.
2023-12-21T22:12:52,651+0000 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10000d9e80d00a8
2023-12-21T22:12:52,943+0000 [main] INFO  org.apache.pulsar.functions.worker.PulsarWorkerService - Function worker service setup completed
2023-12-21T22:12:52,964+0000 [main] INFO  org.apache.pulsar.functions.worker.PulsarWorkerService - /** Starting worker id=c-pulsar-fw-pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local-8080 **/
2023-12-21T22:12:53,551+0000 [main] INFO  org.apache.pulsar.functions.worker.PulsarWorkerService - Worker Configs: WorkerConfig(workerId=c-pulsar-fw-pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local-8080, workerHostname=pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local, workerPort=8080, workerPortTls=null, authenticateMetricsEndpoint=true, includeStandardPrometheusMetrics=false, jvmGCMetricsLoggerClassName=null, numHttpServerThreads=8, httpRequestsLimitEnabled=false, httpRequestsMaxPerSecond=100.0, maxConcurrentHttpRequests=1024, httpServerThreadPoolQueueSize=8192, httpServerAcceptQueueSize=8192, maxHttpServerConnections=2048, configurationStoreServers=localhost:2181, configurationMetadataStoreUrl=pulsar-zookeeper:2181, metadataStoreSessionTimeoutMillis=30000, metadataStoreOperationTimeoutSeconds=30, metadataStoreCacheExpirySeconds=300, metadataStoreAllowReadOnlyOperations=false, zooKeeperSessionTimeoutMillis=-1, zooKeeperOperationTimeoutSeconds=-1, zooKeeperCacheExpirySeconds=-1, zooKeeperAllowReadOnlyOperations=false, connectorsDirectory=./connectors, narExtractionDirectory=/tmp, validateConnectorConfig=false, uploadBuiltinSinksSources=true, functionsDirectory=./functions, functionMetadataTopicName=metadata, useCompactedMetadataTopic=false, functionWebServiceUrl=http://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080, pulsarServiceUrl=pulsar://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:6650, pulsarWebServiceUrl=http://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080, clusterCoordinationTopicName=coordinate, pulsarFunctionsNamespace=public/functions, pulsarFunctionsCluster=pulsar, numFunctionPackageReplicas=1, functionsWorkerEnablePackageManagement=false, downloadDirectory=download/pulsar_functions, stateStorageServiceUrl=null, stateStorageProviderImplementation=org.apache.pulsar.functions.instance.state.BKStateStoreProviderImpl, functionAssignmentTopicName=assignments, schedulerClassName=org.apache.pulsar.functions.worker.scheduler.RoundRobinScheduler, failureCheckFreqMs=30000, rescheduleTimeoutMs=60000, rebalanceCheckFreqSec=-1, workerListProbeIntervalSec=60, initialBrokerReconnectMaxRetries=60, assignmentWriteMaxRetries=60, instanceLivenessCheckFreqMs=30000, brokerClientAuthenticationEnabled=null, brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationDisabled, brokerClientAuthenticationParameters=, bookkeeperClientAuthenticationPlugin=null, bookkeeperClientAuthenticationParametersName=null, bookkeeperClientAuthenticationParameters=null, topicCompactionFrequencySec=1800, tlsEnabled=false, tlsCertificateFilePath=null, tlsKeyFilePath=null, tlsTrustCertsFilePath=, tlsAllowInsecureConnection=false, tlsRequireTrustedClientCertOnConnect=false, useTls=false, tlsEnableHostnameVerification=false, tlsCertRefreshCheckDurationSec=300, tlsEnabledWithKeyStore=false, tlsProvider=Conscrypt, tlsKeyStoreType=JKS, tlsKeyStore=null, tlsTrustStoreType=JKS, tlsTrustStore=null, webServiceTlsProtocols=null, webServiceTlsCiphers=null, authenticationEnabled=false, authenticationProviders=[], authorizationEnabled=false, authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider, superUserRoles=[], proxyRoles=[], saslJaasClientAllowedIds=.*pulsar.*, saslJaasServerSectionName=PulsarFunction, saslJaasServerRoleTokenSignerSecretPath=null, kinitCommand=/usr/bin/kinit, properties={}, initializedDlogMetadata=false, brokerClientTrustCertsFilePath=, functionRuntimeFactoryClassName=org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory, functionRuntimeFactoryConfigs={changeConfigMap=pulsar-functions-worker-config, changeConfigMapNamespace=pulsar, expectedMetricsCollectionInterval=30, extraFunctionDependenciesDir=null, installUserCodeDependencies=true, javaInstanceJarLocation=null, jobNamespace=pulsar, logDirectory=logs/, pulsarAdminUrl=http://pulsar-broker:8080/, pulsarDockerImageName=apachepulsar/pulsar-all:3.0.2, pulsarRootDir=/pulsar, pulsarServiceUrl=pulsar://pulsar-broker:6650/, pythonInstanceLocation=null, submittingInsidePod=true}, secretsProviderConfiguratorClassName=null, secretsProviderConfiguratorConfig=null, functionInstanceMinResources=null, functionInstanceMaxResources=null, functionInstanceResourceGranularities=null, functionInstanceResourceChangeInLockStep=false, functionAuthProviderClassName=org.apache.pulsar.functions.auth.KubernetesSecretsTokenAuthProvider, runtimeCustomizerClassName=null, runtimeCustomizerConfig={}, maxPendingAsyncRequests=1000, forwardSourceMessageProperty=true, additionalJavaRuntimeArguments=[], functionsWorkerServiceNarPackage=, functionsWorkerServiceCustomConfigs={}, exposeAdminClientEnabled=false, threadContainerFactory=null, processContainerFactory=null, kubernetesContainerFactory=WorkerConfig.KubernetesContainerFactory(), clientAuthenticationParameters=null, clientAuthenticationPlugin=null)
2023-12-21T22:12:53,739+0000 [main] INFO  org.apache.distributedlog.api.namespace.NamespaceBuilder - No feature provider is set. All features are disabled now.
2023-12-21T22:12:53,767+0000 [main] INFO  org.apache.distributedlog.impl.BKNamespaceDriver - Created shared zooKeeper client builder dlzk:distributedlog://pulsar-zookeeper:2181/pulsar/functions:factory_writer_shared: zkServers = pulsar-zookeeper:2181, numRetries = 3, sessionTimeout = 30000, retryBackoff = 5000, maxRetryBackoff = 30000, zkAclId = null.
2023-12-21T22:12:53,837+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-zookeeper:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@4c627575
2023-12-21T22:12:53,840+0000 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
2023-12-21T22:12:53,840+0000 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2023-12-21T22:12:53,842+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper/10.108.1.167:2181.
2023-12-21T22:12:53,842+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-12-21T22:12:53,844+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.108.1.180:40446, server: pulsar-zookeeper/10.108.1.167:2181
2023-12-21T22:12:53,848+0000 [main-SendThread(pulsar-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper/10.108.1.167:2181, session id = 0x10000d9e80d00a9, negotiated timeout = 30000
2023-12-21T22:12:53,849+0000 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
2023-12-21T22:12:53,860+0000 [main] INFO  org.apache.distributedlog.impl.BKNamespaceDriver - Created shared client builder bk:distributedlog://pulsar-zookeeper:2181/pulsar/functions:factory_writer_shared : zkServers = pulsar-zookeeper:2181, ledgersPath = /ledgers, numIOThreads = 1
2023-12-21T22:12:53,938+0000 [main] INFO  org.apache.distributedlog.impl.metadata.BKDLConfig - Propagate BKDLConfig to DLConfig : encodeRegionID = false, firstLogSegmentSequenceNumber = 1, createStreamIfNotExists = true, isFederated = false.
2023-12-21T22:12:53,957+0000 [main] INFO  org.apache.distributedlog.impl.BKNamespaceDriver - Initialized BK namespace driver: clientId = function-worker-c-pulsar-fw-pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local-8080, regionId = 0, federated = false.
2023-12-21T22:12:54,045+0000 [main] INFO  org.apache.distributedlog.logsegment.LogSegmentMetadataCache - Log segment cache is enabled = true
2023-12-21T22:12:54,046+0000 [main] INFO  org.apache.pulsar.functions.worker.WorkerUtils - Create Pulsar Admin to service url http://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080: authPlugin = null, authParams = null, tlsTrustCerts = , allowTlsInsecureConnection = false, enableTlsHostnameVerification = false
2023-12-21T22:12:57,250+0000 [main] INFO  org.apache.pulsar.functions.worker.WorkerUtils - Create Pulsar Admin to service url http://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080: authPlugin = null, authParams = null, tlsTrustCerts = , allowTlsInsecureConnection = false, enableTlsHostnameVerification = false
2023-12-21T22:12:59,047+0000 [pulsar-web-38-3] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:12:59,048+0000 [pulsar-web-38-7] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:12:59,549+0000 [pulsar-web-38-3] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:12:58 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 711
2023-12-21T22:12:59,638+0000 [pulsar-web-38-7] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:12:58 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 800
2023-12-21T22:13:00,038+0000 [pulsar-2-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 1 brokers being considered for assignment of public/functions/0x40000000_0x80000000
2023-12-21T22:13:00,044+0000 [pulsar-2-1] INFO  org.apache.pulsar.broker.namespace.OwnershipCache - Trying to acquire ownership of public/functions/0x40000000_0x80000000
2023-12-21T22:13:00,142+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl - Acquired resource lock on /namespace/public/functions/0x40000000_0x80000000
2023-12-21T22:13:00,143+0000 [metadata-store-10-1] INFO  org.apache.pulsar.broker.namespace.OwnershipCache - Successfully acquired ownership of OwnedBundle(bundle=public/functions/0x40000000_0x80000000, isActive=1)
2023-12-21T22:13:00,153+0000 [metadata-store-10-1] INFO  org.apache.pulsar.broker.systopic.NamespaceEventsSystemTopicFactory - Create topic policies system topic client persistent://public/functions/__change_events
2023-12-21T22:13:01,847+0000 [pulsar-io-4-2] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xc56a747b, L:/10.108.1.180:46572 - R:pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local/10.108.1.180:6650]] Connected to server
2023-12-21T22:13:02,148+0000 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.108.1.180:46572] connected with clientVersion=Pulsar-Java-v3.0.2, clientProtocolVersion=21, proxyVersion=null
2023-12-21T22:13:02,737+0000 [pulsar-2-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 1 brokers being considered for assignment of public/functions/0x00000000_0x40000000
2023-12-21T22:13:02,739+0000 [pulsar-2-1] INFO  org.apache.pulsar.broker.namespace.OwnershipCache - Trying to acquire ownership of public/functions/0x00000000_0x40000000
2023-12-21T22:13:02,747+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl - Acquired resource lock on /namespace/public/functions/0x00000000_0x40000000
2023-12-21T22:13:02,748+0000 [metadata-store-10-1] INFO  org.apache.pulsar.broker.namespace.OwnershipCache - Successfully acquired ownership of OwnedBundle(bundle=public/functions/0x00000000_0x40000000, isActive=1)
2023-12-21T22:13:02,752+0000 [pulsar-2-1] INFO  org.apache.pulsar.broker.PulsarService - Loading all topics on bundle: public/functions/0x00000000_0x40000000
2023-12-21T22:13:02,761+0000 [pulsar-io-4-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/__change_events][reader-27b6544e0e] Subscribing to topic on cnx [id: 0xc56a747b, L:/10.108.1.180:46572 - R:pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local/10.108.1.180:6650], consumerId 0
2023-12-21T22:13:02,947+0000 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [[id: 0xe45bebc8, L:/10.108.1.180:6650 - R:/10.108.1.180:46572]] Subscribing on topic persistent://public/functions/__change_events / reader-27b6544e0e. consumerId: 0
2023-12-21T22:13:03,854+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/functions/persistent/__change_events
2023-12-21T22:13:03,939+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN  org.apache.bookkeeper.common.util.OrderedExecutor - Runnable java.util.concurrent.CompletableFuture$UniAccept@49a93576:class java.util.concurrent.CompletableFuture$UniAccept took too long 1089767 micros to execute.
2023-12-21T22:13:04,445+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xbfd4d8eb, L:/10.108.1.180:60124 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,447+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xb08b836a, L:/10.108.1.180:60138 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,448+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xb08b836a, L:/10.108.1.180:60138 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,450+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x57448bf4, L:/10.108.1.180:60114 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,450+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x57448bf4, L:/10.108.1.180:60114 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,453+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xbfd4d8eb, L:/10.108.1.180:60124 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,454+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xcef4b5ba, L:/10.108.1.180:60150 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,450+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x79264ad0, L:/10.108.1.180:60098 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,455+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x79264ad0, L:/10.108.1.180:60098 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,455+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xf5a7c99b, L:/10.108.1.180:60132 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,456+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xf5a7c99b, L:/10.108.1.180:60132 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,454+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xcef4b5ba, L:/10.108.1.180:60150 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,539+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x08a9ae7b, L:/10.108.1.180:60156 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,539+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x08a9ae7b, L:/10.108.1.180:60156 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,539+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x44c99840, L:/10.108.1.180:60168 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,540+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x00772d7b, L:/10.108.1.180:60136 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,543+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x00772d7b, L:/10.108.1.180:60136 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,544+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xd3ae8578, L:/10.108.1.180:60102 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,544+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xd3ae8578, L:/10.108.1.180:60102 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,544+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x5462e826, L:/10.108.1.180:60196 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,544+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x5462e826, L:/10.108.1.180:60196 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,544+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xc080ec3f, L:/10.108.1.180:60116 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,545+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xc080ec3f, L:/10.108.1.180:60116 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,643+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x44c99840, L:/10.108.1.180:60168 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,646+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xde8fa246, L:/10.108.1.180:60182 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,647+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xde8fa246, L:/10.108.1.180:60182 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,649+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x7c35a855, L:/10.108.1.180:60210 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,649+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x7c35a855, L:/10.108.1.180:60210 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,649+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xa8ba62be, L:/10.108.1.180:60218 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,650+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xa8ba62be, L:/10.108.1.180:60218 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,651+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x6a5eccb7, L:/10.108.1.180:60234 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:04,651+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x6a5eccb7, L:/10.108.1.180:60234 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:04,755+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.client.ReadOnlyLedgerHandle - Closing recovered ledger 38 at entry -1
2023-12-21T22:13:05,037+0000 [metadata-store-10-1] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 42
2023-12-21T22:13:05,042+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/__change_events] Created ledger 42 after closed null
2023-12-21T22:13:05,137+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/__change_events] Loading cursor __compaction
2023-12-21T22:13:05,147+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/__change_events] Deleted empty ledger ledgerId=38 rc=0
2023-12-21T22:13:05,244+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/__change_events] Recovering from bookkeeper ledger cursor: __compaction
2023-12-21T22:13:05,259+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/__change_events] Cursor __compaction recovered to position 42:-1
2023-12-21T22:13:05,343+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/__change_events] Recovery for cursor __compaction completed. pos=42:-1 -- todo=0
2023-12-21T22:13:05,344+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [public/functions/persistent/__change_events] Successfully initialize managed ledger
2023-12-21T22:13:05,646+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.BrokerService - Created topic persistent://public/functions/__change_events - dedup is disabled
2023-12-21T22:13:05,743+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.PulsarService - Loaded 1 topics on public/functions/0x00000000_0x40000000 -- time taken: 2.99 seconds
2023-12-21T22:13:05,841+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x50949586, L:/10.108.1.180:60248 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,842+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x50949586, L:/10.108.1.180:60248 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,844+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection null authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,845+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xf65ac894, L:/10.108.1.180:60254 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,846+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xf65ac894, L:/10.108.1.180:60254 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,846+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x9eb63c22, L:/10.108.1.180:60270 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,849+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xff7cd23f, L:/10.108.1.180:60286 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,850+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xff7cd23f, L:/10.108.1.180:60286 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,850+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x0887d3c9, L:/10.108.1.180:60296 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,852+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x0887d3c9, L:/10.108.1.180:60296 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,852+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x86717fbf, L:/10.108.1.180:60304 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,853+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x86717fbf, L:/10.108.1.180:60304 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,855+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x9d8346f6, L:/10.108.1.180:60316 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,857+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x9d8346f6, L:/10.108.1.180:60316 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,859+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xfbcfcb3b, L:/10.108.1.180:60324 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,859+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xfbcfcb3b, L:/10.108.1.180:60324 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,860+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x1ab97476, L:/10.108.1.180:60334 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,860+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x1ab97476, L:/10.108.1.180:60334 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,937+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xc31b3e26, L:/10.108.1.180:60348 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,937+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xc31b3e26, L:/10.108.1.180:60348 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,939+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection null authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,940+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xfe6bea38, L:/10.108.1.180:60362 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,941+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x46b200c6, L:/10.108.1.180:60374 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,944+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x46b200c6, L:/10.108.1.180:60374 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,946+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xca62d863, L:/10.108.1.180:60394 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,946+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xca62d863, L:/10.108.1.180:60394 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,947+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0xfb0c2111, L:/10.108.1.180:60398 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,947+0000 [pulsar-io-4-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xfb0c2111, L:/10.108.1.180:60398 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,951+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x619b64d7, L:/10.108.1.180:60384 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,952+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x619b64d7, L:/10.108.1.180:60384 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:05,954+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x6f0f2668, L:/10.108.1.180:60410 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181]
2023-12-21T22:13:05,955+0000 [pulsar-io-4-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x6f0f2668, L:/10.108.1.180:60410 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.108.1.168:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2023-12-21T22:13:06,343+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/functions/__change_events][reader-27b6544e0e] Creating non-durable subscription at msg id -1:-1:-1:-1 - {}
2023-12-21T22:13:06,345+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.NonDurableCursorImpl - [public/functions/persistent/__change_events] Created non-durable cursor read-position=42:0 mark-delete-position=42:-1
2023-12-21T22:13:06,345+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/__change_events] Opened new cursor: NonDurableCursorImpl{ledger=public/functions/persistent/__change_events, cursor=reader-27b6544e0e, ackPos=42:-1, readPos=42:0}
2023-12-21T22:13:06,452+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.108.1.180:46572] Created subscription on topic persistent://public/functions/__change_events / reader-27b6544e0e
2023-12-21T22:13:06,455+0000 [pulsar-io-4-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/__change_events][reader-27b6544e0e] Subscribed to topic on pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local/10.108.1.180:6650 -- consumer: 0
2023-12-21T22:13:06,637+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/functions/persistent/assignments
2023-12-21T22:13:06,737+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.client.ReadOnlyLedgerHandle - Closing recovered ledger 39 at entry -1
2023-12-21T22:13:06,842+0000 [metadata-store-10-1] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 43
2023-12-21T22:13:06,843+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/assignments] Created ledger 43 after closed null
2023-12-21T22:13:06,854+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [public/functions/persistent/assignments] Successfully initialize managed ledger
2023-12-21T22:13:06,858+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/assignments] Deleted empty ledger ledgerId=39 rc=0
2023-12-21T22:13:06,860+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.BrokerService - Created topic persistent://public/functions/assignments - dedup is disabled
2023-12-21T22:13:06,862+0000 [broker-topic-workers-OrderedExecutor-0-0] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Topic persistent://public/functions/assignments already exists
2023-12-21T22:13:06,937+0000 [broker-topic-workers-OrderedExecutor-0-0] ERROR org.apache.pulsar.broker.admin.v2.PersistentTopics - [null] Failed to create non-partitioned topic persistent://public/functions/assignments
java.util.concurrent.CompletionException: org.apache.pulsar.broker.web.RestException: This topic already exists
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
	at org.apache.pulsar.broker.service.BrokerService$2.lambda$openLedgerComplete$5(BrokerService.java:1771) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:787) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:795) ~[?:?]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
	at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:4.16.3]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.pulsar.broker.web.RestException: This topic already exists
	at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.lambda$internalCreateNonPartitionedTopicAsync$41(PersistentTopicsBase.java:402) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
	... 12 more
2023-12-21T22:13:06,952+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.180 - - [21/Dec/2023:22:12:58 +0000] "PUT /admin/v2/persistent/public/functions/assignments HTTP/1.1" 409 38 "-" "Pulsar-Java-v3.0.2" 8114
2023-12-21T22:13:06,985+0000 [AsyncHttpClient-55-1] WARN  org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/admin/v2/persistent/public/functions/assignments] Failed to perform http put request: javax.ws.rs.ClientErrorException: HTTP 409 {"reason":"This topic already exists"}
2023-12-21T22:13:06,989+0000 [main] WARN  org.apache.pulsar.functions.worker.PulsarWorkerService - Failed to create topic 'persistent://public/functions/assignments': This topic already exists
2023-12-21T22:13:07,155+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/functions/persistent/coordinate
2023-12-21T22:13:07,252+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.client.ReadOnlyLedgerHandle - Closing recovered ledger 40 at entry -1
2023-12-21T22:13:07,352+0000 [metadata-store-10-1] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 44
2023-12-21T22:13:07,354+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/coordinate] Created ledger 44 after closed null
2023-12-21T22:13:07,438+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/coordinate] Loading cursor participants
2023-12-21T22:13:07,439+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/coordinate] Recovering from bookkeeper ledger cursor: participants
2023-12-21T22:13:07,442+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/coordinate] Deleted empty ledger ledgerId=40 rc=0
2023-12-21T22:13:07,444+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/coordinate] Cursor participants meta-data recover from ledger 25
2023-12-21T22:13:07,449+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/coordinate] Opened ledger 25 for cursor participants. rc=0
2023-12-21T22:13:07,459+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/coordinate] Cursor participants recovered to position 44:-1
2023-12-21T22:13:07,459+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/coordinate] Recovery for cursor participants completed. pos=44:-1 -- todo=0
2023-12-21T22:13:07,459+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [public/functions/persistent/coordinate] Successfully initialize managed ledger
2023-12-21T22:13:07,462+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.BrokerService - Created topic persistent://public/functions/coordinate - dedup is disabled
2023-12-21T22:13:07,463+0000 [broker-topic-workers-OrderedExecutor-0-0] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Topic persistent://public/functions/coordinate already exists
2023-12-21T22:13:07,464+0000 [broker-topic-workers-OrderedExecutor-0-0] ERROR org.apache.pulsar.broker.admin.v2.PersistentTopics - [null] Failed to create non-partitioned topic persistent://public/functions/coordinate
java.util.concurrent.CompletionException: org.apache.pulsar.broker.web.RestException: This topic already exists
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
	at org.apache.pulsar.broker.service.BrokerService$2.lambda$openLedgerComplete$5(BrokerService.java:1771) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:787) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:795) ~[?:?]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
	at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:4.16.3]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.pulsar.broker.web.RestException: This topic already exists
	at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.lambda$internalCreateNonPartitionedTopicAsync$41(PersistentTopicsBase.java:402) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
	... 12 more
2023-12-21T22:13:07,541+0000 [AsyncHttpClient-55-1] WARN  org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/admin/v2/persistent/public/functions/coordinate] Failed to perform http put request: javax.ws.rs.ClientErrorException: HTTP 409 {"reason":"This topic already exists"}
2023-12-21T22:13:07,542+0000 [main] WARN  org.apache.pulsar.functions.worker.PulsarWorkerService - Failed to create topic 'persistent://public/functions/coordinate': This topic already exists
2023-12-21T22:13:07,543+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.180 - - [21/Dec/2023:22:13:07 +0000] "PUT /admin/v2/persistent/public/functions/coordinate HTTP/1.1" 409 38 "-" "Pulsar-Java-v3.0.2" 405
2023-12-21T22:13:07,638+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/functions/persistent/metadata
2023-12-21T22:13:07,741+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.client.ReadOnlyLedgerHandle - Closing recovered ledger 41 at entry -1
2023-12-21T22:13:07,837+0000 [metadata-store-10-1] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 45
2023-12-21T22:13:07,839+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/metadata] Created ledger 45 after closed null
2023-12-21T22:13:07,855+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [public/functions/persistent/metadata] Successfully initialize managed ledger
2023-12-21T22:13:07,859+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.BrokerService - Created topic persistent://public/functions/metadata - dedup is disabled
2023-12-21T22:13:07,861+0000 [broker-topic-workers-OrderedExecutor-0-0] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Topic persistent://public/functions/metadata already exists
2023-12-21T22:13:07,863+0000 [broker-topic-workers-OrderedExecutor-0-0] ERROR org.apache.pulsar.broker.admin.v2.PersistentTopics - [null] Failed to create non-partitioned topic persistent://public/functions/metadata
java.util.concurrent.CompletionException: org.apache.pulsar.broker.web.RestException: This topic already exists
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
	at org.apache.pulsar.broker.service.BrokerService$2.lambda$openLedgerComplete$5(BrokerService.java:1771) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:787) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:795) ~[?:?]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
	at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:4.16.3]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.pulsar.broker.web.RestException: This topic already exists
	at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.lambda$internalCreateNonPartitionedTopicAsync$41(PersistentTopicsBase.java:402) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
	... 12 more
2023-12-21T22:13:07,865+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/metadata] Deleted empty ledger ledgerId=41 rc=0
2023-12-21T22:13:07,939+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.180 - - [21/Dec/2023:22:13:07 +0000] "PUT /admin/v2/persistent/public/functions/metadata HTTP/1.1" 409 38 "-" "Pulsar-Java-v3.0.2" 393
2023-12-21T22:13:07,942+0000 [AsyncHttpClient-55-1] WARN  org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/admin/v2/persistent/public/functions/metadata] Failed to perform http put request: javax.ws.rs.ClientErrorException: HTTP 409 {"reason":"This topic already exists"}
2023-12-21T22:13:07,943+0000 [main] WARN  org.apache.pulsar.functions.worker.PulsarWorkerService - Failed to create topic 'persistent://public/functions/metadata': This topic already exists
2023-12-21T22:13:08,039+0000 [main] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Searching for connectors in /pulsar/./connectors
2023-12-21T22:13:08,041+0000 [main] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Loading 35 connector definitions with a thread pool of size 1
2023-12-21T22:13:08,045+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-jdbc-openmldb-3.0.2.nar-unpacked
2023-12-21T22:13:08,638+0000 [pulsar-web-38-3] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:13:08,638+0000 [pulsar-web-38-4] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:13:08,649+0000 [pulsar-web-38-3] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:13:08 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 94
2023-12-21T22:13:08,652+0000 [pulsar-web-38-4] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:13:08 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 97
2023-12-21T22:13:09,481+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-jdbc-openmldb-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-jdbc-openmldb-3.0.2.nar-unpacked/bDd6HM5n3D2qJfesIv3biQ
2023-12-21T22:13:11,369+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=jdbc-openmldb, description=JDBC sink for OpenMLDB, sourceClass=null, sinkClass=org.apache.pulsar.io.jdbc.OpenMLDBJdbcAutoSchemaSink, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-jdbc-openmldb-3.0.2.nar
2023-12-21T22:13:11,370+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-jdbc-postgres-3.0.2.nar-unpacked
2023-12-21T22:13:11,446+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-jdbc-postgres-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-jdbc-postgres-3.0.2.nar-unpacked/E8FKjnXPhkSJy6HOG3Dv7Q
2023-12-21T22:13:11,568+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=jdbc-postgres, description=JDBC sink for PostgreSQL, sourceClass=null, sinkClass=org.apache.pulsar.io.jdbc.PostgresJdbcAutoSchemaSink, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-jdbc-postgres-3.0.2.nar
2023-12-21T22:13:11,637+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-hdfs2-3.0.2.nar-unpacked
2023-12-21T22:13:11,949+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-hdfs2-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-hdfs2-3.0.2.nar-unpacked/yK29y7GCD-r1ki-DcABGzA
2023-12-21T22:13:12,742+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=hdfs2, description=Writes data into HDFS 2.x, sourceClass=null, sinkClass=org.apache.pulsar.io.hdfs2.sink.text.HdfsStringSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.hdfs2.sink.HdfsSinkConfig) from /pulsar/./connectors/pulsar-io-hdfs2-3.0.2.nar
2023-12-21T22:13:12,771+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-elastic-search-3.0.2.nar-unpacked
2023-12-21T22:13:12,962+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-elastic-search-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-elastic-search-3.0.2.nar-unpacked/DaEmhc9OljAqNzwdJXYJlw
2023-12-21T22:13:13,559+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=elastic_search, description=Writes data into Elastic Search, sourceClass=null, sinkClass=org.apache.pulsar.io.elasticsearch.ElasticSearchSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.elasticsearch.ElasticSearchConfig) from /pulsar/./connectors/pulsar-io-elastic-search-3.0.2.nar
2023-12-21T22:13:13,640+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-influxdb-3.0.2.nar-unpacked
2023-12-21T22:13:14,582+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-influxdb-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-influxdb-3.0.2.nar-unpacked/bcHndIwxBO_Qgdh-FXUeIQ
2023-12-21T22:13:16,650+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=influxdb, description=Writes data into InfluxDB database, sourceClass=null, sinkClass=org.apache.pulsar.io.influxdb.InfluxDBGenericRecordSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.influxdb.v2.InfluxDBSinkConfig) from /pulsar/./connectors/pulsar-io-influxdb-3.0.2.nar
2023-12-21T22:13:16,653+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-debezium-mysql-3.0.2.nar-unpacked
2023-12-21T22:13:17,642+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-debezium-mysql-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-debezium-mysql-3.0.2.nar-unpacked/D1PH6F00l6ht3lJCIe6bkA
2023-12-21T22:13:18,167+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=debezium-mysql, description=Debezium MySql Source, sourceClass=org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource, sinkClass=null, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-debezium-mysql-3.0.2.nar
2023-12-21T22:13:18,168+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-dynamodb-3.0.2.nar-unpacked
2023-12-21T22:13:18,354+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-dynamodb-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-dynamodb-3.0.2.nar-unpacked/5oQK3fl4A9_He8WtOCxzzQ
2023-12-21T22:13:18,557+0000 [pulsar-web-38-5] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:13:18,562+0000 [pulsar-web-38-8] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:13:18,566+0000 [pulsar-web-38-5] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:13:18 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 10
2023-12-21T22:13:18,566+0000 [pulsar-web-38-8] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:13:18 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 8
2023-12-21T22:13:18,671+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=dynamodb, description=DynamoDB connectors, sourceClass=org.apache.pulsar.io.dynamodb.DynamoDBSource, sinkClass=null, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-dynamodb-3.0.2.nar
2023-12-21T22:13:18,671+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-hdfs3-3.0.2.nar-unpacked
2023-12-21T22:13:18,957+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-hdfs3-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-hdfs3-3.0.2.nar-unpacked/Si5fnXr-0W4RRBvwNFc8tQ
2023-12-21T22:13:19,649+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=hdfs3, description=Writes data into HDFS 3.x, sourceClass=null, sinkClass=org.apache.pulsar.io.hdfs3.sink.text.HdfsStringSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.hdfs3.sink.HdfsSinkConfig) from /pulsar/./connectors/pulsar-io-hdfs3-3.0.2.nar
2023-12-21T22:13:19,652+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-netty-3.0.2.nar-unpacked
2023-12-21T22:13:19,746+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-netty-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-netty-3.0.2.nar-unpacked/3atPC1N9VSDv6YqDKV3Z1Q
2023-12-21T22:13:20,038+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=netty, description=Netty Tcp or Udp Source Connector, sourceClass=org.apache.pulsar.io.netty.NettySource, sinkClass=null, sourceConfigClass=org.apache.pulsar.io.netty.NettySourceConfig, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-netty-3.0.2.nar
2023-12-21T22:13:20,040+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-cassandra-3.0.2.nar-unpacked
2023-12-21T22:13:20,466+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-cassandra-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-cassandra-3.0.2.nar-unpacked/quar1-TQlQRXw3s3bPii7A
2023-12-21T22:13:20,566+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=cassandra, description=Writes data into Cassandra, sourceClass=null, sinkClass=org.apache.pulsar.io.cassandra.CassandraStringSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.cassandra.CassandraSinkConfig) from /pulsar/./connectors/pulsar-io-cassandra-3.0.2.nar
2023-12-21T22:13:20,569+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-aerospike-3.0.2.nar-unpacked
2023-12-21T22:13:20,968+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-aerospike-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-aerospike-3.0.2.nar-unpacked/5dge4QfmsVti9ynWc4tLiw
2023-12-21T22:13:21,055+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=aerospike, description=Aerospike database sink, sourceClass=null, sinkClass=org.apache.pulsar.io.aerospike.AerospikeStringSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.aerospike.AerospikeSinkConfig) from /pulsar/./connectors/pulsar-io-aerospike-3.0.2.nar
2023-12-21T22:13:21,058+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-http-3.0.2.nar-unpacked
2023-12-21T22:13:21,069+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-http-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-http-3.0.2.nar-unpacked/cMbbMJOOKCVeSguwuRprDw
2023-12-21T22:13:21,242+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=http, description=Writes data to an HTTP server (Webhook), sourceClass=null, sinkClass=org.apache.pulsar.io.http.HttpSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.http.HttpSinkConfig) from /pulsar/./connectors/pulsar-io-http-3.0.2.nar
2023-12-21T22:13:21,245+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-file-3.0.2.nar-unpacked
2023-12-21T22:13:21,255+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-file-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-file-3.0.2.nar-unpacked/FQfJyk6UzoENyo4QtEg6-g
2023-12-21T22:13:21,338+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=file, description=Reads data from local filesystem, sourceClass=org.apache.pulsar.io.file.FileSource, sinkClass=null, sourceConfigClass=org.apache.pulsar.io.file.FileSourceConfig, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-file-3.0.2.nar
2023-12-21T22:13:21,340+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-mongo-3.0.2.nar-unpacked
2023-12-21T22:13:21,360+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-mongo-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-mongo-3.0.2.nar-unpacked/LYbYD89U9Zn7YjBqexnCHg
2023-12-21T22:13:21,545+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=mongo, description=MongoDB source and sink connector, sourceClass=org.apache.pulsar.io.mongodb.MongoSource, sinkClass=org.apache.pulsar.io.mongodb.MongoSink, sourceConfigClass=org.apache.pulsar.io.mongodb.MongoSourceConfig, sinkConfigClass=org.apache.pulsar.io.mongodb.MongoSinkConfig) from /pulsar/./connectors/pulsar-io-mongo-3.0.2.nar
2023-12-21T22:13:21,550+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-hbase-3.0.2.nar-unpacked
2023-12-21T22:13:22,580+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-hbase-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-hbase-3.0.2.nar-unpacked/ZZbN952NlslccBuifiTILw
2023-12-21T22:13:22,645+0000 [pulsar-web-38-1] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:13:22,650+0000 [pulsar-web-38-1] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:13:22 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 7
2023-12-21T22:13:24,668+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=hbase, description=Writes data into hbase table, sourceClass=null, sinkClass=org.apache.pulsar.io.hbase.sink.HbaseGenericRecordSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.hbase.sink.HbaseSinkConfig) from /pulsar/./connectors/pulsar-io-hbase-3.0.2.nar
2023-12-21T22:13:24,672+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-debezium-mssql-3.0.2.nar-unpacked
2023-12-21T22:13:25,635+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-debezium-mssql-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-debezium-mssql-3.0.2.nar-unpacked/o0xE5VapeZge8lPgltccsw
2023-12-21T22:13:26,141+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=debezium-mssql, description=Debezium Microsoft SQL Server Source, sourceClass=org.apache.pulsar.io.debezium.mssql.DebeziumMsSqlSource, sinkClass=null, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-debezium-mssql-3.0.2.nar
2023-12-21T22:13:26,141+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-jdbc-clickhouse-3.0.2.nar-unpacked
2023-12-21T22:13:26,358+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-jdbc-clickhouse-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-jdbc-clickhouse-3.0.2.nar-unpacked/n0VR3SomSQ96JcpwTVl_Vw
2023-12-21T22:13:26,842+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=jdbc-clickhouse, description=JDBC sink for ClickHouse, sourceClass=null, sinkClass=org.apache.pulsar.io.jdbc.ClickHouseJdbcAutoSchemaSink, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-jdbc-clickhouse-3.0.2.nar
2023-12-21T22:13:26,843+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-jdbc-mariadb-3.0.2.nar-unpacked
2023-12-21T22:13:26,862+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-jdbc-mariadb-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-jdbc-mariadb-3.0.2.nar-unpacked/HS57PKMgvEjsSr2Ld56pZw
2023-12-21T22:13:26,968+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=jdbc-mariadb, description=JDBC sink for MariaDB, sourceClass=null, sinkClass=org.apache.pulsar.io.jdbc.MariadbJdbcAutoSchemaSink, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-jdbc-mariadb-3.0.2.nar
2023-12-21T22:13:26,968+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-twitter-3.0.2.nar-unpacked
2023-12-21T22:13:27,045+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-twitter-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-twitter-3.0.2.nar-unpacked/jIziU365jO1hbhCmRIuD6w
2023-12-21T22:13:27,160+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=twitter, description=Ingest data from Twitter firehose, sourceClass=org.apache.pulsar.io.twitter.TwitterFireHose, sinkClass=null, sourceConfigClass=org.apache.pulsar.io.twitter.TwitterFireHoseConfig, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-twitter-3.0.2.nar
2023-12-21T22:13:27,163+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-debezium-postgres-3.0.2.nar-unpacked
2023-12-21T22:13:27,381+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-debezium-postgres-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-debezium-postgres-3.0.2.nar-unpacked/i942MU2NzOuBza-NgI7iyQ
2023-12-21T22:13:28,556+0000 [pulsar-web-38-3] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:13:28,640+0000 [pulsar-web-38-4] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:13:28,643+0000 [pulsar-web-38-3] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:13:28 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 89
2023-12-21T22:13:28,647+0000 [pulsar-web-38-4] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:13:28 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 10
2023-12-21T22:13:28,658+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=debezium-postgres, description=Debezium Postgres Source, sourceClass=org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource, sinkClass=null, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-debezium-postgres-3.0.2.nar
2023-12-21T22:13:28,659+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-batch-data-generator-3.0.2.nar-unpacked
2023-12-21T22:13:29,028+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-batch-data-generator-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-batch-data-generator-3.0.2.nar-unpacked/vYV4QaKr_ZZLhj6_jP05Qw
2023-12-21T22:13:29,248+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=batch-data-generator, description=Test batch data generator source, sourceClass=org.apache.pulsar.io.batchdatagenerator.BatchDataGeneratorSource, sinkClass=org.apache.pulsar.io.batchdatagenerator.BatchDataGeneratorPrintSink, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-batch-data-generator-3.0.2.nar
2023-12-21T22:13:29,248+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-redis-3.0.2.nar-unpacked
2023-12-21T22:13:30,047+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-redis-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-redis-3.0.2.nar-unpacked/Do76DMC1ecMQa9dNAVANbw
2023-12-21T22:13:31,577+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=redis, description=Writes data into Redis, sourceClass=null, sinkClass=org.apache.pulsar.io.redis.sink.RedisSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.redis.sink.RedisSinkConfig) from /pulsar/./connectors/pulsar-io-redis-3.0.2.nar
2023-12-21T22:13:31,669+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-rabbitmq-3.0.2.nar-unpacked
2023-12-21T22:13:32,442+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-rabbitmq-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-rabbitmq-3.0.2.nar-unpacked/uvZ_zgWqn-NcoWhZS5Op3A
2023-12-21T22:13:33,943+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=rabbitmq, description=RabbitMQ source and sink connector, sourceClass=org.apache.pulsar.io.rabbitmq.RabbitMQSource, sinkClass=org.apache.pulsar.io.rabbitmq.RabbitMQSink, sourceConfigClass=org.apache.pulsar.io.rabbitmq.RabbitMQSourceConfig, sinkConfigClass=org.apache.pulsar.io.rabbitmq.RabbitMQSinkConfig) from /pulsar/./connectors/pulsar-io-rabbitmq-3.0.2.nar
2023-12-21T22:13:34,047+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-flume-3.0.2.nar-unpacked
2023-12-21T22:13:34,158+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-flume-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-flume-3.0.2.nar-unpacked/WN9aChJf7-ZmoFvdfUpuow
2023-12-21T22:13:34,572+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=flume, description=flume source and sink connector, sourceClass=org.apache.pulsar.io.flume.source.StringSource, sinkClass=org.apache.pulsar.io.flume.sink.StringSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.flume.FlumeConfig) from /pulsar/./connectors/pulsar-io-flume-3.0.2.nar
2023-12-21T22:13:34,575+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-kafka-3.0.2.nar-unpacked
2023-12-21T22:13:34,864+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-kafka-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-kafka-3.0.2.nar-unpacked/7x4sMbPdaFk_6W6teMAjrw
2023-12-21T22:13:35,648+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=kafka, description=Kafka source and sink connector, sourceClass=org.apache.pulsar.io.kafka.KafkaBytesSource, sinkClass=org.apache.pulsar.io.kafka.KafkaBytesSink, sourceConfigClass=org.apache.pulsar.io.kafka.KafkaSourceConfig, sinkConfigClass=org.apache.pulsar.io.kafka.KafkaSinkConfig) from /pulsar/./connectors/pulsar-io-kafka-3.0.2.nar
2023-12-21T22:13:35,652+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-kinesis-3.0.2.nar-unpacked
2023-12-21T22:13:36,045+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-kinesis-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-kinesis-3.0.2.nar-unpacked/8uCGVHDYzm8xeV2UzHQAxA
2023-12-21T22:13:36,556+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=kinesis, description=Kinesis connectors, sourceClass=org.apache.pulsar.io.kinesis.KinesisSource, sinkClass=org.apache.pulsar.io.kinesis.KinesisSink, sourceConfigClass=org.apache.pulsar.io.kinesis.KinesisSourceConfig, sinkConfigClass=org.apache.pulsar.io.kinesis.KinesisSinkConfig) from /pulsar/./connectors/pulsar-io-kinesis-3.0.2.nar
2023-12-21T22:13:36,564+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-solr-3.0.2.nar-unpacked
2023-12-21T22:13:36,643+0000 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Writing local data to metadata store because maximum change Infinity% exceeded threshold 10%; time since last report written is 45.39 seconds
2023-12-21T22:13:36,660+0000 [metadata-store-10-1] INFO  org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl - Acquired resource lock on /loadbalance/brokers/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080
2023-12-21T22:13:37,536+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-solr-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-solr-3.0.2.nar-unpacked/e2ut92Yv10ud-xLc3WIfPA
2023-12-21T22:13:38,557+0000 [pulsar-web-38-8] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:13:38,561+0000 [pulsar-web-38-8] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:13:38 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 6
2023-12-21T22:13:38,565+0000 [pulsar-web-38-6] WARN  org.apache.pulsar.common.configuration.VipStatus - Failed to access "status.html". The service is not ready
2023-12-21T22:13:38,568+0000 [pulsar-web-38-6] INFO  org.eclipse.jetty.server.RequestLog - 10.108.1.129 - - [21/Dec/2023:22:13:38 +0000] "GET /status.html HTTP/1.1" 404 508 "-" "kube-probe/1.27" 12
2023-12-21T22:13:39,161+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=solr, description=Writes data into solr collection, sourceClass=null, sinkClass=org.apache.pulsar.io.solr.SolrGenericRecordSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.solr.SolrSinkConfig) from /pulsar/./connectors/pulsar-io-solr-3.0.2.nar
2023-12-21T22:13:39,164+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-canal-3.0.2.nar-unpacked
2023-12-21T22:13:39,664+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-canal-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-canal-3.0.2.nar-unpacked/XhaGcacgdlf3Si8VXDtZ-w
2023-12-21T22:13:39,954+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=canal, description=canal source and read data from mysql, sourceClass=org.apache.pulsar.io.canal.CanalStringSource, sinkClass=null, sourceConfigClass=org.apache.pulsar.io.canal.CanalSourceConfig, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-canal-3.0.2.nar
2023-12-21T22:13:39,957+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-debezium-mongodb-3.0.2.nar-unpacked
2023-12-21T22:13:40,294+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.SchedulerManager - Closing scheduler manager
2023-12-21T22:13:40,295+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:6650
2023-12-21T22:13:40,577+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-debezium-mongodb-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-debezium-mongodb-3.0.2.nar-unpacked/wn3-WDC_NVDhZZgXp8QsjQ
2023-12-21T22:13:41,241+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=debezium-mongodb, description=Debezium MongoDb Source, sourceClass=org.apache.pulsar.io.debezium.mongodb.DebeziumMongoDbSource, sinkClass=null, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-debezium-mongodb-3.0.2.nar
2023-12-21T22:13:41,241+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-debezium-oracle-3.0.2.nar-unpacked
2023-12-21T22:13:42,361+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-debezium-oracle-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-debezium-oracle-3.0.2.nar-unpacked/CeZG3cIE0CvOuUG4OLCvhg
2023-12-21T22:13:42,437+0000 [pulsar-service-shutdown] INFO  org.apache.distributedlog.BookKeeperClient - BookKeeper Client closed bk:distributedlog://pulsar-zookeeper:2181/pulsar/functions:factory_writer_shared
2023-12-21T22:13:42,437+0000 [pulsar-service-shutdown] INFO  org.apache.distributedlog.ZooKeeperClient - Close zookeeper client dlzk:distributedlog://pulsar-zookeeper:2181/pulsar/functions:factory_writer_shared.
2023-12-21T22:13:42,437+0000 [pulsar-service-shutdown] INFO  org.apache.distributedlog.ZooKeeperClient - Closing zookeeper client dlzk:distributedlog://pulsar-zookeeper:2181/pulsar/functions:factory_writer_shared.
2023-12-21T22:13:42,543+0000 [pulsar-service-shutdown] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x10000d9e80d00a9 closed
2023-12-21T22:13:42,543+0000 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10000d9e80d00a9
2023-12-21T22:13:42,544+0000 [pulsar-service-shutdown] INFO  org.apache.distributedlog.ZooKeeperClient - Closed zookeeper client dlzk:distributedlog://pulsar-zookeeper:2181/pulsar/functions:factory_writer_shared.
2023-12-21T22:13:42,546+0000 [pulsar-service-shutdown] INFO  org.apache.distributedlog.impl.BKNamespaceDriver - Release external resources used by channel factory.
2023-12-21T22:13:42,546+0000 [pulsar-service-shutdown] INFO  org.apache.distributedlog.impl.BKNamespaceDriver - Stopped request timer
2023-12-21T22:13:42,548+0000 [pulsar-service-shutdown] INFO  org.apache.distributedlog.BKDistributedLogNamespace - Executor Service Stopped.
2023-12-21T22:13:42,548+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.PulsarBrokerStarter - Shut down functions worker service successfully.
2023-12-21T22:13:43,254+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=debezium-oracle, description=Debezium Oracle Source, sourceClass=org.apache.pulsar.io.debezium.oracle.DebeziumOracleSource, sinkClass=null, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-debezium-oracle-3.0.2.nar
2023-12-21T22:13:43,254+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-kafka-connect-adaptor-3.0.2.nar-unpacked
2023-12-21T22:13:43,536+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-kafka-connect-adaptor-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-kafka-connect-adaptor-3.0.2.nar-unpacked/QhwMu4QvIML8Wbo6-Ro5fw
2023-12-21T22:13:43,970+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=kafka-connect-adaptor, description=Kafka source connect adaptor, sourceClass=org.apache.pulsar.io.kafka.connect.KafkaConnectSource, sinkClass=org.apache.pulsar.io.kafka.connect.KafkaConnectSink, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-kafka-connect-adaptor-3.0.2.nar
2023-12-21T22:13:43,971+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-alluxio-3.0.2.nar-unpacked
2023-12-21T22:13:44,451+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-alluxio-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-alluxio-3.0.2.nar-unpacked/CWJfCvTTI5PUsjHgSoZiFg
2023-12-21T22:13:45,177+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=alluxio, description=Writes data into Alluxio, sourceClass=null, sinkClass=org.apache.pulsar.io.alluxio.sink.AlluxioSink, sourceConfigClass=null, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-alluxio-3.0.2.nar
2023-12-21T22:13:45,178+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-data-generator-3.0.2.nar-unpacked
2023-12-21T22:13:45,263+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-data-generator-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-data-generator-3.0.2.nar-unpacked/yfCML_VOPPgr7A-_8BIv-A
2023-12-21T22:13:45,479+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=data-generator, description=Test data generator source, sourceClass=org.apache.pulsar.io.datagenerator.DataGeneratorSource, sinkClass=org.apache.pulsar.io.datagenerator.DataGeneratorPrintSink, sourceConfigClass=org.apache.pulsar.io.datagenerator.DataGeneratorSourceConfig, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-data-generator-3.0.2.nar
2023-12-21T22:13:45,484+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-jdbc-sqlite-3.0.2.nar-unpacked
2023-12-21T22:13:45,576+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-jdbc-sqlite-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-jdbc-sqlite-3.0.2.nar-unpacked/tMexyfcTWUVYCxF_hqW6UA
2023-12-21T22:13:45,759+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=jdbc-sqlite, description=JDBC sink for SQLite, sourceClass=null, sinkClass=org.apache.pulsar.io.jdbc.SqliteJdbcAutoSchemaSink, sourceConfigClass=null, sinkConfigClass=org.apache.pulsar.io.jdbc.JdbcSinkConfig) from /pulsar/./connectors/pulsar-io-jdbc-sqlite-3.0.2.nar
2023-12-21T22:13:45,766+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar-nar/pulsar-io-nsq-3.0.2.nar-unpacked
2023-12-21T22:13:45,842+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.common.nar.NarUnpacker - Extracting /pulsar/./connectors/pulsar-io-nsq-3.0.2.nar to /tmp/pulsar-nar/pulsar-io-nsq-3.0.2.nar-unpacked/OW8Bs-lkl0frMCALDAobXg
2023-12-21T22:13:45,951+0000 [connector-extraction-executor-0] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=nsq, description=Ingest data from an NSQ topic, sourceClass=org.apache.pulsar.io.nsq.NSQSource, sinkClass=null, sourceConfigClass=org.apache.pulsar.io.nsq.NSQSourceConfig, sinkConfigClass=null) from /pulsar/./connectors/pulsar-io-nsq-3.0.2.nar
2023-12-21T22:13:45,957+0000 [main] INFO  org.apache.pulsar.functions.utils.functions.FunctionUtils - Searching for functions in /pulsar/./functions
2023-12-21T22:13:45,959+0000 [main] WARN  org.apache.pulsar.functions.utils.functions.FunctionUtils - Functions archive directory not found
2023-12-21T22:13:45,961+0000 [main] ERROR org.apache.pulsar.functions.worker.PulsarWorkerService - Error Starting up in worker
org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
	at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252) ~[org.apache.pulsar-pulsar-client-admin-api-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:354) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:456) ~[org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1801) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:894) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	Suppressed: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
		at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:300) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:354) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:456) ~[org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1801) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:894) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	Caused by: java.lang.IllegalStateException: Client instance has been closed.
		at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1224) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:587) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$22(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:342) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		... 6 more
Caused by: java.lang.IllegalStateException: Client instance has been closed.
	at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1224) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:587) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$22(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:342) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	... 6 more
2023-12-21T22:13:45,962+0000 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:584) ~[org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1801) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:894) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
	at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252) ~[org.apache.pulsar-pulsar-client-admin-api-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:354) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:456) ~[org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:3.0.2]
	... 4 more
	Suppressed: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
		at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:300) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:354) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:456) ~[org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1801) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:894) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	Caused by: java.lang.IllegalStateException: Client instance has been closed.
		at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1224) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:587) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$22(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:342) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		... 6 more
Caused by: java.lang.IllegalStateException: Client instance has been closed.
	at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1224) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:587) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$22(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:342) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:456) ~[org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:3.0.2]
	... 4 more
2023-12-21T22:13:46,037+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.broker.PulsarService - Closing PulsarService
2023-12-21T22:13:46,036+0000 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:931) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
Caused by: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:584) ~[org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1801) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:894) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	... 2 more
Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
	at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252) ~[org.apache.pulsar-pulsar-client-admin-api-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:354) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:456) ~[org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1801) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:894) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	... 2 more
	Suppressed: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.IllegalStateException: Client instance has been closed.
		at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:300) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:354) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:456) ~[org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1801) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:894) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
		at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	Caused by: java.lang.IllegalStateException: Client instance has been closed.
		at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1224) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:587) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$22(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:342) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
		... 6 more
Caused by: java.lang.IllegalStateException: Client instance has been closed.
	at org.glassfish.jersey.internal.guava.Preconditions.checkState(Preconditions.java:169) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyClient.checkNotClosed(JerseyClient.java:257) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.checkNotClosed(JerseyWebTarget.java:118) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:129) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.path(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.topicPath(TopicsImpl.java:1224) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptionsAsync(TopicsImpl.java:587) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.lambda$getSubscriptions$22(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:342) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:581) ~[org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:3.0.2]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:456) ~[org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1801) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:894) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	... 2 more
2023-12-21T22:13:46,040+0000 [main] WARN  org.apache.pulsar.common.util.ShutdownUtil - Triggering immediate shutdown of current process with status 1
java.lang.Exception: Stacktrace for immediate shutdown
	at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:55) ~[org.apache.pulsar-pulsar-common-3.0.2.jar:3.0.2]
	at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:50) ~[org.apache.pulsar-pulsar-common-3.0.2.jar:3.0.2]
	at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:82) ~[org.apache.pulsar-pulsar-common-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:359) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
2023-12-21T22:13:46,048+0000 [pulsar-service-shutdown] INFO  org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@9679750{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2023-12-21T22:13:46,049+0000 [pulsar-service-shutdown] INFO  org.eclipse.jetty.server.session - node0 Stopped scavenging
2023-12-21T22:13:46,051+0000 [pulsar-service-shutdown] INFO  org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.h.ContextHandler@794366a5{/static,null,STOPPED}

lhotari · 2023-12-22T07:12:23Z

Thanks @theoribeiro , this gives a clear picture of what happened. I won't be able to work on this until next year.

theoribeiro · 2023-12-22T13:10:57Z

No worries, @lhotari. Any chance I can help?

lhotari · 2023-12-22T14:08:37Z

No worries, @lhotari. Any chance I can help?

Having clear repro steps would help so that I could follow the same steps and reproduce on GKE. Perhaps you already provided this info somewhere?

lhotari · 2024-01-17T20:56:48Z

@theoribeiro Please share your values file (or helm get values for your deployment)

lhotari · 2024-01-26T17:04:34Z

@theoribeiro Do you have a chance to test with latest Apache Pulsar Helm Chart release 3.2.0 ?

lhotari · 2024-06-05T11:02:00Z

@theoribeiro Do you have a chance to test with latest Apache Pulsar Helm Chart release 3.4.1 . Please reopen the ticket if the problem reproduces with 3.4.1.

lhotari · 2024-06-05T11:04:26Z

I'm assuming that this was a duplicate of issue #473, fixed by #474.

lhotari closed this as completed Jun 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Brokers fail starting on GKE Autopilot #416

Brokers fail starting on GKE Autopilot #416

theoribeiro commented Dec 21, 2023

lhotari commented Dec 21, 2023

theoribeiro commented Dec 21, 2023

lhotari commented Dec 21, 2023

theoribeiro commented Dec 21, 2023 •

edited

Loading

lhotari commented Dec 21, 2023

lhotari commented Dec 21, 2023

lhotari commented Dec 21, 2023

lhotari commented Dec 21, 2023

theoribeiro commented Dec 21, 2023

theoribeiro commented Dec 21, 2023

lhotari commented Dec 22, 2023

theoribeiro commented Dec 22, 2023

lhotari commented Dec 22, 2023

lhotari commented Jan 17, 2024

lhotari commented Jan 26, 2024

lhotari commented Jun 5, 2024

lhotari commented Jun 5, 2024

Brokers fail starting on GKE Autopilot #416

Brokers fail starting on GKE Autopilot #416

Comments

theoribeiro commented Dec 21, 2023

lhotari commented Dec 21, 2023

theoribeiro commented Dec 21, 2023

lhotari commented Dec 21, 2023

theoribeiro commented Dec 21, 2023 • edited Loading

lhotari commented Dec 21, 2023

lhotari commented Dec 21, 2023

lhotari commented Dec 21, 2023

lhotari commented Dec 21, 2023

theoribeiro commented Dec 21, 2023

theoribeiro commented Dec 21, 2023

lhotari commented Dec 22, 2023

theoribeiro commented Dec 22, 2023

lhotari commented Dec 22, 2023

lhotari commented Jan 17, 2024

lhotari commented Jan 26, 2024

lhotari commented Jun 5, 2024

lhotari commented Jun 5, 2024

theoribeiro commented Dec 21, 2023 •

edited

Loading