one of namespaces seems having issue and deleting the topics got 504 error. #23112
Replies: 1 comment 2 replies
-
|
@JingTyagi-ICON What happens if you try the pulsar-admin topic delete with the |
Beta Was this translation helpful? Give feedback.
-
|
I tried that too, didn't work. |
Beta Was this translation helpful? Give feedback.
-
|
Reason: HTTP 504 Gateway Timeout Reason: HTTP 504 Gateway Timeout |
Beta Was this translation helpful? Give feedback.
-
we are using pulsar APPVersion 3.0.3 and one of our namespace Research/RemoveExpiredDownloadFilesWatcherService seems have issues as below,
[persistent://Research/RemoveExpiredDownloadFilesWatcherService/__change_events] [reader-237329c40b]
Error connecting to broker
: org.apache.pulsar.client.api.PulsarClientException: {"errorMsg":"Error while reading ledger - ledger=4585 - operation=Failed to read entry - entry=0","reqId":2561804311684148649, "remote":"pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local/100.64.7.26:6650", "local":"/100.64.7.26:35282"}
I'm not sure how to fix the above and tried to delete the namespace, tried to delete the topics first, got the 504 error.
Could you help me out?
Thanks,
Jing
$ kg svc -n pulsar
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
pulsar-bookie ClusterIP None 3181/TCP,8000/TCP 88d
pulsar-broker ClusterIP None 8080/TCP,6650/TCP 88d
pulsar-proxy ClusterIP 172.20.42.173 80/TCP,6650/TCP 88d
pulsar-pulsar-manager ClusterIP 172.20.44.191 80/TCP 88d
pulsar-pulsar-manager-admin ClusterIP 172.20.60.37 7750/TCP 88d
pulsar-recovery ClusterIP None 8000/TCP 88d
pulsar-toolset ClusterIP None 88d
pulsar-zookeeper ClusterIP None 8000/TCP,2888/TCP,3888/TCP,2181/TCP 88d
pulsar@pulsar-toolset-0:/pulsar$ ./bin/pulsar-admin topics list Research/RemoveExpiredDownloadFilesWatcherService
persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents-chunk
persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents-chunk--DLQ
persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents--DLQ
persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents
pulsar@pulsar-toolset-0:/pulsar$ ./bin/pulsar-admin topics subscriptions Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents-chunk
--- An unexpected error occurred in the server ---
Message: Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents-chunk error_message={"errorMsg":"Error while reading ledger - ledger=4585 - operation=Failed to read
entry - entry=0","reqId":541356355081583279, "remote":"pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local/100.64.7.26:6650", "local":"/100.64.1.22:51962"}
Stacktrace:
org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents-chunk error_message={"errorMsg":"Error while reading ledger - ledger=4585 - operation=Failed to read entry - entry=0","reqId":541356355081583279, "remote":"pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local/100.64.7.26:6650", "local":"/100.64.1.22:51962"}
at org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$28(BrokerService.java:1065)
at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at org.apache.pulsar.client.impl.PulsarClientImpl.lambda$createSingleTopicReaderAsync$14(PulsarClientImpl.java:687)
at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at org.apache.pulsar.client.impl.ConsumerImpl.lambda$connectionOpened$17(ConsumerImpl.java:893)
at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at org.apache.pulsar.client.impl.ClientCnx.handleError(ClientCnx.java:793)
at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:192)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
Reason:
--- An unexpected error occurred in the server ---
Message: Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents-chunk error_message={"errorMsg":"Error while reading ledger - ledger=4585 - operation=Failed to read
entry - entry=0","reqId":541356355081583279, "remote":"pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local/100.64.7.26:6650", "local":"/100.64.1.22:51962"}
Stacktrace:
org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents-chunk error_message={"errorMsg":"Error while reading ledger - ledger=4585 - operation=Failed to read entry - entry=0","reqId":541356355081583279, "remote":"pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local/100.64.7.26:6650", "local":"/100.64.1.22:51962"}
at org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$28(BrokerService.java:1065)
at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at org.apache.pulsar.client.impl.PulsarClientImpl.lambda$createSingleTopicReaderAsync$14(PulsarClientImpl.java:687)
at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at org.apache.pulsar.client.impl.ConsumerImpl.lambda$connectionOpened$17(ConsumerImpl.java:893)
at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at org.apache.pulsar.client.impl.ClientCnx.handleError(ClientCnx.java:793)
at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:192)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
pulsar@pulsar-toolset-0:/pulsar$ ./bin/pulsar-admin namespaces delete Research/RemoveExpiredDownloadFilesWatcherService
2024-07-31T21:18:57,276+0000 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-proxy:80/admin/v2/namespaces/Research/RemoveExpiredDownloadFilesWatcherService?force=false] Failed to perform http delete request: javax.ws.rs.ClientErrorException: HTTP 409 {"reason":"Cannot delete non empty namespace"}
Cannot delete non empty namespace
Reason: Cannot delete non empty namespace
pulsar@pulsar-toolset-0:/pulsar$ ./bin/pulsar-admin topics delete persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents-chunk
2024-07-31T21:24:01,705+0000 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-proxy:80/admin/v2/persistent/Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents-chunk?force=false&deleteSchema=true] Failed to perform http delete request: javax.ws.rs.ServerErrorException: HTTP 504 Gateway Timeout
HTTP 504 Gateway Timeout
Reason: HTTP 504 Gateway Timeout
pulsar@pulsar-toolset-0:/pulsar$ ./bin/pulsar-admin topics delete persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents
2024-07-31T21:25:55,459+0000 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-proxy:80/admin/v2/persistent/Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents?force=false&deleteSchema=true] Failed to perform http delete request: javax.ws.rs.ServerErrorException: HTTP 504 Gateway Timeout
HTTP 504 Gateway Timeout
Reason: HTTP 504 Gateway Timeout
pulsar@pulsar-toolset-0:/pulsar$ ./bin/pulsar-admin topics delete persistent://Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents --force
2024-07-31T21:27:54,752+0000 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://pulsar-proxy:80/admin/v2/persistent/Research/RemoveExpiredDownloadFilesWatcherService/BusinessEvents?force=true&deleteSchema=true] Failed to perform http delete request: javax.ws.rs.ServerErrorException: HTTP 504 Gateway Timeout
HTTP 504 Gateway Timeout
Reason: HTTP 504 Gateway Timeout
pulsar@pulsar-toolset-0:/pulsar$
Beta Was this translation helpful? Give feedback.
All reactions