Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata #14372

Open
Abdulsamirkhan opened this issue Nov 4, 2024 · 6 comments
Open

Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata #14372

Abdulsamirkhan opened this issue Nov 4, 2024 · 6 comments
Labels
ingestion real-time troubleshooting

Comments

@Abdulsamirkhan
Copy link

Abdulsamirkhan commented Nov 4, 2024 •
edited
Loading

can you please help me to fix this error, i tried all possible combinations

I am able to able to read topic metadata, produce message and consume message from c# code.

Pinot docker image: [apachepinot/pinot:latest]

  Command 
  bin/pinot-admin.sh AddTable -tableConfigFile  /tmp/SamirTestTable1.json -schemaFile /tmp/SamirTestSchema.json -controllerHost localhost -controllerPort 9000 -exec

Kafka :
server.properties :
message.max.bytes=2200000000
listeners=PLAINTEXT://localhost:9092,PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
zookeeper.connect=localhost:2181

Consumer.properties
    bootstrap.servers=localhost:29092
    group.id=test-consumer-group
    message.max.bytes=2200000000
    max.request.size=1200000000
    metadata.max.age.ms=300000
    heartbeat.interval.ms=1000
    request.timeout.ms=160000
    max.partition.fetch.bytes =2200000000
    fetch.metadata.timeout.ms =600000 
    session.timeout.ms = 145000 
    fetch.max.bytes = 2200000000

Compose Yaml
kafka:
image: confluentinc/cp-kafka:5.5.0
depends_on:
- zookeeper
ports:
- "29092:29092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

Schema Json file
{
"schemaName": "SamirTest",
"dimensionFieldSpecs": [
{"name": "uuid", "dataType": "STRING"}
],
"metricFieldSpecs": [
{"name": "count", "dataType": "INT"}
],
"dateTimeFieldSpecs": [
{"name": "SamirTestTime", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS"}
]
}

Table Json file
{
"tableName": "SamirTest_REALTIME",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "SamirTestTime",
"schemaName": "SamirTest",
"replication": "1"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"metadata": {},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "simple",
"stream.kafka.topic.name": "samirtestNew",
"stream.kafka.broker.list": "localhost:29092",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.consumer.prop.fetch.metadata.timeout.ms": "120000",
"stream.kafka.consumer.prop.session.timeout.ms": "60000",
"stream.kafka.consumer.prop.request.timeout.ms": "120000",
"realtime.segment.flush.threshold.rows": "50000",
"realtime.segment.flush.threshold.time": "24h"
}
}
}

Code used to get metadata and worked

              var config1 = new ConsumerConfig
              {
                  BootstrapServers = "localhost:29092",
                  GroupId = "metadata-fetcher-group",
                  AutoOffsetReset = AutoOffsetReset.Earliest,
                  EnableAutoCommit = false
              };
              
              using (var adminClient = new AdminClientBuilder(config).Build())
              {
                  try
                  {
                      // Fetch metadata for all topics
                      var metadata = adminClient.GetMetadata(TimeSpan.FromSeconds(30));
              
              
                      foreach (var topic in metadata.Topics)
                      {
                          Console.WriteLine($"Topic: {topic.Topic}");
                          Console.WriteLine($"Partition Count: {topic.Partitions.Count}");
                          foreach (var partition in topic.Partitions)
                          {
                              Console.WriteLine($"Partition: {partition.PartitionId}, Leader: {partition.Leader}, Replicas: {string.Join(",", partition.Replicas)}, ISR: {string.Join(",", partition.InSyncReplicas)}");
                          }
                      }
                  }
                  catch (KafkaException e)
                  {
                      Console.WriteLine($"An error occurred: {e.Error.Reason}");
                  }
              }

Error details :
2024-11-04 17:39:19 2024/11/04 12:09:19.104 ERROR [PinotTableIdealStateBuilder] [grizzly-http-server-5] Could not get PartitionGroupMetadata for topic: samirtestNew of table: SamirTest_REALTIME
2024-11-04 17:39:19 org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:19 at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchPartitionCount(KafkaStreamMetadataProvider.java:72) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(StreamMetadataProvider.java:76) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:70) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:31) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.getNewPartitionGroupMetadataList(PinotLLCRealtimeSegmentManager.java:799) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.setUpNewTable(PinotLLCRealtimeSegmentManager.java:325) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.addTable(PinotHelixResourceManager.java:1746) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.api.resources.TableConfigsRestletResource.addConfig(TableConfigsRestletResource.java:233) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2024-11-04 17:39:19 at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
2024-11-04 17:39:19 Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:19 2024/11/04 12:09:19.302 ERROR [PinotHelixResourceManager] [grizzly-http-server-5] Caught exception while setting up table: SamirTest_REALTIME, cleaning it up
2024-11-04 17:39:19 java.lang.RuntimeException: org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:99) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.getNewPartitionGroupMetadataList(PinotLLCRealtimeSegmentManager.java:799) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.setUpNewTable(PinotLLCRealtimeSegmentManager.java:325) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.addTable(PinotHelixResourceManager.java:1746) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.api.resources.TableConfigsRestletResource.addConfig(TableConfigsRestletResource.java:233) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2024-11-04 17:39:19 at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
2024-11-04 17:39:19 Caused by: org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:19 at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchPartitionCount(KafkaStreamMetadataProvider.java:72) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(StreamMetadataProvider.java:76) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:70) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:31) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 ... 30 more
2024-11-04 17:39:19 Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:20 2024/11/04 12:09:20.044 ERROR [TableConfigsRestletResource] [grizzly-http-server-5] org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:20 java.lang.RuntimeException: org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:20 at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:99) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.getNewPartitionGroupMetadataList(PinotLLCRealtimeSegmentManager.java:799) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.setUpNewTable(PinotLLCRealtimeSegmentManager.java:325) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.addTable(PinotHelixResourceManager.java:1746) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.controller.api.resources.TableConfigsRestletResource.addConfig(TableConfigsRestletResource.java:233) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2024-11-04 17:39:20 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
2024-11-04 17:39:20 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2024-11-04 17:39:20 at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
2024-11-04 17:39:20 Caused by: org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:20 at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchPartitionCount(KafkaStreamMetadataProvider.java:72) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(StreamMetadataProvider.java:76) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:70) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:31) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 ... 30 more
2024-11-04 17:39:20 Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
@Abdulsamirkhan
Copy link
Author

@Jackie-Jiang, any help much appreciated

@Jackie-Jiang
Copy link
Contributor

cc @swaminathanmanish @KKcorps to help take a look

@Abdulsamirkhan
Copy link
Author

@Jackie-Jiang @swaminathanmanish @KKcorps, Still i am not able to solve this any help much appreciated.

@Abdulsamirkhan
Copy link
Author

@Jackie-Jiang @swaminathanmanish @KKcorps, gentle reminder

@Jackie-Jiang
Copy link
Contributor

Can you double check the topic existence and kafka settings? This seems a Kafka issue to me.
You may also get more help in the slack troubleshooting channel

@Abdulsamirkhan
Copy link
Author

I double checked Topic existence and kafka settings, but not able to point out the issues, let me move to slack if it helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ingestion real-time troubleshooting
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Jackie-Jiang @Abdulsamirkhan