Transiting to abortable error state

[MohTaheri]

Hello

My spring application setup involves three instances where the Kafka producers and consumers are within the same service. i'm using the @Transactional annotation for consumers with a read_committed isolation level (read message + insert into DB + publish output). We are encountering a CommitFailedException when re-deploying to Kubernetes.
However, with each deployment in Kubernetes, after creating new pods and before terminating the old ones, we encounter these errors.

[Producer clientId=a-5489cf557d-gjpvk-producer-17, transactionalId=tx-31afb89f-b79f-4a17-bf89-e0300ef7dfe716] 
Transiting to abortable error state due to org.apache.kafka.clients.consumer.CommitFailedException: 
Transaction offset Commit failed due to consumer group metadata mismatch: Specified group generation id is not valid. |  
-- | --
 
 thread=[org.springframework.kafka.KafkaListenerEndpointContainer#10-1-C-1] message=Transaction rolled back  org.apache.kafka.clients.consumer.CommitFailedException: 
 Transaction offset Commit failed due to consumer group metadata mismatch: Specified group generation id is not valid. 	
 at org.apache.kafka.clients.producer.internals.TransactionManager$TxnOffsetCommitHandler.handleResponse(TransactionManager.java:1656) 	at 
 org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1207) 	
 at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) 	at 
 org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:583) 	at 
 org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575) 	at 
 org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:426) 	at 
 org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:316) 	at 
 org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) 	at java.base/java.lang.Thread.run(Unknown Source)

It seems this could be related to the transaction-id-prefix configuration, causing different transaction IDs for old and new instances.
Should we consider removing the randomness from this configuration?

spring:
  kafka:
    producer:
      transaction-id-prefix: tx-${random.uuid}

• spring-kafka: 3.0.10

Thanks

[garyrussell]

I think, if you do that, you will just get a different error (ProducerFencedException) when the old producer tries to commit. It's best to stop the old producers first to avoid any such errors.

[MohTaheri]

ok thanks
So this answer from you is not valid now.

[garyrussell]

That is four years old; KIP-447 changed all that https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics

[garyrussell]

I added a note to that answer.

[MohTaheri]

None of the solutions can work for us because stopping all previous pods causes a delay in our processing, and we don't want to lose our real-time availability.
It's strange that exceptions happened for both the producer and consumer sides in both previous and new instances.

ERROR thread=[org.springframework.kafka.KafkaListenerEndpointContainer#10-0-C-1] logger=o.s.k.l.KafkaMessageListenerContainer trace_id= span_id= message=Transaction rolled back

org.apache.kafka.clients.consumer.CommitFailedException: Transaction offset Commit failed due to consumer group metadata mismatch: Specified group generation id is not valid.
	at org.apache.kafka.clients.producer.internals.TransactionManager$TxnOffsetCommitHandler.handleResponse(TransactionManager.java:1656)
	at org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1207)
	at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
	at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:583)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575)
	at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:426)
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:316)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)

ERROR thread=[kafka-producer-network-thread | x-595d98c54b-k5f52-producer-22] logger=o.s.k.s.LoggingProducerListener trace_id= span_id= message=Exception thrown when sending a message with key='078622be-c0f2-4e89-adb8-6e6eae5526ea' and payload='{"is_i...' to topic x:

org.apache.kafka.clients.consumer.CommitFailedException: Transaction offset Commit failed due to consumer group metadata mismatch: Specified group generation id is not valid.
	at org.apache.kafka.clients.producer.internals.TransactionManager$TxnOffsetCommitHandler.handleResponse(TransactionManager.java:1656)
	at org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1207)
	at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
	at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:583)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575)
	at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:426)
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:316)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
	at java.base/java.lang.Thread.run(Unknown Source)

Another point that might be related is that we have some recurring consumers producing messages to the topics that are also consumed by another consumer in the same service.

[garyrussell]

There is nothing that Spring can do to help with this; the behavior is due to the underlying kafka-clients library and the broker itself. I suggest you raise your concerns over there; either via their JIRA or on one of their mailing lists.