Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

checkAutoCommitIsDisabled fails for kafka-clients v3.7.0 #721

Closed
nscuro opened this issue Feb 27, 2024 · 2 comments
Closed

checkAutoCommitIsDisabled fails for kafka-clients v3.7.0 #721

nscuro opened this issue Feb 27, 2024 · 2 comments
Labels
enhancement New feature or request verified bug Something isn't working

Comments

@nscuro
Copy link
Contributor

nscuro commented Feb 27, 2024

When using PC with kafka-clients v3.7.0, PC fails while trying to determine whether the provided Consumer has auto-commit enabled:

Caused by: java.lang.IllegalStateException: Cannot check auto commit is disabled for consumer type: org.apache.kafka.clients.consumer.KafkaConsumer
	at io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.checkAutoCommitIsDisabled(AbstractParallelEoSStreamProcessor.java:493)
	at io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.validateConfiguration(AbstractParallelEoSStreamProcessor.java:330)
	at io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.<init>(AbstractParallelEoSStreamProcessor.java:283)
	at io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.<init>(AbstractParallelEoSStreamProcessor.java:266)
	at io.confluent.parallelconsumer.ParallelEoSStreamProcessor.<init>(ParallelEoSStreamProcessor.java:46)
	at io.confluent.parallelconsumer.ParallelStreamProcessor.createEosStreamProcessor(ParallelStreamProcessor.java:26)
...
Caused by: java.lang.NoSuchFieldException: coordinator
	at java.base/java.lang.Class.getDeclaredField(Class.java:2610)
	at io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.checkAutoCommitIsDisabled(AbstractParallelEoSStreamProcessor.java:473)

Upon surface-level investigation, the failure is caused by https://issues.apache.org/jira/browse/KAFKA-15277. KafkaConsumer no longer holds a coordinator field.

Instead, KafkaConsumer now has a ConsumerDelegate<K, V> delegate field, which resolves to either LegacyKafkaConsumer, or AsyncKafkaConsumer.

LegacyKafkaConsumer still has a coordinator field, but AsyncKafkaConsumer does not.

@rkolesnev
Copy link
Contributor

I will need to work on making the reflection checks to account for this, but i am thinking that it may be worth to add an override option to disable this check in edge cases like when custom delegate / wrapper for Consumer is used or when things break due to dependency changes until Parallel Consumer code catches up etc.

@rkolesnev
Copy link
Contributor

Closing as fixed.
Please re-open if still an issue after the #762.

Thanks @nscuro and @ondryaso!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request verified bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants