Autowired does not support EmbeddedKafkaZKBroker

[TheMagicNacho]

In what version(s) of Spring for Apache Kafka are you seeing this issue?
Spring-Kafka: 3.2.4
Spring-Boot: 3.3.4

Describe the bug
Because of the upgrade from older Spring-Kafka versions, we need to use EmbeddedKafkaZKBroker.

In doing so, @Autowired does not support EmbeddedKafkaZKBroker and yields the error: "Cannot find symbol" for getBrokerAddress().

The current fix is to instantiate EmbeddedKafkaZKBroker object.
This fix doesn't truly work. It allows the app to compile but it fails at runtime.

To Reproduce
Try to autowire the broker.

@Autowired
private EmbeddedKafkaZKBroker kafkaBroker;

// Remainder of code

Error will say:

[ERROR] <path> cannot find symbol
[ERROR] symbol: getBrokerAdress(int)

Expected behavior

Upgrading from EmbededKafkaBroker to EmbeddedKafkaZKBroker should be a drop in replacement.

Sample

This is what the current work around looks like

private EmbeddedKafkaZKBroker kafkaBroker = new EmbeddedKafkaZKBroker(10);

[artembilan]

It is hard to help without more info from the test configuration.
For example, the EmbeddedKafkaZKBroker could be registered as a bean only if alongside with an @EmbeddedKafka there is a @SpringJUnitConfig which exactly indicated that there is a Spring application context to create.
Another observation, in the versions prior 3.2, the @EmbeddedKafka registered brokers inthe KRaft mode, and therefore a bean was exactly EmbeddedKafkaKraftBroker.
So, if you set @EmbeddedKafka(kraft = false), that might fix your current problem.
More info in docs: https://docs.spring.io/spring-kafka/reference/testing.html#embedded-kafka-annotation

[TheMagicNacho]

Hey @artembilan !

Here's a snipit if this helps:

@RunWith(SpringRunner.class)
@SpringBootTest(classes = ManagerApp.class)
@DirtiesContext
@EmbeddedKafka(topics = {"topic.1","topic.2"}, brokerProperties = {"listeners=PLAINTEXT://localhost:9092"}, kraft = false)

public class KafkaTimeSorterTest {


    @Autowired
    private EmbeddedKafkaZKBroker kafkaBroker;

// NOTE : I tried instanciating the broker directly, but this also did not work.
//   private final EmbeddedKafkaZKBroker kafkaBroker = new EmbeddedKafkaZKBroker(1, true, 10, "topic.1", "topic.2");

< REST OF CODE > 

I read through the docs you sent, and for the most part everything looks correct.

[artembilan]

So, what is the problem when you run this?
Do you mean that EmbeddedKafkaZKBroker is not injected or what?
Just wrote this test, works as expected:

@RunWith(SpringRunner.class)
@DirtiesContext
@EmbeddedKafka(topics = {"topic.1","topic.2"}, brokerProperties = {"listeners=PLAINTEXT://localhost:9092"}, kraft = false)
public class Gh3585Tests {

	@Autowired
	private EmbeddedKafkaZKBroker kafkaBroker;

	@Test
	public void testEmbeddedKafkaZKBroker() {
		assertThat(this.kafkaBroker.getBrokerAddress(0)).isEqualTo(new BrokerAddress("127.0.0.1", 9092));
	}

}

[TheMagicNacho]

I'm getting this Error: Could not autowire. No beans of 'EmbeddedKafkaZKBroker' type found.

Rolling back to a branch which uses Spring 3.1.X works fine, granted we were using EmbeddedKafkaBroker for the other branch. No other changes were made during the upgrade, I was expecting upgrading to ..ZKBroker to be a drop in replacement.

[artembilan]

Something is strange...
I wonder if you try to autowire EmbeddedKafkaZKBroker outside of the KafkaTimeSorterTest somewhere else.

So, no any chances that you can share with us that simple project to reproduce on our side?

[muralibasani]

Even I am trying to upgrade to 3.2.4, and notice the the same. EmbeddedKafkaZKBroker cannot be autowired.

I also have another strange issue. About log.dirs issue 3416 even though it's fixed, probably not in this release 3.2.4 ?

[artembilan]

@muralibasani ,

That's still not enough info to dig further.
The issue you are mentioning is about upgrade to Kafka Client 3.8.0 and indeed for Spring for Apache Kafka 3.3.0.
Has nothing to do with log.dirs.
Sorry, but that is confusing.

If you can share with us a simple failing project, then it would make progress in our investigation.
I'm not sure how to reproduce it on my side since the test I've shared above confirms opposite: the EmbeddedKafkaZKBroker can be injected into the test class.

[muralibasani]

@artembilan sorry, I mixed up things. Regarding log.dirs issue, I was referring to #3417 which is not part of 3.2.4 release yet.

Regarding autowired issue, EmbeddedKafkaZKBroker gets injected into the test class properly, just that the ide (intellij) displays this error "Could not autowire. No beans of 'EmbeddedKafkaZKBroker' type found. " only after this upgrade to 3.2.4

[artembilan]

I still don't understand the story about log.dirs. Sounds more like this issue hijacking and not relevant.
And that Kafka 3.8.0 upgrade cannot go to the 3.2.x generation since that contradicts point-minor versioning policy.

Re. IntelliJ IDEA and its error: it has nothing to do with this project.
We don't do any tooling over here.
Perhaps more like a question to JetBrains.

An EmbeddedKafkaBroker bean is registered by the EmbeddedKafkaContextCustomizer. And it always was.
Not sure why IDEA was able to find it before we have introduced KRaft support, but that is still outside of this project scope.

Closed as Works as Designed.

Please, consider in the future be more specific with the question and issues you are trying to get help on from us.