Support regex topic pattern in consumer group topic subscribe?

[jeffwidman]

I'm not seeing anything in the docs about how to pass a regex topic pattern for a consumer group subscription?

My coworker checked a few months back and didn't see it, but in this morning's blog post @jkreps said the Python client has feature parity (I assume with Java) so did we just miss it?

We're redoing our internal Kafka python wrapper and trying to decide whether to bet on the Confluent python library or the Kafka-python library.

[edenhill]

The next version of librdkafka (0.9.2, or master) that will be released within a week (or so.., there is an 0.9.2-RC1 for the restless) provides proper support for regex subscriptions.

Any subscribe()d topic name that begins with "^" will be treated as a regexp and matched with the full set of topics in the cluster.

[jeffwidman]

Great. When this is released, it'd be nice to add an example of how to do this to the docs for this Python wrapper... it's no fun having to look at Python docs and then librdkafka docs and then back again.

[jianbin-wei]

Does this support topics created in the future?

[edenhill]

@jianbin-wei Yes it does

@jeffwidman Absolutely, will see to it.

[edenhill]

Make sure to use librdkafka 0.9.2 or later (master) with regex subscriptions.

[jeffwidman]

The docstring update isn't appearing in the actual docs... I'm guessing Sphinx just needs to do a rebuild?

[edenhill]

By actual docs you mean http://docs.confluent.io/...?

This commit was added after v0.9.2 which the confluent docs are based on.

[jeffwidman]

That makes sense. I couldn't find a way to switch the version I was viewing to master so wasn't sure if I was just missing it or was an error.

[edenhill]

The syntax is standard .subscribe(topic_list) but prefix regexp-patterns with "^", e.g.:
c.subscribe(["fixedtopic", "^a_regex.*", "otherfixed", "^or_this_[0-9]"])

[Kashyap23]

@edenhill I want to match a pattern which has changing pre-fix.
Eg :

testsetup-test-topic
prodsetup-test-topic

I tried the following

c.subscribe(["*test-topic"])

But didnt get the message at the consumer.

Then

c.subscribe(["^*test-topic"])

But got the following error

KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to set subscription: Local: Invalid argument or configuration"}

Can you please help me in the above ?

Thanks in advance.

[edenhill]

^.*-test-topic$

[shubhang93]

^.*-test-topic$

What if I want to use the ^ character to match the start of the string? Can I do this ^^.*-test-topic$

[edenhill]

It behaves like a standard regexp as-is, ^ anchoring the beginning of the string.
^a.* - must start with a
^.*a.* - must contain a