Create partitions for a partitioned topic even if topic auto-creation disabled #9173
Comments
codelipenghui
added
type/bug
codelipenghui
changed the title
|
Move this to the next release |
|
I'll be happy to participate |
|
@codelipenghui can you please share more details on the way you proceed to reproduce the issue ? Here is how I've tested and the results. Auto Creation EnabledBy keeping the auto-creation enabled, After a certain time of inactivity, the regular topic, auto-created, is deleted. Auto Creation DisabledBy disabling the auto-creation, Hence the partitions are auto-created even when the auto-creation is disabled. The "odd" behavior that I noticed is the auto-creation of that regular topic with the same name as the partitioned topic. Pulsar version: 2.7.0 |
|
got that thanks. And I can reproduce it. @codelipenghui What if the partitions are created automatically when a non-persistent partitioned topic is created? Like for persistent topics? |
|
@mlyahmed For the persistent topic, we can create all partitions when creating the partitioned topic, the metadata for each partition also maintains in the zookeeper. But for the non-persistent topic, we don't need to store any metadata for the partition, so we can't keep consistent with the current persistent topic behavior I think we can fix the non-persistent topic case directly, if the non-persistent partitioned topic is created, we should allow creating partitions even if the topic auto-creation disabled so that the client able to connect to partitions. |
|
@codelipenghui when the partitions, for non persistent topics, created are those metadata created? |
|
@mlyahmed No, Pulsar does not store any metadata for the non-persistent topic. |
|
@codelipenghui got that thanks ! |
|
Here is a suggestion |
… the auto-creation is disabled (#9786) Related to #9173 Master Issue: #9173 ### Motivation Please see #9173 ### Modifications When the BrokerService is about to create a non-persistent topic it checks if it is a partition and if so, even the auto-creation is disabled, it creates it. (cherry picked from commit a19dc65)
|
It seems non-persistent part of this issue is fixed. |
|
@codelipenghui Here is how I reproduce this case for persistent topics.
Here is the main error log in server:
|
Describe the bug
Currently, for the persistent topic. If create a new partitioned persistent topic, we also create partitions. So even if users disabled the topic auto-creation, the producers and consumers also can connect to the partitions. But for the non-persistent topic. It needs to enable the topic auto-creation, otherwise, the partitions will not be created since we don't maintain any partition metadata in the metadata service. See #9107
But, enabled the topic auto-creation will allow any new topics from the producers and consumers. From the Pulsar maintainer perspective, if the partitioned topic created, this means allows to create the partitons of the partitioned topic.
Expected behavior
Even if the topic auto-creation disabled, the partitions should be created automatically if the partitioned topic created.
The text was updated successfully, but these errors were encountered: