Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Leader election seems elect in endless loop #10522

Closed
linlinnn opened this issue May 9, 2021 · 2 comments · Fixed by #10530
Closed

Leader election seems elect in endless loop #10522

linlinnn opened this issue May 9, 2021 · 2 comments · Fixed by #10530
Labels
type/bug The PR fixed a bug or issue reported a bug type/flaky-tests
Milestone
2.8.0

Comments

@linlinnn
Copy link
Contributor

linlinnn commented May 9, 2021 •
edited
Loading

Describe the bug
TransactionEndToEndTest was broken/flaky (Related to #10457)
Leader election seems elect in endless loop.
I think it can work well when connecting real zk.

To Reproduce
run setup in TransactionEndToEndTest

Expected behavior
pass test

Additional context

18:14:54.813 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.813 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.814 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.814 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.814 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.817 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.817 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.817 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.819 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.820 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.821 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.821 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.821 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.822 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.823 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.823 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.823 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.824 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.824 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.825 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.825 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.825 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.826 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.827 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.827 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.828 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.828 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.828 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.829 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.830 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.830 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.830 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.831 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.831 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.831 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.832 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.833 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.833 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.834 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.834 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.834 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.835 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.835 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.836 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.837 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.837 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.837 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.837 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.838 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.838 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.839 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.839 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.840 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.842 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.842 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.842 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.843 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.843 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.843 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.844 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.844 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.845 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.847 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.847 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.820 [main] ERROR org.apache.pulsar.broker.PulsarService - org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /loadbalance/leader
java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /loadbalance/leader
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_262]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_262]
	at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:714) ~[?:1.8.0_262]
	at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:701) ~[?:1.8.0_262]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_262]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_262]
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$null$21(ZKMetadataStore.java:269) ~[classes/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_262]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_262]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.63.Final.jar:4.1.63.Final]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262]
Caused by: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /loadbalance/leader
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:302) ~[classes/:?]
	... 5 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /loadbalance/leader
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) ~[zookeeper-3.6.2.jar:3.6.2]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.6.2.jar:3.6.2]
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:296) ~[classes/:?]
	... 5 more
18:14:54.847 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.848 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.848 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.848 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.848 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.848 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.849 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.849 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.849 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.849 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.850 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.850 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.850 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.850 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.850 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.851 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.851 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.851 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.851 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.851 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.851 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.852 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.852 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.852 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.853 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.853 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.853 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.853 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.853 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.853 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.854 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.854 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.854 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.854 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.855 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.855 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.855 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.855 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.855 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.856 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.856 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.856 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.856 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.856 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.857 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.857 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.857 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.858 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.858 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.858 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.858 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.858 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader

18:14:54.858 [metadata-store-4-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.859 [metadata-store-4-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.859 [metadata-store-44-1] INFO  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Acquired leadership on /loadbalance/leader
18:14:54.859 [metadata-store-4-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader
18:14:54.859 [metadata-store-44-1] INFO  org.apache.pulsar.broker.PulsarService - This broker was elected leader
18:14:54.859 [metadata-store-44-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Leadership released for /loadbalance/leader


org.apache.pulsar.broker.PulsarServerException: java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /loadbalance/leader

	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:806)
	at org.apache.pulsar.broker.transaction.TransactionTestBase.startBroker(TransactionTestBase.java:138)
	at org.apache.pulsar.broker.transaction.TransactionTestBase.init(TransactionTestBase.java:105)
	at org.apache.pulsar.broker.transaction.TransactionTestBase.internalSetup(TransactionTestBase.java:84)
	at org.apache.pulsar.client.impl.TransactionEndToEndTest.setup(TransactionEndToEndTest.java:94)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61)
	at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:366)
	at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:320)
	at org.testng.internal.TestInvoker.runConfigMethods(TestInvoker.java:701)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:527)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.testng.TestRunner.privateRun(TestRunner.java:764)
	at org.testng.TestRunner.run(TestRunner.java:585)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
	at org.testng.TestNG.runSuites(TestNG.java:1069)
	at org.testng.TestNG.run(TestNG.java:1037)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:110)
Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /loadbalance/leader
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:714)
	at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:701)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$null$21(ZKMetadataStore.java:269)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /loadbalance/leader
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:302)
	... 5 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /loadbalance/leader
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:296)
	... 5 more
@linlinnn linlinnn added the type/bug The PR fixed a bug or issue reported a bug label May 9, 2021
@linlinnn linlinnn changed the title TransactionEndToEndTest was broken TransactionEndToEndTest was broken by recent merged May 9, 2021
@sijie sijie mentioned this issue May 9, 2021
Closed
@linlinnn linlinnn changed the title TransactionEndToEndTest was broken by recent merged TransactionEndToEndTest was broken/flaky May 9, 2021
@linlinnn linlinnn changed the title TransactionEndToEndTest was broken/flaky Leader election seems elect in endless loop May 9, 2021
@linlinnn
Copy link
Contributor Author

linlinnn commented May 10, 2021 •
edited
Loading

This issue caused by mockZookeeper, because all brokers share the same mockZookeeper and same SessionId

if (res.getStat().isCreatedBySelf()) {  // true because of same `SessionId`, should be false if we connect a real zk
    // The existing value is different but was created from the same session
    return store.delete(path, Optional.of(res.getStat().getVersion()))
            .thenCompose(__ -> tryToBecomeLeader());
}

@linlinnn
Copy link
Contributor Author

linlinnn commented May 10, 2021 •
edited
Loading

Any suggestion? @eolivelli @lhotari

@linlinnn linlinnn mentioned this issue May 10, 2021
Merged
@lhotari lhotari added this to the 2.8.0 milestone May 13, 2021
codelipenghui pushed a commit that referenced this issue May 13, 2021
@linlinnn
[Issue 10522] support mock zookeeper with different session (#10530)
1926dee 
fixes #10522

**Motivation**
support mock zookeeper with different session

**Modification**
wrap `MockZookeeper` to support mock zookeeper with different session
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug The PR fixed a bug or issue reported a bug type/flaky-tests
Projects
None yet
Milestone
2.8.0
Development

Successfully merging a pull request may close this issue.

[Issue 10522] support mock zookeeper with different session linlinnn/pulsar
2 participants
@lhotari @linlinnn