Commit 2bdbc87
[SPARK-19564][SPARK-19559][SS][KAFKA] KafkaOffsetReader's consumers should not be in the same group
## What changes were proposed in this pull request?
In `KafkaOffsetReader`, when error occurs, we abort the existing consumer and create a new consumer. In our current implementation, the first consumer and the second consumer would be in the same group (which leads to SPARK-19559), **_violating our intention of the two consumers not being in the same group._**
The cause is that, in our current implementation, the first consumer is created before `groupId` and `nextId` are initialized in the constructor. Then even if `groupId` and `nextId` are increased during the creation of that first consumer, `groupId` and `nextId` would still be initialized to default values in the constructor for the second consumer.
We should make sure that `groupId` and `nextId` are initialized before any consumer is created.
## How was this patch tested?
Ran 100 times of `KafkaSourceSuite`; all passed
Author: Liwei Lin <lwlin7@gmail.com>
Closes #16902 from lw-lin/SPARK-19564-.1 parent bc0a0e6 commit 2bdbc87
File tree
1 file changed
+7
-4
lines changed- external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010
1 file changed
+7
-4
lines changedLines changed: 7 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
67 | 74 | | |
68 | 75 | | |
69 | 76 | | |
| |||
76 | 83 | | |
77 | 84 | | |
78 | 85 | | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | 86 | | |
84 | 87 | | |
85 | 88 | | |
| |||
0 commit comments