Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIx KAAP Issues #181

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

FIx KAAP Issues #181

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a3d4f22
Fix NPE when auth is disabled in pulsar cluster
rithick-ctds Oct 23, 2024
91d33dc
Remove ensemblePlacementPolicy incase if it's empty.
rithick-ctds Oct 23, 2024
61eedf9
Disable Applying lables to VolumeClainTemplates
rithick-ctds Oct 23, 2024
b1f6c9b
Fix Test Cases
rithick-ctds Oct 29, 2024
c01b01a
Fix Migration Tool Test Cases
rithick-ctds Oct 30, 2024
80b1c36
Add Flag Enable/Disable Labels
rithick-ctds Nov 11, 2024
46d5380
Revert "Fix Test Cases"
rithick-ctds Nov 11, 2024
464b1e6
Fix Tests
rithick-ctds Nov 11, 2024
352fdf0
Skip Volume Claim Labels
rithick-ctds Nov 11, 2024
b85f160
Update docs
rithick-ctds Nov 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions docs/api-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10903,6 +10903,13 @@ Resources requirements.
Sidecar containers<br/>
</td>
<td>false</td>
</tr><tr>
<td><b>skipVolumeClaimLabels</b></td>
<td>boolean</td>
<td>
Skip Applying Labels to Statefulset.VolumeClaimTemplate<br/>
</td>
<td>false</td>
</tr><tr>
<td><b><a href="#pulsarclusterspecbookkeepertolerationsindex">tolerations</a></b></td>
<td>[]object</td>
Expand Down Expand Up @@ -17529,6 +17536,13 @@ Service configuration.
Sidecar containers<br/>
</td>
<td>false</td>
</tr><tr>
<td><b>skipVolumeClaimLabels</b></td>
<td>boolean</td>
<td>
Skip Applying Labels to Statefulset.VolumeClaimTemplate<br/>
</td>
<td>false</td>
</tr><tr>
<td><b><a href="#pulsarclusterspecbookkeepersetskeytolerationsindex">tolerations</a></b></td>
<td>[]object</td>
Expand Down Expand Up @@ -29167,6 +29181,13 @@ Indicates if a StorageClass is used. The operator will create the StorageClass i
Sidecar containers<br/>
</td>
<td>false</td>
</tr><tr>
<td><b>skipVolumeClaimLabels</b></td>
<td>boolean</td>
<td>
Skip Applying Labels to Statefulset.VolumeClaimTemplate<br/>
</td>
<td>false</td>
</tr><tr>
<td><b><a href="#pulsarclusterspecbrokertolerationsindex">tolerations</a></b></td>
<td>[]object</td>
Expand Down Expand Up @@ -35810,6 +35831,13 @@ Service configuration.
Sidecar containers<br/>
</td>
<td>false</td>
</tr><tr>
<td><b>skipVolumeClaimLabels</b></td>
<td>boolean</td>
<td>
Skip Applying Labels to Statefulset.VolumeClaimTemplate<br/>
</td>
<td>false</td>
</tr><tr>
<td><b><a href="#pulsarclusterspecbrokersetskeytolerationsindex">tolerations</a></b></td>
<td>[]object</td>
Expand Down Expand Up @@ -47214,6 +47242,13 @@ Update strategy for the StatefulSet.
Sidecar containers<br/>
</td>
<td>false</td>
</tr><tr>
<td><b>skipVolumeClaimLabels</b></td>
<td>boolean</td>
<td>
Skip Applying Labels to Statefulset.VolumeClaimTemplate<br/>
</td>
<td>false</td>
</tr><tr>
<td><b><a href="#pulsarclusterspecfunctionsworkertolerationsindex">tolerations</a></b></td>
<td>[]object</td>
Expand Down Expand Up @@ -57926,6 +57961,13 @@ TLS configurations related to the ZooKeeper component.
Sidecar containers<br/>
</td>
<td>false</td>
</tr><tr>
<td><b>skipVolumeClaimLabels</b></td>
<td>boolean</td>
<td>
Skip Applying Labels to Statefulset.VolumeClaimTemplate<br/>
</td>
<td>false</td>
</tr><tr>
<td><b>standaloneFunctionsWorker</b></td>
<td>boolean</td>
Expand Down Expand Up @@ -64450,6 +64492,13 @@ Service configuration.
Sidecar containers<br/>
</td>
<td>false</td>
</tr><tr>
<td><b>skipVolumeClaimLabels</b></td>
<td>boolean</td>
<td>
Skip Applying Labels to Statefulset.VolumeClaimTemplate<br/>
</td>
<td>false</td>
</tr><tr>
<td><b>standaloneFunctionsWorker</b></td>
<td>boolean</td>
Expand Down Expand Up @@ -76049,6 +76098,13 @@ Resources requirements.
Sidecar containers<br/>
</td>
<td>false</td>
</tr><tr>
<td><b>skipVolumeClaimLabels</b></td>
<td>boolean</td>
<td>
Skip Applying Labels to Statefulset.VolumeClaimTemplate<br/>
</td>
<td>false</td>
</tr><tr>
<td><b><a href="#pulsarclusterspeczookeepertolerationsindex">tolerations</a></b></td>
<td>[]object</td>
Expand Down
6 changes: 6 additions & 0 deletions helm/kaap/crds/bookkeepers.kaap.oss.datastax.com-v1.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -904,6 +904,9 @@ spec:
type: string
type: object
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down Expand Up @@ -3083,6 +3086,9 @@ spec:
type: string
type: object
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down
6 changes: 6 additions & 0 deletions helm/kaap/crds/brokers.kaap.oss.datastax.com-v1.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -855,6 +855,9 @@ spec:
Default is 'false'.
type: boolean
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down Expand Up @@ -3000,6 +3003,9 @@ spec:
is 'false'.
type: boolean
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down
3 changes: 3 additions & 0 deletions helm/kaap/crds/functionsworkers.kaap.oss.datastax.com-v1.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -787,6 +787,9 @@ spec:
should be used.
type: string
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down
6 changes: 6 additions & 0 deletions helm/kaap/crds/proxies.kaap.oss.datastax.com-v1.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -850,6 +850,9 @@ spec:
Default is 'false'.
type: boolean
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down Expand Up @@ -2992,6 +2995,9 @@ spec:
is 'false'.
type: boolean
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down
24 changes: 24 additions & 0 deletions helm/kaap/crds/pulsarclusters.kaap.oss.datastax.com-v1.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -850,6 +850,9 @@ spec:
Default is 'false'.
type: boolean
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down Expand Up @@ -2992,6 +2995,9 @@ spec:
is 'false'.
type: boolean
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down Expand Up @@ -5077,6 +5083,9 @@ spec:
should be used.
type: string
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down Expand Up @@ -9619,6 +9628,9 @@ spec:
should be used.
type: string
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down Expand Up @@ -11763,6 +11775,9 @@ spec:
Default is 'false'.
type: boolean
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down Expand Up @@ -13908,6 +13923,9 @@ spec:
is 'false'.
type: boolean
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down Expand Up @@ -16101,6 +16119,9 @@ spec:
type: string
type: object
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down Expand Up @@ -18280,6 +18301,9 @@ spec:
type: string
type: object
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down
3 changes: 3 additions & 0 deletions helm/kaap/crds/zookeepers.kaap.oss.datastax.com-v1.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,9 @@ spec:
should be used.
type: string
type: object
skipVolumeClaimLabels:
description: Skip Applying Labels to Statefulset.VolumeClaimTemplate
type: boolean
imagePullSecrets:
description: Image pull secrets.
items:
Expand Down
5 changes: 3 additions & 2 deletions ...ool/src/main/java/com/datastax/oss/kaap/migrationtool/PulsarClusterResourceGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
import java.util.function.Predicate;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.BooleanUtils;
import org.apache.commons.lang3.StringUtils;

@Slf4j
Expand Down Expand Up @@ -210,11 +211,11 @@ private AuthConfig getAuthConfig() {
)
.contains(spec.getSpecName()) && spec.isEnabled();

final boolean authEnabled = getValueAssertSame(
final boolean authEnabled = BooleanUtils.isTrue(getValueAssertSame(
filterComponents,
spec -> parseConfigValueBool(spec.getConfig(), "authenticationEnabled"),
false,
"authenticationEnabled").booleanValue();
"authenticationEnabled"));
if (!authEnabled) {
log.info("Found authenticationEnabled=false, auth will be disabled in the CRD");
return AuthConfig.builder()
Expand Down
6 changes: 4 additions & 2 deletions ...ol/src/main/java/com/datastax/oss/kaap/migrationtool/specs/AutorecoverySpecGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
import java.util.Map;
import java.util.Objects;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;

@Slf4j
public class AutorecoverySpecGenerator extends BaseSpecGenerator<AutorecoverySpec> {
Expand Down Expand Up @@ -82,8 +83,9 @@ public void internalGenerateSpec() {
isRestartOnConfigMapChange = isPodDependantOnConfigMap(deploymentSpec.getTemplate());
priorityClassName = spec.getPriorityClassName();
config = convertConfigMapData(configMap);
if (!config.containsKey("ensemblePlacementPolicy")) {
config.put("ensemblePlacementPolicy", "");
Object ensemblePlacementPolicy = config.get("ensemblePlacementPolicy");
if (ensemblePlacementPolicy instanceof String policy && StringUtils.isBlank(policy)) {
config.remove("ensemblePlacementPolicy");
}
tlsEntryConfig = createTlsEntryConfig(deployment);

Expand Down
1 change: 1 addition & 0 deletions ...l/src/main/java/com/datastax/oss/kaap/migrationtool/specs/BookKeeperSetSpecGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ public void internalGenerateSpec() {
.nodeAffinity(nodeAffinity)
.pdb(podDisruptionBudgetConfig)
.labels(statefulSet.getMetadata().getLabels())
.skipVolumeClaimLabels(true)
.podLabels(statefulSetSpec.getTemplate().getMetadata().getLabels())
.matchLabels(matchLabels)
.config(config)
Expand Down
1 change: 1 addition & 0 deletions ...src/main/java/com/datastax/oss/kaap/migrationtool/specs/FunctionsWorkerSpecGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ public void internalGenerateSpec() {
.nodeAffinity(nodeAffinity)
.probes(createProbeConfig(mainContainer))
.labels(statefulSet.getMetadata().getLabels())
.skipVolumeClaimLabels(true)
.podLabels(statefulSetSpec.getTemplate().getMetadata().getLabels())
.matchLabels(matchLabels)
.config(config)
Expand Down
1 change: 1 addition & 0 deletions ...-tool/src/main/java/com/datastax/oss/kaap/migrationtool/specs/ZooKeeperSpecGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ public void internalGenerateSpec() {
.nodeAffinity(nodeAffinity)
.pdb(podDisruptionBudgetConfig)
.labels(statefulSet.getMetadata().getLabels())
.skipVolumeClaimLabels(true)
.podLabels(statefulSetSpec.getTemplate().getMetadata().getLabels())
.matchLabels(matchLabels)
.config(config)
Expand Down
10 changes: 6 additions & 4 deletions ...src/test/java/com/datastax/oss/kaap/migrationtool/PulsarClusterResourceGeneratorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ public void test() throws Exception {
final DiffCollectorOutputWriter diff = generate(client, tmpDir);
final File outputDir = new File(tmpDir.toFile(), CONTEXT);
assertValue(outputDir);
assertDiff(diff, 159);
assertDiff(diff, 155);
}

@Test
Expand All @@ -74,7 +74,7 @@ public void testNoFunctions() throws Exception {
final DiffCollectorOutputWriter diff = generate(client, tmpDir);
final File outputDir = new File(tmpDir.toFile(), CONTEXT);
final PulsarCluster pulsar = getPulsarClusterFromOutputdir(outputDir);
assertDiff(diff, 120);
assertDiff(diff, 117);
Assert.assertEquals(pulsar.getSpec().getFunctionsWorker().getReplicas(), 0);
}

Expand All @@ -91,7 +91,7 @@ public void testNoBastion() throws Exception {
final DiffCollectorOutputWriter diff = generate(client, tmpDir);
final File outputDir = new File(tmpDir.toFile(), CONTEXT);
final PulsarCluster pulsar = getPulsarClusterFromOutputdir(outputDir);
assertDiff(diff, 152);
assertDiff(diff, 148);
Assert.assertEquals(pulsar.getSpec().getBastion().getReplicas(), 0);
}

Expand Down Expand Up @@ -202,6 +202,7 @@ private void assertValue(File tmpDir) {
component: zookeeper
heritage: Helm
release: pulsar-cluster
skipVolumeClaimLabels: true
podLabels:
app: pulsar
cluster: pulsar-cluster
Expand Down Expand Up @@ -284,6 +285,7 @@ private void assertValue(File tmpDir) {
component: bookkeeper
heritage: Helm
release: pulsar-cluster
skipVolumeClaimLabels: true
podLabels:
app: pulsar
cluster: pulsar-cluster
Expand Down Expand Up @@ -892,7 +894,6 @@ private void assertValue(File tmpDir) {
PULSAR_EXTRA_OPTS: -Dpulsar.log.root.level=info
PULSAR_LOG_LEVEL: info
PULSAR_LOG_ROOT_LEVEL: info
ensemblePlacementPolicy: ""
zkServers: pulsar-cluster-zookeeper-ca:2181
replicas: 1
annotations: {}
Expand Down Expand Up @@ -1013,6 +1014,7 @@ private void assertValue(File tmpDir) {
component: function
heritage: Helm
release: pulsar-cluster
skipVolumeClaimLabels: true
podLabels:
app: pulsar
cluster: pulsar-cluster
Expand Down
Loading
Loading