Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dm: fix chaos test #11686

Merged
merged 23 commits into from
Oct 30, 2024
2 changes: 1 addition & 1 deletion .github/workflows/dataflow_engine_chaos.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -333,7 +333,7 @@ jobs:
# Upload logs as artifact seems not stable, so we set `continue-on-error: true` here.
- name: Upload logs
continue-on-error: true
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: chaos-base-logs.${{ matrix.chaos-obj }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/dm_binlog_999999.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ jobs:
# Update logs as artifact seems not stable, so we set `continue-on-error: true` here.
- name: Upload logs
continue-on-error: true
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: test-binlog-999999
Expand Down
31 changes: 20 additions & 11 deletions .github/workflows/dm_chaos.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,19 @@ jobs:
path: tools/bin
key: ${{ runner.os }}-ticdc-tools-${{ hashFiles('tools/check/go.sum') }}

- name: Create k8s Kind Cluster
uses: helm/kind-action@v1.4.0
- name: install k3s
run: |
curl -fsSL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 \
"${k3s_disable_command:---disable}" metrics-server \
"${k3s_disable_command:---disable}" traefik \
--flannel-backend=none \
--docker
shell: bash

- name: Export KUBECONFIG environment variable
run: |
echo 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> $GITHUB_ENV
shell: bash

- name: Print cluster information
run: |
Expand Down Expand Up @@ -96,12 +107,7 @@ jobs:
cp -r $GITHUB_WORKSPACE/dm/chaos/cases/conf/ $GITHUB_WORKSPACE/bin/
docker build -f $GITHUB_WORKSPACE/dm/chaos/manifests/Dockerfile -t dm:chaos $GITHUB_WORKSPACE/bin
docker image list

# Load DM docker image into KIND, see https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster
- name: Load DM docker image into KIND
run: |
kind load docker-image dm:chaos --name chart-testing


# Set up upstream instances
- name: Set up sources
run: |
Expand Down Expand Up @@ -252,18 +258,21 @@ jobs:
- name: Wait for chaos test case complete
run: |
$GITHUB_WORKSPACE/dm/chaos/scripts/check-case.sh

- name: Setup tmate session
if: ${{ failure() }}
uses: mxschmitt/action-tmate@v3

- name: Copy logs to hack permission
if: ${{ always() }}
run: |
mkdir ./logs
kubectl get pods --no-headers -o custom-columns=":metadata.name"|grep -E "dm-"|xargs -I{} sudo kubectl cp {}:/log/{}.log ./logs/{}.log || true
kind export logs ./logs/kind --name chart-testing
kubectl get pods --no-headers -o custom-columns=":metadata.name"|grep -E "dm-"|xargs -I{} kubectl cp {}:/log/{}.log ./logs/{}.log || true
sudo chown -R runner ./logs
# Update logs as artifact seems not stable, so we set `continue-on-error: true` here.
- name: Upload logs
continue-on-error: true
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: chaos-base-logs.${{ matrix.chaos-obj }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/dm_mariadb_master_down_and_up.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
# Update logs as artifact seems not stable, so we set `continue-on-error: true` here.
- name: Upload logs
continue-on-error: true
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: upstream-switch-logs
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/dm_upstream_switch.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ jobs:
# Update logs as artifact seems not stable, so we set `continue-on-error: true` here.
- name: Upload logs
continue-on-error: true
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: upstream-switch-logs
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/upgrade_dm_via_tiup.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ jobs:
# Update logs as artifact seems not stable, so we set `continue-on-error: true` here.
- name: Upload logs
continue-on-error: true
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
if: ${{ failure() }}
with:
name: upgrade-via-tiup-${{ matrix.previous_v2 }}
Expand Down
2 changes: 1 addition & 1 deletion dm/chaos/cases/cases.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ var (
// NOTE: items in `doSchemas` should be specified in the corresponding task files (`filenames`).
// TODO: can not support optimistic shard DDL now because go-sqlsmith will generated some statements like
// `ALTER TABLE `db_optimistic`.`dxeyydwb` ADD COLUMN (`zuhxfgsce` INT(18) NOT NULL)` which has NOT NULL column without DEFAULT value.
filenames = []string{"task-single.yaml", "task-pessimistic.yaml", "task-optimistic.yaml"}
filenames = []string{"task-single.yaml"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pass it by command line argument?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, will update it in next pr

doSchemas = []string{"db_single", "db_pessimistic", "db_optimistic"}
)

Expand Down
2 changes: 1 addition & 1 deletion dm/chaos/cases/task.go
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@
if err != nil {
return err
} else if !resp.Result && !strings.Contains(resp.Msg, "already exist") { // imprecise match
return fmt.Errorf("fail to start task: %s", resp.Msg)
return fmt.Errorf("fail to start task: %v", resp)

Check warning on line 285 in dm/chaos/cases/task.go

View check run for this annotation

Codecov / codecov/patch

dm/chaos/cases/task.go#L285

Added line #L285 was not covered by tests
}
return nil
}
Expand Down
2 changes: 1 addition & 1 deletion dm/chaos/manifests/dm-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ spec:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storage: 100Mi
- metadata:
name: dm-master-log
spec:
Expand Down
2 changes: 1 addition & 1 deletion dm/chaos/manifests/dm-worker.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ spec:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storage: 100Mi
- metadata:
name: dm-worker-log
spec:
Expand Down
4 changes: 2 additions & 2 deletions dm/chaos/manifests/sources.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ spec:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storage: 500Mi

---
apiVersion: apps/v1
Expand All @@ -87,7 +87,7 @@ spec:
spec:
containers:
- name: mysql8
image: mysql:latest
image: mysql:8.0
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: "/var/lib/mysql"
Expand Down
2 changes: 1 addition & 1 deletion dm/chaos/manifests/tidb.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,4 @@ spec:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storage: 500Mi
Loading