Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] text_encoder failing kfp_ray test #678

Open
1 of 2 tasks
daw3rd opened this issue Oct 7, 2024 · 1 comment
Open
1 of 2 tasks

[Bug] text_encoder failing kfp_ray test #678

daw3rd opened this issue Oct 7, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@daw3rd
Copy link
Member

daw3rd commented Oct 7, 2024

Search before asking

  • I searched the issues and found no similar issues.

Component

Transforms/Other

What happened + What you expected to happen

With the recent PR to properly enabled kfp workflow tests (PR #672) on transforms, we found that repo_level_ordering was not passing (see log here. To get the PR to pass, text_encoder/kfp_ray/Makefile was renamed to Makefile.disable-cicd.

Reproduction script

To reproduce

cd transforms/language/text_encoder/kfp_ray
git mv Makefile.disable-cicd Makefile
git commit -s -a -m "restore kfp_ray test for text_encoder"
git push

Make a PR and watch the cicd

Anything else

 make[5]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
rm -rf data-processing-lib-ray
rm -rf data-processing-lib-python
rm -rf shared_workflow_support_lib
rm -rf workflow_support_lib
make[4]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
make[3]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
make[3]: Entering directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
kind load docker-image quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest --name=dataprep
Image: "quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" with ID "sha256:000fdabe3bf3f95eabb3f5aef098fed7e57e047bda1046df696ecf299eaeb7d0" not yet present on node "dataprep-control-plane", loading...
Image: "quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" with ID "sha256:000fdabe3bf3f95eabb3f5aef098fed7e57e047bda1046df696ecf299eaeb7d0" not yet present on node "dataprep-worker", loading...
Image: "quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" with ID "sha256:000fdabe3bf3f95eabb3f5aef098fed7e57e047bda1046df696ecf299eaeb7d0" not yet present on node "dataprep-worker2", loading...
ERROR: command "docker save -o /tmp/images-tar359392491/images.tar quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" failed with error: exit status 1

Command Output: write /tmp/images-tar359392491/.docker_temp_4286091745: no space left on device
make[3]: *** [Makefile:57: kind-load-image] Error 1
make[3]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
make[2]: *** [/home/runner/work/data-prep-kit/data-prep-kit/transforms/language/text_encoder/kfp_ray/../../../..//transforms/.make.workflows:45: .workflows.test-pipeline] Error 2
make[2]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/language/text_encoder/kfp_ray'
make[1]: *** [Makefile:45: workflow-test] Error 2
make[1]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/language/text_encoder/kfp_ray'
make: *** [Makefile:64: workflow-test] Error 2
make: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/language/text_encoder'
Error: Process completed with exit code 2.

OS

Ubuntu

Python

3.10.x

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!
@daw3rd daw3rd added the bug Something isn't working label Oct 7, 2024
@roytman roytman assigned revit13 and unassigned Mohammad-nassar10 Oct 10, 2024
@revit13
Copy link
Collaborator

revit13 commented Oct 10, 2024

It is added to the kfp black list in #684. Additionally, the kfp_ray/Makefile has been restored to enable local testing and to verify the workflow build in the CI/CD pipeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants