This step is in Error state with this message: failed to save outputs: path /mlpipeline-ui-metadata.json does not exist (or /mlpipeline-ui-metadata.json is empty) in archive /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz #1589
Comments
|
Hello @malixian, Looks like you are using old SDK and kfp versions. This issue is fixed by #1260 and #1289. |
|
Closing, feel free to reopen if upgarde doesn't address it. |
you mean i should upgrade kfp in notebook container? |
|
Yes, that is an option. Either upgrade your KFP SDK and recompile the pipeline, or upgrade the KFP installation. The second option means to bump the images of the corresponding Kubernetes deployments ( |
tanks，this problem is sloved，but another has arisen，argo logs show “This step is in Error state with this message: failed to save outputs: open /argo/secret/mlpipeline-minio-artifact/accesskey: no such file or directory”，what‘s the problem？sad... |
|
Hello! have you found a solution to this? |
i write code with notebook like this in arm node:
"""
@dsl.pipeline( name='Testpipelines',description='shows how to define dsl.Condition.')
def cambricon_demo():
op = dsl.ContainerOp(name='cambricon-1', image='10.18.127.4:5000/cambricon/test/centos:v.3.3')
testOp = op.add_volume_devices(k8s_client.V1VolumeDevice(name="cambricon-device", device_path="/dev/cambricon_c10Dev0"))
camb = cambriconOp().add_volume(
k8s_client.V1Volume(name='cambricon-mlu', nfs=k8s_client.V1NFSVolumeSource(path='/mnt/xfs/Cambricon-MLU100',server='10.18.129.161'))).add_volume_mount(k8s_client.V1VolumeMount(mount_path='/home/Cambricon-Test',name='cambricon-mlu'))
camb.add_resource_limit("cambricon.com/mlu", "1")
device_name = "dev-cambricon"
#camb.add_volume(k8s_client.V1Volume(name=device_name, host_path=k8s_client.V1HostPathVolumeSource(path="/dev/cambricon_c10Dev0"))).add_volume_mount(k8s_client.V1VolumeMount(name=device_name, mount_path="/dev/cambricon_c10Dev0"))
#camb._container.set_security_context(k8s_client.V1SecurityContext(privileged=True))
camb.add_node_selector_constraint('beta.kubernetes.io/arch','arm64')
""""
but the code failed, and logs we can see:
This step is in Error state with this message: failed to save outputs: path /mlpipeline-ui-metadata.json does not exist (or /mlpipeline-ui-metadata.json is empty) in archive /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz
then i run command "kubectl logs task-pod-id -n kubeflow -c wait" i find:
time="2019-07-03T10:26:38Z" level=info msg="Creating a docker executor"
time="2019-07-03T10:26:38Z" level=info msg="Executor (version: v2.4.0+c9f6d96, build_date: 2019-06-21T07:52:00Z) initialized (pod: kubeflow/testpipelines-94wqg-3058698834) with template:\n{"name":"cambricon-test","inputs":{},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"runs/0908f320-9d7d-11e9-9f5c-ac1f6bac1d10/testpipelines-94wqg-3058698834/mlpipeline-ui-metadata.tgz"}},{"name":"mlpipeline-metrics","path":"/mlpipeline-metrics.json","s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"runs/0908f320-9d7d-11e9-9f5c-ac1f6bac1d10/testpipelines-94wqg-3058698834/mlpipeline-metrics.tgz"}}]},"nodeSelector":{"beta.kubernetes.io/arch":"arm64"},"metadata":{},"container":{"name":"","image":"10.18.127.1:5000/argoexec:v0.1","command":["sh","-c"],"args":["sleep 300"],"resources":{},"volumeMounts":[{"name":"cambricon-mlu","mountPath":"/home/Cambricon-Test"}]},"archiveLocation":{"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/testpipelines-94wqg/testpipelines-94wqg-3058698834"}}}"
time="2019-07-03T10:26:38Z" level=info msg="Waiting on main container"
time="2019-07-03T10:26:39Z" level=info msg="main container started with container ID: 13a6281936bc2e8afe85d5a8b4f06b967ec7cda7d71b971c528b8354e80d6972"
time="2019-07-03T10:26:39Z" level=info msg="Starting annotations monitor"
time="2019-07-03T10:26:39Z" level=info msg="docker wait 13a6281936bc2e8afe85d5a8b4f06b967ec7cda7d71b971c528b8354e80d6972"
time="2019-07-03T10:26:39Z" level=info msg="Starting deadline monitor"
time="2019-07-03T10:31:37Z" level=info msg="Main container completed"
time="2019-07-03T10:31:37Z" level=info msg="Killing sidecars"
time="2019-07-03T10:31:37Z" level=info msg="Annotations monitor stopped"
time="2019-07-03T10:31:37Z" level=info msg="No output parameters"
time="2019-07-03T10:31:37Z" level=info msg="Saving output artifacts"
time="2019-07-03T10:31:37Z" level=info msg="Staging artifact: mlpipeline-ui-metadata"
time="2019-07-03T10:31:37Z" level=info msg="Copying /mlpipeline-ui-metadata.json from container base image layer to /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz"
time="2019-07-03T10:31:37Z" level=info msg="Archiving 13a6281936bc2e8afe85d5a8b4f06b967ec7cda7d71b971c528b8354e80d6972:/mlpipeline-ui-metadata.json to /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz"
time="2019-07-03T10:31:37Z" level=info msg="sh -c docker cp -a 13a6281936bc2e8afe85d5a8b4f06b967ec7cda7d71b971c528b8354e80d6972:/mlpipeline-ui-metadata.json - | gzip > /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz"
time="2019-07-03T10:31:38Z" level=warning msg="path /mlpipeline-ui-metadata.json does not exist (or /mlpipeline-ui-metadata.json is empty) in archive /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz"
time="2019-07-03T10:31:38Z" level=error msg="executor error: path /mlpipeline-ui-metadata.json does not exist (or /mlpipeline-ui-metadata.json is empty) in archive /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz\ngithub.com/argoproj/argo/errors.New\n\t/go/src/github.com/argoproj/argo/errors/errors.go:49\ngithub.com/argoproj/argo/errors.Errorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:55\ngithub.com/argoproj/argo/workflow/executor/docker.(*DockerExecutor).CopyFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/docker/docker.go:66\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).stageArchiveFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:347\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).saveArtifact\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:248\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:234\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:54\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1114"
time="2019-07-03T10:31:38Z" level=info msg="Alloc=3736 TotalAlloc=11667 Sys=70590 NumGC=6 Goroutines=8"
time="2019-07-03T10:31:38Z" level=fatal msg="path /mlpipeline-ui-metadata.json does not exist (or /mlpipeline-ui-metadata.json is empty) in archive /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz\ngithub.com/argoproj/argo/errors.New\n\t/go/src/github.com/argoproj/argo/errors/errors.go:49\ngithub.com/argoproj/argo/errors.Errorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:55\ngithub.com/argoproj/argo/workflow/executor/docker.(*DockerExecutor).CopyFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/docker/docker.go:66\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).stageArchiveFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:347\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).saveArtifact\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:248\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:234\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:54\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1114"
The text was updated successfully, but these errors were encountered: