Reconcile PyTorch Job error Operation cannot be fulfilled on pytrochjobs.kubeflow.org

[jazka]

After creating Pytorch Job, the stauts of the job pods will always be pending, and the training-operator controller throws error as below:

Reconcile PyTorch Job error Operation cannot be fulfilled on pytrochjobs.kubeflow.org "xxx-pytorchjob": the object has been modifyed; please apply your changes to the latest version and try again

The gang schedule is enabled with volcano.

[gaocegege]

It should not affect the logic since we will retry. Could you please show us the full log? Which version are you using?

[jazka]

It should not affect the logic since we will retry. Could you please show us the full log? Which version are you using?

Yes, it tries to "Reconciling", but the job pods hang on "pending" status, and there is no any logs about this job on volcano-scheduer pod.

[gaocegege]

May I ask which version you are using?

[jazka]

The container image of training operator is "public.ecr.aws/j1r0q0g6/training/training-operator:760ac1171dd30039a7363ffa03c77454bd714da5". I am not sure if it is okay to check the version, or any other way to get the version?

[gaocegege]

/cc @Jeffwan @PatrickXYS Do you know about it?

Are you using it in AWS?

[jazka]

If it is not the official version, I can change it with official version and try it again.

[jazka]

It should not affect the logic since we will retry. Could you please show us the full log? Which version are you using?

Attach the error logs:

[gaocegege]

Is there any pod created?

[jazka]

Yes, the pods are created but hang on pending status.

[gaocegege]

Can you please show kubectl describe pods? I need to know why it hangs.

[jazka]

I could find any clue from pod describe. It is different to attatch large image directly for me, please open it with link url.

master-0 pod:
https://l4x826wg3c.feishu.cn/file/boxcnvhEHBXXtnjE1s4qcMWSOme
https://l4x826wg3c.feishu.cn/file/boxcnMa1sYsODn7Weu3QRrT1Bqd

worker-0 pod:
https://l4x826wg3c.feishu.cn/file/boxcnDBuVOyg7YPUTA2ERxxXHNe
https://l4x826wg3c.feishu.cn/file/boxcn5pLXEE9kbuywU1P7yReRt4

[gaocegege]

I did not understand why there is no event of the pods. It's weird.

But I do not think it is related to the operator. Seems that the pods are already created.

[jazka]

Yes, it is weird. Some things I noticed may be helpful：

• The pending pytroch job will fail after long time(last one job, it is abount 60min). One info log is "Ignoring inactive pod xxxx-pytorchjob-worker-0 in state Failed, deleteion time "
• It is still pending if restarting training-operator pod
• It will begin to running if restart volcano-scheduler pod. But there is no any log abount the pod scheduling error of this pytorch job

[gaocegege]

Personally, I think it is may be related to volcano. Since it can work after restarting the volcano.

[Jeffwan]

The container image of training operator is "public.ecr.aws/j1r0q0g6/training/training-operator:760ac1171dd30039a7363ffa03c77454bd714da5". I am not sure if it is okay to check the version, or any other way to get the version?

760ac1171dd30039a7363ffa03c77454bd714da5 is the commit id, you can search it in git logs.
We probably can change to version tag later for easy debugging.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.