Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Katib cannot obtain trail task information after istio is enabled. After completion, the pod status is NotReady #1716

Closed
longpi1 opened this issue Oct 15, 2021 · 5 comments
Labels

Comments

@longpi1
Copy link

longpi1 commented Oct 15, 2021

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
Since my training container needs to communicate with ML-pipeline, I need to enable istioto add Kubeflow-userid and the corresponding AuthorizationPolicy. However, automL cannot obtain the status information of Taial after istio is opened.Is there any way to solve this problem? Is there any other way to solve the communication problem between the training container and ml-pipeline?

@ewilderj @tmielika @mhausenblas @marcoceppi @pineking
At present,I don't open istio, I have added the corresponding request header in the source code, the following is the error message:

I1015 01:26:39.579167      54 main.go:136] kfp_server_api.exceptions.ApiException: (403)
I1015 01:26:39.579191      54 main.go:136] Reason: Forbidden
I1015 01:26:39.579198      54 main.go:136] HTTP response headers: HTTPHeaderDict({'content-length': '19', 'content-type': 'text/plain', 'date': 'Fri, 15 Oct 2021 01:26:39 GMT', 'server': 'istio-envoy', 'x-envoy-decorator-operation': 'ml-pipeline.kubeflow.svc.cluster.local:8888/*'})
I1015 01:26:39.579222      54 main.go:136] HTTP response body: RBAC: access denied

Thank you for your guidance

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

@anencore94
Copy link
Member

Here's the main reason of this issue and temporal solution if you need to use istio enable in trial:
#1638

You could build your own metrics-collector image with such code and use it.

@longpi1
Copy link
Author

longpi1 commented Oct 18, 2021

如果您需要在试用中使用 istio enable,这是此问题的主要原因和临时解决方案: #1638

您可以使用此类代码构建自己的指标收集器图像并使用它。

Thank you very much, but I haven't solved the problem after trying. Where should you add this part of code? Is there a modified image sample?
image

@anencore94
Copy link
Member

@longpi1
I'm sorry I couldn't share the modified sample docker image since I added it to my company's private repository.

Just for your question, I added such part to here

Also, sending quitquitquit API to istio-proxy by defer like this:
image

@longpi1
Copy link
Author

longpi1 commented Oct 20, 2021

@longpi1 很抱歉我无法分享修改后的示例 docker 镜像,因为我将它添加到我公司的私有存储库中。

只是为了你的问题,我在这里添加了这样的部分

此外,通过如下方式将 quitquitquit API 发送到 istio-proxy defer图片

I'm glad you replied to my question. I have solved the above problem in my own way, but thank you very much

@longpi1 longpi1 closed this as completed Oct 20, 2021
@anencore94
Copy link
Member

anencore94 commented Oct 21, 2021

| I'm glad you replied to my question. I have solved the above problem in my own way, but thank you very much

Thanks for the reply ! If you are ok, how about share your solution? Your solution may help other people who suffer with this istio-stuff. @longpi1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants