Ensure that images are always pulled

[GeorgesLorre]

No description provided.

[PhilippeMoussalli]

This setting does not seem to be currently available in KFP V2. Created an issue for this

#9980kubeflow/pipelines#9980

[RobbeSneyders]

Two notes:

• This is the case for :latest images by default, so this would mainly be an issue for our contributors who use :dev images.
• There is a workaround by enabling the AlwaysPullImages admission controller.

[RobbeSneyders]

Also wondering if we can't just add it to the container part of the KfP v2 spec.

[RobbeSneyders]

Adding it to the spec would probably be the best option, as Vertex depends on the same spec.

[PhilippeMoussalli]

I looked into it and it doesn't seem to be feasible through the component object directly since it does not include an attribute for it nor an integration in the v2 kfp compiler to write it to the k8s workflow.

https://github.com/kubeflow/pipelines/blob/2131dfaf6da3eff1c577a8cd2777c4606b121545/sdk/python/kfp/dsl/structures.py#L289

We might need to modify the workflow ourselves after compilation by inserting it manually which is feasible but less elegant.

Link to compiler step in v1 for the image policy: https://github.com/kubeflow/pipelines/blob/2131dfaf6da3eff1c577a8cd2777c4606b121545/sdk/python/kfp/deprecated/compiler/compiler.py#L869

[RobbeSneyders]

We already construct the IR YAML definition manually for a large part, so it would be straightforward to add (https://github.com/ml6team/fondant/pull/477/files#diff-bd5f20c5d8fb87f062b2a177ad783c129563de771a5a54512cd6deafdf3cbca8R341)

We would still have to test if this gets translated to k8s correctly though.

[PhilippeMoussalli]

We already construct the IR YAML definition manually for a large part, so it would be straightforward to add (https://github.com/ml6team/fondant/pull/477/files#diff-bd5f20c5d8fb87f062b2a177ad783c129563de771a5a54512cd6deafdf3cbca8R341)

We would still have to test if this gets translated to k8s correctly though.

It doesn't seem to translate. I tried inserting it into the IR yaml and checked the pod specs (link). The image set policy is not set to the one I defined.

https://docs.bridgecrew.io/docs/bc_k8s_14

I'm not sure why, might be missing from the IR yaml to argo workflow conversion in v2.

[RobbeSneyders]

Ok thanks for checking!

Let's follow up on the issue on KfP and in the meantime enable the AlwaysPullImages admission controller on our development cluster.

For users this should be less relevant.

[PhilippeMoussalli]

Tried with the admission controller as well but ran into some issues. It seems like it's not possible to modify it's configs on GKE

https://stackoverflow.com/questions/63483346/how-to-enable-defaultstorageclass-admission-controllers-on-a-gke-cluster
https://stackoverflow.com/questions/55283432/enable-podtolerationrestriction-on-gcloud-k8s-cluster?rq=4
https://serverfault.com/questions/980544/list-kubernetes-api-server-flags-on-gke

Seems to be the same on AWS and other managed k8s deployments

https://stackoverflow.com/questions/62152355/how-to-enable-podnodeselector-admission-controller-in-eks-version-1-15

Tried looking for other solutions but could not find anything concrete. Do you see any other way to enable this?