[Feature] Integrated OpenDataHub Centralized Custom Certificate

[Jooho]

Description

OpenDataHub Centralized Cert PR has been merged.
KServe can now integrate with the centralized certificate as a result of this change.
This PR facilitates the update of the storage-config secret, which includes the custom certificate.
More details about the issue can be found here.

How Has This Been Tested?

Setup
Manual

1. install ossm/serverless operators
2. create this catalogsource

cat << EOF|oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: rhods-catalog-dev
  namespace: openshift-marketplace
spec:
  displayName: Red Hat OpenShift Data Science
  publisher: RHODS Development Catalog
  image: quay.io/opendatahub/opendatahub-operator-catalog:v2.8.0-incubation
  sourceType: grpc
EOF

3. install rhods v2.8.0-incubation operator
4. create dsci
5. update dsci

kubectl patch DSCInitialization default-dsci --type='json' -p='[{"op": "add", "path": "/spec/trustedCABundle", "value": {"managementState": "Managed", "customCABundle": ""}}]'

6. create dsc (Managed for dashboard/kserve)
7. deploy ssl minio for test

# SSL Minio 
export MINIO_DEMO_HOME=/tmp/demo
mkdir $MINIO_DEMO_HOME
cd $MINIO_DEMO_HOME
git clone git@github.com:Jooho/jhouse_openshift.git
cd jhouse_openshift/Minio/minio-tls-kserve/
source env.sh
./1.setup.sh 
./2.generate-cert.sh 

./3.deploy-minio.sh 
cp ${DEMO_HOME}/minio_certs/root.crt   ${DEMO_HOME}/minio_certs/cabundle.crt
cp ${DEMO_HOME}/minio_certs/cabundle.crt $MINIO_DEMO_HOME/.

8. create a DS namespace

oc new-project kserve-demo 
oc label ns/kserve-demo opendatahub.io/dashboard="true"

9. Deploy Sample Caikit ISVC for KServe (script

Script
(NOTE) This was tested on fedora only.

git clone   git@github.com:Jooho/loopy.git
cd loopy
make init
cat cluster_info.sh
CLUSTER_CONSOLE_URL=https://console-openshift-console.XXXX
CLUSTER_API_URL=https://api.XXX:443
CLUSTER_ADMIN_ID=admin
CLUSTER_ADMIN_PW=password
CLUSTER_TOKEN=sha256~XXX
CLUSTER_TYPE=ROSA

./loopy playbooks run setup-rhoai-for-global-cert -i ./cluster_info.sh 

Set root.crt path

ls /tmp/ms_cli
20240222_XXX

#replace the XXX to yours
export crt_content=/tmp/ms_cli/20240222_XXX/artifacts/8-cert-generate/root.crt

TEST
1. Create a dataconnection

1-1. without global cert

#Create a dataconnection
cat <<EOF |oc apply -f -
kind: Secret
apiVersion: v1
metadata:
  name: aws-connection-minio
  labels:
    opendatahub.io/dashboard: 'true'
    opendatahub.io/managed: 'true'
  annotations:
    opendatahub.io/connection-type: s3
    openshift.io/display-name: minio
stringData:
  AWS_ACCESS_KEY_ID: admin
  AWS_DEFAULT_REGION: us-east-1
  AWS_S3_BUCKET: modelmesh-example-models
  AWS_S3_ENDPOINT: https://minio.minio.svc:9000
  AWS_SECRET_ACCESS_KEY: password
EOF

# It is the usual storage-config secret
oc get secret storage-config -o yaml

1-2. with global cert

# Clear data connection
oc delete secret aws-connection-minio

# Update global cert in dsci
kubectl patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"'"$(awk '{printf "%s\\n", $0}' $crt_content)"'"}]'

# new configmap created for kserve
oc get cm odh-kserve-custom-ca-bundle 

# Create a dataconnection
cat <<EOF |oc apply -f -
kind: Secret
apiVersion: v1
metadata:
  name: aws-connection-minio
  labels:
    opendatahub.io/dashboard: 'true'
    opendatahub.io/managed: 'true'
  annotations:
    opendatahub.io/connection-type: s3
    openshift.io/display-name: minio
stringData:
  AWS_ACCESS_KEY_ID: admin
  AWS_DEFAULT_REGION: us-east-1
  AWS_S3_BUCKET: modelmesh-example-models
  AWS_S3_ENDPOINT: https://minio.minio.svc:9000
  AWS_SECRET_ACCESS_KEY: password
EOF

# storage-config will have certificate information
oc get secret storage-config -o yaml

2. Update global cert

#update global custom cert
kubectl patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"ABC"}]'
# 2-1. updated kserve cm 
oc get cm odh-kserve-custom-ca-bundle -o yaml
# 2-2. updated storage-config
oc get secret storage-config -o yaml

3. KServe test

kubectl patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"'"$(awk '{printf "%s\\n", $0}' $crt_content)"'"}]'
oc delete pod --all -n kserve-demo

4. ModelMesh test

oc label namespace kserve-demo modelmesh-enabled=true --overwrite=true

oc apply -f https://raw.githubusercontent.com/Jooho/jhouse_openshift/main/Kserve/docs/Common/manifests/openvino-serving-runtime.yaml -n kserve-demo
oc patch servingruntime/ovms-1.x -p '{"spec":{"replicas":1}}' --type=merge -n kserve-demo
cat <<EOF| oc apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: example-onnx-mnist
  namespace: kserve-demo
  annotations:
    serving.kserve.io/deploymentMode: ModelMesh
spec:
  predictor:
    model:
      modelFormat:
        name: onnx
      runtime: ovms-1.x
      storage:
        key: aws-connection-minio
        path: onnx/mnist.onnx
EOF

# You can see that it successfully pull a model from ssl minio       
oc logs deploy/modelmesh-serving-ovms-1.x -c ovms-adapter

Clean up demo

oc delete isvc,servingruntime,pod --all -n kserve-demo --force
oc delete ns kserve-demo minio --force --grace-period=0
oc delete all --all --force --grace-period=0 -n opendatahub
oc patch DataScienceCluster rhoai -p '{"metadata":{"finalizers":[]}}' --type=merge
oc delete dsc,dsci --all --force --grace-period=0

oc delete subs serverless-operator -n rhoai-serverless
oc delete subs servicemeshoperator -n openshift-operators 
oc delete subs opendatahub-operator -n openshift-operators

oc delete csv serverless-operator.v1.31.1 servicemeshoperator.v2.4.5 opendatahub-operator.v2.8.0-incubation -n openshift-operators
oc delete catalogsource rhods-catalog-dev -n openshift-marketplace
oc delete pod --all -n rhoai-serverless --force --grace-period=0
oc delete validatingwebhookconfiguration openshift-operators.servicemesh-resources.maistra.io
oc delete mutatingwebhookconfiguration openshift-operators.servicemesh-resources.maistra.io 
oc patch smmr default -p '{"metadata":{"finalizers":[]}}' --type=merge -n istio-system
oc delete ns opendatahub redhat-ods-monitoring rhoai-serverless istio-system 

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jooho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Jooho]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vaibhavjainwiz]

Current reconciliation logic looks fine but could I request you to please write in similiar pattern that we follow in other reconciler like below. It help us to maintain uniformity in code and its more robust.

https://github.com/opendatahub-io/odh-model-controller/blob/ba123e578471dabfd6154cb6a9588342abb2662c/controllers/reconcilers/kserve_istio_peerauthentication_reconciler.go#L15-L14

[vaibhavjainwiz]

@Jooho I am not following whole centralized cert story from begining, Could I ask Who is responsible to create odhGlobalCACertConfigMap? I mean does some operator create it or user will create it manually?

If its created by some other operator like odh-operator then ideally that operator should maintain the kserveCustomCACertConfigMap as well.

[Jooho]

Current reconciliation logic looks fine but could I request you to please write in similiar pattern that we follow in other reconciler like below. It help us to maintain uniformity in code and its more robust.
https://github.com/opendatahub-io/odh-model-controller/blob/ba123e578471dabfd6154cb6a9588342abb2662c/controllers/reconcilers/kserve_istio_peerauthentication_reconciler.go#L15-L14

@vaibhavjainwiz I agree. Therefore, I have accepted your recommendation and made significant changes to the source code accordingly. Please review.

Could I ask Who is responsible to create odhGlobalCACertConfigMap? I mean does some operator create it or user will create it manually? If its created by some other operator like odh-operator then ideally that operator should maintain the kserveCustomCACertConfigMap as well.

The odhGlobalCACertConfigMap is created by the opendatahub operator. Each component owner must use this centralized certificate appropriately for their respective components. This is because each component applies certificates in very different ways. Although kserve and modelmesh both apply certificates, their methods are different from each other. The kserveCustomCACertConfigMap is created specifically for utilizing kserve's functionality, therefore, it needs to be managed by us.

[danielezonca]

Overall the PR looks good to me, just a couple of comments to clarify impact on existing code

[vaibhavjainwiz]

changes looks good to me as well. I had reviewed it by dry run on code.

[vaibhavjainwiz]

/lgtm

[Jooho]

@danielezonca I found the root cause why the old logic was working. I fixed it with #167 and it will be included.

[Jooho]

This pr had to be squeezed and merged. I will do it manually.