Skip to content

DB can not be created on a normal namespace in OpenShift (security!) #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
rbaumgar opened this issue May 17, 2024 · 20 comments
Closed

DB can not be created on a normal namespace in OpenShift (security!) #100

rbaumgar opened this issue May 17, 2024 · 20 comments
Assignees
@IshaanDesai45
Labels
documentation Improvements or additions to documentation sidb Single Instance Database

Comments

@rbaumgar
Copy link

When I try to deploy the xe-sample to a namespace like "oracle". Operator is not able to create deployment/pod.

runAsUser=54321 is not allowed by default.

2024-05-17T13:12:15Z	INFO	singleinstancedatabase-resource	default	{"name": "xedb"}
2024-05-17T13:12:15Z	INFO	singleinstancedatabase-resource	validate create	{"name": "xedb"}
2024-05-17T13:12:15Z	INFO	controllers.database.SingleInstanceDatabase	Reconcile requested
2024-05-17T13:12:15Z	INFO	singleinstancedatabase-resource	default	{"name": "xedb"}
2024-05-17T13:12:16Z	INFO	singleinstancedatabase-resource	validate update	{"name": "xedb"}
2024-05-17T13:12:16Z	INFO	singleinstancedatabase-resource	validate create	{"name": "xedb"}
2024-05-17T13:12:16Z	INFO	controllers.database.SingleInstanceDatabase	Entering reconcile validation
2024-05-17T13:12:16Z	INFO	controllers.database.SingleInstanceDatabase	Completed reconcile validation
2024-05-17T13:12:16Z	INFO	controllers.database.SingleInstanceDatabase	Creating a new PVC	{"createPVC Datafiles-Vol": {"name":"xedb","namespace":"oracle"}, "PVC.Namespace": "oracle", "PVC.Name": "xedb"}
2024-05-17T13:12:16Z	INFO	No xedb Pod is Ready 	{"controller": "singleinstancedatabase", "controllerGroup": "database.oracle.com", "controllerKind": "SingleInstanceDatabase", "SingleInstanceDatabase": {"name":"xedb","namespace":"oracle"}, "namespace": "oracle", "name": "xedb", "reconcileID": "3c0e7686-7d38-4654-8ef4-4bbc76e8fbd7", "FindPods": {"name":"xedb","namespace":"oracle"}}
2024-05-17T13:12:16Z	INFO	xedb Pods Available ( Other Than Ready Pod )	{"controller": "singleinstancedatabase", "controllerGroup": "database.oracle.com", "controllerKind": "SingleInstanceDatabase", "SingleInstanceDatabase": {"name":"xedb","namespace":"oracle"}, "namespace": "oracle", "name": "xedb", "reconcileID": "3c0e7686-7d38-4654-8ef4-4bbc76e8fbd7", "FindPods": {"name":"xedb","namespace":"oracle"}, " Names :": []}
2024-05-17T13:12:16Z	INFO	Total No Of xedb PODS	{"controller": "singleinstancedatabase", "controllerGroup": "database.oracle.com", "controllerKind": "SingleInstanceDatabase", "SingleInstanceDatabase": {"name":"xedb","namespace":"oracle"}, "namespace": "oracle", "name": "xedb", "reconcileID": "3c0e7686-7d38-4654-8ef4-4bbc76e8fbd7", "FindPods": {"name":"xedb","namespace":"oracle"}, "Count": 0}
2024-05-17T13:12:16Z	INFO	controllers.database.SingleInstanceDatabase	Replica Info	{"createPods": {"name":"xedb","namespace":"oracle"}, "Found": 0, "Required": 1}
2024-05-17T13:12:16Z	INFO	controllers.database.SingleInstanceDatabase	Creating a new xedb POD	{"createPods": {"name":"xedb","namespace":"oracle"}, "POD.Namespace": "oracle", "POD.Name": "xedb-5qt1e"}
2024-05-17T13:12:16Z	ERROR	controllers.database.SingleInstanceDatabase	Failed to create new xedb POD	{"createPods": {"name":"xedb","namespace":"oracle"}, "pod.Namespace": "oracle", "POD.Name": "xedb-5qt1e", "error": "pods \"xedb-5qt1e\" is forbidden: unable to validate against any security context constraint: [provider \"anyuid\": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{54321}: 54321 is not an allowed group, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 54321: must be in the ranges: [1000700000, 1000709999], provider restricted-v2: .containers[0].runAsUser: Invalid value: 54321: must be in the ranges: [1000700000, 1000709999], provider restricted-v2: .containers[0].capabilities.add: Invalid value: \"SYS_NICE\": capability may not be added, provider \"restricted\": Forbidden: not usable by user or serviceaccount, provider \"nonroot-v2\": Forbidden: not usable by user or serviceaccount, provider \"nonroot\": Forbidden: not usable by user or serviceaccount, provider \"hostmount-anyuid\": Forbidden: not usable by user or serviceaccount, provider \"machine-api-termination-handler\": Forbidden: not usable by user or serviceaccount, provider \"hostnetwork-v2\": Forbidden: not usable by user or serviceaccount, provider \"hostnetwork\": Forbidden: not usable by user or serviceaccount, provider \"hostaccess\": Forbidden: not usable by user or serviceaccount, provider \"hostpath-provisioner\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]"}
github.com/oracle/oracle-database-operator/controllers/database.(*SingleInstanceDatabaseReconciler).createPods
	/workspace/controllers/database/singleinstancedatabase_controller.go:2151
github.com/oracle/oracle-database-operator/controllers/database.(*SingleInstanceDatabaseReconciler).createOrReplacePods
	/workspace/controllers/database/singleinstancedatabase_controller.go:1915
github.com/oracle/oracle-database-operator/controllers/database.(*SingleInstanceDatabaseReconciler).Reconcile
	/workspace/controllers/database/singleinstancedatabase_controller.go:189
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:227
@rbaumgar
Copy link
Author

How can I remove runAsUser property?

@yunus-qureshi
Copy link
Member

@rbaumgar for openshift envs, you must apply this yaml

https://github.com/oracle/oracle-database-operator/blob/main/config/samples/sidb/openshift_rbac.yaml

and specify the service account name "sidb-sa" in the SIDB yaml

@rbaumgar
Copy link
Author

This might be a workaround, but is never a solution. Every normal pod has to run with an arbitrary uid. Sorry, a database is a normal pod and does not require special security requirements.
You will find much more information on this and several other links
https://developers.redhat.com/articles/2021/11/11/best-practices-building-images-pass-red-hat-container-certification#

@yunus-qureshi
Copy link
Member

Agreed. The latest v1.1.0 has an attribute called setWritePermissions. Set it to false

https://github.com/oracle/oracle-database-operator/blob/main/config/samples/sidb/singleinstancedatabase.yaml

@rbaumgar
Copy link
Author

does not work on singleinstancedatabase_express.yaml

Invalid value: []int64{54321}: 54321 is not an allowed group, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 54321: must be in the ranges: ...

spec:
  adminPassword:
    keepSecret: true
    secretKey: oracle_pwd
    secretName: xedb-admin-secret
  createAs: primary
  edition: express
  image:
    prebuiltDB: true
    pullFrom: 'container-registry.oracle.com/database/express:latest'
  pdbName: XEPDB1
  persistence:
    accessMode: ReadWriteOnce
    setWritePermissions: false
    size: 50Gi
    storageClass: oci-bv
  replicas: 1
  sid: XE

@yunus-qureshi
Copy link
Member

@rbaumgar also set the attribute prebuiltDB to false

@andbos
Copy link

andbos commented May 27, 2024

Hi,

When I try to apply openshift_rbac.yaml I get the following error:

$ oc apply -f https://raw.githubusercontent.com/oracle/oracle-database-operator/main/config/samples/sidb/openshift_rbac.yaml
serviceaccount/sidb-sa created
role.rbac.authorization.k8s.io/use-sidb-scc created
rolebinding.rbac.authorization.k8s.io/use-sidb-scc created
error: resource mapping not found for name: "sidb-scc" namespace: "default" from "https://raw.githubusercontent.com/oracle/oracle-database-operator/main/config/samples/sidb/openshift_rbac.yaml": no matches for kind "SecurityContextConstraints" in version "v1"
ensure CRDs are installed first

Installation of the operator went fine:

$ oc apply -f https://raw.githubusercontent.com/oracle/oracle-database-operator/main/oracle-database-operator.yaml
namespace/oracle-database-operator-system created
customresourcedefinition.apiextensions.k8s.io/autonomouscontainerdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabasebackups.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabaserestores.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/cdbs.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/databaseobservers.observability.oracle.com created
customresourcedefinition.apiextensions.k8s.io/dataguardbrokers.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/dbcssystems.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/oraclerestdataservices.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/pdbs.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/shardingdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/singleinstancedatabases.database.oracle.com created
role.rbac.authorization.k8s.io/oracle-database-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-leader-election-rolebinding created
rolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/oracle-database-operator-proxy-rolebinding created
service/oracle-database-operator-controller-manager-metrics-service created
service/oracle-database-operator-webhook-service created
certificate.cert-manager.io/oracle-database-operator-serving-cert created
issuer.cert-manager.io/oracle-database-operator-selfsigned-issuer created
mutatingwebhookconfiguration.admissionregistration.k8s.io/oracle-database-operator-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/oracle-database-operator-validating-webhook-configuration created
deployment.apps/oracle-database-operator-controller-manager created

$ oc -n oracle-database-operator-system get pods
NAME                                                           READY   STATUS    RESTARTS   AGE
oracle-database-operator-controller-manager-7f84b7dc4b-994lm   1/1     Running   0          18s
oracle-database-operator-controller-manager-7f84b7dc4b-t5j7r   1/1     Running   0          18s
oracle-database-operator-controller-manager-7f84b7dc4b-twf7d   1/1     Running   0          18s

@rbaumgar
Copy link
Author

rbaumgar commented May 27, 2024 •
edited
Loading

@andbos this works only on OpenShift. Openshift has an SCC object:

$ oc get crd securitycontextconstraints.security.openshift.io -o yaml|grep storedVersion -A2
  storedVersions:
  - v1

@andbos
Copy link

andbos commented May 27, 2024

Yes, started testing in OpenShift.

$ oc version
Client Version: 4.14.11
Kustomize Version: v5.0.1
Server Version: 4.14.12
Kubernetes Version: v1.27.10+28ed2d7

The instance was installed properly anyway...

$ oc -n default get singleinstancedatabase
NAME            EDITION      STATUS    ROLE      VERSION      CONNECT STR                                                                            TCPS CONNECT STR   OEM EXPRESS URL
sinchdb11rhos   Enterprise   Healthy   PRIMARY   21.3.0.0.0   605682735.eu-west-1.elb.amazonaws.com:1521/RHOSDB11   Unavailable        https://605682735.eu-west-1.elb.amazonaws.com:5500/em

No errors in the operator logs.

@rbaumgar
Copy link
Author

rbaumgar commented May 27, 2024 •
edited
Loading

Oh, I see. The SCC is completely wrong formated and the api version is wrong. :-(

@IshaanDesai45 IshaanDesai45 added documentation Improvements or additions to documentation sidb Single Instance Database labels Jun 24, 2024
@IshaanDesai45 IshaanDesai45 mentioned this issue Jul 4, 2024
Merged
@IshaanDesai45
Copy link
Contributor

@rbaumgar updated the steps to deploy a sidb in a normal namespace/project in openshift . kindly check the PR above

@rbaumgar
Copy link
Author

rbaumgar commented Jul 4, 2024

@IshaanDesai45 I created a new NS oracle.
updated the deployment of the operator.
created a new sid.
nothing happens

operator.log```

2024-07-04T08:40:28Z INFO singleinstancedatabase-resource default {"name": "freedb"}
2024-07-04T08:40:28Z INFO singleinstancedatabase-resource validate create {"name": "freedb"}
W0704 08:40:52.733694 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1alpha1.PDB: pdbs.database.oracle.com is forbidden: User "system:serviceaccount:oracle-database-operator-system:default" cannot list resource "pdbs" in API group "database.oracle.com" in the namespace "oracle"
E0704 08:40:52.733826 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.PDB: failed to list *v1alpha1.PDB: pdbs.database.oracle.com is forbidden: User "system:serviceaccount:oracle-database-operator-system:default" cannot list resource "pdbs" in API group "database.oracle.com" in the namespace "oracle"

@IshaanDesai45
Copy link
Contributor

The pdb controller is causing this issue. Can you tell me how you are deploying the operator in the namespaced-scope or the cluster-scope

@rbaumgar
Copy link
Author

rbaumgar commented Jul 4, 2024

I am using namespace based installation and added your newly created openshift-rbac.

@rbaumgar
Copy link
Author

rbaumgar commented Jul 4, 2024

BTW it is a bad design when the operator runs with SA default and has such a rolebinding. should be a nondefault SA.

@IshaanDesai45
Copy link
Contributor

I am using namespace based installation and added your newly created openshift-rbac.

for using namespace based installation did you also apply the file /rbac/default-ns-rolebinding.yaml with the corresponding namespace ?

@IshaanDesai45
Copy link
Contributor

BTW it is a bad design when the operator runs with SA default and has such a rolebinding. should be a nondefault SA.

You mean the operator pods that is currently using serviceaccount:oracle-database-operator-sytem:default should use serviceaccount:oracle-database-operator-system:

@rbaumgar
Copy link
Author

rbaumgar commented Jul 5, 2024

yes

@rbaumgar
Copy link
Author

rbaumgar commented Jul 5, 2024

the problem is fixed, typo when applying rbac/default-ns-rolebinding.yaml, therefor I recommended a different approach.
Having an environment variable for the namespace would allow to apply the same file for multiple namespaces.

@IshaanDesai45
Copy link
Contributor

@rbaumgar we plan to add support of helm charts for the very purpose that user wouldn't need to go and manually change the config/deployment files. So when that is published this your problem of changing the yaml files would be solved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@IshaanDesai45 IshaanDesai45

Labels
documentation Improvements or additions to documentation sidb Single Instance Database
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yunus-qureshi @rbaumgar @IshaanDesai45 @andbos