mongodb persistent template does not come up

[gabemontero]

@bparees @rhcarvalho @mfojtik - FYI

I can bring up the mongodb ephemeral template fine, as well as the mysql peristent template and postgresql persistent templates.

However, when bringing up the mongodb persistent template, the deploy fails with the following in the pod log:
=> Waiting for container IP address ... 172.17.0.45:27017
=> Waiting for MongoDB service startup ...
note: noprealloc may hurt performance in many applications
2016-01-27T19:01:23.944+0000 [initandlisten] MongoDB starting : pid=29 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=mongodb-1-dl9uq
2016-01-27T19:01:23.945+0000 [initandlisten] db version v2.6.9
2016-01-27T19:01:23.945+0000 [initandlisten] git version: nogitversion
2016-01-27T19:01:23.945+0000 [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-01-27T19:01:23.945+0000 [initandlisten] build info: Linux c1bg.rdu2.centos.org 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 BOOST_LIB_VERSION=1_53
2016-01-27T19:01:23.945+0000 [initandlisten] allocator: tcmalloc
2016-01-27T19:01:23.945+0000 [initandlisten] options: { config: "/etc/mongod.conf", net: { http: { enabled: false }, port: 27017 }, processManagement: { pidFilePath: "/var/lib/mongodb/mongodb.pid" }, replication: { oplogSizeMB: 64 }, storage: { dbPath: "/var/lib/mongodb/data", preallocDataFiles: false, smallFiles: true }, systemLog: { quiet: true } }
2016-01-27T19:01:23.946+0000 [initandlisten] exception in initAndListen: 10309 Unable to create/open lock file: /var/lib/mongodb/data/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
2016-01-27T19:01:23.946+0000 [initandlisten] dbexit:
2016-01-27T19:01:23.946+0000 [initandlisten] shutdown: going to close listening sockets...
2016-01-27T19:01:23.946+0000 [initandlisten] shutdown: going to flush diaglog...
2016-01-27T19:01:23.946+0000 [initandlisten] shutdown: going to close sockets...
2016-01-27T19:01:23.946+0000 [initandlisten] shutdown: waiting for fs preallocator...
2016-01-27T19:01:23.946+0000 [initandlisten] shutdown: lock for final commit...
2016-01-27T19:01:23.946+0000 [initandlisten] shutdown: final commit...
2016-01-27T19:01:23.946+0000 [initandlisten] shutdown: closing all files...
2016-01-27T19:01:23.947+0000 [initandlisten] closeAllFiles() finished
2016-01-27T19:01:23.947+0000 [initandlisten] shutdown: removing fs lock...
2016-01-27T19:01:23.947+0000 [initandlisten] couldn't remove fs lock errno:9 Bad file descriptor
2016-01-27T19:01:23.947+0000 [initandlisten] dbexit: really exiting now
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Waiting for MongoDB service startup ...
=> Giving up: Failed to start MongoDB service!

I'm defining the PV with the following:
oc create --config=/home/gmontero/go/src/github.com/openshift/origin/_output/local/bin/linux/amd64/openshift.local.config/master/admin.kubeconfig -f - <<EOF
{
"kind": "PersistentVolume",
"apiVersion": "v1",
"metadata": {
"name": "mongodb"
},
"spec": {
"capacity": {
"storage": "512Mi"
},
"hostPath": {
"path": "mktemp -d --tmpdir pg-data.XXXXX | tee >(xargs chmod a+rwx)"
},
"accessModes": [
"ReadWriteOnce"
]
}
}
EOF

And I clear out /tmp/pg-data.* prior to trying the scenario.

[bparees]

Given the error: Unable to create/open lock file: /var/lib/mongodb/data/mongod.lock errno:13 Permission denied

i'd confirm the hostpath directory that's getting created for the mount path has the correct permissions on your host (777) as a starting point.

after that you can oc rsh into the pod and see what the permission of the dir are there. it seems like they are wrong.

[gabemontero]

the permissions on my host are rwxrwxrwt, where t is the sticky bit, which I thought meant that the world could not delete files, but could open/create, hence I opened the issue ....
that said, i'll experiment with those particulars as you've outlined and see what's up and report here

[bparees]

not sure why it's got the sticky bit but in theory that shouldn't matter.

On Wed, Jan 27, 2016 at 2:18 PM, Gabe Montero notifications@github.com
wrote:

the permissions on my host are rwxrwxrwt, where t is the sticky bit, which
I thought meant that the world could not delete files, but could
open/create, hence I opened the issue ....
that said, i'll experiment with those particulars as you've outlined and
see what's up and report here

—
Reply to this email directly or view it on GitHub
#6856 (comment).

Ben Parees | OpenShift

[gabemontero]

Leaving the host vol with rwxrwxrwt
/var/lib/mongodb was 775:
drwxrwxr-x. 3 mongodb root 4096 Jan 27 19:21 mongodb
so the user id for the image could not create the mongod.lock file

Forcing the host vol to be 777 had no effect

Any other ideas on how the permissions for /var/lib/mongodb could be adjusted, aside from creating a new image based off of the image the template currently uses?

Should the accessModes in the PV def be changed or added to .... I'll experiment with that in the interim

[bparees]

775 should be fine, your uid is a member of the root group.

i'm wondering if there's an selinux problem or something else going on here.

when you rsh'd in, were you able to create files in that dir?

[gabemontero]

I was able to touch a file in that dir.

I in fact had to disable sellinux (sudo setenforce 0) before I could even oc rsh into the pod's container. I've since left it disabled for subsequent runs.

[bparees]

hm. so you can, as the container user, create files in that directory. but the mongo process itself cannot. i'm running out of ideas.

i assume that file doesn't actually exist when you rsh in?

[gabemontero]

... and additionally, it seems to be staying up now :-) ... I'll do a few more runs to confirm, then close this out.