Can't deploy apps that have persist volume claims

[ryannix123]

I finally got my cluster to build correctly with the service catalog installed. I believe what was hanging my install up was that port 80 and 443 were not available to the outside world. Once that was cleared up, my cluster installed with the service broker installed.

My new issue is that I can't deploy any apps from the service catalog that have persistence i.e. cake-php-mysql non-ephemeral. Ephermal apps deploy just fine.

Recreate:

1. sh createPV.sh, which is located in /tmp of installcentos
2. Run oc get pv to confirm the existence of pv claims. Confirmed to work
3. Deploy app that needs persistence. e.g. Jenkins, php-mysql, etc
4. Run oc get pv again to confirm app has made a claim. Confirmed to work

Sample output:

=> sourcing 20-validate-variables.sh ... => sourcing 25-validate-replication-variables.sh ... => sourcing 30-base-config.sh ... ---> 18:23:21 Processing basic MySQL configuration files ... => sourcing 60-replication-config.sh ... => sourcing 70-s2i-config.sh ... ---> 18:23:21 Processing additional arbitrary MySQL configuration provided by s2i ... => sourcing 40-paas.cnf ... => sourcing 50-my-tuning.cnf ... ---> 18:23:21 Initializing database ... ---> 18:23:21 Running mysqld --initialize-insecure ... mysqld: Can't create directory '/var/lib/mysql/data/' (Errcode: 17 - File exists) 2018-04-14T18:23:21.872387Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2018-04-14T18:23:21.890075Z 0 [Warning] Duplicate ignore-db-dir directory name 'lost+found' found in the config file(s). Ignoring the duplicate. 2018-04-14T18:23:21.890196Z 0 [ERROR] Aborting

Apps fail to deploy each time, although there is a claim. What could be happening?

[DominikPieper]

Same problem here. I can't deploy any pods with storage.

[hhorak]

There are obviously some files in the MySQL directory /var/lib/mysql/data/. We saw errors like this and it was always some wrong configuration of PV, so I'd suggest to focus on that part.

[ryannix123]

@hhorak @marekjelen I thought we solved this issue at Summit, however, there is an inheritance problem on the directory. I have to chmod 777 the pod to get the persistent volume claim to work. I even tried chmod g+s data/ on the directory, but the 777 perms won't inherit.

[marekjelen]

Nope, they will not inherit, for each dir that is being used as a PV you need to set the selinux context and 777 explicitly.

[hhorak]

@ryannix123 is it possible to export the resulting OpenShift objects as a JSON file, so we can just take to reproduce the issue?

[hhorak]

@kubco2 FYI

[marekjelen]

Updated the script to automatically create 200 PVs with Retain policy. This should be enough for playground clusters. There were two problems

a) the PV script was faulty
b) the Reclaim policy seems to remove the permissions set by the script

And so a) was an easy fixe, as for b) the Retain policy should help.