Add Glassfish Statefulsets to OpenShift/Kubernetes

[MichaelClifford]

As students in Boston University's EC528 Cloud Computing Course, my team has been working with @pdurbin @danmcp & @DirectXMan12 to further work on #4040 & #4168.

I have been working on scaling the Glassfish pods and am ready to open a pull request. My solution entails creating a statefulset and providing updates to the Glassfish installation script to differentiate between the initial master pod, dataverse-glassfish-0, and any subsequently deployed pods.

[pdurbin]

@MichaelClifford thanks for opening this issue and working on this project! You're editingopenshift.json, right? I'm asking because pull request #4599 by @patrickdillon has edits to that file and I'm close to merging that pull request. I don't mean to complicate your life but you might want to branch from his branch before you make your pull request. That way, we can test both changes together.

[MichaelClifford]

@pdurbin <https://github.com/pdurbin>, yes, that's correct. I have made some updates to the openshift.json configuration file. I was just about to submit my pull request in the next hour or so here (doing some final checks), but if you want me to hold off until #4599 is merged, that's fine. Just let me know.

…

On Wed, Apr 25, 2018 at 6:38 PM, Philip Durbin ***@***.***> wrote: @MichaelClifford <https://github.com/MichaelClifford> thanks for opening this issue and working on this project! You're editingopenshift.json, right? I'm asking because pull request #4599 <#4599> by @patrickdillon <https://github.com/patrickdillon> has edits to that file and I'm close to merging that pull request. I don't mean to complicate your life but you might want to branch from his branch before you make your pull request. That way, we can test both changes together. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4617 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AESWOq0Pq0Ap2-HRzB5Ycn6UISRdD1yFks5tsPr5gaJpZM4TkETi> .

[pdurbin]

@MichaelClifford right, it's the final checks that I don't want you to waste your time on if I'm going to ask you to retest with the combined solution. Does that make sense? I'm also leaving comments for you in #4598 just to make this extra hard. 😉

[pdurbin]

@MichaelClifford ok I just merged #4599 so please makes sure your (future) pull request has those changes in it before you spend a lot of time testing. @patrickdillon and I think his changes and your changes to openshift.json should be compatible. Please let me know if you have questions. Thanks!

[pdurbin]

@MichaelClifford thanks for making pull request #4626! I requested a few minor changes and I'm going to ask @landreev to take a look at changes you made to glassfish-setup.sh.

If you have any questions, please let me know!

[pdurbin]

@MichaelClifford how much memory do you recommend for minishift? I'm been using 4GB like this:

minishift start --vm-driver=virtualbox --memory=4GB

But as of your latest fixes (ae115fd) I'm getting "insufficient memory" errors:

[MichaelClifford]

@pdurbin I would recommend changing line 156 of the openshift.json file "resources": {"limits": {"memory": "3072Mi"} to "resources": {"limits": {"memory": "512Mi"} when deploying locally.

This seems to work on minishift, but when deployed to the MOC we needed additional resources, about 3Gb, to successful run the glassfish pods. Let me know if that helps.

[pdurbin]

@MichaelClifford Whoops! I wrote 8GB and 4GB above but I'm really using 4GB, I just edited my comment above to reflect reality.

Does that change anything? It seems a bit counterintuitive to give it less memory (512Mi instead of 3072Mi), but I'll give it a shot!

[pdurbin]

Ok, I tried a few things. I switched to 8GB like this:

minishift start --vm-driver=virtualbox --memory=8GB

I'm not messing with memory in openshift.json. The only changes I made locally are to switch to my Docker Hub account ("pdurbin") rather than "iqss". I won't be pushing this change, of course:

murphy:dataverse pdurbin$ git diff conf/openshift/openshift.json
diff --git a/conf/openshift/openshift.json b/conf/openshift/openshift.json
index b49f7fe..684b1d1 100644
--- a/conf/openshift/openshift.json
+++ b/conf/openshift/openshift.json
@@ -100,7 +100,7 @@
         "name": "dataverse-plus-glassfish"
       },
       "spec": {
-        "dockerImageRepository": "iqss/dataverse-glassfish"
+        "dockerImageRepository": "pdurbin/dataverse-glassfish"
       }
     },
     {
@@ -120,7 +120,7 @@
         "name": "iqss-dataverse-solr"
       },
       "spec": {
-        "dockerImageRepository": "iqss/dataverse-solr"
+        "dockerImageRepository": "pdurbin/dataverse-solr"
       }
     },
     {
@@ -146,7 +146,7 @@
             "containers": [
               {
                 "name": "dataverse-plus-glassfish",
-                "image": "iqss/dataverse-glassfish:latest",
+                "image": "pdurbin/dataverse-glassfish:latest",
                 "ports": [
                   {
                     "containerPort": 8080,
murphy:dataverse pdurbin$ 

The good news is that oc new-app conf/openshift/openshift.json "just worked" in the sense that I got a working Dataverse installation, was able to log in, create a dataverse, see that it was indexed into Solr, etc. Then I thought I'd try a similar trick as at #4598 (comment) to scale the number of replicas to 3:

oc scale statefulset/dataverse-glassfish --replicas=3

For some reason, I'm getting "The pod has been stuck in the pending state for more than five minutes." Here are screenshots:

Before (1 replica)

After (immediately)

After (five minutes later)

I'm not really sure what's going on. Dataverse is still up, running off dataverse-glassfish-0, but I'm not sure how to scale up to additional replicas.

Any thoughts for me @MichaelClifford @danmcp @DirectXMan12 @patrickdillon @rockash? I tried looking at the log for dataverse-glassfish-1 but this pod seems not well:

I'm on commit ae115fd

[danmcp]

Anything going on in events?

[MichaelClifford]

@pdurbin, Not sure if you tired this yet or not, but did you try using my glassfish image jcliffbu/glassboat:test2 or did you rebuild the entire application and push it to your docker hub?

[pdurbin]

@danmcp shoot, I forgot to check and I'm no longer at my desk. I'll take a look next week.

@MichaelClifford I rebuilt the entire application and pushed it to my docker hub. There's a new option I added in e7a56c7 so I can run ./build.sh huborg pdurbin or whatever to push to a non-iqss Docker Hub account.

[pdurbin]

@danmcp @MichaelClifford ah, for dataverse-glassfish-1 the "Events" page says "Failed Scheduling... 0/1 nodes are available: 1 Insufficient memory. 450 times in the last 2 days".

Here are some screenshots:

Again, I'm using 8 GB like this:

minishift start --vm-driver=virtualbox --memory=8GB

@MichaelClifford you were mentioning before that I should lower the amount of memory from 3072Mi to 512Mi. I'll include the "diff" for openshift.json below, which also includes my changes to getting the image I built and pushed to my "pdurbin" account on Docker Hub. Is lowering the amount of memory what you want me to try try next? It seems counterintuitive to me.

Please advise. Thanks. I'm still on ae115fd and and standup today I'll let @landreev know about your changes to the Dataverse installer.

murphy:dataverse pdurbin$ git diff conf/openshift/openshift.json 
diff --git a/conf/openshift/openshift.json b/conf/openshift/openshift.json
index b49f7fe..0eabbe7 100644
--- a/conf/openshift/openshift.json
+++ b/conf/openshift/openshift.json
@@ -100,7 +100,7 @@
         "name": "dataverse-plus-glassfish"
       },
       "spec": {
-        "dockerImageRepository": "iqss/dataverse-glassfish"
+        "dockerImageRepository": "pdurbin/dataverse-glassfish"
       }
     },
     {
@@ -120,7 +120,7 @@
         "name": "iqss-dataverse-solr"
       },
       "spec": {
-        "dockerImageRepository": "iqss/dataverse-solr"
+        "dockerImageRepository": "pdurbin/dataverse-solr"
       }
     },
     {
@@ -146,14 +146,14 @@
             "containers": [
               {
                 "name": "dataverse-plus-glassfish",
-                "image": "iqss/dataverse-glassfish:latest",
+                "image": "pdurbin/dataverse-glassfish:latest",
                 "ports": [
                   {
                     "containerPort": 8080,
                     "protocol": "TCP"
                   }
                 ],
-                "resources": {"limits": {"memory": "3072Mi"
+                "resources": {"limits": {"memory": "512Mi"
                   }
                   
                   },
murphy:dataverse pdurbin$ 

[pdurbin]

I'm still on ae115fd and and standup today I'll let @landreev know about your changes to the Dataverse installer.

I just ran vagrant up on the commit above, which is a way to test the installer outside the context of OpenShift (it run in a CentOS 7 VM running in VirtualBox). The installer worked fine and I got a running version of Dataverse on http://localhost:8888 as expected and as documented at http://guides.dataverse.org/en/4.8.6/developers/tools.html#vagrant

[patrickdillon]

The change you made to openshift.json to lower the memory looks good. I would try that. The 3GB limit is intended for running on the MOC. I could deploy the 512MB version on my VM which has 4GB.

[pdurbin]

@patrickdillon ok, 512 MB for Glassfish seems like a very small amount of memory I'll try what you're suggesting, including 4GB or the VM. Thanks. Please stay tuned.

[MichaelClifford]

@pdurbin, I ran a similar version on my local minishift this morning with 1024MB as the glassfish resource limit and that seemed to work.

[MichaelClifford]

@pdurbin if you try to deploy 3 3GB pods on a VM that only has 8Gb of memory I think that will necessarily run into memory issues, right? As @patrickdillon mentioned, the 3GB limit was needed when deploying on the MOC. Let me know if you continue to have memory issues after updating the memory resources.

[patrickdillon]

@MichaelClifford even when replicas is set to 1, I still get issues with the 3GB limit.

[MichaelClifford]

@patrickdillon, you need to account for the total memory on your VM, right? If GlassFish is 3GB, PostgreSql is 1 GB and Solr is 1 GB, that will exceed the memory allocated to your entire Minishift VM. And since, Glassfish can only successfully deploy after Solr and PostgreSql. I think it makes sense that it would not deploy correctly without updating the memory allocation.

[pdurbin]

I just tried a 4GB VM and 512 MB Glassfish and the default config of 1 replica I just got the dreaded "Remote server does not listen for requests on [localhost:4848]. Is the server up?" error, which probably means that Glassfish wasn't able to start because it didn't have enough memory. Here's the log:

What's you're saying makes sense that my VM needs to have enough memory to all the replicas. Maybe I'll switch to a 8 GB VM for further testing. And perhaps 1024 MB for Glassfish. I'm pretty much ok with merging the pull request as-is, especially since vagrant up worked, but it might be nice to explain in the dev guide what to expect with the current config of 1 replica (worked fine in my initial test: #4617 (comment) ) and how setting will need to be adjusted if you want to play with replicas=3 or whatever. I'll keep playing with it. Thanks for the chatter.

[pdurbin]

@MichaelClifford @patrickdillon et al. success! I'm going to push this change to openshift.json unless there are any objections:

murphy:dataverse pdurbin$ git diff conf/openshift/openshift.json
diff --git a/conf/openshift/openshift.json b/conf/openshift/openshift.json
index b49f7fe..4f43f5c 100644
--- a/conf/openshift/openshift.json
+++ b/conf/openshift/openshift.json
@@ -153,7 +153,7 @@
                     "protocol": "TCP"
                   }
                 ],
-                "resources": {"limits": {"memory": "3072Mi"
+                "resources": {"limits": {"memory": "1024Mi"
                   }
                   
                   },
murphy:dataverse pdurbin$ 

I'm also changing the part in the doc where I recommend a 4 GB Minishift VM. I'm going to make it 8 GB because I'm having much more success with three Glassfish servers.

I'm going to document some of the stuff I'm doing to test in the dev guide as well as some info on StatefulSets in general. This will help @landreev and others code review this pull request as well, I think.

[pdurbin]

@MichaelClifford @patrickdillon I just pushed 1324e39 to the pull request and made the change above to Glassfish memory and documented some stuff about replicas. Please take a look and let me know if you spot a typo or anything weird. It's kind of a brain dump. Thanks!

[pdurbin]

I have some screenshots I took while writing up my thoughts in 1324e39 and will put them below.

First, here's the out of the box config with a single replica for Glassfish:

Then you run the following command to add two more Glassfish servers for a total of three:

oc scale statefulset/dataverse-glassfish --replicas=3

Here's how a dataverse logo looks when it can be found on the filesystem:

Here's a broken image of a dataverese logo that can't be found because you are on a different Glassfish server than the one you uploaded it on:

As discussed in 1324e39 and IQSS/dataverse-aws#10 and http://irclog.iq.harvard.edu/dataverse/2016-10-21 and https://help.hmdc.harvard.edu/Ticket/Display.html?id=240995 I believe the current work around is to put the "logos" directory on a shared file system. On a single host the directory the default directory is /usr/local/glassfish4/glassfish/domains/domain1/docroot/logos.

[pdurbin]

@landreev and I looked over pull request #4626 and I just merged it. Thanks, all!

For more details, I highly recommend watching the 23 minute video at https://github.com/BU-NU-CLOUD-SP18/Dataverse-Scaling#our-project-video

Closing. For more on OpenShift, keep and eye on #4040.