Merge pull request #87 from IQSS/64-use-develop-branch

64 - Run branches in K8s deployment

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "dataverse"]
+	path = dataverse
+	url = https://github.com/IQSS/dataverse
+	branch = develop

README.md

@@ -26,14 +26,17 @@ and https://github.com/IQSS/dataverse/issues/5448
 
 ## Usage
 
-For quick and easy demo purposes, you can use one of the examples:
+*Quick'n'dirty on naked cluster:*
+```
+kubectl apply -k .
+```
+(This will of course need a recent `kubectl` and a configured cluster context.)
 
-* [Quick demo with Minikube](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/minikube.md)
-* [Usage with minimized k3s](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/k3s.md)
-* [Deploy to an Amazon EC2 based custom K8s cluster](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/aws-kops.md)
+A number of utilities have been added for your convienience:
+have a look at [Little Helpers](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/little-helpers.md).
 
-For production usage, you should make yourself familiar with a series of
-documentation articles, linked below:
+#### Production usage
+You should make yourself familiar with a series of documentation articles, linked below:
 
 * [Container images](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/images.md)
 * [Detailed insight into inner workings](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/how-it-works.md)
@@ -45,8 +48,35 @@ Please be aware that this project currently only offers images and support
 for basic usage. Integrations are not yet part of this, but may be added as needed.
 See also relevant docs within Dataverse guides and upstream projects.
 
-A number of utilities have been added for your convienience:
-have a look at [Little Helpers](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/little-helpers.md).
+#### Development usage
+First, you will need to read up and get familiar with all of the above about production usage.
+More details about usage for developing Dataverse below.
+
+* [Development container images](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/images.md#development-images)
+* [Prepare toolchain](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/rundev.md#prepare-toolchain)
+* [Using local cluster](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/rundev.md#local-cluster)
+* [Using remote cluster](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/rundev.md#remote-cluster) (not yet supported)
+
+<small>
+<details>
+<summary>If you think this is weird and/or cumbersome:</summary>
+As long as K8s usage is not a first class citizen for IQSS, this project should
+not (or cannot) be included in Dataverse upstream.
+
+```diff
++ We don't have to deal with upstream merge process for PRs and can move quicker.
++ We can use tools like Skaffold, Kustomization, etc only usable when living at the topmost level.
+- We have to deal with `git submodules` and somewhat bloated image builds.
+- We cannot use fancy Maven tools like JIB and others.
+```
+</details>
+</small>
+
+#### Examples
+
+* [Quick demo with Minikube](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/minikube.md)
+* [Usage with minimized k3s](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/k3s.md)
+* [Deploy to an Amazon EC2 based custom K8s cluster](https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/aws-kops.md)
 
 ### Upgrading your installation
 

docker/dataverse-k8s/glassfish-dev/Dockerfile

@@ -0,0 +1,29 @@
+# Copyright 2019 Forschungszentrum Jülich GmbH
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# http://www.apache.org/licenses/LICENSE-2.0
+
+############# BUILD DATAVERSE #############
+
+FROM maven:3.5-jdk-8 as builder
+# copy the project files
+COPY dataverse/local_lib ./local_lib
+COPY dataverse/pom.xml ./pom.xml
+# build all dependencies for offline use
+RUN mvn de.qaware.maven:go-offline-maven-plugin:resolve-dependencies
+# copy your other files
+COPY dataverse/src ./src
+# build for release (in offline mode)
+RUN mvn -o package -DskipTests
+
+############# BUILD IMAGE #############
+FROM iqss/dataverse-k8s
+LABEL maintainer="FDM FZJ <forschungsdaten@fz-juelich.de>"
+
+# Copy files for the application
+COPY --chown=glassfish:glassfish --from=builder /target/dataverse-*.war ${HOME_DIR}/dvinstall/dataverse.war
+COPY --chown=glassfish:glassfish dataverse/scripts/api/data ${HOME_DIR}/dvinstall/data
+COPY --chown=glassfish:glassfish dataverse/scripts/api/*.sh ${HOME_DIR}/dvinstall/
+COPY --chown=glassfish:glassfish dataverse/scripts/database/reference_data.sql ${HOME_DIR}/dvinstall/
+COPY --chown=glassfish:glassfish dataverse/conf/jhove/* ${HOME_DIR}/dvinstall/

docker/dataverse-k8s/Dockerfile → docker/dataverse-k8s/glassfish/Dockerfile

@@ -94,7 +94,7 @@ RUN wget --no-verbose -O dvinstall.zip ${DATAVERSE_PKG} && \
     rm dvinstall.zip
 
 # Copy across docker scripts
-COPY --chown=glassfish:glassfish bin/*.sh bin/default.config ${SCRIPT_DIR}/
+COPY --chown=glassfish:glassfish docker/dataverse-k8s/bin/* docker/dataverse-k8s/glassfish/bin/* ${SCRIPT_DIR}/
 RUN mkdir -p ${SCRIPT_DIR}/init.d && \
     chmod +x ${SCRIPT_DIR}/*
 

docs/aws-demo/kustomization.yaml

@@ -1,4 +1,7 @@
 ---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
 bases:
   - ../../k8s/dataverse
   - ../../k8s/solr

docs/images.md

@@ -7,16 +7,59 @@ images to be used for the Dataverse deployment.
 2. Derive those and push your customized image to a place you like.
 3. Build the genuine images yourself and push to a registry of your choice.
 
-## Building is easy
+## Tooling
+
+*Currently, only Docker is supported for image building.*
+
+You might try to build with [Podman](https://podman.io) or [Buildah](https://buildah.io/), too.
+This has the advantage of no need to run a Docker daemon, which might be easier
+when using your custom CI.
+
+## Production images
 
 Simple with Docker after cloning and accessing the source folder:
 ```
-docker build -t iqss/dataverse-k8s:4.15.1 docker/dataverse-k8s
+docker build -t iqss/dataverse-k8s:4.15.1 -f docker/dataverse-k8s/glassfish/Dockerfile .
 docker build -t iqss/solr-k8s:4.15.1 docker/solr-k8s
 ```
 *Please remember to change the tag above as appropriate. You should be*
 *using tagged images as best practice, not 'latest'.*
 
-You might try to build with [Podman](https://podman.io) or [Buildah](https://buildah.io/), too.
-This has the advantage of no need to run a Docker daemon, which might be easier
-when using your custom CI.
+## Development images
+
+*NOTE: You don't need to worry about anything related to Dataverse like compilation.*
+*Image building will take care of it. So if you don't have Maven installed,*
+*don't use an IDE or just want to try out sth.: you are all set. Go ahead.*
+
+### Prepare Dataverse sources
+For building images from any branch or commit of Dataverse, you need to have
+it in your Docker build context. Easiest way to achieve this, after cloning the
+K8s repo, run:
+
+```
+git submodule sync --recursive
+git submodule update --init --recursive
+```
+
+This will checkout the upstream Dataverse `develop` branch into `./dataverse`.
+If you want a feature branch, add your fork, whatever: just follow normal `git submodule`
+routines. (For example, goto `./dataverse` and `git checkout` your branch.)
+
+For more on submodules, have a look at
+  - https://medium.com/@porteneuve/mastering-git-submodules-34c65e940407
+  - https://chrisjean.com/git-submodules-adding-using-removing-and-updating/
+  - https://gist.github.com/gitaarik/8735255
+  - https://lmgtfy.com/?q=git+submodule
+
+You can even point your IDE to this new subfolder and it will work like a champ.
+
+### Build snappy images manually
+Simply use Docker again (almost like above, but different path):
+```
+docker build -t iqss/dataverse-k8s:test -f docker/dataverse-k8s/glassfish-dev/Dockerfile .
+```
+*NOTE: currently there is no Solr development image. This is likely to change.*
+
+### Build images automatically (with Skaffold)
+Please see [development usage docs](rundev.md) for this. It will make your life
+easier, I suppose.

docs/k3s-demo/kustomization.yaml

@@ -1,4 +1,7 @@
 ---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
 bases:
   - ../../k8s/dataverse
   - ../../k8s/solr

docs/release-notes/4.16-64-develop-in-k8s.md

@@ -0,0 +1,6 @@
+# Support for running any branch in K8s
+
+This release adds support for running any branch in Kubernetes by using
+local Kind clusters and Skaffold for building images.
+
+For more information, see [the docs](docs/rundev.md).

docs/rundev.md

@@ -0,0 +1,90 @@
+# Running Dataverse development snapshots
+
+Images on Docker Hub are meant for production usage or quick demos.
+When developing Dataverse, testing a new feature not yet shipped in a release or
+running integration tests you have a need to deploy all moving parts into
+a (more or less) ephemeral environment.
+
+**Be warned:** deployment times of Dataverse are a nightmare as of writing (Sept. 2019).
+For any change to the codebase, you most likely will have to wait about five
+to ten minutes to redeploy from compilation to reload of webpage.
+
+This is due to a really big WAR file (which needs to be loaded into the cluster)
+and being stuck on old technology in combination with a tremendous monolith.
+
+**YOU HAVE BEEN WARNED.**
+
+## Prepare toolchain
+For efficient workflows, tools make life easier. Please install:
+
+1. [skaffold](https://skaffold.dev/docs/getting-started/#installing-skaffold)
+2. [kustomize](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/INSTALL.md)
+    * necessary as long as GoogleContainerTools/skaffold#1781 isn't resolved
+
+When you opt for using a local cluster, add:
+
+1. [Docker](https://docs.docker.com/install/)
+2. [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
+   <small>(be nice to yourself, download a release, don't compile from source)</small>
+
+*Note: you do have `kubectl` already installed, don't you?*
+
+## Workflow
+While you can build and deploy everything manually, it will be easier to
+let Skaffold take care of everything.
+
+Running `skaffold run` or `skaffold dev` will build, tag and deploy for you.
+For a deeper insight, read docs at https://skaffold.dev/docs.
+
+As you will need to access services, be sure to add `--port-forward`.
+See also [port forward docs](https://skaffold.dev/docs/how-tos/portforward).
+
+**Before running Skaffold for the first time** be sure to have a cluster at hands.
+You can check via `kubectl`. Currently only using a local cluster with `kind` is
+supported by this project, see next section. PRs welcome.
+
+### A word on waiting
+Be aware that initial builds and deployments take lots of time due to cold caches.
+Recurring builds and deployments will be much faster, although you will still
+suffer from Glassfish WAR deployment times.
+
+Typically, when there is no change to `pom.xml` and caches are warmed up,
+* building will take ~1 minute,
+* loading images into cluster ~45 secs,
+* deploying to K8s ~30 secs
+* and Glassfish startup + WAR deployment ~3 minutes.
+
+<small>*How about some coffee?*</small>
+
+### Local cluster
+The easiest way to work with a local cluster is using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/), which is an abbrev for "Kubernetes IN Docker".
+
+Skaffold supports this out of the box now. Using `kind`, your context will be set
+to sth. like "@kind", which triggers loading images into a local `kind` cluster
+instead of pushing to remote registry.
+
+After installing Docker and `kind`, you simply need to run:
+```
+kind create cluster
+```
+or - if your prefer a specific K8s version, e.g. `1.14.6`:
+```
+kind create cluster --image kindest/node:v1.14.6
+```
+
+<small><i>
+Please note that `kind` might have some troubles with changing networks and
+switching to a new DNS resolver. You might need to rebuild the cluster, which is
+no big deal.
+</i></small>
+
+If you want to use `k3s`, `minikube`, `microk8s` or similar, please consult the
+Skaffold docs, search via Google, etc.
+
+### Remote cluster
+Currently only using a [local cluster](#local-cluster) with `kind` is supported by this project.
+PRs welcome. Hint: we'll need [Kaniko](https://github.com/GoogleContainerTools/kaniko)
+for that, as Docker Hub must not be cluttered.
+
+## Future ideas
+- Test using [telepresence](https://www.telepresence.io/) - it might lower dev cycle time

k8s/dataverse/deployment.yaml

@@ -24,7 +24,7 @@ spec:
     spec:
       containers:
         - name: dataverse
-          image: iqss/dataverse-k8s:4.15.1
+          image: iqss/dataverse-k8s
           ports:
             - containerPort: 8080
           envFrom:

k8s/dataverse/kustomization.yaml

@@ -1,4 +1,6 @@
 ---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
 resources:
   - deployment.yaml
   - configmap.yaml

k8s/solr/deployment.yaml

@@ -24,7 +24,7 @@ spec:
     spec:
       containers:
         - name: solr
-          image: iqss/solr-k8s:4.15.1
+          image: iqss/solr-k8s
           ports:
             - containerPort: 8983
           volumeMounts:

k8s/solr/kustomization.yaml

@@ -1,4 +1,6 @@
 ---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
 resources:
   - deployment.yaml
   - pvc.yaml

k8s/utils/postgresql/kustomization.yaml

@@ -1,4 +1,6 @@
 ---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
 resources:
   - deployment.yaml
   - pvc.yaml

kustomization.yaml

@@ -0,0 +1,16 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+bases:
+  - k8s/utils/postgresql
+  - k8s/solr
+  - k8s/dataverse
+
+resources:
+  - k8s/utils/demo-secrets.yaml
+
+images:
+  - name: iqss/dataverse-k8s
+    newTag: 4.15.1
+  - name: iqss/solr-k8s
+    newTag: 4.15.1

skaffold.yaml

@@ -0,0 +1,13 @@
+apiVersion: skaffold/v1beta14
+kind: Config
+build:
+  artifacts:
+  - image: iqss/dataverse-k8s
+    context: .
+    docker:
+      dockerfile: docker/dataverse-k8s/glassfish-dev/Dockerfile
+  - image: iqss/solr-k8s
+    context: docker/solr-k8s
+  local: {}
+deploy:
+  kustomize: {}