Use any commit or version of Dataverse in container and Kubernetes

[poikilotherm]

Currently, the upstream project does not contain up-to-date Docker images nor does IQSS provide officiall support these installation adventures (which is why this project came to live in the first place).

With the current rise of attention for salvation of IQSS/dataverse#4172, a demand to try things on K8s is rising, too. In addendum, it should be possible to develop, run and test any commit or code change on containers (enhanced by K8s), locally or within CI.

A current solution to this is usage of IQSS/dataverse-ansible in combination with ec2-create-instance.sh. This is fine, but requires an Amazon EC2 account and payment. Also, DataverseEU is going to run things on K8s and the word goes it might be a bad idea to have too much differences between development and production (coming from the principles of CD), especially when talking about technology stack.

Thus, a workflow is needed, which preferably is adaptable to different scenarios and use cases.
To all readers: any feedback is welcome on this. Please join the discussion.
Pinging @pdurbin, @4tikhonov, @donsizemore, @pameyer here.

[poikilotherm]

Let me post a first idea about a possible workflow.

First, let us make some assumptions for this:

1. People interested in developing things on K8s most certainly will be ok with installing a tool or two.
2. As far as I understood it, interest in running Dataverse on K8s is relatively low at IQSS. Thus I will try to keep upstream code as much as possible out of this for now - the less code changes the better. (There will be more than enough for getting things like Config API in place...)
3. People don't like waiting. Try to cache as much as possible.
4. Anybody hacking on Dataverse will want to build and execute unit tests locally. It should be safe to assume Maven is installed.

A possible flow:

1. git clone dataverse-kubernetes
2. git submodule update --init will download upstream code in a subfolder.
   This submodule can be versioned in the branch of the kubernetes project. It can point to any fork of upstream, etc. Thus people can develop in their forks/branches and follow their guidelines. Obviously, using submodules has its pitfalls. Everything comes at a price. See also this simple reference
3. skaffold dev triggers the build and more (see https://skaffold.dev):
   1. Build source code in multistage (so keep Maven and Java version in sync with app server container)
   2. Build application image with build artifacts
   3. Test with container-structure-test
   4. Tag with commit id or branch. Maybe overrideable via variable?
   5. Push to registry
   6. Deploy to K8s
   7. Watch for changes :-)

[poikilotherm]

I just pushed commits with initial groundwork on this.
When you are feeling lucky, you should already have a usable image building manually.
Checkout the branch and do a docker build -t iqss/dataverse-k8s:4.14-test -f docker/dataverse-k8s/dev-payara/Dockerfile . from the source folder of this project. The initial build will take about 4 minutes. Succeding builds should be MUCH faster due to caching in Docker layers 😄

[poikilotherm]

Blockers:

• Ignore kustomization.yaml when using kubectl deployer GoogleContainerTools/skaffold#2432
• Deployers need better documentation for tag behaviour GoogleContainerTools/skaffold#2547