Dataverse container-based development environment

[GPortas]

Overview of the Feature Request

The goal is making Dataverse development environment setup and management easier, from the initial installation to the day-to-day development. The following points have been considered when opting for a container-based development environment:

• SPA frontend developers don’t have to worry so much about backend installation details.
• Environment initialization/shut down is done in a simple way, considering we use docker-compose.
• Easier process and with fewer installation dependencies than the current process based on installation scripts and dependencies directly installed on the localhost machine.

In addition, introducing this, currently development-oriented, offers the following advantages, which are transferable and beneficial for a future container-based production environment.

• Greater scalability and modularity, by self-containing each dependency service in its own container.
• Greater testability, by having more control over the different dependencies of Dataverse and the application itself.

For Dataverse, a custom Docker image is required. As for now is just for development purposes only, we can resort to fake properties or dependencies when needed (such as fake DOI values or SMTP server). Ideally this image will be evolved for productive use as well.

For Dataverse dependencies (PostgreSQL, Solr, etc) we should use official images where possible or create custom ones if the official ones are not enough. In the case of Keycloak, it would become part of these services too, based on the existing docker (and docker-compose) configuration recently added to the code base.

The ultimate goal is to have a docker-compose file that we can manage to have a full dev environment deployed locally (with persistent volumes when necessary). We can take inspiration from this: https://github.com/harvard-lts/drs-import-management-service/blob/main/docker-compose-dev.yml

What kind of user is the feature intended for?

Developers

What inspired the request?

• A demo from @ekraffmiller of docker-compose-cypressdev.yml from https://github.com/opendp/dpcreator
  • docs at https://github.com/opendp/dpcreator/blob/develop/README-DEVELOPMENT.md
  • docker-compose up --build - to build dependencies and start the containers
  • docker-compose run server ./migrate.sh - for dev, it's ok to run a script to do some config after the container is running
  • docker-compose down --rmi - to delete all containers
• The recent addition of a Container Guide, previewable at http://preview.guides.gdcc.io/en/develop/container/index.html
• SPA development work
• Difficulties with the current installation process
• Other applications (this approach is widely used today) Example
• Add docker-compose build system #8709
• As a developer, I'd like Keycloak or similar so I can test OIDC login #9228
• 9228 - add OIDC development setup for OIDC login feature testing #9234
• Add Testcontainers support for more integration tests #6799
• Refactor API tests to using JUnit5 + custom extensions #8251
• Running integration tests for more than just API #5068
• Integration testing: Experimentation with Arquillian and Testcontainers #9237

What existing behavior do you want changed?

Dataverse development environment setup

Any brand new behavior do you want to add to Dataverse?

No, should be the same.

Any related open or closed issues to this feature request?

• Add docker-compose build system #8709
• As a developer, I'd like Keycloak or similar so I can test OIDC login #9228
• Epic: small footprint container usable for development, testing and production purposes #5292
• Get Dataverse running on OpenShift (Docker and Kubernetes) #4040
• Docker for production #4665
• Feature Request/Idea: create a base container image providing a Dataverse-tuned Payara application server #8932 and 8932 container base image #8933
• Feature Request/Idea: provide a Dataverse container image #8934
• 7662 make solrconfig less static #8320 and Solr 8.8 upgrade - remaining issues with solrconfig.xml #7662

[mreekie]

This deliverable Includes:

• Containerize Dataverse service dependencies for development #9414
• Analyze Dataverse application containerization for development #9415

[pdurbin]

I just tried docker-compose up at https://github.com/GPortas/dataverse/blob/9353-docker-dev-env/conf/docker-dev-env/docker-compose.yml (as of c620fe2) and it "just worked!" 🎉

That is to say, it spun up Docker containers for PostgreSQL, Solr, and a mail server. Then installed Payara locally, ran the installer, and now I have a working Dataverse installation! With a good portion of it running in containers!

This is actually really convenient for me because I just bought a new Macbook at home and I only use PostgreSQL for Dataverse so running it in a container seems fine.

I'd say we're well on our way toward containerizing the services (for dev):

• Containerize Dataverse service dependencies for development #9414

Next we'll need to think hard about an approach for containerizing Dataverse itself (for dev):

• Analyze Dataverse application containerization for development #9415

Great job, @GPortas!! 🎉 🚀

Since the mail server for dev is new (I guess it's called MailDev), here's a screenshot. This was me clicking the "Support" link in the header and filling out a form.

[pdurbin]

Should we close this issue in favor of this one...

• Update dev guide to make use of containers the default #9616

... and then just fix stuff along the way as more developers use Docker?

[pdurbin]

@GPortas and I agree. Close it! See https://preview.guides.gdcc.io/en/develop/container/dev-usage.html for how to use containers! 🎉