scripts for rebuilding a dev environment (with sample data)

[pdurbin]

Yesterday in tech hours we talked about things to help developers and I brought up what I think of as the "rebuild with sample data" scripts I've been using for years and years. (I stopped using them only recently because we switched to Payara.)

There are lots of reasons developers might want to rebuild from time to time:

• You've been running integration tests and all your "nice" datasets are buried beneath them.
• You're ready for a clean start with known (or no) data.
• You're having trouble with database upgrade scripts.

By rebuild I don't mean that every single dependency is removed. In fact, the application server (Glassfish or Payara) is largely untouched in my scripts. That said, a number of major changes are made:

• The database is dropped.
• The data files are deleted.
• Solr is cleared out.

These scripts eventually evolved into the ones we used on the phoenix server for many years. This server was rebuilt as described above on every run. The scripts can be found at https://github.com/IQSS/dataverse/tree/develop/scripts/deploy/phoenix.dataverse.org

After "rebuild" has run, the "post" scripts gets executed, and starts with some setup...

• Run setup-all.sh.
• Run SQL scripts (reference data and create sequence).
• Set DOI provider to FAKE.

... and then continues on to load some sample data. As I mentioned on the call, these scripts create some "birds and trees" users, dataverses, and datasets. (Even though our sample data repo is newer, it doesn't create users.) The Spruce Goose dataset (screenshot below) might be familiar.

When estimating this issue, here are some questions to consider:

• Do we want the "birds and trees" data? Or would we rather have the sample data? Or both?
• Should we defer worrying about sample data until a later issue?
• Should we consider adding a rebuild or reinstallation of Payara as part of this? Or should we stick to the model above?
• Should we use the bash scripts above as a starting point? Or should this be a (dangerous!) feature of the installer?

[poikilotherm]

Some ideas:

1. Move all SQL stuff to a Flyway baseline and/or migration. Flyway: disable DDL generation from EclipseLink #5871
2. Make DOI providers configurable via MicroProfile Config API (just like everything else) As a sysadmin I want to use MicroProfile Config API to configure my installation #7000
3. Move all from setup.sh to bootstrapping code Bootstrapping config on first deployment #5361 while keeping it configurable with As a sysadmin I want to use MicroProfile Config API to configure my installation #7000
4. Load test/sample data depending on environment/configuration with the same trick or with a Maven profile using Flyway from there.

"Birds and tress" data sounds good. It should be quick to load. dataverse-sample-data sometimes is a bit slow, yet we could change that.

[djbrooke]

• This would be updating existing scripts to work with Payara
• A better outline/index to exemplify the characteristics of the sample data (New Index/Outline of Characteristics + Review/Revise/Iterate Data Files dataverse-sample-data#23)

[pdurbin]

I've made two pull requests:

• Pull request add script for rebuilding dev environment #7256 #7363 - add script for rebuilding dev environment
• allow API token to be retrieved from environment variable #24 dataverse-sample-data#25 - allow API token to be retrieved from environment variable