Reimplementation of ccdmodel according to the linkml model template

.gitignore

@@ -1,3 +1,8 @@
+# MAM 20210805
+# this directory is for organizing local scratch files along with this project
+# could also be called something like temp
+local
+
 # Outputs from various generators.
 /generators/google-sheets/output/*.yaml
 /docs

AUTHORS

@@ -0,0 +1,11 @@
+Dazhi Jiao <djiao@jhu.edu>
+Gaurav Vaidya <gaurav@ggvaidya.com>
+Gaurav Vaidya <gaurav@renci.org>
+Harold Solbrig <solbrig@jhu.edu>
+Jim Balhoff <balhoff@gmail.com>
+Mark A. Miller <MAM@lbl.gov>
+Mark Andrew Miller <MAM@lbl.gov>
+Matthew Brush <mhb120@gmail.com>
+Monica Munoz-Torres <monicacecilia@users.noreply.github.com>
+hsolbrig <solbrig@jhu.edu>
+jiaola <djiao@jhu.edu>

CHANGELOG.md

@@ -7,6 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+_In August 2021, This repo was updated based on the [linkml model template](https://github.com/linkml/linkml-model-template)_
+
+Some linkml model template files, like `MakeConfig` hav been moved into `from_template`. Note that `MakeConfig` makes extensive changes to numerous files including the model, tests, docs and PyPI metadata. These updates are based on `model/CONFIG.yaml`. Updating with `MakeConfig` could be very convenient but a strategy combining `MakeConfig` with manual updates to these files could lose important work. It's probably best to commit before running `MakeConfig`.
+
+Most recently, several steps were taken to keep the `squeaky-clean` Makefile recipe for erroring out. 
+- `PKG_T_JSON_SCHEMA` was set to `json_schema` instead of `jsonschema`.
+- `docs` is being cleaned with `rm -rf docs/*`. That will delete any `README.*` or `.keep` file placed in there.
+
+There is some conflict in terms of the Python version require specifications.
+
+- `setup.cfg` says `python_requires = >=3.7`
+- `setup.py` says `if sys.version_info < (3, 7, 0):`
+    - The reporting of warning string was modified due to a error when trying to run `setup.py` under 2.7
+- `tox.ini` says `envlist = py37, py38, py39`
+- `./Pipfile says` inherited `python_version = "3.8"` from pre-template code
+
+TODO: 
+- Aggregate Markdown/documentation files like `from_template/ABOUT.md` and `from_template/ABOUT.md` into `MAINTAINERS.md`
+- A verbose `./ChangeLog` is being created in `make pypi`, Disable that or move the output somewhere less conspicuous.
+
 ## [v1.0.1] - 2021-06-08
 
 * Model changes

MAINTAINERS.md

@@ -1,29 +1,181 @@
-# How to make a release
+# Regenerating the CRDC-H model
 
-See the instructions at the top of [Makefile](Makefile)
+_In August 2021, This repo was updated based on the [linkml model template](https://github.com/linkml/linkml-model-template)_
 
-First install the linkml python package and mkdocs:
+## Requirements
+* __Python >= 3.7.1__
+* __GNU make__ - Windows users can use [cygwin](https://www.cygwin.com/). `make` comes built into MacOS and most Linuxes.
+* __pip__ - bundled with most python distros
+* __virtualenv__ - can be added via `pip install virtualenv`
+* __pipenv__ - can be added via `pip install pipenv`
 
+See also the linkml model template's [original ABOUT.md](https://github.com/cancerDHC/ccdhmodel/blob/5ec6c203edd871a72c87aea03a894bbb3c798c79/from_template/ABOUT.md) file.
+
+## The following is a reasonable start from a fresh repo
+
+- `make regen-google-sheets` 
+  - Requires some environment setup. Partially documented in `generators/google-sheets/README.md`
+  - Recreates `model/schema/crdch_model.yaml` from the ["CDM_Dictionary_v1 (Active)" Google Sheet](https://docs.google.com/spreadsheets/d/1oWS7cao-fgz2MKWtyr8h2dEL9unX__0bJrWKv6mQmM4/edit#gid=1262189248) using `generators/google-sheets/sheet2linkml.py`
+- `make`
+  - Installs `make-venv` environment if necessary 
+  - Builds artifacts in `crdch_model` based on `model/schema/crdch_model.yaml` 
+  - Also builds documentation in `docs` based on `model/schema/crdch_model.yaml` and `model/docs` 
+  - Run this every time the model is modified, whether manually or via `make regen-google-sheets`
+- `make test`
+  - Minimal test input data in `tests/input/test_diagnosis.json`
+  - Test script in `tests/test_input_against_model.py`
+    - Like many other parts of this repo, would be regenerated by making the `from_template/MakeConfig`, which takes its configuration from `from_template/MakeConfig`
+- git add, commit, push if desired
+
+## Initial PyPI setup
+
+### Prerequisites
+- A PyPi username and password (or PyPI OAUTH2 token)
+- `make` and `make test` run to completion
+- `mkdir dist` if necessary
+- `pipenv install --dev` if necessary
+  - `pip install wheel` if necessary
+  - `pip install twine` if necessary
+
+Pushes to PyPI can be authenticated with a password or a token. The `twine` application will ask for a username and password when it is first run.
+
+Optionally create an OAUTH2 token from the PyPi site
+- log in to pypi
+- select the Manage button for your project
+- select _Settings_
+- select _Create a token for crdch_model_
+- give the token a label and set the scope to `crdch_model`
+- copy the token to the clipboard and save it somewhere safe
+    -Note that this is the only time you will see this token. Once you leave this web page, the token cannot be retrieved.
+
+In order to run `make pypi` in the future, a `~/.pypirc` file with one of the following formats should be created.  See https://packaging.python.org/specifications/pypirc/
+
+```
+[pypi]
+username = <SECRET>
+password = <SECRET>
+```
+or
+```
+[pypi]
+username = __token__
+password = <OAUTH2 token>
+```
+
+In some cases, the PyPI username and password may be stored in a keyring. That can be disabled with `keyring --disable`. See https://twine.readthedocs.io/en/latest/
+
+### Process
+
+1) Commit all changes
 ```bash
-. environment.sh
-pip install -r requirements.txt
+> git commit -am "Ready for testing..."
+> git status
+On branch main
+Your branch is up to date with 'origin/main'.
+
+nothing to commit, working tree clean
+>
 ```
 
-Then every time you change the source schema, run:
+2) Make an interactive push to PyPI
+
+- Check the previous tag at https://pypi.org/project/crdch-model/#history
+    - `git tag` and _https://github.com/cancerDHC/ccdhmodel/releases show tagged releases from the `main` branch (or explicity pushed tags?). `git describe` shows the latest local/dev version._
+- Set the next tag to something incremental with a command like `git tag v0.0.1`
 
 ```bash
-make all
+> rm -f dist/*
+> pipenv shell
+> python setup.py sdist bdist_wheel
+...
+adding 'crdch_model-0.2.2.data/data/model/docs/home.md'
+adding 'crdch_model-0.2.2.data/data/model/schema/README.md'
+adding 'crdch_model-0.2.2.data/data/model/schema/crdch_model.yaml'
+adding 'crdch_model-0.2.2.dist-info/AUTHORS'
+adding 'crdch_model-0.2.2.dist-info/METADATA'
+adding 'crdch_model-0.2.2.dist-info/WHEEL'
+adding 'crdch_model-0.2.2.dist-info/pbr.json'
+adding 'crdch_model-0.2.2.dist-info/top_level.txt'
+adding 'crdch_model-0.2.2.dist-info/RECORD'
+removing build/bdist.macosx-10.14.6-x86_64/wheel
+> ls dist
+crdch_model-0.2.2-py3-none-any.whl	crdch_model-0.2.2.tar.gz
+>
 ```
 
-This will generate files in:
+_If the tags above show up with a `dev` suffix on them, something has changed since the tag.  If this occurs, you can go ahead and find what got missed in the commit, fix and commit it and grab the next tag (v0.2.3 in this example) and redo the setup._
+
+3) Upload the commit to pypi
+
+```bash
+> twine upload dist/*
+Uploading distributions to https://upload.pypi.org/legacy/
+Enter your username: <pypi user name>
+Uploading crdch_model-0.2.2-py3-none-any.whl
+100%|██████████████████████████████████████| 659k/659k [00:02<00:00, 275kB/s]
+Uploading crdch_model-0.2.2.tar.gz
+100%|██████████████████████████████████████| 1.54M/1.54M [00:04<00:00, 372kB/s]
+
+View at:
+https://pypi.org/project/crdch-model/0.2.2/
+```
+4) Push the tag info to github
+```bash
+> git commit -am "Created initial distro"
+> git push --tags
+```
+
+**After this initial setup**, `make pypi` should be sufficient, as long as the project is clean and a new tag has been asserted.
+- _This is currently going into @turbomam's PyPI account_
+
+### Cleanup options
+
+- `make clean`
+  - Just cleans up `target`, which doesn't get synced anyway
+    - `rm -rf target/*`
+
+- `make squeaky-clean` (includes `make clean`)
+  - uninstalls the  `make-venv` environment
+    - `pipenv --rm` does the same thing. It can be run in the root of the project **and** in `make-venv`
+  - Removes artifacts in the synced output directory `crdch_model`
+
+## Optionally using `tox`
+
+- Install globally with `pip3 install tox`
+  - will be dependent on the globally available python versions
+- **Or** `pipenv install --dev` if necessary and `pipenv run tox4`
+  - _Would tox purists object to running it inside a virtual environment?!_
+- Runs tests the same way that they are run by `make test` **but under multiple versions of Python**
+  - `pipenv run python -m unittest`
+- Can also be configured to build and publish? Those aren't really unmet needs here.
+
+## Regarding virtual environments
+
+The environments for using and building this package are primarily managed with pipenv now. There are multiple virtual environments. The Makefile recipes install and enter into the environments as necessary. Manually installing the environment shouldn’t be necessary for the steps above, but couldn’t hurt.
+
+### All of the following come from the linkml-model-template:
+- `./make-venv/Pipfile` is used by most of the `Makefile` recipes
+- There is also a smaller `./Pipfile`, including a `dev` section, which can be used for running Python code in `crdch_model`
+- `from_template/MakeConfig` uses `virtualenv` to install `template-configurator`
+- `make test` seems to install a virtual environment... from `./Pipfile`, with teh `--dev` flag?
+- `tox` (which is currently optionally) creates its own virtual environments with pipenv. (Using which `Pipfile`?)
+
+`regen-google-sheets` (which precedes linkml-model-template) has its own environment
+
+virtualenv style requirements files can be generated as follows:
+
+```
+pipenv lock --dev --requirements 
+pipenv lock --dev --requirements > requirements-dev.txt
+```
+
+## Regarding documentation
+
+The previous doc genration/publishing approach and the approach supported by the linkml-model-template have not been completely aligned yet.
 
- * [docs]
- * [jsonschema]
- * [shex]
- * [owl]
- * [rdf]
+The template does require git adding and pushing the `./docs` folder (or making commits within GH actions), but that was previously discouraged.
 
-Do **not** git add the files in docs
+MAM: I'm not 100% sure about using mike to make the docs, although mike is specified as a plugin in the mkdocs setup file. I’m not sure where github tagging tagging fits into this. A gh-pages branch may not be _necessary_ for publishing the docs to https://cancerdhc.github.io/ccdhmodel, but it may be preferable since we're using mike.
 
 ## Using mike to maintain multiple versions
 
@@ -39,7 +191,7 @@ created and populated correctly, you can run:
 
 ```
 $ make clean      # This will delete the `docs/` directory. 
-$ make gen-docs	  # This will generate the docs in `target/docs`.
+$ make gen-docs   # This will generate the docs in `target/docs`.
 $ make stage-docs # This will copy `target/docs` into `docs/`.
 ```