diff --git a/docs/Development Guide.rst b/docs/Development Guide.rst index b50f24cb..783af1e7 100644 --- a/docs/Development Guide.rst +++ b/docs/Development Guide.rst @@ -358,7 +358,7 @@ How to Contribute Development of fmdtools is coordinated by the `fmdtools team <../CONTRIBUTORS.md>`_ at NASA Ames Research Center. As an NASA-developed and maintained open-source tool, outside contributions are welcomed. To be able to submit contributions (e.g., pull requests), external contributors should first submit a contributors license agreement (`Individual CLA `_ , `Corporate CLA `_). -Repo Structure +Repository Structure -------------------------------- .. image:: /docs/figures/uml/repo_structure.svg @@ -379,29 +379,48 @@ There are additionally a few scripts/config files with specific purposes to serv - ``MAKE``, which is used to build the sphinx documentation. -Remote Structure -******************************** +Git Structure and Setup +-------------------------------- .. image:: /docs/figures/uml/git_structure.svg :width: 800 Development of fmdtools uses a two-track development model, in which contributions are provided within NASA as well as by external collaborators. To support this, there are multiple repositories which must be managed in the development process, as shown above. Essentially, there is: -- An internal bitbucket, where NASA coordination and development takes place, -- A public GitHub, where collaboration with outside developers takes place (and where documentation is hosted), and -- A PyPI repository which contains stable versions of fmdtools which can be readily installed via ``pip``. +- An internal bitbucket, ``origin``, where NASA coordination and development takes place, +- A public GitHub, ``public``, where collaboration with outside developers takes place (and where documentation is hosted), and +- A PyPI repository, which contains stable versions of fmdtools which can be readily installed via ``pip``. This repository is automatically updated when a new version is released on GithHub. +- The fmdtools GitHub Pages site, which updates from the ``gh-pages`` branch. The fmdtools team is responsible for coordinating the development between the internal and external git repositories. Managing multiple repositories can best be coordinated by: -- setting up multiple remotes on a single git repo on your machine using `git remote` and -- propagating changes between repositories during development using `git push` and `git pull` from each repository +- setting up the ``public`` and ``origin`` remotes on a single git repo on your machine +- using the ``dev`` branch and feature branches on ``origin`` for development and integration +- releasing to the ``main`` branch on ``origin`` and ``public`` + +To assist with this, the custom git alias below can be helpful:: + + [alias] + up = "!git merge dev main" + tl = "!f() { git tag -s -a \"$1\" -m \"$2\"; }; f" + pp = "!f() { git push public tag \"$1\"; }; f" + po = "!f() { git push origin tag \"$1\"; }; f" + release = "!f() { git checkout main && git up && git tl \"$1\" \"$2\" && git pp \"$1\" && git po \"$1\"; }; f" + fb = "!f() { git fetch origin && git fetch public; }; f" + mm = "!git merge main dev" + sync-into-dev = "!f() { git checkout dev && git fb && git pull origin dev && git pull public dev && mm; }; f" + +Adding this block to your repository's git config file (e.g., ``.git/config``) adds custom git commands which can be used to simplify the release process. Specifically: + +- ``git sync-into-dev`` will merge all main and dev branches (local and remote) into your local dev branch +- ``git release "vX.X.X" "Version X.X.X"`` will merge ``dev`` into ``main``, tag it with the given version, and upload it to ``public`` and ``origin``. + +Note that the ``-s`` option above `signs` the tag, attributing it to your credentials. For this to work, you should `set up commit signing `_ and configure git to sign commits by default. -Development Process --------------------------------- Git Development Workflow -******************************** +-------------------------------- .. image:: /docs/figures/uml/dev_process.svg :width: 800 @@ -424,7 +443,7 @@ The major exceptions to this process are: - minor documentation changes. Release Process -******************************** +-------------------------------- Releases are made to fmdtools to push new features and bugfixes to the open-source community as they are developed. Some important things to remember about the release process are: diff --git a/docs/figures/uml/git_structure.svg b/docs/figures/uml/git_structure.svg index 1fa0f60d..00eea9ab 100644 --- a/docs/figures/uml/git_structure.svg +++ b/docs/figures/uml/git_structure.svg @@ -1,5 +1,5 @@ - + @@ -118,608 +118,872 @@ + + + + + + - + - + - + + - - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + diff --git a/docs/figures/uml/module-reference-diagrams.gaphor b/docs/figures/uml/module-reference-diagrams.gaphor index 9f1c013a..e79bf7dc 100644 --- a/docs/figures/uml/module-reference-diagrams.gaphor +++ b/docs/figures/uml/module-reference-diagrams.gaphor @@ -2245,6 +2245,7 @@ + @@ -2253,7 +2254,6 @@ - @@ -3859,16 +3859,22 @@ - - + + + + + + + + @@ -3877,6 +3883,10 @@ + + + + @@ -3887,8 +3897,16 @@ -git repo +local git repo + + + + + + + + @@ -3897,17 +3915,25 @@ -(1.0, 0.0, 0.0, 1.0, 51.651327397181475, 168.86538897309788) +(1.0, 0.0, 0.0, 1.0, 79.55496380532384, 170.2307779461956) -(0.0, 0.0) +(-248.72951515495959, 0.0) -100.0 +521.0186059305324 -64.0 +140.53574848929003 + + + + + + + + @@ -3922,7 +3948,7 @@ -internal bitbucket +origin: <internal bitbucket> @@ -3932,7 +3958,7 @@ -(1.0, 0.0, 0.0, 1.0, -246.0, 163.36538897309794) +(1.0, 0.0, 0.0, 1.0, -447.17332080111385, 188.75136625219264) (0.0, 0.0) @@ -3957,12 +3983,12 @@ -github.com/nasa/fmdtools +public: github.com/nasa/fmdtools - - + + @@ -3976,20 +4002,14 @@ (1.0, 0.0, 0.0, 1.0, 147.53593839333305, -1.9999999999999432) -(-155.669014695667, -44.63461102690218) +(-155.669014695667, 18.26323701636369) -327.7146379090009 +321.97713088323064 -130.266564507125 +67.36871646385913 - - - - - - @@ -4020,7 +4040,7 @@ (1.0, 0.0, 0.0, 1.0, 270.0, 254.0) -[(-373.0, -53.13461102690212), (-218.3486726028185, -58.7692220538043)] +[(-574.1733208011138, -27.748633747807446), (-439.17455134963575, -25.87422715465479)] @@ -4093,7 +4113,7 @@ (1.0, 0.0, 0.0, 1.0, 582.0, 242.0) -[(-480.3486726028186, -158.36804651977712), (-480.3486726028185, -73.13461102690212)] +[(-482.2707374564517, -158.36804651977712), (-490.6652483843695, -71.76922205380441)] @@ -4111,8 +4131,8 @@ - + @@ -4150,7 +4170,7 @@ -pypi.org/project/fmdtools/ +PyPi: pypi.org/project/fmdtools/ @@ -4160,7 +4180,7 @@ -(1.0, 0.0, 0.0, 1.0, 459.22478025941854, -13.15023694396029) +(1.0, 0.0, 0.0, 1.0, 448.6612372700639, 14.444196743213388) (0.0, 0.0) @@ -4204,7 +4224,7 @@ (1.0, 0.0, 0.0, 1.0, 459.0, 452.9999999999999) -[(-139.41843839333308, -426.5183654539824), (0.2247802594185373, -426.51836545398226)] +[(-145.15594541910332, -398.92393176680844), (-10.338762729936093, -398.92393176680855)] @@ -4215,7 +4235,7 @@ - + @@ -4233,7 +4253,7 @@ -(1.0, 0.0, 0.0, 1.0, 48.676800253225665, 18.11350603599565) +(1.0, 0.0, 0.0, 1.0, 153.29603363381366, 57.98576737260092) (0.0, 0.0) @@ -4248,7 +4268,7 @@ - + @@ -4256,7 +4276,7 @@ - + @@ -4274,7 +4294,7 @@ -(1.0, 0.0, 0.0, 1.0, -114.36765570469582, 15.068804935145417) +(1.0, 0.0, 0.0, 1.0, 39.24090897964419, 57.98576737260083) (0.0, 0.0) @@ -4289,7 +4309,7 @@ - + @@ -4312,7 +4332,7 @@ -(1.0, 0.0, 0.0, 1.0, -8.133076302333919, -198.0895240408207) +(1.0, 0.0, 0.0, 1.0, -8.13307630233389, -93.08952404082066) (0.0, 0.0) @@ -4344,7 +4364,7 @@ (1.0, 0.0, 0.0, 1.0, 71.86473844339707, -150.0000000000001) -[(-21.5914212311788, 19.137609870290845), (-16.678247515486788, 103.36538897309799)] +[(-21.5914212311788, 124.13760987029085), (-17.786823147537547, 166.26323701636386)] @@ -4370,7 +4390,7 @@ -(1.0, 0.0, 0.0, 1.0, -313.67029510220044, 10.113506035995643) +(1.0, 0.0, 0.0, 1.0, 171.8509974391376, 370.86538897309794) (0.0, 0.0) @@ -4437,7 +4457,7 @@ (1.0, 0.0, 0.0, 1.0, 562.6270859787093, 243.00000000000006) -[(6.408733883955392, -176.8864939640044), (6.038323248773281, -126.1346110269021)] +[(-4.154809105399181, -149.2920602768307), (6.038323248773281, -126.1346110269021)] @@ -4463,7 +4483,7 @@ -(1.0, 0.0, 0.0, 1.0, -113.13307630233396, 300.2124096525419) +(1.0, 0.0, 0.0, 1.0, -313.6488967244532, 328.86538897309805) (-195.29173565556994, 0.0) @@ -4495,7 +4515,7 @@ (1.0, 0.0, 0.0, 1.0, -163.57425838390674, 229.64795914656005) -[(0.0, 8.717429826537881), (-0.4999999999999929, 70.56445050598185)] +[(-201.17332080111385, 34.103407105632556), (-201.0158204221193, 99.217429826538)] @@ -16730,7 +16750,7 @@ (1.0, 0.0, 0.0, 1.0, -169.1247158570257, 29.558491664293058) -[(79.45442075482526, 16.00021233389279), (202.29299854566295, 8.510313270852407)] +[(452.9757132961633, 341.3068973088049), (321.9687704379224, 248.65805365450336)] @@ -16756,7 +16776,7 @@ -(1.0, 0.0, 0.0, 1.0, 183.30663228770774, 173.8653889730979) +(1.0, 0.0, 0.0, 1.0, 430.6163505627207, 370.86538897309794) (0.0, 0.0) @@ -16790,7 +16810,6 @@ - @@ -16814,9 +16833,9 @@ - - - + + + @@ -16835,7 +16854,7 @@ (1.0, 0.0, 0.0, 1.0, 302.0868799675055, 194.6756697093645) -[(-28.420106848474347, -20.810280736266606), (-34.182024331097935, -128.56216367336881)] +[(218.88961142653852, 176.18971926373342), (2.4562344614808467, 83.54087560943202)] @@ -16844,4 +16863,373 @@ + + + + + + + + + + +dev + + + + + + + + + +(1.0, 0.0, 0.0, 1.0, -73.74316476182723, 57.98576737260083) + + +(-6.293705841514964, 0.0) + + +106.29370584151496 + + +50.0 + + + + + + + + + + + + + + + + + + +Used for integration and testing + + + + + + + + + +(1.0, 0.0, 0.0, 1.0, 37.43637934126993, 373.865388973098) + + +(0.0, 0.0) + + +125.0 + + +59.0 + + + + + + + + + + + + + +0 + + +0 + + +(1.0, 0.0, 0.0, 1.0, -28.30838090548417, 263.93900526693403) + + +[(102.18736413226321, 14.277540051862559), (144.29794726024318, 109.92638370616396)] + + + + + + + + + + + + + + + +GitHub Pages: https://nasa.github.io/fmdtools/ + + + + + + + + + +(1.0, 0.0, 0.0, 1.0, -435.0719705987537, -8.05240475170671) + + +(0.0, 0.0) + + +281.3565715310435 + + +101.76034447487609 + + + + + + + + + + + + + + + + +0 + + +0 + + +1 + + + + + + + + +(1.0, 0.0, 0.0, 1.0, 20.457716641983325, 51.052768099137836) + + +[(-28.590792944317258, 0.5820757923167932), (-174.1731157096935, 0.4999999999999929)] + + + + + + + + + + + + + + + + +Release + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +<feature branches> + + + + + + + + + +(1.0, 0.0, 0.0, 1.0, -223.99487150387887, 53.98576737260086) + + +(0.0, 0.0) + + +100.0 + + +58.0 + + + + + + + + + + + + + + + + + + +Used to develop features and fix bugs + + + + + + + + + +(1.0, 0.0, 0.0, 1.0, -86.72573917936663, 373.7210755817531) + + +(-66.98965988834368, 0.0) + + +174.17311570969366 + + +53.28862678268979 + + + + + + + + + + + + + +0 + + +0 + + +(1.0, 0.0, 0.0, 1.0, -91.87878994000846, 252.02174776416888) + + +[(3.091477890952085, 30.19479755462757), (10.4448826594191, 121.69932781758422)] + + + + + + + + + + + + + + + +Used for documentation + + + + + + + + + +(1.0, 0.0, 0.0, 1.0, -358.393684833232, 116.86538897309804) + + +(0.0, 0.0) + + +128.0 + + +50.0 + + + + + + + + + + + + + +0 + + +0 + + +(1.0, 0.0, 0.0, 1.0, -300.06204066993797, 63.42008992467822) + + +[(2.814855851807579, 30.287849798491123), (4.084745667881805, 53.44529904841981)] + + + + + + + + \ No newline at end of file diff --git a/docs/release_checklist.csv b/docs/release_checklist.csv index 3c08bd3b..1185358d 100644 --- a/docs/release_checklist.csv +++ b/docs/release_checklist.csv @@ -1,7 +1,5 @@ Step,Description,Complete?,Comment -1,Sync appropriate branches into release branch,, -1a," ""Internal branches (e.g. main dev)""",, -1b," ""External branches (e.g. main)""",, +1,Sync appropriate branches into release branch using `git sync-into-dev`,, 2,Run script run_all_tests.py and verify test results are appropriate,, 2a," ""Run full tests""",, 2b," ""Regenerate badge""",, @@ -9,11 +7,10 @@ Step,Description,Complete?,Comment 3a," ""Check files conf.py LICENSE.py README.py Intro_to_fmdtools.md pyproject.toml __init__.py""",, 3b," ""Run a repository-wide search for the version number""",, 4,"Generate the documentation using ""./make html"" ",, -5,Commit and tag branch with the appropriate version.,, -5a," ""git tag -s -a v2.x.x -m 'version 2.x.x'""",, -6,Sync to remotes,, -6a," ""Push release branch to github and internal remotes""",, -6b," ""Push docs using `git subtree split --prefix=_build/html --branch gh-pages`, `git push -f public gh-pages` `git branch -D gh-pages`""",, +5,Commit changes.,, +6,Release to GitHub/remotes,, +6a," ""Release using ``git release "v2.x.x" "Version 2.x.x" `` """,, +6b," ""Push docs using ``git subtree split --prefix=_build/html --branch gh-pages``, ``git push -f public gh-pages`` ``git branch -D gh-pages``""",, 7,Create a release in GitHub,, 7a," ""incorporate a narrative summary of features and changes""",, 7b," ""incorporate the auto-generated bitbucket release notes.""",, diff --git a/run_all_tests.py b/run_all_tests.py index c2f913a4..d0ba3e17 100644 --- a/run_all_tests.py +++ b/run_all_tests.py @@ -1,3 +1,4 @@ +#!/usr/bin/env python # -*- coding: utf-8 -*- """ Created on Tue May 16 15:12:39 2023 @@ -94,7 +95,7 @@ ] # retcode = pytest.main(["--nbmake", "--nbmake-find-import-errors", "--nbmake-timeout=20", *too_slow_notebooks]) - # we ignore the following notebooks for various reasons: + # we ignore the following notebooks for various reasons: # while not included in the testing approach, they should be verified periodically ignore_notebooks = [*too_slow_notebooks, "examples/pump/Tutorial_unfilled.ipynb", # intended to be blank diff --git a/tools/keep.txt b/tools/keep.txt new file mode 100644 index 00000000..e69de29b