Migrate existing pieces to new pluggable component architecture, part 2 the simple pieces #213
Assignees
Comments
Merged
huard
added a commit
that referenced
this issue
Nov 17, 2021
Update Finch to v0.7.7 Changes Non-breaking changes New component version Finch:0.7.7 Related Issue / Discussion Additional Information Revert "added SENTRY_ENV environment variable" by @matprov in #213 Refresh notebooks by @Zeitsperre in bird-house/finch#201 Add SENTRY_ENV param by @matprov in bird-house/finch#203 Multiple rcps for ensemble processes by @aulemahal in bird-house/finch#202 Remove bugging drop after geoserver upgrade by @aulemahal in bird-house/finch#207 Update to xclim 0.31 by @aulemahal in bird-house/finch#209 Fix xarray, dask and gunicorn related hang by @cjauvin in bird-house/finch#204 run load() before switching to single-threaded scheduler by @huard in bird-house/finch#211 Add average flag for ensemble processes by @aulemahal in bird-house/finch#210 Revert "added SENTRY_ENV environment variable" by @matprov in bird-house/finch#213
This was referenced Feb 17, 2023
7 tasks
mishaschwartz
added a commit
that referenced
this issue
Mar 22, 2023
## Overview Makes all components part of the "pluggable architecture" meaning that no components are required to be configured, each components can be added to the stack as needed. See previous discussion for this PR at #295 ## Changes **Non-breaking changes** The components that will be added to the stack are only those whose configuration directory is listed in either `DEFAULT_CONF_DIRS` or `EXTRA_CONF_DIRS`. **Breaking changes** None ## Related Issue / Discussion - Resolves #213 ## Additional Information In a separate PR we should: - components configurations are now stored in `configs`, `components`, and `optional_components` directories. Now that these distinctions are no longer relevant it might make more sense to divide them as follows: - `components`: all components (optional or otherwise; eg: finch, weaver, twitcher) - `shared_dependencies`: docker containers required to run multiple components (eg: postgres, mongodb) - `component_enhancements`: additional configurations that modify an existing component (eg: all-public-access, database-external-ports) ## Tests - Ensure baseline status on `master` branch pass Jenkins to prepare for upgrade test - If env was previous updated to newer branch, run this script to delete new ignored files that branch `master` do not know about, which would break autodeploy [delete-ignore.sh.txt](https://github.com/bird-house/birdhouse-deploy/files/11036499/delete-ignore.sh.txt) - First `./pavics-compose.sh up -d` [master-up.txt](https://github.com/bird-house/birdhouse-deploy/files/11036376/master-up.txt) - First `./pavics-compose.sh ps` not all container healthy [master-ps.txt](https://github.com/bird-house/birdhouse-deploy/files/11036378/master-ps.txt) - Subsequent `./pavics-compose.sh up -d` for all services responding so Weaver can add provider [master2-up.txt](https://github.com/bird-house/birdhouse-deploy/files/11036380/master2-up.txt) - Subsequent `./pavics-compose.sh ps` ready to run Jenkins [master-ps-2.txt](https://github.com/bird-house/birdhouse-deploy/files/11036479/master-ps-2.txt) - Jenkins output, only known expected failure with `esgf-dap.ipynb` and `WMS_example.ipynb` [baseline-master-PAVICS-e2e-workflow-tests-current-production-version-114-consoleText.txt](https://github.com/bird-house/birdhouse-deploy/files/11036409/baseline-master-PAVICS-e2e-workflow-tests-current-production-version-114-consoleText.txt) - Upgrade test via autodeploy to simulate production environment rollout - Autodeploy logs which includes `./pavics-compose.sh up -d` [autodeploy.log](https://github.com/bird-house/birdhouse-deploy/files/11036433/autodeploy.log) - `./pavics-compose.sh ps` after autodeploy [new-ps-after-autodeploy.txt](https://github.com/bird-house/birdhouse-deploy/files/11036463/new-ps-after-autodeploy.txt) - Ensure no regressions, no backward incompatible changes, when using default config after upgrade - Jenkins output, same known expected failure with `esgf-dap.ipynb` and `WMS_example.ipynb` [new-branch-PAVICS-e2e-workflow-tests-current-production-version-115-consoleText.txt](https://github.com/bird-house/birdhouse-deploy/files/11036466/new-branch-PAVICS-e2e-workflow-tests-current-production-version-115-consoleText.txt) - Diff `master-ps-2.txt` and `new-ps-after-autodeploy.txt`, same number of containers so nothing missing (default enabled components are good) - Trigger autodeploy again, it works, no missing `.gitignore` - (not done yet) Test the new ability provided by this PR to pick and choose a subset of components - Proxy, Finch, Weaver only: the subset that CRIM runs but CRIM unable to use PAVICS because it does not want to rest of PAVICS stack - Proxy, Thredds, Twitcher, Magpie, Phoenix only + external birds: the subset that PCIC needs but is stuck with the entire PAVICS stack
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Goal is for the CCDP deployment on CRIM side that a subset of PAVICS (nginx proxy, finch, weaver) to use the actual PAVICS stack and avoid duplicating code (it has
finch-compose.shwhich is sort of a copy ofpavics-compose.shwith the sameenv.localandconfig/dir mechanism).CCDP deployment has a logging monitoring that PAVICS do not have but PAVICS have system metrics monitoring and autodeploy that CCDP do not have.
Basically both sides have new development that are compatible with each other and that can mutually benefit each other.
Simple standalone pieces are thredds, geoserver, jupyterhub.
Includes:
default.envandenv.local.exampleinto each componentdefault.env. Everything related should be within the component, no leaking outside.Backward compat:
DEFAULT_COMPONENTSlist that will activate these by default so this re-architecture is completely transparentThe text was updated successfully, but these errors were encountered: