Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve dmod.client and update CLI #424

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
7f2c2a9
General JobClient for requests.
robertbartel Aug 30, 2023
2563b14
Delete old, deprecated DatasetRequestClient types.
robertbartel Aug 30, 2023
ad5bae1
Add re-implemented DataServiceClient.
robertbartel Aug 30, 2023
7e37727
Uploads in DataServiceClient create_dataset.
robertbartel Aug 31, 2023
9e0292f
Fix DataServiceClient signatures for CLI.
robertbartel Aug 31, 2023
953b31e
Update imports, docstring in request_clients.py.
robertbartel Aug 31, 2023
f570830
Refactor/rework DmodClient.
robertbartel Aug 31, 2023
18822ec
Refactor/rework dmod.client CLI.
robertbartel Aug 31, 2023
75b0d4f
Fix bug with redundant dest value for CLI arg.
robertbartel Aug 31, 2023
07c9362
Update BasicResultIndicator with data attribute.
robertbartel Sep 7, 2023
2474022
Bump core package version to 0.11.0.
robertbartel Sep 7, 2023
632183f
Add unit tests for BasicResultIndicator.
robertbartel Sep 7, 2023
3bd6a74
Redesign dmod.client.ClientConfig using Pydantic.
robertbartel Sep 8, 2023
c72d0ac
Add unit test for dmod.client.ClientConfig.
robertbartel Sep 8, 2023
aff6c7b
Add test config example for ClientConfig.
robertbartel Sep 8, 2023
1447d24
Update .gitignore for modified ClientConfig.
robertbartel Sep 8, 2023
c3f1027
Update dmod.client main func for ClientConfig.
robertbartel Sep 8, 2023
33c8d12
Add protocol-client-type func to dmod_client.py.
robertbartel Sep 8, 2023
b523f6e
Update DmodClient for new ClientConfig class.
robertbartel Sep 8, 2023
4ab1467
Update core version required by comms package.
robertbartel Sep 8, 2023
9edaf50
Add DataTransmitResponse.create_for_received.
robertbartel Sep 8, 2023
6b84664
Fix MaasDatasetManagementMessage.factory_create.
robertbartel Sep 8, 2023
b2b5cfa
Bump comms package version to 0.16.0.
robertbartel Sep 8, 2023
036cdc0
Fix unit tests for [Dataset/DataService]Client.
robertbartel Sep 8, 2023
dfd1759
Add simply unit test for DmodClient.
robertbartel Sep 8, 2023
5812764
Improve dmod.client main func behavior.
robertbartel Sep 8, 2023
7b9f1b7
Update dmod.client package dependencies.
robertbartel Sep 8, 2023
5513678
Bump dmod.client package version to 0.4.0.
robertbartel Sep 8, 2023
2fa6831
Safeguard write to disk when loop through transfer recpts
robertbartel Sep 19, 2023
013cac9
Fix bug in SimpleDataTransferAgent upload.
robertbartel Sep 19, 2023
ea2d946
Refactor exception handling for _process_request.
robertbartel Sep 19, 2023
f79ccea
Rename func to get_dataset_item_names.
robertbartel Sep 19, 2023
8afbc9a
Fix mismatched data service action strings.
robertbartel Sep 20, 2023
8fe5e41
Comment out SchedulerClient init.
robertbartel Sep 20, 2023
95bed86
Fix bug with non-async context manager.
robertbartel Sep 20, 2023
79d70b2
Optional remote debug info in client config.
robertbartel Oct 2, 2023
8b510d3
Support remote debugging on client main().
robertbartel Oct 2, 2023
cd314cb
Fixes in client package __main__.py.
robertbartel Oct 2, 2023
c232cd4
Change GUI Dockerfile to new style client config.
robertbartel Oct 3, 2023
087a5dc
Fix GUI image request service SSL path.
robertbartel Nov 1, 2023
48a6443
Fix RequestClient to handle multiple response types.
robertbartel Nov 6, 2023
8e159b8
Add service debug env flags to example.env.
robertbartel Nov 6, 2023
fd90869
Add debug args from env to GUI service image build.
robertbartel Nov 6, 2023
9e65c6b
Adjust GUI service image to fix client config debug.
robertbartel Nov 6, 2023
fc77675
Fix DataRequestHandler after recent client changes.
robertbartel Nov 6, 2023
a7bfb31
Fix client package clients to use connection context.
robertbartel Nov 6, 2023
51f583d
Add CLI client arg to turn debugging on/off.
robertbartel Nov 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ dist/
build/
*.egg-info/

/.dmod_client_config.yml
/.dmod_client_config.json

/python/pydevd-pycharm.egg

Expand Down
31 changes: 26 additions & 5 deletions docker/nwm_gui/app_server/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -46,15 +46,36 @@ COPY ./python/gui/MaaS ./MaaS
COPY ./docker/nwm_gui/app_server/entrypoint.sh ./
COPY ./docker/nwm_gui/app_server/client_debug_helper.py ./

RUN echo "request-service:" > .dmod_client_config.yml \
&& echo " hostname: 'request-service'" >> .dmod_client_config.yml \
&& echo " port: 3012" >> .dmod_client_config.yml \
&& echo " ssl-dir: '/usr/maas_portal/ssl'" >> .dmod_client_config.yml
ARG PYCHARM_REMOTE_DEBUG_HOST
ARG PYCHARM_REMOTE_DEBUG_PORT
ARG REQUEST_SERVICE_PORT

ENV PYCHARM_DEBUG_EGG=/pydevd-pycharm.egg
ENV REQ_SRV_SSL_DIR=${WORKDIR}/request_service_ssl
# TODO: needs to be fixed ... doesn't mesh with configurability of location
COPY ./ssl/request-service ${REQ_SRV_SSL_DIR}

# TODO: move to heredoc syntax once confirmed it's reasonable to expect all environments run sufficiently recent Docker
RUN echo '{' > .dmod_client_config.json \
&& if [ -n "${PYCHARM_DEBUG_EGG:-}" ] && [ -n "${PYCHARM_REMOTE_DEBUG_PORT:-}" ] && [ -n "${PYCHARM_REMOTE_DEBUG_HOST:-}" ]; then \
echo ' "remote-debug": {' >> .dmod_client_config.json ; \
echo " \"egg-path\": \"${PYCHARM_DEBUG_EGG:?}\"," >> .dmod_client_config.json ; \
echo " \"port\": ${PYCHARM_REMOTE_DEBUG_PORT:?}," >> .dmod_client_config.json ; \
echo " \"host\": \"${PYCHARM_REMOTE_DEBUG_HOST:?}\"" >> .dmod_client_config.json ; \
echo ' },' >> .dmod_client_config.json ; \
fi \
&& echo ' "request-service": {' >> .dmod_client_config.json \
&& echo ' "protocol": "wss",' >> .dmod_client_config.json \
&& echo " \"pem\": \"${REQ_SRV_SSL_DIR}/certificate.pem\"," >> .dmod_client_config.json \
&& echo " \"port\": ${REQUEST_SERVICE_PORT:?}," >> .dmod_client_config.json \
&& echo ' "hostname": "request-service"' >> .dmod_client_config.json \
&& echo ' }' >> .dmod_client_config.json \
&& echo '}' >> .dmod_client_config.json

# TODO: when image tagging/versioning is improved, look at keeping this in a "debug" image only
# Copy this to have access to debugging pydevd egg
COPY --from=sources /dmod /dmod_src
RUN if [ -e /dmod_src/python/pydevd-pycharm.egg ]; then mv /dmod_src/python/pydevd-pycharm.egg /. ; fi \
RUN if [ -e /dmod_src/python/pydevd-pycharm.egg ]; then mv /dmod_src/python/pydevd-pycharm.egg ${PYCHARM_DEBUG_EGG} ; fi \
&& rm -rf /dmod_src

# Set the entry point so that it is run every time the container is started
Expand Down
4 changes: 4 additions & 0 deletions docker/nwm_gui/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ services:
args:
docker_internal_registry: ${DOCKER_INTERNAL_REGISTRY:?Missing DOCKER_INTERNAL_REGISTRY value (see 'Private Docker Registry ' section in example.env)}
comms_package_name: ${PYTHON_PACKAGE_DIST_NAME_COMMS:?}
# Necessary to generate the CLI's ClientConfig to support remote debugging
PYCHARM_REMOTE_DEBUG_HOST: ${PYCHARM_REMOTE_DEBUG_SERVER_HOST:-host.docker.internal}
PYCHARM_REMOTE_DEBUG_PORT: ${PYCHARM_REMOTE_DEBUG_SERVER_PORT_GUI:-55875}
REQUEST_SERVICE_PORT: ${DOCKER_REQUESTS_CONTAINER_PORT:-3012}
networks:
- request-listener-net
# Call this when starting the container
Expand Down
13 changes: 12 additions & 1 deletion example.env
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,19 @@ PYTHON_PACKAGE_NAME_DATA_SERVICE=dmod.dataservice
## server at the level of individual (supported) services. ##
########################################################################
PYCHARM_REMOTE_DEBUG_VERSION=~=211.7628.24
## Right now there is only support for the GUI app service
## Flag to indicate whether remote Pycharm debugging should be active
## in the GUI image and service.
PYCHARM_REMOTE_DEBUG_GUI_ACTIVE=false
## Flag for whether remote Pycharm debugging is active for data service
PYCHARM_REMOTE_DEBUG_DATA_SERVICE_ACTIVE=false
## Flag for whether remote Pycharm debugging is active for evaluation service
PYCHARM_REMOTE_DEBUG_EVALUATION_SERVICE_ACTIVE=false
## Flag for whether remote Pycharm debugging is active for partitioner service
PYCHARM_REMOTE_DEBUG_PARTITIONER_SERVICE_ACTIVE=false
## Flag for whether remote Pycharm debugging is active for request service
PYCHARM_REMOTE_DEBUG_REQUEST_SERVICE_ACTIVE=false
## Flag for whether remote Pycharm debugging is active for scheduler service
PYCHARM_REMOTE_DEBUG_SCHEDULER_SERVICE_ACTIVE=false
## The debug server host for the debugged Python processes to attach to
PYCHARM_REMOTE_DEBUG_SERVER_HOST=host.docker.internal
## The remote debug server port to for debugging request-service
Expand Down
Loading
Loading