From 28cd001150521cca7cd1c9b5705b2843f64576e8 Mon Sep 17 00:00:00 2001 From: Alex Bozarth Date: Wed, 4 May 2022 22:43:02 -0500 Subject: [PATCH 1/5] Develop Elyra against the Jupyterlab source repo Adds documentation and some additions to the build scripts to allow Elyra developers to develope Elyra against a local build of Jupyterlab. Updates include: - Updated documentation with step by step instructions on how to step up the devlopment environment - Updating our `page_config.json` to the new format and location. We were previously using a deprecated format and location uncompatable with this feature. - Added a way to run `clean-jupyterlab.sh` without installing a new version of jupyterlab - Added new npm targets and make targets to leverage a yarn link version of `@jupyterlab/builder`. This was added to the existing support for `make dev-link`, but the order of make targets had to be rearranged to make sure lerna was installed before calling it. - Made a small backwards compatable change to fix a type conflict found when running against jupyterlab 4.0.0 alpha These changes are blocked by https://github.com/jupyterlab/jupyterlab/pull/12533 without that change you also need to update the version of `@jupyterlab/builder` in every package.json to prevent an error --- Makefile | 18 +++++++---- build_requirements.txt | 1 - .../developer_guide/development-workflow.md | 32 ++++++++++++++++++- etc/config/settings/page_config.json | 4 ++- etc/scripts/clean-jupyterlab.sh | 14 +++++--- packages/code-snippet/package.json | 4 ++- packages/code-viewer/package.json | 4 ++- .../metadata-common/src/MetadataEditor.tsx | 4 +-- packages/metadata/package.json | 4 ++- packages/pipeline-editor/package.json | 4 ++- .../src/PipelineEditorWidget.tsx | 2 +- packages/python-editor/package.json | 4 ++- packages/r-editor/package.json | 4 ++- packages/theme/package.json | 4 ++- setup.py | 6 ++-- tsconfig.base.json | 3 ++ 16 files changed, 84 insertions(+), 28 deletions(-) diff --git a/Makefile b/Makefile index 05947d855..7053e4032 100644 --- a/Makefile +++ b/Makefile @@ -17,8 +17,8 @@ .PHONY: help purge uninstall-src uninstall clean .PHONY: lint-dependencies lint-server black-format prettier-check-ui eslint-check-ui prettier-ui eslint-ui lint-ui lint .PHONY: dev-link dev-unlink -.PHONY: build-dependencies yarn-install build-ui package-ui build-server install-server-package install-server -.PHONY: install install-all install-examples install-gitlab-dependency check-install watch release +.PHONY: build-dependencies yarn-install build-ui package-ui package-ui-dev build-server install-server-package install-server +.PHONY: install install-all install-dev install-examples install-gitlab-dependency check-install watch release .PHONY: test-dependencies pytest test-server test-ui-unit test-integration test-integration-debug test-ui test .PHONY: docs-dependencies docs .PHONY: elyra-image publish-elyra-image kf-notebook-image publish-kf-notebook-image @@ -128,12 +128,14 @@ lint: lint-ui lint-server ## Run linters ## Library linking targets dev-link: - yarn link @elyra/pipeline-services - yarn link @elyra/pipeline-editor + - yarn link @elyra/pipeline-services + - yarn link @elyra/pipeline-editor + - lerna run link:dev dev-unlink: - yarn unlink @elyra/pipeline-services - yarn unlink @elyra/pipeline-editor + - yarn unlink @elyra/pipeline-services + - yarn unlink @elyra/pipeline-editor + - lerna run unlink:dev yarn install --force ## Build and install targets @@ -150,6 +152,8 @@ build-ui: # Build packages package-ui: build-dependencies yarn-install lint-ui build-ui +package-ui-dev: build-dependencies yarn-install dev-link lint-ui build-ui + build-server: # Build backend $(PYTHON) -m setup bdist_wheel sdist @@ -165,6 +169,8 @@ install: package-ui install-server check-install ## Build and install install-all: package-ui install-server install-examples install-gitlab-dependency check-install ## Build and install, including examples +install-dev: package-ui-dev install-server install-examples install-gitlab-dependency check-install + install-examples: ## Install example pipeline components # install Kubeflow Pipelines example components # -> https://github.com/elyra-ai/examples/tree/master/component-catalog-connectors/kfp-example-components-connector diff --git a/build_requirements.txt b/build_requirements.txt index eb741431d..84236d6c0 100644 --- a/build_requirements.txt +++ b/build_requirements.txt @@ -1,2 +1 @@ -jupyterlab>=3.0.17 jupyter-packaging>=0.10 diff --git a/docs/source/developer_guide/development-workflow.md b/docs/source/developer_guide/development-workflow.md index 89a6a8ea5..c9a24e0b1 100644 --- a/docs/source/developer_guide/development-workflow.md +++ b/docs/source/developer_guide/development-workflow.md @@ -134,7 +134,7 @@ To specify a JupyterLab version to be installed: You can install Elyra using a local build of @elyra/pipeline-editor with: ```bash -make clean dev-link install +make clean install-dev ``` ### Back-end Development @@ -195,3 +195,33 @@ or you can checkout the respective tagged release and omit the `TAG` parameter. Official container images are published on [Docker Hub](https://hub.docker.com/r/elyra/elyra/tags) and [quay.io](https://quay.io/repository/elyra/elyra?tab=tags). + +### Developing Elyra against the Jupyterlab source repo + +Sometimes it is useful to develop Elyra against a local build of Jupyterlab. To use a local build of Jupyterlab use the +following steps. + +1. Uninstall any pip installations of jupyterlab. You can use `etc/scripts/clean-jupyterlab.sh --version dev` as + mentioned above with `--version dev` to not reinstall jupyterlab at the end of the script. + +2. Build your local repo of jupyterlab. Uninstalling in the previous step will also wipe any previous installations of a + local build. + +3. `cd` to the `builder/` directory in your jupyterlab repo and run `yarn link`. Elyra's `Makefile` will use this yarn + link in step 6. + +4. Uncomment the following line in `tsconfig.base.json` to tell Typescript to use the local jupyterlab packages when building: + + ```"paths": { "@jupyterlab/*": ["../jupyterlab/packages/*"] },``` + +5. Comment out `jupyterlab` and `jupyterlab-lsp` in the `install_requires` section of `setup.py`. This will prevent + jupyterlab being pip installed during the Elyra build. Note: `jupyterlab-lsp` also pip installs jupyterlab when installed + +6. Run `make install-dev` to install Elyra using the linked `@jupyterlab/builder` from step 3. + +7. You can now start jupyterlab by running `jupyter lab --dev-mode --extensions-in-dev-mode`, this will automatically watch + for changes in the jupyterlab repo. To also watch for changes in Elyra run `make watch` in a separate terminal in the + same Python environment. + +When you want to switch back to developing Elyra against a Jupyterlab release, you just have to undo the comments in +steps 4 and 5 and rebuild with `make clean install` diff --git a/etc/config/settings/page_config.json b/etc/config/settings/page_config.json index e6715c9c3..634e0cb29 100644 --- a/etc/config/settings/page_config.json +++ b/etc/config/settings/page_config.json @@ -1,3 +1,5 @@ { - "disabledExtensions": ["@jupyterlab/launcher-extension"] + "disabledExtensions": { + "@jupyterlab/launcher-extension": true + } } diff --git a/etc/scripts/clean-jupyterlab.sh b/etc/scripts/clean-jupyterlab.sh index 3d16d7afa..f54bfd084 100755 --- a/etc/scripts/clean-jupyterlab.sh +++ b/etc/scripts/clean-jupyterlab.sh @@ -79,7 +79,10 @@ echo " " echo "Installing/Updating JupyterLab" pip install --upgrade pip wheel pip install --upgrade tornado -pip install --upgrade "jupyterlab$LAB_VERSION" +if [ "$LAB_VERSION" != "==dev" ] +then + pip install --upgrade "jupyterlab$LAB_VERSION" +fi echo " " echo "Installing Xeus kernel" @@ -101,9 +104,10 @@ jupyter --version echo " " jupyter kernelspec list echo " " -jupyter serverextension list -echo " " jupyter server extension list echo " " -jupyter labextension list -echo " " +if [ "$LAB_VERSION" != "==dev" ] +then + jupyter labextension list + echo " " +fi diff --git a/packages/code-snippet/package.json b/packages/code-snippet/package.json index 897d7fc38..ddd35cec1 100644 --- a/packages/code-snippet/package.json +++ b/packages/code-snippet/package.json @@ -38,7 +38,9 @@ "watch:src": "tsc -w", "watch:labextension": "jupyter labextension watch .", "lab:install": "jupyter labextension install --no-build", - "lab:uninstall": "jupyter labextension uninstall --no-build" + "lab:uninstall": "jupyter labextension uninstall --no-build", + "link:dev": "yarn link @jupyterlab/builder", + "unlink:dev": "yarn unlink @jupyterlab/builder" }, "dependencies": { "@elyra/metadata-common": "3.9.0-dev", diff --git a/packages/code-viewer/package.json b/packages/code-viewer/package.json index 1c747d407..acff4ea91 100644 --- a/packages/code-viewer/package.json +++ b/packages/code-viewer/package.json @@ -38,7 +38,9 @@ "watch:src": "tsc -w", "watch:labextension": "jupyter labextension watch .", "lab:install": "jupyter labextension install --no-build", - "lab:uninstall": "jupyter labextension uninstall --no-build" + "lab:uninstall": "jupyter labextension uninstall --no-build", + "link:dev": "yarn link @jupyterlab/builder", + "unlink:dev": "yarn unlink @jupyterlab/builder" }, "dependencies": { "@jupyterlab/application": "^3.4.0", diff --git a/packages/metadata-common/src/MetadataEditor.tsx b/packages/metadata-common/src/MetadataEditor.tsx index e952702cb..c0e155f13 100644 --- a/packages/metadata-common/src/MetadataEditor.tsx +++ b/packages/metadata-common/src/MetadataEditor.tsx @@ -638,9 +638,7 @@ export class MetadataEditor extends ReactWidget { ''}`; } const error = this.displayName === '' && this.invalidForm; - const onKeyPress: React.KeyboardEventHandler = ( - event: React.KeyboardEvent - ) => { + const onKeyPress = (event: React.KeyboardEvent): void => { const targetElement = event.nativeEvent.target as HTMLElement; if (event.key === 'Enter' && targetElement?.tagName !== 'TEXTAREA') { this.saveMetadata(); diff --git a/packages/metadata/package.json b/packages/metadata/package.json index b575ea942..8462e91bc 100644 --- a/packages/metadata/package.json +++ b/packages/metadata/package.json @@ -38,7 +38,9 @@ "watch:src": "tsc -w", "watch:labextension": "jupyter labextension watch .", "lab:install": "jupyter labextension install --no-build", - "lab:uninstall": "jupyter labextension uninstall --no-build" + "lab:uninstall": "jupyter labextension uninstall --no-build", + "link:dev": "yarn link @jupyterlab/builder", + "unlink:dev": "yarn unlink @jupyterlab/builder" }, "dependencies": { "@elyra/metadata-common": "3.9.0-dev", diff --git a/packages/pipeline-editor/package.json b/packages/pipeline-editor/package.json index 70e581c64..423e1b9b9 100644 --- a/packages/pipeline-editor/package.json +++ b/packages/pipeline-editor/package.json @@ -40,7 +40,9 @@ "watch:src": "tsc -w", "watch:labextension": "jupyter labextension watch .", "lab:install": "jupyter labextension install --no-build", - "lab:uninstall": "jupyter labextension uninstall --no-build" + "lab:uninstall": "jupyter labextension uninstall --no-build", + "link:dev": "yarn link @jupyterlab/builder", + "unlink:dev": "yarn unlink @jupyterlab/builder" }, "dependencies": { "@elyra/metadata-common": "3.9.0-dev", diff --git a/packages/pipeline-editor/src/PipelineEditorWidget.tsx b/packages/pipeline-editor/src/PipelineEditorWidget.tsx index dacfccbce..d050f059e 100644 --- a/packages/pipeline-editor/src/PipelineEditorWidget.tsx +++ b/packages/pipeline-editor/src/PipelineEditorWidget.tsx @@ -99,7 +99,7 @@ export const commandIDs = { submitNotebook: 'notebook:submit', addFileToPipeline: 'pipeline-editor:add-node', refreshPalette: 'pipeline-editor:refresh-palette', - openViewer: 'elyra-code-viewer:open' + openViewer: 'code-viewer:open' }; const getAllPaletteNodes = (palette: any): any[] => { diff --git a/packages/python-editor/package.json b/packages/python-editor/package.json index f5a34ef61..0907109f3 100644 --- a/packages/python-editor/package.json +++ b/packages/python-editor/package.json @@ -38,7 +38,9 @@ "watch:src": "tsc -w", "watch:labextension": "jupyter labextension watch .", "lab:install": "jupyter labextension install --no-build", - "lab:uninstall": "jupyter labextension uninstall --no-build" + "lab:uninstall": "jupyter labextension uninstall --no-build", + "link:dev": "yarn link @jupyterlab/builder", + "unlink:dev": "yarn unlink @jupyterlab/builder" }, "dependencies": { "@elyra/script-editor": "3.9.0-dev", diff --git a/packages/r-editor/package.json b/packages/r-editor/package.json index 3dc2bb5e6..20692a92d 100644 --- a/packages/r-editor/package.json +++ b/packages/r-editor/package.json @@ -38,7 +38,9 @@ "watch:src": "tsc -w", "watch:labextension": "jupyter labextension watch .", "lab:install": "jupyter labextension install --no-build", - "lab:uninstall": "jupyter labextension uninstall --no-build" + "lab:uninstall": "jupyter labextension uninstall --no-build", + "link:dev": "yarn link @jupyterlab/builder", + "unlink:dev": "yarn unlink @jupyterlab/builder" }, "dependencies": { "@elyra/script-editor": "3.9.0-dev", diff --git a/packages/theme/package.json b/packages/theme/package.json index f5d5477c8..f5a4913fa 100644 --- a/packages/theme/package.json +++ b/packages/theme/package.json @@ -38,7 +38,9 @@ "watch:src": "tsc -w", "watch:labextension": "jupyter labextension watch .", "lab:install": "jupyter labextension install --no-build", - "lab:uninstall": "jupyter labextension uninstall --no-build" + "lab:uninstall": "jupyter labextension uninstall --no-build", + "link:dev": "yarn link @jupyterlab/builder", + "unlink:dev": "yarn unlink @jupyterlab/builder" }, "dependencies": { "@elyra/ui-components": "3.9.0-dev", diff --git a/setup.py b/setup.py index 29aff5cab..dab8d379b 100644 --- a/setup.py +++ b/setup.py @@ -32,12 +32,12 @@ data_files_spec = [ ("etc/jupyter/jupyter_notebook_config.d", "etc/config/jupyter_notebook_config.d", "*.json"), ("etc/jupyter/jupyter_server_config.d", "etc/config/jupyter_server_config.d", "*.json"), + ("etc/jupyter/labconfig", "etc/config/settings", "*.json"), ("share/jupyter/metadata/runtime-images", "etc/config/metadata/runtime-images", "*.json"), ("share/jupyter/metadata/component-catalogs", "etc/config/metadata/component-catalogs", "*.json"), # deprecated ("share/jupyter/components", "etc/config/components", "*.json"), # deprecated ("share/jupyter/components/kfp/", "etc/config/components/kfp", "*.yaml"), # deprecated ("share/jupyter/components/airflow/", "etc/config/components/airflow", "*.py"), # deprecated - ("share/jupyter/lab/settings", "etc/config/settings", "*.json"), ("share/jupyter/labextensions", "dist/labextensions", "**"), ] @@ -75,9 +75,9 @@ "jupyter_client>=6.1.7", "jupyter-packaging>=0.10", "jupyter_server>=1.7.0", - "jupyterlab>=3.4.0", + "jupyterlab>=3.4.0", # comment out to use local jupyterlab + "jupyterlab-lsp>=3.8.0", # comment out to use local jupyterlab "jupyterlab-git~=0.32", # Avoid breaking 1.x changes - "jupyterlab-lsp>=3.8.0", "jupyter-resource-usage>=0.5.1", "MarkupSafe>=2.1", "minio>=7.0.0", diff --git a/tsconfig.base.json b/tsconfig.base.json index 6966db86c..82cf4f4ff 100644 --- a/tsconfig.base.json +++ b/tsconfig.base.json @@ -25,6 +25,9 @@ "allowSyntheticDefaultImports": true, "esModuleInterop": true, + /* Uncomment to use local jupyterlab instance */ + // "paths": { "@jupyterlab/*": ["../jupyterlab/packages/*"] }, + /* Logging */ "preserveWatchOutput": true } From bd9da8e9b5562855a34d15b59508a1c919f9bd93 Mon Sep 17 00:00:00 2001 From: Alex Bozarth Date: Wed, 4 May 2022 23:18:33 -0500 Subject: [PATCH 2/5] fix CI --- .github/workflows/build.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 128f121d3..9976d810a 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -114,6 +114,7 @@ jobs: key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }} - name: Build run: | + pip install "jupyterlab>=3.4.0" make build-dependencies make yarn-install make build-ui @@ -143,6 +144,7 @@ jobs: key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }} - name: Build run: | + pip install "jupyterlab>=3.4.0" make build-dependencies make yarn-install make build-ui From 67b0c2fefa3f83e21d0da3343caa22f580ae2964 Mon Sep 17 00:00:00 2001 From: Alex Bozarth Date: Wed, 4 May 2022 23:34:08 -0500 Subject: [PATCH 3/5] only run status prints if jupyter is installed --- etc/scripts/clean-jupyterlab.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/etc/scripts/clean-jupyterlab.sh b/etc/scripts/clean-jupyterlab.sh index f54bfd084..c6087a463 100755 --- a/etc/scripts/clean-jupyterlab.sh +++ b/etc/scripts/clean-jupyterlab.sh @@ -100,14 +100,14 @@ conda install -y r r-essentials r-irkernel conda install -y -c conda-forge r-languageserver echo " " -jupyter --version -echo " " -jupyter kernelspec list -echo " " -jupyter server extension list -echo " " if [ "$LAB_VERSION" != "==dev" ] then + jupyter --version + echo " " + jupyter kernelspec list + echo " " + jupyter server extension list + echo " " jupyter labextension list echo " " fi From 4c382ac4f397d1b0adda60976de8673102468e3e Mon Sep 17 00:00:00 2001 From: Alex Bozarth Date: Thu, 5 May 2022 11:32:58 -0500 Subject: [PATCH 4/5] remove accidentally commited test code --- packages/pipeline-editor/src/PipelineEditorWidget.tsx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/pipeline-editor/src/PipelineEditorWidget.tsx b/packages/pipeline-editor/src/PipelineEditorWidget.tsx index d050f059e..dacfccbce 100644 --- a/packages/pipeline-editor/src/PipelineEditorWidget.tsx +++ b/packages/pipeline-editor/src/PipelineEditorWidget.tsx @@ -99,7 +99,7 @@ export const commandIDs = { submitNotebook: 'notebook:submit', addFileToPipeline: 'pipeline-editor:add-node', refreshPalette: 'pipeline-editor:refresh-palette', - openViewer: 'code-viewer:open' + openViewer: 'elyra-code-viewer:open' }; const getAllPaletteNodes = (palette: any): any[] => { From 49b828123fc825fa5c5651563d93b648a91d9f23 Mon Sep 17 00:00:00 2001 From: Alex Bozarth Date: Mon, 9 May 2022 12:23:42 -0500 Subject: [PATCH 5/5] address doc review --- .../developer_guide/development-workflow.md | 27 ++++++++++--------- 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/docs/source/developer_guide/development-workflow.md b/docs/source/developer_guide/development-workflow.md index c9a24e0b1..a082e6e36 100644 --- a/docs/source/developer_guide/development-workflow.md +++ b/docs/source/developer_guide/development-workflow.md @@ -199,29 +199,32 @@ and [quay.io](https://quay.io/repository/elyra/elyra?tab=tags). ### Developing Elyra against the Jupyterlab source repo Sometimes it is useful to develop Elyra against a local build of Jupyterlab. To use a local build of Jupyterlab use the -following steps. +following steps in the same python environment. -1. Uninstall any pip installations of jupyterlab. You can use `etc/scripts/clean-jupyterlab.sh --version dev` as - mentioned above with `--version dev` to not reinstall jupyterlab at the end of the script. +1. Uninstall any pip installations of Jupyterlab. You can use `etc/scripts/clean-jupyterlab.sh --version dev` as + mentioned above with `--version dev` to not reinstall Jupyterlab at the end of the script. -2. Build your local repo of jupyterlab. Uninstalling in the previous step will also wipe any previous installations of a - local build. +2. Build your local repo of Jupyterlab, step-by-step instructions can be found in the + [Jupyterlab documentation](https://jupyterlab.readthedocs.io/en/latest/developer/contributing.html#installing-jupyterlab). + Uninstalling in the previous step will also wipe any previous installations of a local build. -3. `cd` to the `builder/` directory in your jupyterlab repo and run `yarn link`. Elyra's `Makefile` will use this yarn +3. `cd` to the `builder/` directory in your Jupyterlab repo and run `yarn link`. The Elyra `Makefile` will use this yarn link in step 6. -4. Uncomment the following line in `tsconfig.base.json` to tell Typescript to use the local jupyterlab packages when building: +4. In your Elyra repo, uncomment the following line in `tsconfig.base.json` to tell Typescript to use the local + Jupyterlab packages when building: ```"paths": { "@jupyterlab/*": ["../jupyterlab/packages/*"] },``` -5. Comment out `jupyterlab` and `jupyterlab-lsp` in the `install_requires` section of `setup.py`. This will prevent - jupyterlab being pip installed during the Elyra build. Note: `jupyterlab-lsp` also pip installs jupyterlab when installed +5. Comment out `jupyterlab` and `jupyterlab-lsp` in the `install_requires` section of `setup.py` in your Elyra repo. + This will prevent Jupyterlab from being pip installed during the Elyra build. + Note: `jupyterlab-lsp` also pip installs Jupyterlab when installed 6. Run `make install-dev` to install Elyra using the linked `@jupyterlab/builder` from step 3. -7. You can now start jupyterlab by running `jupyter lab --dev-mode --extensions-in-dev-mode`, this will automatically watch - for changes in the jupyterlab repo. To also watch for changes in Elyra run `make watch` in a separate terminal in the - same Python environment. +7. You can now start Jupyterlab by running `jupyter lab --dev-mode --extensions-in-dev-mode`, this will automatically + watch for changes in the Jupyterlab repo. To also watch for changes in Elyra run `make watch` in a separate terminal + in the same Python environment. When you want to switch back to developing Elyra against a Jupyterlab release, you just have to undo the comments in steps 4 and 5 and rebuild with `make clean install`