ci: update testing pipeline for notebooks #1345

loferris · 2022-02-03T04:38:30Z

This PR updates the notebook template to reflect parallelized tests for notebooks repos. We have also included one sample notebook with which to test the template.

parthea · 2022-03-29T21:10:38Z

Here is the output for the failing session

nox > flake8 synthtool tests autosynth integration_tests
synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/execute_changed_notebooks_helper.py:76:86: E231 missing whitespace after ','
synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/execute_changed_notebooks_helper.py:152:9: F841 local variable 'operation_result' is assigned to but never used
synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/execute_changed_notebooks_helper.py:212:15: F541 f-string is missing placeholders
synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/utils/utils.py:16:1: F401 'datetime.datetime' imported but unused
synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/utils/utils.py:17:1: F401 'typing.Optional' imported but unused
synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/utils/utils.py:18:1: F401 'google.cloud.storage' imported but unused
synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/utils/utils.py:19:1: F401 'google.cloud.aiplatform.utils' imported but unused
synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/utils/utils.py:20:1: F401 'google.auth.credentials as auth_credentials' imported but unused
nox > Command flake8 synthtool tests autosynth integration_tests failed with exit code 1
nox > Session lint-3.6 failed.

parthea · 2022-04-01T14:58:00Z

synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/CheckPythonVersion.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-# # Copyright 2021 Google LLC
+# # Copyright 2022 Google LLC


Please keep original copyright year. Applies throughout

resolved in latest commit

parthea · 2022-04-01T14:59:15Z

synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/cleanup/cleanup.py


+        print("")


Please remove print statement or add a comment to clarify the reason for keeping it.

Resolved in recent commit.

parthea · 2022-04-01T15:00:03Z

synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/cleanup/cleanup.py

+is_dry_run = False
+
+if is_dry_run:
+    print("Starting cleanup in dry run mode...")


is_dry_run is always False, please remove test code or add a comment to clarify the reason for having it.

Removed - I think the setting fixes an immediate problem but the print statement exists for future settings of that variable. However, I tend to not include code until it's supported. @ivanmkc may have more context!

...cp/templates/python_notebooks_testing_pipeline/.cloud-build/execute_changed_notebooks_cli.py

parthea · 2022-04-01T15:22:22Z

...tool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/execute_notebook_remote.py

+from google.api_core import operation
+
+CLOUD_BUILD_FILEPATH = ".cloud-build/notebook-execution-test-cloudbuild-single.yaml"
+TIMEOUT_IN_SECONDS = 86400


Can you confirm that a 24 hour timeout is correct? It's better to have a lower timeout to limit resource usage in case the build is stuck.

Yes, this is a requirement for the tests at present.

parthea · 2022-04-01T15:23:45Z

...tool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/execute_notebook_remote.py

+    # Print the in-progress operation
+    # print("IN PROGRESS:")
+    # print(operation.metadata)
+
+    # Print the completed status
+    # print("RESULT:", result.status)


nit: add a TODO comment if there is action needed for a developer, otherwise remove commented out code

Added in recent commit!

parthea · 2022-04-01T15:24:52Z

...ython_notebooks_testing_pipeline/.cloud-build/notebook-execution-test-cloudbuild-single.yaml

+    - 'python3 -m pip install -U pip && python3 -m pip freeze && python3 .cloud-build/execute_notebook_cli.py --notebook_source "${_NOTEBOOK_GCS_URI}" --output_file_or_uri "${_NOTEBOOK_OUTPUT_GCS_URI}"'
+    env:
+    - 'IS_TESTING=1'
+timeout: 86400s


Similar to above. Could we lower the timeout and increase as needed?

Not as present, but we can check in about it again with notebooks folks!

parthea · 2022-04-01T15:26:20Z

...ython_notebooks_testing_pipeline/.cloud-build/notebook-execution-test-cloudbuild-single.yaml

+timeout: 86400s
+options:
+  pool:
+    name: ${_PRIVATE_POOL_NAME}


Can you clarify where the environment variables are set? It may be helpful to add a README or provide a link to existing documentation to clarify what environment variables are needed.

I'd like to add that as a follow up issue, but I agree.

parthea · 2022-04-01T15:28:48Z

synthtool/protos/metadata_pb2.py

@@ -3,6 +3,7 @@
 # source: metadata.proto


If this change is not necessary for your PR, please could you change it in a separate PR?

Removed in recent commit.

…ud-build/execute_changed_notebooks_cli.py Co-authored-by: Anthonios Partheniou <partheniou@google.com>

* update testing pipeline * kokoro fix * fixing linting errors * Update synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/execute_changed_notebooks_cli.py Co-authored-by: Anthonios Partheniou <partheniou@google.com> * Update synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/execute_changed_notebooks_cli.py Co-authored-by: Anthonios Partheniou <partheniou@google.com> * resolving license year * removing redundant print statements * adding developer todos * reverting protos * resetting protos to main Co-authored-by: Anthonios Partheniou <partheniou@google.com>

* adding notebook template function * changing template path * changing template path * chore: add googleapis/yoshi-python to CODEOWNERS (#1380) * chore(python): add E231 to .flake8 ignore list (#1379) * chore: Enable Size-Label bot in all googleapis Python notebook testing (#1384) * chore: Enable Size-Label bot in all googleapis Python notebook testing repositories Auto-label T-shirt size indicator should be assigned on every new pull request in all googleapis Python notebook testing repositories * Remove product Remove product since it is by default true * chore(python): Enable size-label bot (#1383) * chore: Enable Size-Label bot in all googleapis Python repositories Auto-label T-shirt size indicator should be assigned on every new pull request in all googleapis Python repositories * Remove product Remove product since it is by default true Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(python): update .pre-commit-config.yaml to use black==22.3.0 (#1378) Co-authored-by: Tim Swast <swast@google.com> * chore(python): Enable size-label bot (#1385) * chore: remove dependency on google-api-core (#1372) * chore(deps): update dependency google-api-core to v2.7.1 * chore: remove dependency on google-api-core Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(deps): update dependency numpy to v1.22.3 (#1387) * chore(deps): disable dependency dashboard (#1223) Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(deps): update dependency pandas to v1.4.1 (#1258) Co-authored-by: Anthonios Partheniou <partheniou@google.com> * build(deps): bump ipython (#1336) Bumps [ipython](https://github.com/ipython/ipython) from 7.0 to 7.16.3. - [Release notes](https://github.com/ipython/ipython/releases) - [Commits](ipython/ipython@7.0.0...7.16.3) --- updated-dependencies: - dependency-name: ipython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: remove use of googleapis-discovery (#1280) * chore: remove use of googleapis-discovery * remove constant and method for googleapis-discovery Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore: add classifiers for python 3.7, 3.8 and 3.9 (#1247) added Support for python 3.7, 3.8, 3.9 Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(python): refactor unit / system test dependency install (#1294) * chore(python): refactor unit / system test dependency install Closes #1185. * chore: use editable installs for local deps * chore: don't install deps using '-e' * chore: deprecate 'unit_test_external_dependencies' * fix: install standard + main unit test deps together FBO pip resolver. Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(python_notebooks): update dependency black to v22 (#1396) * chore(deps): update dependency ipython to v8 (#1397) Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore: Enable Size-Label bot in all googleapis NodeJs repositories (#1382) * chore: Enable Size-Label bot in all googleapis NodeJs repositories Auto-label T-shirt size indicator should be assigned on every new pull request in all googleapis NodeJs repositories * Remove product Remove product since it is by default true * chore(deps): update dependency setuptools to v61 (#1398) Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(deps): update dependency org.sonatype.plugins:nexus-staging-maven-plugin to v1.6.12 (#1388) Co-authored-by: Neenu Shaji <Neenu1995@users.noreply.github.com> * ci: update testing pipeline for notebooks (#1345) * update testing pipeline * kokoro fix * fixing linting errors * Update synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/execute_changed_notebooks_cli.py Co-authored-by: Anthonios Partheniou <partheniou@google.com> * Update synthtool/gcp/templates/python_notebooks_testing_pipeline/.cloud-build/execute_changed_notebooks_cli.py Co-authored-by: Anthonios Partheniou <partheniou@google.com> * resolving license year * removing redundant print statements * adding developer todos * reverting protos * resetting protos to main Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(deps): update actions/setup-python action to v3 (#1395) Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore: upgrade black in noxfile.py to 22.3.0 (#1377) * chore: upgrade black to 22.3.0 * update lint also * run blacken session * ci: use click>8.0 for blacken/lint sessions * chore: fix typo Co-authored-by: nicain <nicholascain@google.com> * rebase error Co-authored-by: nicain <nicholascain@google.com> * chore(deps): update dependency pandas to v1.4.2 (#1400) * chore(deps): update dependency setuptools to v61.3.1 (#1399) Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(deps): update dependency nbqa to v1.3.1 (#1391) Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(deps): update dependency protobuf to v3.19.4 (#1389) Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(deps): update dependency protobuf to v3.20.0 (#1403) * chore(deps): update dependency setuptools to v62 (#1406) * chore(python): Use python 3.10.4 base image for post processor (#1405) * chore(deps): update actions/setup-node action to v3 (#1393) Co-authored-by: Jeffrey Rennie <rennie@google.com> * chore(python): add license header to auto-label.yaml (#1404) * chore: improve logic to configure release-please for previous major versions (#1408) * fix: allow version.py files without the library version * chore: remove obsolete file * chore: run black Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore: Enable Size-Label bot in all googleapis Java repositories (#1381) * chore: Enable Size-Label bot in all googleapis Java repositories Auto-label T-shirt size indicator should be assigned on every new pull request in all googleapis Java repositories * Remove product Remove product since it is by default true * add license header Co-authored-by: Neenu Shaji <Neenu1995@users.noreply.github.com> * chore(deps): update actions/checkout action to v3 (#1392) Co-authored-by: Jeffrey Rennie <rennie@google.com> * chore: run tests with Python 3.10 (#1407) Co-authored-by: Anthonios Partheniou <partheniou@google.com> * chore(deps): update dependency python to v3.10.4 (#1386) Co-authored-by: Anthonios Partheniou <partheniou@google.com> Co-authored-by: Jeffrey Rennie <rennie@google.com> * fix: clarify the gax-nodejs usage in README (#1352) Co-authored-by: Benjamin E. Coe <bencoe@google.com> Co-authored-by: Jeffrey Rennie <rennie@google.com> * Revert "fix: clarify the gax-nodejs usage in README (#1352)" (#1409) This reverts commit e1557e4. * chore(deps): update dependency pyupgrade to v2.32.0 (#1412) * chore(deps): update dependency nbconvert to v6.5.0 (#1414) * build: make ci testing conditional on engines field in package.json, move configs to Node 12 (#1418) * build: make ci testing conditional on engines field in package.json, move configs to Node 12 Co-authored-by: Benjamin E. Coe <bencoe@google.com> * build: sdd srs yaml file (#1419) * build: add sync-repo-settings and change branch protection * update names for template * adding type annotation Co-authored-by: Anthonios Partheniou <partheniou@google.com> Co-authored-by: losalex <90795544+losalex@users.noreply.github.com> Co-authored-by: Tim Swast <swast@google.com> Co-authored-by: WhiteSource Renovate <bot@renovateapp.com> Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Brent Shaffer <betterbrent@google.com> Co-authored-by: Anurag Kumar <mailanu98@gmail.com> Co-authored-by: Tres Seaver <tseaver@palladion.com> Co-authored-by: Neenu Shaji <Neenu1995@users.noreply.github.com> Co-authored-by: nicain <nicholascain@google.com> Co-authored-by: Jeffrey Rennie <rennie@google.com> Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Co-authored-by: Summer Ji <summerji@google.com> Co-authored-by: Benjamin E. Coe <bencoe@google.com> Co-authored-by: sofisl <55454395+sofisl@users.noreply.github.com>

update testing pipeline

82eee5d

loferris requested review from bcoe, chingor13 and SurferJeffAtGoogle as code owners February 3, 2022 04:38

loferris requested a review from a team February 3, 2022 04:38

Merge branch 'master' into notebooks_parallel_ci

1ba8a4e

loferris changed the title ~~feat: update testing pipeline for notebooks~~ WIP feat: update testing pipeline for notebooks Feb 3, 2022

loferris changed the title ~~WIP feat: update testing pipeline for notebooks~~ feat: update testing pipeline for notebooks Feb 3, 2022

loferris requested a review from ivanmkc February 3, 2022 04:45

kokoro fix

0d02565

fixing linting errors

f1b5fef

loferris requested a review from a team as a code owner March 31, 2022 21:03

Merge branch 'master' into notebooks_parallel_ci

77c969e

loferris requested a review from parthea March 31, 2022 22:28

parthea self-assigned this Apr 1, 2022

parthea approved these changes Apr 1, 2022

View reviewed changes

parthea changed the title ~~feat: update testing pipeline for notebooks~~ ci: update testing pipeline for notebooks Apr 1, 2022

parthea assigned loferris and unassigned parthea Apr 1, 2022

loferris and others added 9 commits April 1, 2022 11:19

Merge branch 'master' into notebooks_parallel_ci

e48179e

Update synthtool/gcp/templates/python_notebooks_testing_pipeline/.clo…

9b2580a

…ud-build/execute_changed_notebooks_cli.py Co-authored-by: Anthonios Partheniou <partheniou@google.com>

Update synthtool/gcp/templates/python_notebooks_testing_pipeline/.clo…

538d391

…ud-build/execute_changed_notebooks_cli.py Co-authored-by: Anthonios Partheniou <partheniou@google.com>

resolving license year

1c286b3

removing redundant print statements

a43dcd4

adding developer todos

0d2f80c

reverting protos

777282b

resetting protos to main

2e04f04

Merge branch 'master' into notebooks_parallel_ci

7c20a33

loferris merged commit ddcf429 into googleapis:master Apr 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: update testing pipeline for notebooks #1345

ci: update testing pipeline for notebooks #1345

loferris commented Feb 3, 2022 •

edited

Loading

parthea commented Mar 29, 2022

parthea Apr 1, 2022

loferris Apr 1, 2022

parthea Apr 1, 2022

loferris Apr 1, 2022

parthea Apr 1, 2022

loferris Apr 1, 2022

parthea Apr 1, 2022

loferris Apr 1, 2022

parthea Apr 1, 2022

loferris Apr 1, 2022

parthea Apr 1, 2022

loferris Apr 1, 2022

parthea Apr 1, 2022

loferris Apr 1, 2022

parthea Apr 1, 2022

loferris Apr 1, 2022

ci: update testing pipeline for notebooks #1345

ci: update testing pipeline for notebooks #1345

Conversation

loferris commented Feb 3, 2022 • edited Loading

parthea commented Mar 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

loferris commented Feb 3, 2022 •

edited

Loading