Rebaseline from master #2
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Don't explicitly break py2 support * Typo * Typo
…the Env Escape server (#960) We were currently not reflecting possible values passed using PYTHONPATH or programmatically added to the interpreter launching the environment escape server.
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.7 to 1.14.9. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.14.7...v1.14.9) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#972) * Fix "Too many symbolic links" error when using Conda + Batch on MacOS * Ran black
* Fixed dual distribution import in some cases; fixed get_plugin_cli extension If sys.path contains the same directory multiple times, metadata.distributions() will list a package multiple times which breaks the import mechanism we have. This is addressed by skipping duplicate packages. get_plugin_cli for extensions wasn't evaluated late enough therefore not allowing additional parameters to be added. * Fix issue with metaflow_extensions and Conda environment In a Conda environment, it is not possible to re-resolve all the metaflow_extension packages since they appear as a single directory. In that case, we resolve on the INFO file to give us the proper information. Also fixed the way metaflow_extensions are added to the Conda environment to avoid leaking more information than needed. * Validate configuration files for extensions and de-duplicate for internal ns packages * Update packaging of metaflow_extensions packages Packaging is now handled directly by extension_support.py and each distribution/package can define its own suffixes to include. * Rework the import mechanism for modules contained in metaflow_extensions This change does away with the deprecated load_module and also improves handling of loading children modules and _orig modules * Address comments * Squash merge of origin/master * Fix issue when packages were left out * Merge get_pinned_conda_libs * Fix issue with __init__.py in non-distribution packages * Moving card import changes on top of Romain's branch (#973) * Moving card import changes on top of Romain's branch - tiny bug fix to extensions. - Added card related refactor to support mfextinit and regular package. - removing card packaging related code in decorator + everywhere else. * Tiny bug fix on top of romain's new changes. * removing unneccessary logic * Added a bit more inline documentation and addressed documentation comments * Properly handle parsing of package requirements * Fix Pylint errors in some case of metaflow_extensions module aliasing * Properly handle case of files at the root of PYTHONPATH package * Tests for Extensions (#978) * Added tests for extensions. Checking if they work. * dummy commit to see if things work. * fix * debug * bug fix. * possible fix. * tweeking context * Bug fix to tests. * dummy commit * bug fix * Added extension test to core tests. - remove seperate gh action * removing files. * fix * added extension test to py3 context - remove redundant complexity. Co-authored-by: Valay Dave <valaygaurang@gmail.com>
* Fixing some hacky plumbing for card tests. - Added `--file` arg to card list cli - Using files to get the information written by cli - changes to test to use files instead of stdout - Piping stderrors to stderrr to capture card not found errors. * Removing `\n` from all card tests assertions - `\n` was there because earlier we read stdout. - Now since we read files, it will not be needed.
… 3.6+ (#988) * Use importlib_metadata 2.8.3 for Python 3.4 and 3.5 but 4.8.3 for Python 3.6+ This is required because in importlib_metadata 3.4.0, a field called `_normalized_name` was introduced and is relied on by later versions of importlib_metadata. When importlib_metadata looks for distributions, it queries all registered/installed importlib_metadata and if an older version (like 2.8.3) returns something that does not have this field, it causes a crash. * Add __init__.py files to make the vendored package non namespace
…from S3. (#991) Co-authored-by: kgullikson <kgullikson@sparkcognition.com>
* Add an option to pass a `role` argument to S3() to provide a role for the S3 client * Move to partial to be able to pickle ops * Address comments
* Bug fix for datetime_indexes. * ran black
Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6. - [Release notes](https://github.com/substack/minimist/releases) - [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6) --- updated-dependencies: - dependency-name: minimist dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* first attempt * fix _set_env param typo * add assertions for tags to current singleton test
* Dispatch Metaflow flows to Argo Workflows * Remove spurious files * Make black happy * Make black happy again * Remove spurious TODOs * Handle non-numeric task ids * Fix typos * don't actively automount service account tokens * make black happy * Update @kubernetes * remove spurious commits * fix issue with hyphens * raise memory for tests * change limits to requests * pin back test resources * update refresh timeout * stylistic nits * Add lint check * make black happy * support gpus * drop extra bracket * fix
* Don't sleep if not retrying S3 operations * Remove unused env variable * VSCode is too smart
* Alternate way of adding metadata to cloned tasks This adds it on the client side * Clean up metadata passing. Also optimize use within the client * Typo * Fix black test * Keep black happy * Added tests * Ignore namespace when getting origin task
…1027) Bumps [cross-fetch](https://github.com/lquixada/cross-fetch) from 3.1.4 to 3.1.5. - [Release notes](https://github.com/lquixada/cross-fetch/releases) - [Commits](lquixada/cross-fetch@v3.1.4...v3.1.5) --- updated-dependencies: - dependency-name: cross-fetch dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* current.pathspec to return None when used outside Flow * formatted with black
* Enable EMCECS * Enable MinIO & EMC-ECS blob stores for Metaflow * fix InvalidRange issue
* stash system info as metadata (in addition to system_tags today) * improve * avoid parsing on ":" for user id * remove date - it's not well defined and created_at is available * python_version is in metadata now, and can vary between orig and resume runs * address CR comments * add the comments
…1180) * issue 1040 fix: apply _sanitize to template names in Argo workflows * apply _sanitize to foreach step template names in Argo workflows
Log the message and the exit code.
…1194) Bumps [loader-utils](https://github.com/webpack/loader-utils) from 3.2.0 to 3.2.1. - [Release notes](https://github.com/webpack/loader-utils/releases) - [Changelog](https://github.com/webpack/loader-utils/blob/master/CHANGELOG.md) - [Commits](webpack/loader-utils@v3.2.0...v3.2.1) --- updated-dependencies: - dependency-name: loader-utils dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Fix `._orig` access for submodules for MF extensions This addresses an issue where submodules like `mymodule._orig.submodule` would not load properly. This will properly load things like `from .foofa import xyz` from a module or sub-module in the original location but currently does not work if the import path is absolute. * Add test that orig module is accessible (#1192) * Black format Co-authored-by: Tom Furmston <tfurmston@googlemail.com>
Co-authored-by: Adam Merberg <adam.merberg@equilibriumenergy.com>
* Fixed typos, spelling, and grammar * Fixed several simple PEP warnings * Reverted changes to _vendor folder * Black formatting * Black linting * Reset _vendor folder after black formatting
* Pin GH tests to Ubuntu 20.04 Ubuntu latest which maps to 22.04 doesn't support Py 3.5/3.6 on GH actions * Update test.yml
* Set gpu resources correctly "--with kubernetes" GPU resources were not getting propagated when used "--with kubernetes". Now, it does. Testing Done: - Reproduced the problem creating a step with a decorator @resources(gpu=1) and running it "--with kubernetes". The pods were scheduled without gpus. - Verified that after the fix, the pods were scheduled with gpus (request and limit). * Incorporate review comments + fix black.
* Clean up configuration variables * Fix bug in argo/airflow; update default value of AZURE_STORAGE_WORKLOAD_TYPE * Forgot file * Clean up what values are passed down Now, if a value is the default, it is not propagated by default since it can be reconstructed simply from the code. With this change, we basically propagate (by default) only those values that require some external knowledge (env vars, configuration file) to set. This reduces the set of values propagated to only the ones the user overrides. * Remove propagate flag; change name of SERVICE_URL for batch * Fixed more instances of INTERNAL_SERVICE_URL; fixed comments
* GCP * card rendering to tolerate runs not tagged with metaflow_version * patch GCP impl to match new includefile impl * fix validate function bug * fix GS includefile merge * fix GS workload type config
* Bump version; remove R tests * Remove more R test
…1186) * Deal with transient errors (like SlowDowns) more effectively for S3 In the previous incantantion, SlowDown errors were treated as a regular error and everything was retried (making things worse). With this change, we continue retrying the operations that were unsuccessful (and only those). Also use a better internal boto retry policy (unless specified by the user). Modified the tests to inject failures to be able to test the functionality. Tests now have a failure injection rate of 0, 10, 50 or 90 percent. * Added more comments * Addressed comments; added more comments in code
* Move files -- no code change * Fixups for datatools and datastores * Pure moving of code around -- no code modification This commit can be safely ignored when reviewing code; it has no semantic change although it clearly does not work. * Modify CLI commands to make functional
* Fix regression causing CL tool to not work. Update version for immediate release * Fix tox to keep github actions happy
Bumps [qs](https://github.com/ljharb/qs) from 6.5.2 to 6.5.3. - [Release notes](https://github.com/ljharb/qs/releases) - [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md) - [Commits](ljharb/qs@v6.5.2...v6.5.3) --- updated-dependencies: - dependency-name: qs dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add support for Kubernetes tolerations * Revert get_docker_registry import * Fix KUBERNETES_TOLERATIONS default value * Update toleration example * Remove KUBERNETES_NODE_SELECTOR in kubernetes_job.py * Fix black code style * Add param doc to KubernetesDecorator * Fix typo * Serialize tolerations in runtime_step_cli * Fix KUBERNETES_TOLERATIONS config * Fix node_selector env var in the kubernetes decorator * JSON loads KUBERNETES_TOLERATIONS in kubernetes_decorator init * Parse node_selector and tolerations in the decorator * Update comment * Validate tolerations object in kubernetes_decorator.py * Use hard coded tolerations attribute_map * Fix black code style * String formatting compatible with python 3.5 * Use V1Toleration.attribute_map to validate tolerations * Fix black lint * Improve error handling * Fix black lint
Co-authored-by: Dan <daniel.corvesor@medtronic.com>
* Fix CVE-2007-4559 (tar.extractall) See #1177 for more details * Address comment
* Support .conda packages * fix black issues
npow
added a commit
that referenced
this pull request
Jan 12, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.