Merge 2.5.4 #163
Closed
Merge 2.5.4 #163
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ge (Netflix#614) * Allow Metaflow Custom packages to define additional suffixes to package This enables you to package non-python files in your metaflow_custom package and have them included when Metaflow creates the code package. * Addressed comment
* Add inputs to task_pre_step * Addressed comments * Forgot to remove an import
You can now specify a function that returns an AWS client. This is useful if you want to use something other than boto3 and can be used with the metaflow_custom mechanism to provide your own authentication
* Add S3 tests * Addressed comments
…flix#630) A parameter specification like - ``` Parameter(name="test_param", type=int, default=None) ``` will result in an error even though the default has been specified ``` Flow failed: The value of parameter test_param is ambiguous. It does not have a default and it is not required. ``` This PR fixes this issue.
* DRAFT: IncludeFile now returns the included file in the client and CLI THIS IS NOT FINISHED; DO NOT MERGE AS IS. * Fix the tests * Forgot to update type check for multiple encoding
* Add resource tags to AWS Batch jobs This PR assumes that Batch:TagResource is whitelisted for the role which submits the AWS Batch job. * propagate tags * add env var * better commens * add production token
…lix#607)" (Netflix#637) This reverts commit 0575f8d.
…flix#623) * Properly allow full toplevel module overrides in metaflow_custom * Modified test to follow new format * Improve module aliasing for metaflow_custom (Netflix#625)
* Handle parameters as `foreach vars` * Change fix to work more upstream The reason _find_input was not seeing the proper value for the parameters was because _init_parameters, which is supposed to initialize them, calls _get_parameters which needlessly calls input which calls _find_input. This removes the needless call to input which solves the problem and has the added benefit of being slightly more efficient. Co-authored-by: Romain Cledat <rcledat@netflix.com>
…flix#660) * Conda environment now applys underlying environment's decorators * Add tuples instead of unpacking with * for py2 compatibility
* Added test for exception handling. - Added Repr to fix error representation problem * Fix test doc string * nit fixes.
…x#674) * Fixes recursion error when METAFLOW_DEFAULT_ENVIRONMENT=conda * Add explanation
* Add README.md to PyPI package description * make black happy
* Don't explicitly break py2 support * Typo * Typo
…the Env Escape server (Netflix#960) We were currently not reflecting possible values passed using PYTHONPATH or programmatically added to the interpreter launching the environment escape server.
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.7 to 1.14.9. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.14.7...v1.14.9) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…Netflix#972) * Fix "Too many symbolic links" error when using Conda + Batch on MacOS * Ran black
* Fixed dual distribution import in some cases; fixed get_plugin_cli extension If sys.path contains the same directory multiple times, metadata.distributions() will list a package multiple times which breaks the import mechanism we have. This is addressed by skipping duplicate packages. get_plugin_cli for extensions wasn't evaluated late enough therefore not allowing additional parameters to be added. * Fix issue with metaflow_extensions and Conda environment In a Conda environment, it is not possible to re-resolve all the metaflow_extension packages since they appear as a single directory. In that case, we resolve on the INFO file to give us the proper information. Also fixed the way metaflow_extensions are added to the Conda environment to avoid leaking more information than needed. * Validate configuration files for extensions and de-duplicate for internal ns packages * Update packaging of metaflow_extensions packages Packaging is now handled directly by extension_support.py and each distribution/package can define its own suffixes to include. * Rework the import mechanism for modules contained in metaflow_extensions This change does away with the deprecated load_module and also improves handling of loading children modules and _orig modules * Address comments * Squash merge of origin/master * Fix issue when packages were left out * Merge get_pinned_conda_libs * Fix issue with __init__.py in non-distribution packages * Moving card import changes on top of Romain's branch (Netflix#973) * Moving card import changes on top of Romain's branch - tiny bug fix to extensions. - Added card related refactor to support mfextinit and regular package. - removing card packaging related code in decorator + everywhere else. * Tiny bug fix on top of romain's new changes. * removing unneccessary logic * Added a bit more inline documentation and addressed documentation comments * Properly handle parsing of package requirements * Fix Pylint errors in some case of metaflow_extensions module aliasing * Properly handle case of files at the root of PYTHONPATH package * Tests for Extensions (Netflix#978) * Added tests for extensions. Checking if they work. * dummy commit to see if things work. * fix * debug * bug fix. * possible fix. * tweeking context * Bug fix to tests. * dummy commit * bug fix * Added extension test to core tests. - remove seperate gh action * removing files. * fix * added extension test to py3 context - remove redundant complexity. Co-authored-by: Valay Dave <valaygaurang@gmail.com>
* Fixing some hacky plumbing for card tests. - Added `--file` arg to card list cli - Using files to get the information written by cli - changes to test to use files instead of stdout - Piping stderrors to stderrr to capture card not found errors. * Removing `\n` from all card tests assertions - `\n` was there because earlier we read stdout. - Now since we read files, it will not be needed.
… 3.6+ (Netflix#988) * Use importlib_metadata 2.8.3 for Python 3.4 and 3.5 but 4.8.3 for Python 3.6+ This is required because in importlib_metadata 3.4.0, a field called `_normalized_name` was introduced and is relied on by later versions of importlib_metadata. When importlib_metadata looks for distributions, it queries all registered/installed importlib_metadata and if an older version (like 2.8.3) returns something that does not have this field, it causes a crash. * Add __init__.py files to make the vendored package non namespace
…from S3. (Netflix#991) Co-authored-by: kgullikson <kgullikson@sparkcognition.com>
…tflix#987) * Add an option to pass a `role` argument to S3() to provide a role for the S3 client * Move to partial to be able to pickle ops * Address comments
* Bug fix for datetime_indexes. * ran black
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolving conflicts from upstream.
Most of conflicts are result of netflix/metaflow embracing black formatting 🎉
Major changes that resulted in code changes: