Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[release v1.2] Backport v1.6 (#1645)
* Data subsystem (#1526) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Pluck retry from flytekit and into sagemaker (#1411) * Remove retry from flytekit's setup.py and regenerate requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add to sagemaker Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove retry from sagemaker plugin requirements file Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Restore doc-requirements.txt Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Fix bad merge Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Update the pypi wait (#1554) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Stream Directories and Files using Flyte (#1512) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Make `FlyteFile` compatible with `Annotated[..., HashMethod]` (#1544) * fix: Make FlyteFile compatible with Annotated[..., HashMethod] See issue #3424 Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> * tests: Add test case for FlyteFile with HashMethod annotation Issue: #3424 Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> * fix: Use typing_extensions.Annotated for py3.8 compatibility Issue: #3424 Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> * fix: Use `get_args` and `get_origin` from typing_extensions for py3.8 compatibility Issue: #3424 Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> * fix(tests): Use fixture for local dummy file Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> --------- Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> * move FlyteSchema deprecation warning to initialization method (#1558) * move FlyteSchema deprecation warning to initialization method Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add pod_template and pod_template_name arguments for ContainerTask (#1515) * add pod_template and pod_template_name arguments for ContainerTask Signed-off-by: Felix Ruess <felix.ruess@roboception.de> * factor out _serialize_pod_spec into separate util function Signed-off-by: Felix Ruess <felix.ruess@roboception.de> * model file changes, couple other changes Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * minor cleanup Signed-off-by: Felix Ruess <felix.ruess@roboception.de> * add unit test for container_task pod_template Signed-off-by: Felix Ruess <felix.ruess@roboception.de> * bump min version of flyteidl to 1.3.12 for pod template data config support Signed-off-by: Felix Ruess <felix.ruess@roboception.de> * require flyteidl==1.3.12 in doc-requirements.txt Signed-off-by: Felix Ruess <felix.ruess@roboception.de> --------- Signed-off-by: Felix Ruess <felix.ruess@roboception.de> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Pass locally defined scopes to RemoteClientConfigStore (#1553) Signed-off-by: franco-bocci <franco.bocci@wolt.com> * TypeTransformer for TensorFlow model (#1562) * TypeTransformer for TensorFlow model Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * clean up Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * clean up Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * fix lint Signed-off-by: Samhita Alla <aallasamhita@gmail.com> --------- Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * Remove fsspec flytekit plugin from main Dockerfile (#1561) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Device auth flow / Headless auth (#1552) * Device auth flow Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Device AuthFlow is now available in flytekit Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * unit tests Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * test added Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Fixed test Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Fixed unit test Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Fixed lint errors Signed-off-by: Ketan Umare <ketan.umare@gmail.com> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * url encode secret in client credentials flow (#1566) * url encode secret first Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * nit Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Python run multiple files (#1559) * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * update comment Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * General Partial support in flytekit and multi-list support in flytekit (#1556) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fix: Silence keyring warnings by changing to debug (#1568) Co-authored-by: ggydush-fn <greg.gydush@freenome.com> * Support GCP secrets (#1571) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Automatically remove unused import (#1574) * Automatically remove unused import Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Disallow partial lists in map tasks (#1577) * Disallow partial lists in map tasks Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Remove duplicate reporting logic (#1578) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * [Core feature] Convert List[Any] to a single pickle file (#1535) * Convert List[Any] to a single pickle file * remove redundant code * keep batchSize only if type contain flytePickle * fix error * add batch support to translate_inputs_to_literals * add ci test Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * improve comment Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * add more ci test Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * improve Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * handle HashMethod case Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * improve format Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * improve Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * improve Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * improve Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * add test_is_batchable Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * Add BatchSize Signed-off-by: Kevin Su <pingsutw@apache.org> * test_batch_size Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: root <root@ip-172-31-10-58.ec2.internal> Co-authored-by: Kevin Su <pingsutw@apache.org> * Improve authoring structure documentation (#1572) Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * Unify sqlalchemy Dockerfiles (#1585) * Unify sqlalchemy Dockerfiles Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Simplifyte sqlalchemy Dockerfile Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Set pythonpath to /root Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Pyflyte build imageSpec (#1555) * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * pyflyte build Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * Support serialize and package Signed-off-by: Kevin Su <pingsutw@apache.org> * more tests Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * move to plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * fixed tested Signed-off-by: Kevin Su <pingsutw@apache.org> * fixed tested Signed-off-by: Kevin Su <pingsutw@apache.org> * more tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Add support passing yaml in pyflyte run Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Feature/3506: Improve Type Conversion errors, use rich to prettify error messages (#1582) * improve input type conversion error Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add tests Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add tests Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add rich Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * remove prototyping script, update loggers Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update __init__.py Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update logger Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update logger Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix GE and pandera tests Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * Add support nested FlyteFile in pyflyte run (#1587) * Add support nested FlyteFile in pyflyte run Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Set cache_regions=True in the case of s3fs (#1592) * Set cache_regions=True in the case of s3fs * Linting Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Fix tests Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * More linting Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Set default ComSpec when running on Windows (#1595) * set default ComSpec when running on Windows Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Feature: add activate-launchplan command to pyflyte (#1588) * add activate-launchplan command to pyflyte Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com> * fix linting Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com> * rework launchplan command Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com> * fix linting Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com> --------- Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com> * nit (#1596) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Add log streaming to papermill plugin (#1129) * checkpoint Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Experimental implementation works well. Instead of messing with the class, we utilize the interpolation and context to dynamically generate a wrapper around the desired script. In the wrapper, we cd to the ctx.working_directory, export the env variables (handled by an additional method to convert dict to str), and then pass the arguments to the script Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * fix spacing Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * remove breakpoint Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Output ctx.working_directory as single output Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * more doc strings Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * more comments Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Added comments Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Add tests and test files from other branch Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * minor fix, have function return the instance rather than create. It seems to get registered when flyte packages your project Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * fix tests Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * remove set flags not supported by sh Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * fix spellcheck lint errors Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * don't have a windows equivalent test script so bypassing those tests for now Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Add typing to make_export_string_from_env_dict params Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * fixup doc string Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Refactored the new behavior out into a new class. Did not change implementation details at all Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Address linter issues and up test cov Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Skip test on windows, no equivalent script Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Run black and isort, address SC2236 Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Refactored class name to remove _, make utility function require name so multiple uses in a workflow don't break Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * fix utility function Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * fix utility function call in tests Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * Fix isort on test_shell.py Signed-off-by: Mike Zhong <mzhong@embarkvet.com> * adding logging settings to papermill plugin Signed-off-by: Calvin Leather <cleather@embarkvet.com> * set level to info logging * improved comments/docs --------- Signed-off-by: Mike Zhong <mzhong@embarkvet.com> Signed-off-by: Calvin Leather <cleather@embarkvet.com> Co-authored-by: Mike Zhong <mzhong@embarkvet.com> Co-authored-by: Mike Zhong <mike.zhong89@gmail.com> * Set lower bound for s3fs (#1598) * set lower bound for s3fs Signed-off-by: Kevin Su <pingsutw@apache.org> * update doc-requirements.txt Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Implement horovod task in mpi plugin (#1575) * Add horovod task to mpi plugin Signed-off-by: byhsu <byhsu@linkedin.com> * Remove unused Signed-off-by: byhsu <byhsu@linkedin.com> * add test Signed-off-by: byhsu <byhsu@linkedin.com> * inherit from mpifunctiontask Signed-off-by: byhsu <byhsu@linkedin.com> * fix format Signed-off-by: byhsu <byhsu@linkedin.com> * fix format Signed-off-by: byhsu <byhsu@linkedin.com> --------- Signed-off-by: byhsu <byhsu@linkedin.com> Co-authored-by: byhsu <byhsu@linkedin.com> * add ImageRender class for Flyte Decks (#1599) Signed-off-by: esad <esad@peridotml.io> * Bump tensorflow from 2.10.0 to 2.11.1 in /plugins/flytekit-mlflow (#1586) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.10.0 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](tensorflow/tensorflow@v2.10.0...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Handle batchable lists of length 0 (#1600) Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com> * pyflyte run imperative workflow (#1597) * pyflyte run imperative workflow Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Backfill should fill in the right input vars (#1593) * Backfill fix - Backfill was using incorrect arguments - backfill should use the argument that user provides or none at all Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Updated code Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed unit test Signed-off-by: Ketan Umare <ketan.umare@gmail.com> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Pyflyte prettified (#1602) * Pyflyte prettified Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * added dependency Signed-off-by: Ketan Umare <ketan.umare@gmail.com> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * pyflyte run now supports json/yaml files (#1606) * pyflyte run now supports json files Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * added yaml support Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed parsing Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed windows test Signed-off-by: Ketan Umare <ketan.umare@gmail.com> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Add is_inside func to ImageSpec (#1601) * Add is_inside func to ImageSpec Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * Add is_inside func to ImageSpec Signed-off-by: Kevin Su <pingsutw@apache.org> * Add tests Signed-off-by: Kevin Su <pingsutw@apache.org> * rename Signed-off-by: Kevin Su <pingsutw@apache.org> * rename Signed-off-by: Kevin Su <pingsutw@apache.org> * check execution mode Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Measuring the time taken for each component when executing a task (#1581) * add time measurement framework Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * add time measurement framework Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * nit Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * nit Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * nit Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * nit Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * nit Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> --------- Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * Skip the image building process if the check for its existence fails (#1614) * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * add lru cache Signed-off-by: Kevin Su <pingsutw@apache.org> * update Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Enable torch elastic training (torchrun) (#1603) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> This PR brings [torch elastic training (`torchrun`)](https://pytorch.org/docs/stable/elastic/run.html) to the pytorch plugin: ```python from flytekitplugins.kfpytorch import Elastic @task( task_config=Elastic( replicas=4, nproc_per_node=4, ... ), ... ) def train(...): ... ``` flyteorg/flyte#3614 * update sphinx youtube plugin (#1619) * update sphinx youtube plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Read polar dataframe without copying to local (#1618) Signed-off-by: Kevin Su <pingsutw@apache.org> * Add default spark image (#1616) * Add default spark image Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Lazy load modules (#1590) * lazy load module Signed-off-by: Kevin Su <pingsutw@apache.org> * lazy load module Signed-off-by: Kevin Su <pingsutw@apache.org> * import Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * keep structured dataset in flytekit init Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * fixed tess Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * move import pandas to __init__ Signed-off-by: Kevin Su <pingsutw@apache.org> * use lazy import loader instead Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * regular import Signed-off-by: Kevin Su <pingsutw@apache.org> * fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Override node name with a name containing underscore (#1608) * override node_name with a name containing underscore Signed-off-by: Kevin Su <pingsutw@apache.org> * add comment Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * External Plugin Service (grpc) (#1524) External Plugin Service * Include traceback in errors from admin (#1621) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Add support for copying all the files in source root (#1622) * Add support for copying all the files in source root Signed-off-by: Kevin Su <pingsutw@apache.org> * Add tests Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * fix PipelineModel transformer issue 3648 (#1623) Signed-off-by: esad <esad@peridotml.io> * update GH workflow for external plugin service (#1624) Signed-off-by: Kevin Su <pingsutw@apache.org> * [Issue-3617] Enables FlyteFiles, FlyteDirectors, and StructuredDatasets inputs in papermill plugin (#1612) * Add support for using a list as an input for a subworkflow (#1605) Signed-off-by: Kevin Su <pingsutw@apache.org> * Improve task decorator type hints with overload (#1631) Without the overload, the decorated function does not have the proper type of PythonFunctionTask, leading to spurious type errors when attempting to register the task on a FlyteRemote object Signed-off-by: Matthew Hoffman <matthew@protopia.ai> * add annotation option for serialization (#1615) * add annotation option for serialization * add support for guessing type * formatting * test both directions * remove guessing for annotated types * clarify usage in test * Update tests/flytekit/unit/core/test_type_engine.py --------- Co-authored-by: Eli Bixby <eli@cradle.bio> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Delete removed data persistence classes from docs (#1633) Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * Improve workflow decorator type hints with overload (#1635) Previously, the workflow decorator is hinted as always returning a WorkflowBase, which is not true when _workflow_function is None; similar to #1631, we propose using typing.overload to differentiate the return type of workflow based on the value of _workflow_function Signed-off-by: Matthew Hoffman <matthew@protopia.ai> * Fix mypy errors (#1313) * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix mypy errors Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix mypy errors Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * wip Signed-off-by: Kevin Su <pingsutw@apache.org> * fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * fix test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * Update type Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * update dev-requirements.txt Signed-off-by: Kevin Su <pingsutw@apache.org> * Address comment Signed-off-by: Kevin Su <pingsutw@apache.org> * upgrade torch Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Kevin Su <pingsutw@gmail.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * type-protobuf<4 Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Allow for python 3.7 Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Fix protobuf transformer merge error * Fix bad merge of s/_task_models/task_models/ * Add maxsize=128 to @lru_cache to cover python 3.7 Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Regenerate doc-requirements.txt Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Add missing call to lru_cache Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Bump version and regenerate requirements in a few plugins Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Revert "Fix mypy errors (#1313)" This reverts commit af49155. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove docs Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * More linting Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Fix type of get_type_for_output_var Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Remove tensorflow test_model * Replace use of shutil.copytree Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Other use of copytree Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Stringify paths in test_script_mode Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * os.path.isfile instead in imagespec Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Remove tests Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * More linting Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Remove flaky test Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> Signed-off-by: Felix Ruess <felix.ruess@roboception.de> Signed-off-by: franco-bocci <franco.bocci@wolt.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com> Signed-off-by: Mike Zhong <mzhong@embarkvet.com> Signed-off-by: Calvin Leather <cleather@embarkvet.com> Signed-off-by: byhsu <byhsu@linkedin.com> Signed-off-by: esad <esad@peridotml.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Matthew Hoffman <matthew@protopia.ai> Signed-off-by: Kevin Su <pingsutw@gmail.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Ketan Umare <16888709+kumare3@users.noreply.github.com> Co-authored-by: Adrian Rumpold <a.rumpold@gmail.com> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: Felix Ruess <felix.ruess@roboception.de> Co-authored-by: Franco Bocci <121866694+franco-bocci@users.noreply.github.com> Co-authored-by: Samhita Alla <aallasamhita@gmail.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: ggydush <35151789+ggydush@users.noreply.github.com> Co-authored-by: ggydush-fn <greg.gydush@freenome.com> Co-authored-by: Yicheng-Lu-llll <51814063+Yicheng-Lu-llll@users.noreply.github.com> Co-authored-by: root <root@ip-172-31-10-58.ec2.internal> Co-authored-by: Artem <58334441+wckdman@users.noreply.github.com> Co-authored-by: Calvin Leather <cleather@embarkvet.com> Co-authored-by: Mike Zhong <mzhong@embarkvet.com> Co-authored-by: Mike Zhong <mike.zhong89@gmail.com> Co-authored-by: ByronHsu <byronhsu1230@gmail.com> Co-authored-by: byhsu <byhsu@linkedin.com> Co-authored-by: peridotml <106936600+peridotml@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Fabio M. Graetz, Ph.D <fabiograetz@googlemail.com> Co-authored-by: Matthew Hoffman <matthew@protopia.ai> Co-authored-by: Eli Bixby <elibixby@google.com> Co-authored-by: Eli Bixby <eli@cradle.bio>
- Loading branch information
26 people
authored
May 17, 2023
1 parent
f7b59bc
commit 4fd5d81