diff --git a/README.md b/README.md index c76b77e6c2..031002a925 100644 --- a/README.md +++ b/README.md @@ -117,25 +117,25 @@ To dig deeper into Flyte, refer to the [Documentation](https://docs.flyte.org/en - Supports multiple **[data types](https://docs.flyte.org/projects/cookbook/en/latest/core.html)** for machine learning and data processing pipelines, such as Blobs (images, arbitrary files), Directories, Schema (columnar structured data), collections, maps etc. - Memoization and Lineage tracking - Workflow features: - - Start with one task, convert to a pipeline, attach **[multiple schedules](https://docs.flyte.org/projects/cookbook/en/latest/auto_core_remote_flyte/lp_schedules.html)**, trigger using a programmatic API, or on-demand + - Start with one task, convert to a pipeline, attach **[multiple schedules](https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/workflow/lp_schedules.html)**, trigger using a programmatic API, or on-demand - Parallel step execution - - Extensible backend to add [customized plugin](https://docs.flyte.org/projects/cookbook/en/latest/auto_core_advanced/custom_task_plugin.html) experience (with simplified user experience) - - **[Branching](https://docs.flyte.org/projects/cookbook/en/latest/auto_core_intermediate/run_conditions.html)** - - Inline **[subworkflows](https://docs.flyte.org/projects/cookbook/en/latest/auto_core_intermediate/subworkflows.html)** (a workflow can be embeded within one node of the top level workflow) + - Extensible backend to add **[customized plugin](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/extend_flyte/custom_task_plugin.html)** experience (with simplified user experience) + - **[Branching](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/run_conditions.html)** + - Inline **[subworkflows](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/subworkflows.html)** (a workflow can be embeded within one node of the top level workflow) - Distributed **remote child workflows** (a remote workflow can be triggered and statically verified at compile time) - - **[Array Tasks](https://docs.flyte.org/projects/cookbook/en/latest/auto_core_intermediate/map_task.html)** (map a function over a large dataset -- ensures controlled execution of thousands of containers) - - **[Dynamic workflow](https://docs.flyte.org/projects/cookbook/en/latest/auto_core_intermediate/dynamics.html)** creation and execution with runtime type safety + - **[Array Tasks](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/map_task.html)** (map a function over a large dataset -- ensures controlled execution of thousands of containers) + - **[Dynamic workflow](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/dynamics.html)** creation and execution with runtime type safety - Container side [plugins](https://docs.flyte.org/projects/cookbook/en/latest/plugins.html) with first class support in Python - - _PreAlpha_: Arbitrary flytekit-less containers supported ([RawContainer](https://docs.flyte.org/projects/cookbook/en/latest/auto_core_intermediate/raw_container.html)) -- Guaranteed **[reproducibility](https://docs.flyte.org/projects/cookbook/en/latest/auto_core_basic/task_cache.html)** of pipelines via: + - _PreAlpha_: Arbitrary flytekit-less containers supported ([RawContainer](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html)) +- Guaranteed **[reproducibility](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html)** of pipelines via: - Versioned data, code and models - Automatically tracked executions - Declarative pipelines - **Multi cloud support** (AWS, GCP and others) - Extensible core, modularized, and deep observability - Automated notifications to Slack, Email, and Pagerduty -- [Multi K8s cluster support](https://docs.flyte.org/projects/cookbook/en/latest/auto_plugins_pod/index.html) -- Out of the box support to run **[Spark jobs on K8s](https://docs.flyte.org/projects/cookbook/en/latest/auto_plugins_k8s_spark/index.html)**, **[Hive queries](https://docs.flyte.org/projects/cookbook/en/latest/auto_plugins_hive/index.html)**, etc. +- [Multi K8s cluster support](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/index.html) +- Out of the box support to run **[Spark jobs on K8s](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html)**, **[Hive queries](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/external_services/hive/index.html)**, etc. - Snappy Console - Python CLI and Golang CLI (flytectl) - Written in **Golang** and optimized for large running jobs' performance @@ -150,17 +150,17 @@ To dig deeper into Flyte, refer to the [Documentation](https://docs.flyte.org/en ## 🔌 Available Plugins - Containers -- [K8s Pods](https://docs.flyte.org/projects/cookbook/en/latest/auto_plugins_pod/index.html) +- [K8s Pods](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/index.html) - AWS Batch Arrays - K8s Pod Arrays -- K8s Spark (native [Pyspark](https://docs.flyte.org/projects/cookbook/en/latest/auto_plugins_k8s_spark/index.html) and Java/Scala) +- K8s Spark (native [Pyspark](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html) and Java/Scala) - AWS Athena -- [Qubole Hive](https://docs.flyte.org/projects/cookbook/en/latest/auto_plugins_hive/index.html) +- [Qubole Hive](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/external_services/hive/index.html) - Presto Queries -- Distributed Pytorch (K8s Native) -- [Pytorch Operator](https://docs.flyte.org/projects/cookbook/en/latest/auto_plugins_kfpytorch/index.html) -- Sagemaker([builtin algorithms](https://docs.flyte.org/projects/cookbook/en/latest/auto_plugins_sagemaker_training/sagemaker_builtin_algo_training.html) & [custom models](https://docs.flyte.org/projects/cookbook/en/latest/auto_plugins_sagemaker_training/sagemaker_custom_training.html)) -- Distributed Tensorflow (K8s Native) - TFOperator -- Papermill notebook execution ([Python](https://github.com/lyft/flytekit/tree/master/plugins/papermill) and Spark) +- Distributed Pytorch (K8s Native) -- [Pytorch Operator](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/kfpytorch/index.html) +- Sagemaker([builtin algorithms](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/aws/sagemaker_training/sagemaker_builtin_algo_training.html) & [custom models](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/aws/sagemaker_training/sagemaker_custom_training.html)) +- Distributed Tensorflow (K8s Native) - [TFOperator](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/kftensorflow/index.html) +- Papermill notebook execution ([Python](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/papermilltasks/index.html) and Spark) - Type safe and data checking for Pandas dataframe using Pandera ### In Queue diff --git a/rsts/howto/enable_and_use_schedules.rst b/rsts/howto/enable_and_use_schedules.rst index 9bc719282f..9cf9dd91bd 100644 --- a/rsts/howto/enable_and_use_schedules.rst +++ b/rsts/howto/enable_and_use_schedules.rst @@ -98,7 +98,7 @@ assigned to the ``triggered_time`` input you could define the following launch p fixed_inputs={"an_input": 3}, ) -Please see a more complete example in the :std:ref:`cookbook `. +Please see a more complete example in the :std:ref:`User Guide `. Activating a schedule ===================== diff --git a/rsts/howto/launchplans.rst b/rsts/howto/launchplans.rst index 017e642325..511af6f8a1 100644 --- a/rsts/howto/launchplans.rst +++ b/rsts/howto/launchplans.rst @@ -13,7 +13,7 @@ When to use launchplans? - I want to share my workflow to another user but also make sure that some inputs can be overridden if needed. - I want to share my workflow with another user but make sure that some inputs are never changed. -For preliminary examples on using launch plans in code, check out the canonical :std:ref:`cookbook ` examples. +For preliminary examples on using launch plans in code, check out the canonical :std:ref:`User Guide ` examples. Partial Inputs for Launchplans ============================== diff --git a/rsts/howto/notifications.rst b/rsts/howto/notifications.rst index d38d1702b9..e85414da00 100644 --- a/rsts/howto/notifications.rst +++ b/rsts/howto/notifications.rst @@ -30,7 +30,7 @@ For example # This launch plan triggers email notifications when the workflow execution it triggered reaches the phase `SUCCEEDED`. my_notifiying_lp = LaunchPlan.create( "my_notifiying_lp", - my_workflow_defintiion, + my_workflow_definition, default_inputs={"a": 4}, notifications=[ Email( @@ -41,7 +41,7 @@ For example ) -See detailed usage examples in the :std:ref:`cookbook ` +See detailed usage examples in the :std:ref:`User Guide ` Notifications can be combined with schedules to automatically alert you when a scheduled job succeeds or fails. diff --git a/rsts/index.rst b/rsts/index.rst index b3b0f6af9e..fa8fbe137e 100644 --- a/rsts/index.rst +++ b/rsts/index.rst @@ -56,7 +56,7 @@ Meet Flyte .. raw:: html -

The workflow automation platform for complex, mission-critical data and ML processes at scale

+

The workflow automation platform for complex, mission-critical data and ML processes at scale

Flyte is an open-source, container-native, structured programming and distributed processing platform. It enables highly concurrent, scalable and maintainable workflows for machine learning and data processing. diff --git a/rsts/plugins/spark_k8s.rst b/rsts/plugins/spark_k8s.rst index 772da4e598..7acc84342d 100644 --- a/rsts/plugins/spark_k8s.rst +++ b/rsts/plugins/spark_k8s.rst @@ -48,7 +48,7 @@ You can optionally configure the Plugin as per the - `backend Config Structure < Spark in Flytekit ======================== -For a more complete example refer to :std:ref:`Cookbook Spark Plugin ` +For a more complete example refer to the :std:ref:`User Guide ` #. Ensure you have ``flytekit>=0.16.0`` #. Enable Spark in backend, following the previous section.