Consider adding support for running TFX w/Beam in the components

[holdenk]

Looking at the current TFX components, it seems like they only run in local mode. I think it would be useful to consider adding support for running on top of Beam w/Dataflow (and later Flink & Spark).

[numerology]

Actually you have the option to author the pipeline in TFX and run on KubeflowDagRunner, see
https://github.com/tensorflow/tfx/blob/master/tfx/orchestration/kubeflow/kubeflow_dag_runner.py

You can specify Dataflow runner by specifying beam args, see example here https://github.com/tensorflow/tfx/blob/848063c2c84e60f368994edf9ace01d017120dbc/tfx/examples/chicago_taxi_pipeline/taxi_pipeline_kubeflow_gcp.py#L100

[numerology]

/assign @numerology

[holdenk]

So that’s only if I make a TFX pipeline, if I make a kubeflow pipeline w/TFX components it doesn’t let me specify the runner as far as I can see.

[rmgogogo]

Jiaxiao, this match the email we discussed before.
Add Ajay to get more input.

[numerology]

Actually in KubeflowDagRunner all TFX components are converted to KFP containerOp under the hood.

I'm hesitating in supporting a codepath like KFP-TFX-KFP-Argo. This looks unnecessarily redundant.

[holdenk]

So let’s say I have a mixed pipeline, like needing to do some non-TFX data munging first, how do I do my TFX work on beam as part of that? Or is that just not something in scope?

(Also if we don’t want folks to use the KFP TFX components should we delete them? I know KF has a lot of dead ends it might be good to clean up some of them if this is one of those cases).

[Ark-kun]

it would be useful to consider adding support for running on top of Beam w/Dataflow

Well, we had Dataflow components, but someone cleaned them up

So let’s say I have a mixed pipeline, like needing to do some non-TFX data munging first, how do I do my TFX work on beam as part of that? Or is that just not something in scope?

KFP uses Argo and runs on Kubernetes. Beam is a different orchestrator. Can you please tell us about scenarios that require using Beam on Kubernetes?

When using TFX framework (not components), you can choose a runner - either Beam or Kubeflow+Argo+Kubernetes.
There is a choice - to use Beam orchestrator or not.

Of course, KFP allows you to launch code on Dataflow. But that would be opaque task - a single block that executes some beam pipeline. Same with any other pair of orchestrators.

[holdenk]

So my scenario is I fetch data from the world outside of TFX and store it on GCS as part 1 of my pipeline. Then in part 2 I want to train a tensorflow model, ideally with TFT for data transformations.

Like if you don’t imagine TFX being used inside of KF pipelines that’s fine; I’m just a little confused.

[holdenk]

Just following up: do we not expect folks to use TFX as part of a Kubeflow pipeline? I want to make sure I capture this correctly in the book I'm working on.

[jlewi]

See also kubeflow/kubeflow#1583

My understanding is that KFP will be the recommended way to run TFX pipelines especially on K8s. To achieve that my understanding is that at some point the TFX implementation on KFP will support scaling out the processing(e.g. TF.Transform) using Beam. I believe using Dataflow is already supported and we have at least one issue (kubeflow/kubeflow#1583) related to supporting other runners like Flink.

@rmgogogo @jessiezcc @Bobgy Can anyone from pipelines provide some guidance on terms of the roadmap for TFX and KFP? Could we get the relevant information added to
https://github.com/kubeflow/pipelines/blob/master/ROADMAP.md

[jlewi]

/cc @katsiapis

[Ark-kun]

Looking at the current TFX components, it seems like they only run in local mode. I think it would be useful to consider adding support for running on top of Beam w/Dataflow

Good idea.
Currently we utilize the executor code and the component code of the TFX components, but we do not use the launchers. Using the InProcessComponentLauncher would allows some of the TFX components to use Beam and Dataflow.
I think I can make this work.

[numerology]

@Ark-kun
Actually, if by all means the aforementioned TFX components is the way to go, shall we consider implementing them based on this entrypoint? That way, we can get Dataflow without needing to play with TFX launcher/driver.

[Ark-kun]

@Ark-kun
Actually, if by all means the aforementioned TFX components is the way to go, shall we consider implementing them based on this entrypoint? That way, we can get Dataflow without needing to play with TFX launcher/driver.

Thanks for the pointer. It looks like it is very easy to add support for beam_pipeline_args. I did an experiment, and the component starts using Dataflow. Unfortunately, I'm then hit by this bug: tensorflow/tfx#649 Waiting for some fix or workaround for it.

[neuromage]

Hi @holdenk,

I recommend using the official TFX components on KFP instead [1] [2]. Those components are updated, maintained and have been tested with Dataflow. This is the supported path, we'll be happy to help you with any issues here. Thanks!

[1] https://www.tensorflow.org/tfx/tutorials/tfx/cloud-ai-platform-pipelines
[2] https://github.com/kubeflow/pipelines/tree/master/samples/core/parameterized_tfx_oss

[Ark-kun]

Hi @holdenk,

I recommend using the official TFX components on KFP instead [1] [2]. Those components are updated, maintained and have been tested with Dataflow. This is the supported path, we'll be happy to help you with any issues here. Thanks!

[1] https://www.tensorflow.org/tfx/tutorials/tfx/cloud-ai-platform-pipelines
[2] https://github.com/kubeflow/pipelines/tree/master/samples/core/parameterized_tfx_oss

+1.
For pipelines that mostly use TFX components we suggest using the official TFX components available through the TFX SDK.

[holdenk]

Ok but if I have non TFX tasks that doesn’t work

[Ark-kun]

Ok but if I have non TFX tasks that doesn’t work

That's true.
Short-term solution for mixed-component pipelines: The KFP-style TFX components will soon support Beam/Dataflow (after the the official TFX version >0.21.4 is released). There was an issue in new versions of Beam that has affected the ability of core TFX components to run on Dataflow. tensorflow/tfx#649 This bug is already fixed in TFX and the fix will be released as part of the next TFX release. tensorflow/tfx#1699 After that I will update the KFP-style TFX components and support for the beam_pipeline_args which can be used to specify the DataflowRunner.

Longer-term solution:
I'm also working on making it possible to use KFP components with TFX SDK. When the work is finished, you'll also be able to create mixed-component pipelines in TFX SDK as well.

[holdenk]

That’s great news! Thank you for working on these :)

[Ark-kun]

That’s great news! Thank you for working on these :)

@holdenk
The URI-IO-based KFP-style TFX components now support Dataflow. You just pass beam_pipeline_args that specifies DataflowRunner and they start using Dataflow. Check this sample pipeline.