Update RunInference documentation #22250

rszper · 2022-07-12T20:40:47Z

Adding documentation for the RunInference API.

Added a ML page in the Python section, linked to from the Python SDK page.
Added a RunInference transform page, linked to from the Python Transforms page.
Added snippets and output for the RunInference transforms page examples.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

rszper · 2022-07-12T20:41:10Z

R: @yeandy @rezarokni

yeandy · 2022-07-12T20:43:25Z

R: @AnandInguva @ryanthompson591

github-actions · 2022-07-12T20:52:12Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

website/www/site/content/en/documentation/sdks/python-machine-learning.md

rezarokni · 2022-07-12T21:07:51Z

website/www/site/content/en/documentation/sdks/python.md

@@ -48,6 +48,11 @@ language-specific implementation guidance.

 ## Using Beam Python SDK in your ML pipelines

+To use the Beam Python SDK with your machine learning pipelines, you can either use the RunInference API or TensorFlow.


change: use the RunInference API or TensorFlow
to: use the RunInference API for PyTorch and Sklearn models. If using Tensorflow model you can make use of the library from tfx_bsl. Further integrations for tensorflow are planned.

Updated, but I'm not sure if we'll need to update this again based on the email thread?

AnandInguva · 2022-07-13T11:28:01Z

Added the RunInference examples -> #22254

yeandy

Good work so far! Left some feedback on changes.

Let me know when to take another pass, especially with the example snippets updated.

website/www/site/content/en/documentation/sdks/python-machine-learning.md

website/www/site/content/en/documentation/sdks/python.md

website/www/site/content/en/documentation/sdks/python-machine-learning.md

website/www/site/content/en/documentation/transforms/python/elementwise/runinference.md

rszper · 2022-07-13T17:52:13Z

@yeandy I believe all updates are made based on your comments, except I haven't updated the examples yet.

yeandy

Thanks for the updates! A few more :)

website/www/site/content/en/documentation/sdks/python-machine-learning.md

ryanthompson591 · 2022-07-13T17:25:35Z

website/www/site/content/en/documentation/sdks/python-machine-learning.md

+
+To import models, you need to wrap them around a `ModelHandler object`. Add one or more of the following lines of code, depending on the framework and type of data structure that holds the data:
+
+```


These little chunks of code below here seem out of place.

@yeandy How do you want to handle this?

Would it make sense to reword it to something like this, and keep the (refactored) code block?

To import models, you need to wrap them around a ModelHandler object. The ModelHandler you import will depend on the framework and type of data structure that contains the inputs. See the following examples on which ones you may want to import.

from apache_beam.ml.inference.sklearn_inference import SklearnModelHandlerNumpy from apache_beam.ml.inference.sklearn_inference import SklearnModelHandlerPandas from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerTensor from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerKeyedTensor

I made some updates. Take a look and let me know if we need more changes.

Thanks. By the way, the imports I originally wrote had some typos, so I fixed them

from apache_beam.ml.inference.sklearn_inference import SklearnModelHandlerNumpy from apache_beam.ml.inference.sklearn_inference import SklearnModelHandlerPandas from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerTensor from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerKeyedTensor

Updated to fix the typos

website/www/site/content/en/documentation/sdks/python-machine-learning.md

yeandy

@rszper A last minute addition on the batching issue.

website/www/site/content/en/documentation/sdks/python-machine-learning.md

Because of max depth recursion error while pickling

Inference snippets

codecov · 2022-07-15T14:07:34Z

Codecov Report

Merging #22250 (4160d1b) into master (9cf8cf5) will increase coverage by 9.28%.
The diff coverage is 0.00%.

❗ Current head 4160d1b differs from pull request most recent head 4e89126. Consider uploading reports for the commit 4e89126 to get more accurate results

@@            Coverage Diff             @@
##           master   #22250      +/-   ##
==========================================
+ Coverage   74.25%   83.54%   +9.28%     
==========================================
  Files         702      474     -228     
  Lines       92999    65934   -27065     
==========================================
- Hits        69058    55085   -13973     
+ Misses      22674    10849   -11825     
+ Partials     1267        0    -1267

Flag	Coverage Δ
go	`?`
python	`83.54% <0.00%> (-0.10%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...es/snippets/transforms/elementwise/runinference.py	`0.00% <0.00%> (ø)`
sdks/python/apache_beam/io/source_test_utils.py	`88.01% <0.00%> (-1.39%)`	⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py	`93.28% <0.00%> (-0.75%)`	⬇️
...eam/runners/portability/fn_api_runner/execution.py	`92.44% <0.00%> (-0.65%)`	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	`89.09% <0.00%> (-0.32%)`	⬇️
sdks/python/apache_beam/utils/annotations.py	`100.00% <0.00%> (ø)`
...hon/apache_beam/runners/worker/bundle_processor.py	`93.54% <0.00%> (ø)`
sdks/go/pkg/beam/core/graph/coder/int.go
sdks/go/pkg/beam/x/debug/debug.shims.go
sdks/go/pkg/beam/core/graph/xlang.go
... and 227 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9cf8cf5...4e89126. Read the comment docs.

AnandInguva · 2022-07-15T17:24:04Z

R: @tvalentyn @pabloem we did the review on the docs and the snippets. Would you be able to do a final review and merge the PR?

yeandy · 2022-07-15T17:25:54Z

@tvalentyn @pabloem This one too please #22069. The tests are queued, but it should be fine since it's only the .md file that was modified.

tvalentyn · 2022-07-15T20:11:39Z

website/www/site/content/en/documentation/sdks/python-machine-learning.md

+
+### Shared helper class
+
+Instead of loading a model for each thread in the process, we use the `Shared` class, which allows us to load one model that is shared across all threads of each worker in a DoFn. For more information, see the


How about:

Using the Shared class within RunInference implementation allows us to load the model only once per process and share it with all DoFn instances created in that process. This reduces the memory consumption and model loading time.

tvalentyn · 2022-07-15T20:15:18Z

website/www/site/content/en/documentation/sdks/python-machine-learning.md

+```
+Where `model_handler` is the model handler setup code.
+
+To import models, you need to wrap them around a `ModelHandler` object. Which `ModelHandler` you import depends on the framework and type of data structure that contains the inputs. The following examples show some ModelHandlers that you might want to import.


To import models, you need to wrap them around a ModelHandler object

Consider instead:
To import models, you need to configure a ModelHandler object that will wrap the underlying model

tvalentyn · 2022-07-15T20:25:52Z

website/www/site/content/en/documentation/sdks/python-machine-learning.md

+
+Disable batching by overriding the `batch_elements_kwargs` function in your ModelHandler and setting the maximum batch size (`max_batch_size`) to one: `max_batch_size=1`. For more information, see
+[BatchElements PTransforms](/documentation/sdks/python-machine-learning/#batchelements-ptransform).
+


How about we also link apache_beam/examples/inference/pytorch_language_modeling.py as an example that does this?

tvalentyn · 2022-07-15T22:13:47Z

website/www/site/content/en/documentation/sdks/python-machine-learning.md

@@ -171,7 +171,7 @@ In some cases, the `PredictionResults` output might not include the correct pred

 The RunInference API currently expects outputs to be an `Iterable[Any]`. Example return types are `Iterable[Tensor]` or `Iterable[Dict[str, Tensor]]`. When RunInference zips the inputs with the predictions, the predictions iterate over the dictionary keys instead of the batch elements. The result is that the key name is preserved but the prediction tensors are discarded. For more information, see the [Pytorch RunInference PredictionResult is a Dict](https://github.com/apache/beam/issues/22240) issue in the Apache Beam GitHub project.

-To work with the current RunInference implementation, you can create a wrapper class that overrides the `model(input)` call. In PyTorch, for example, your wrapper would override the `forward()` function and return an output with the appropriate format of `List[Dict[str, torch.Tensor]]`. For more information, see our [HuggingFace language modeling example](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_language_modeling.py#L49).
+To work with the current RunInference implementation, you can create a wrapper class that overrides the `model(input)` call. In PyTorch, for example, your wrapper would override the `forward()` function and return an output with the appropriate format of `List[Dict[str, torch.Tensor]]`. For more information, see our [HuggingFace language modeling example](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_language_modeling.py#L49) and our [Bert language modeling example](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_language_modeling.py).


these are the same links, looks like not the change we intended to make?

my last comment referred to disable batching section

Oops. This should be fixed now.

website/www/site/layouts/partials/section-menu/en/sdks.html

Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com> Co-authored-by: Andy Ye <andyye333@gmail.com> Co-authored-by: Anand Inguva <anandinguva98@gmail.com> Co-authored-by: Anand Inguva <anandinguva@google.com>

github-actions bot added examples python website labels Jul 12, 2022

rezarokni reviewed Jul 12, 2022

View reviewed changes

rszper force-pushed the rszper-runInferenceDocs branch from 767a4c2 to 74c7ac4 Compare July 12, 2022 23:25

yeandy suggested changes Jul 13, 2022

View reviewed changes

rszper added 16 commits July 13, 2022 17:47

starting RunInference documentation drafts

c7b0d99

starting RunInference documentation drafts

eb89211

RunInference documentation updates

1a29d9e

RunInference documentation updates

2be4739

Skipping notebook because it doesn't exist

f5a0f0c

Added troubleshooting section

1d404b6

Updated examples in snippets

2d954de

Updated test

245dc49

Updated troubleshooting section

99b1308

Updated troubleshooting section

b0f430a

Moved batching elements content out of the RunInference transform page

b02fd09

Added related links to the ML page

1234657

Updated docs based on comments

0973e45

Changed tensorflow to TensorFlow

5dd713e

Updated content based on Andy and Anand's comments

8a50ee8

Updated content based on Andy and Anand's comments

e49ebec

rszper force-pushed the rszper-runInferenceDocs branch from 817b42e to e49ebec Compare July 13, 2022 17:48

yeandy reviewed Jul 13, 2022

View reviewed changes

ryanthompson591 reviewed Jul 13, 2022

View reviewed changes

rszper and others added 4 commits July 14, 2022 16:26

Updates based on comments

489fce7

Add import try and catch for unit tests that use torch

4b962b4

fixup lint

9e98188

fixup: formatting

46f5ebd

yeandy reviewed Jul 14, 2022

View reviewed changes

website/www/site/content/en/documentation/sdks/python-machine-learning.md Outdated Show resolved Hide resolved

website/www/site/content/en/documentation/sdks/python-machine-learning.md Show resolved Hide resolved

AnandInguva and others added 10 commits July 14, 2022 20:21

fixup lint

1f7ce97

Added error message to troubleshooting section

07a99d7

Add skip when GCP deps are not found

49b0a7f

Added new pages to the TOC

2d988c2

move LinearRegression out of the module

abde489

Because of max depth recursion error while pickling

Add uses_pytorch marker to the torch tests

c340531

change importing order

a3786c9

Modify torch tests

cde3380

fixup torch test string

67b5d42

Merge pull request #18 from AnandInguva/inference-snippets

4160d1b

Inference snippets

AnandInguva approved these changes Jul 15, 2022

View reviewed changes

yeandy approved these changes Jul 15, 2022

View reviewed changes

tvalentyn approved these changes Jul 15, 2022

View reviewed changes

tvalentyn changed the title ~~Rszper run inference docs~~ Update RunInference documentation Jul 15, 2022

Updates based on comments

4aa492c

tvalentyn reviewed Jul 15, 2022

View reviewed changes

rszper added 2 commits July 15, 2022 22:22

Fixed link location

c1d5643

Added a period

4e89126

tvalentyn merged commit fa028d3 into apache:master Jul 15, 2022

rszper deleted the rszper-runInferenceDocs branch July 18, 2022 15:57

tvalentyn reviewed Jul 18, 2022

View reviewed changes

website/www/site/layouts/partials/section-menu/en/sdks.html Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update RunInference documentation #22250

Update RunInference documentation #22250

rszper commented Jul 12, 2022

rszper commented Jul 12, 2022

yeandy commented Jul 12, 2022

github-actions bot commented Jul 12, 2022

rezarokni Jul 12, 2022

rszper Jul 12, 2022

AnandInguva commented Jul 13, 2022

yeandy left a comment

rszper commented Jul 13, 2022

yeandy left a comment

ryanthompson591 Jul 13, 2022

rszper Jul 13, 2022

yeandy Jul 13, 2022

rszper Jul 13, 2022

yeandy Jul 13, 2022

rszper Jul 13, 2022

yeandy left a comment

codecov bot commented Jul 15, 2022 •

edited

Loading

AnandInguva commented Jul 15, 2022 •

edited

Loading

yeandy commented Jul 15, 2022

tvalentyn Jul 15, 2022 •

edited

Loading

rszper Jul 15, 2022

tvalentyn Jul 15, 2022

rszper Jul 15, 2022

tvalentyn Jul 15, 2022 •

edited

Loading

rszper Jul 15, 2022

tvalentyn Jul 15, 2022

tvalentyn Jul 15, 2022

rszper Jul 15, 2022

		@@ -48,6 +48,11 @@ language-specific implementation guidance.

		## Using Beam Python SDK in your ML pipelines

		To use the Beam Python SDK with your machine learning pipelines, you can either use the RunInference API or TensorFlow.


		### Shared helper class

		Instead of loading a model for each thread in the process, we use the `Shared` class, which allows us to load one model that is shared across all threads of each worker in a DoFn. For more information, see the


		Disable batching by overriding the `batch_elements_kwargs` function in your ModelHandler and setting the maximum batch size (`max_batch_size`) to one: `max_batch_size=1`. For more information, see
		[BatchElements PTransforms](/documentation/sdks/python-machine-learning/#batchelements-ptransform).

Update RunInference documentation #22250

Update RunInference documentation #22250

Conversation

rszper commented Jul 12, 2022

GitHub Actions Tests Status (on master branch)

rszper commented Jul 12, 2022

yeandy commented Jul 12, 2022

github-actions bot commented Jul 12, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnandInguva commented Jul 13, 2022

yeandy left a comment

Choose a reason for hiding this comment

rszper commented Jul 13, 2022

yeandy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yeandy left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 15, 2022 • edited Loading

Codecov Report

AnandInguva commented Jul 15, 2022 • edited Loading

yeandy commented Jul 15, 2022

tvalentyn Jul 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvalentyn Jul 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 15, 2022 •

edited

Loading

AnandInguva commented Jul 15, 2022 •

edited

Loading

tvalentyn Jul 15, 2022 •

edited

Loading

tvalentyn Jul 15, 2022 •

edited

Loading