feat: Add tensor type enforcement for converters #2324

gs-olive · 2023-09-18T19:48:13Z

Description

Add utilities to enforce tensor types for aten converters via direct reference
Add robust testing for utilities
Update decorator schemas throughout code

Fixes #2306

Type of change

New feature (non-breaking change which adds functionality)

Checklist:

[ x ] My code follows the style guidelines of this project (You can use the linters)
[ x ] I have performed a self-review of my own code
[ x ] I have commented my code, particularly in hard-to-understand areas and hacks
[ x ] I have made corresponding changes to the documentation
[ x ] I have added tests to verify my fix or my feature
[ x ] New and existing unit tests pass locally with my changes
[ x ] I have added the relevant labels to my PR in so that relevant reviewers are notified

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

py/torch_tensorrt/dynamo/conversion/converter_utils.py

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

gs-olive · 2023-09-19T23:04:44Z

py/torch_tensorrt/dynamo/conversion/converter_utils.py

+    # If the input is 64-bit, cast it to 32-bit for TRT freezing
+    if isinstance(input_val, torch.Tensor):
+        if input_val.dtype == torch.int64:
+            input_val = input_val.to(torch.int32)
+        elif input_val.dtype == torch.float64:
+            input_val = input_val.to(torch.float32)
+    elif isinstance(input_val, np.ndarray):
+        if input_val.dtype == np.int64:
+            input_val = input_val.astype(np.int32)
+        elif input_val.dtype == np.float64:
+            input_val = input_val.astype(np.float32)


Adds automatic type-casting for get_attr operators (cast to 32-bit). This function currently does not check the truncate_long_and_double argument since it applies to weights and additionally cannot easily access compiler-context variables without many code changes.

Additionally, this PR removes autocasting of boolean values to integers for constant freezing, since TRT documentation (link 1, link 2) indicates that boolean values are supported in this layer.

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

gs-olive · 2023-09-22T18:42:11Z

py/torch_tensorrt/dynamo/conversion/converter_utils.py

+    # If the input is 64-bit, cast it to 32-bit for TRT freezing
+    if isinstance(input_val, torch.Tensor):
+        if input_val.dtype == torch.int64:
+            input_val = input_val.to(torch.int32)
+        elif input_val.dtype == torch.float64:
+            input_val = input_val.to(torch.float32)


Augment TRTNetwork to store user settings

Conversion Context class --> TRTNetwork, CompileSettings

INetwork + Context Class, Network inputs // CTX inputs

ctx.net replace network calls in converters

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

gs-olive · 2023-09-29T18:27:29Z

py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py

+        0: (TRTTensor,),
+        1: (TRTTensor, np.ndarray, torch.Tensor),
+        2: (TRTTensor, np.ndarray, torch.Tensor),


Consider tuple ordering as priority for the converter

(np.ndarray, TRTTensor) implies np array preferred

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

- Add utilities to enforce tensor types for aten converters via direct reference - Add robust testing for utilities - Update decorator schemas throughout code

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

gs-olive self-assigned this Sep 18, 2023

facebook-github-bot added the cla signed label Sep 18, 2023

github-actions bot added component: api [Python] Issues re: Python API component: conversion Issues re: Conversion stage component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: tests Issues re: Tests labels Sep 18, 2023

github-actions bot requested a review from narendasan September 18, 2023 19:48

gs-olive force-pushed the tensor_type_enforcement branch from feea927 to 419348b Compare September 18, 2023 19:49

github-actions bot approved these changes Sep 18, 2023

View reviewed changes

gs-olive force-pushed the tensor_type_enforcement branch from 419348b to 568a531 Compare September 18, 2023 21:00

github-actions bot approved these changes Sep 18, 2023

View reviewed changes

gs-olive force-pushed the tensor_type_enforcement branch from 568a531 to 38b970c Compare September 19, 2023 02:09

github-actions bot approved these changes Sep 19, 2023

View reviewed changes

gs-olive force-pushed the tensor_type_enforcement branch from 38b970c to 25c337b Compare September 19, 2023 17:08

github-actions bot approved these changes Sep 19, 2023

View reviewed changes

gs-olive commented Sep 19, 2023

View reviewed changes

py/torch_tensorrt/dynamo/conversion/converter_utils.py Outdated Show resolved Hide resolved

gs-olive force-pushed the tensor_type_enforcement branch from 25c337b to 7fd8187 Compare September 19, 2023 23:01

github-actions bot approved these changes Sep 19, 2023

View reviewed changes

gs-olive commented Sep 19, 2023

View reviewed changes

gs-olive marked this pull request as ready for review September 19, 2023 23:05

github-actions bot approved these changes Sep 19, 2023

View reviewed changes

gs-olive commented Sep 22, 2023

View reviewed changes

gs-olive force-pushed the tensor_type_enforcement branch from 8e5ea07 to d3c8d9c Compare September 27, 2023 21:42

github-actions bot approved these changes Sep 27, 2023

View reviewed changes

gs-olive mentioned this pull request Sep 27, 2023

Update enforce_tensor_types to function with more granular typing #2349

Open

gs-olive force-pushed the tensor_type_enforcement branch from d3c8d9c to 5f02996 Compare September 28, 2023 01:26

github-actions bot approved these changes Sep 28, 2023

View reviewed changes

gs-olive force-pushed the tensor_type_enforcement branch from 0b568d3 to db1b7af Compare September 29, 2023 00:23

github-actions bot approved these changes Sep 29, 2023

View reviewed changes

gs-olive commented Sep 29, 2023

View reviewed changes

gs-olive force-pushed the tensor_type_enforcement branch 2 times, most recently from 6091c6a to be84ba5 Compare September 29, 2023 19:32

github-actions bot approved these changes Sep 29, 2023

View reviewed changes

gs-olive force-pushed the tensor_type_enforcement branch from be84ba5 to b92be0d Compare September 29, 2023 20:08

github-actions bot approved these changes Sep 29, 2023

View reviewed changes

gs-olive added 3 commits September 29, 2023 13:41

feat: Add tensor type enforcement for converters

6d68d5d

- Add utilities to enforce tensor types for aten converters via direct reference - Add robust testing for utilities - Update decorator schemas throughout code

fix: Full revamp of Conversion system

efc5bd7

fix: Upgrade to_numpy to allow boolean constants

45e9184

gs-olive force-pushed the tensor_type_enforcement branch from b92be0d to 45e9184 Compare September 29, 2023 20:46

github-actions bot approved these changes Sep 29, 2023

View reviewed changes

gs-olive merged commit 42e514b into main Sep 29, 2023

gs-olive deleted the tensor_type_enforcement branch September 29, 2023 21:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add tensor type enforcement for converters #2324

feat: Add tensor type enforcement for converters #2324

gs-olive commented Sep 18, 2023

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

gs-olive Sep 19, 2023

github-actions bot left a comment

github-actions bot left a comment

gs-olive Sep 22, 2023

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

gs-olive Sep 29, 2023

gs-olive Sep 29, 2023

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

feat: Add tensor type enforcement for converters #2324

feat: Add tensor type enforcement for converters #2324

Conversation

gs-olive commented Sep 18, 2023

Description

Type of change

Checklist:

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

gs-olive Sep 19, 2023

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

gs-olive Sep 22, 2023

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

gs-olive Sep 29, 2023

Choose a reason for hiding this comment

gs-olive Sep 29, 2023

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment