Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Triton ensemble export #2251

Merged
merged 65 commits into from
Aug 16, 2022
Merged

Triton ensemble export #2251

Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
db994c6
adding basic logic and dataclasses
Jul 8, 2022
66c078c
modified `InferenceModule.postprocessor_forward` to return what the p…
Jul 8, 2022
11a6cec
undo flattening change.
Jul 8, 2022
fdc8398
finalized export details for preprocessor, predictor, and postprocessor
Jul 9, 2022
bc8cc43
debugged export. properly works now
Jul 9, 2022
a561517
adding ensemble config
Jul 9, 2022
c15c371
added triton ensemble exporting
Jul 10, 2022
4571dec
fixed formatting and commenting
Jul 10, 2022
7c14761
added docstrings
Jul 11, 2022
a0235d4
fixed docstring
Jul 11, 2022
22f1564
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 11, 2022
06d2999
Merge branch 'master' into triton-ensemble-export
Jul 15, 2022
3586245
using `to_inference_module_input_from_dataframe`
Jul 15, 2022
574aed4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 19, 2022
fc7d08c
Merge branch 'master' into triton-ensemble-export
Jul 19, 2022
4e5e706
bug fix in wrapper export
Jul 20, 2022
0f43ce7
support model with single input/output feature
Jul 21, 2022
b8980c7
making sure forward of the wrapper module uses `torch.no_grad()`
Jul 21, 2022
978f4ae
adding `load_paths=True` to support image and audio loading
Jul 21, 2022
4132e05
returning [-1] for variable size features (e.g. set feature)
Jul 21, 2022
7cc1c10
need head(1) to get the right dimensions
Jul 21, 2022
a33a3bc
adding ensemble dummy model
Jul 22, 2022
2ad3eab
merging with master
Jul 22, 2022
f7cee0f
fixed input types for preprocessor to those accepted by triton
Jul 25, 2022
8f8a2de
TO BE REMOVED (for experimentin only): returning flattened outputs fo…
Jul 25, 2022
a3498ab
experimental: supporting dynamic batching
Jul 26, 2022
7a7508f
properly exporting models to work on CPU/GPU
Jul 28, 2022
5ddcbcf
pass instead of raising error because it fails on torch.script
Aug 4, 2022
78cdf2b
adding support for max_batch_size, reshape, dynamic batching export
Aug 4, 2022
9fca6eb
fixed reshape in triton config
Aug 9, 2022
11ff13f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 9, 2022
66c50d1
Merge branch 'master' into triton-ensemble-export
Aug 9, 2022
821ec23
flake8 fixes
Aug 9, 2022
84cc83f
not prepending ensemble to the ensemble model
Aug 9, 2022
8045eec
merging master
Aug 9, 2022
9fac336
removed batching/squeezing in `inference.py`
Aug 9, 2022
70ee332
changed ensemble config so that input and output feature names match …
Aug 9, 2022
0a68bd8
update to feature output name in ensemble config
Aug 9, 2022
78051b1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 9, 2022
d634d24
added `max_batch_size` as an argument to `export_triton`
Aug 9, 2022
01735c8
added `max_queue_delay_microseconds` as an argument to `export_triton`
Aug 9, 2022
5737ff2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 9, 2022
199e842
updated default value of `max_batch_size``
Aug 9, 2022
a0d92aa
formatting of exported configs
Aug 9, 2022
8c404bf
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 9, 2022
95df21a
remove empty lines for config string formatting
Aug 9, 2022
10ac81c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 9, 2022
49b2419
changed order of args to triton export
Aug 10, 2022
5d30014
editing input params
Aug 15, 2022
584f2de
made input data optional
Aug 15, 2022
b32af1a
added model size
Aug 15, 2022
1c43a73
adding docstrings
Aug 15, 2022
747b72e
updating docstrings
Aug 15, 2022
8f631ef
Merge branch 'master' into triton-ensemble-export
abidwael Aug 15, 2022
77b393b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 15, 2022
f08a889
added named arguments to TritonMaster
Aug 16, 2022
4144a72
added tests
Aug 16, 2022
aed003d
removing to see where the tests problem is
Aug 16, 2022
b0c4a66
remove testing for triton
Aug 16, 2022
d55c35b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 16, 2022
741312f
reduced input types in the test
Aug 16, 2022
8952224
returning List[TritonArtifacts]
Aug 16, 2022
f76c9dc
style improvements
Aug 16, 2022
2f052db
added cli export
Aug 16, 2022
e34d411
artifact size edits
Aug 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ludwig/utils/torch_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,8 @@ def input_dtype(self):
@abstractmethod
def input_shape(self) -> torch.Size:
"""Returns size of the input tensor without the batch dimension."""
raise NotImplementedError("Abstract class.")
pass
# raise NotImplementedError("Abstract class.")

@property
def output_shape(self) -> torch.Size:
Expand Down
Loading