You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docsrc/tutorials/notebooks.rst
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,13 +23,13 @@ and running it to test the speedup obtained.
23
23
* `Torch-TensorRT Getting Started - CitriNet <https://github.com/pytorch/TensorRT/blob/master/notebooks/CitriNet-example.ipynb>`_
24
24
25
25
26
-
Compiling EfficentNet with Torch-TensorRT
26
+
Compiling EfficientNet with Torch-TensorRT
27
27
********************************************
28
28
29
-
EfficentNet is a feedforward CNN designed to achieve better performance and accuracy than alternative architectures
29
+
EfficientNet is a feedforward CNN designed to achieve better performance and accuracy than alternative architectures
30
30
by using a "scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient".
31
31
32
-
This notebook demonstrates the steps for optimizing a pretrained EfficentNet model with Torch-TensorRT,
32
+
This notebook demonstrates the steps for optimizing a pretrained EfficientNet model with Torch-TensorRT,
33
33
and running it to test the speedup obtained.
34
34
35
35
* `Torch-TensorRT Getting Started - EfficientNet-B0 <https://github.com/pytorch/TensorRT/blob/master/notebooks/EfficientNet-example.ipynb>`_
@@ -43,7 +43,7 @@ This way, the model learns an inner representation of the English language that
43
43
features useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train
44
44
a standard classifier using the features produced by the BERT model as inputs." (https://huggingface.co/bert-base-uncased)
45
45
46
-
This notebook demonstrates the steps for optimizing a pretrained EfficentNet model with Torch-TensorRT,
46
+
This notebook demonstrates the steps for optimizing a pretrained EfficientNet model with Torch-TensorRT,
47
47
and running it to test the speedup obtained.
48
48
49
49
* `Masked Language Modeling (MLM) with Hugging Face BERT Transformer <https://github.com/pytorch/TensorRT/blob/master/notebooks/Hugging-Face-BERT.ipynb>`_
@@ -73,7 +73,7 @@ Using Dynamic Shapes with Torch-TensorRT
73
73
74
74
Making use of Dynamic Shaped Tensors in Torch TensorRT is quite simple. Let's say you are
75
75
using the ``torch_tensorrt.compile(...)`` function to compile a torchscript module. One
76
-
of the args in this function in this function is ``input``: which defines an input to a
76
+
of the args in this function is ``input``: which defines an input to a
77
77
module in terms of expected shape, data type and tensor format: ``torch_tensorrt.Input.``
78
78
79
79
For the purposes of this walkthrough we just need three kwargs: `min_shape`, `opt_shape`` and `max_shape`.
@@ -96,8 +96,8 @@ In this example, we are going to use a simple ResNet model to demonstrate the us
96
96
Using the FX Frontend with Torch-TensorRT
97
97
********************************************
98
98
99
-
The purpose of this example is to demostrate the overall flow of lowering a PyTorch model to TensorRT
100
-
conveniently with using FX.
99
+
The purpose of this example is to demonstrate the overall flow of lowering a PyTorch model to TensorRT
100
+
conveniently using FX.
101
101
102
102
* `Using the FX Frontend with Torch-TensorRT <https://github.com/pytorch/TensorRT/blob/master/notebooks/getting_started_with_fx_path_lower_to_trt.ipynb>`_
Copy file name to clipboardExpand all lines: docsrc/user_guide/dynamic_shapes.rst
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,9 +6,9 @@ Dynamic shapes with Torch-TensorRT
6
6
By default, you can run a pytorch model with varied input shapes and the output shapes are determined eagerly.
7
7
However, Torch-TensorRT is an AOT compiler which requires some prior information about the input shapes to compile and optimize the model.
8
8
In the case of dynamic input shapes, we must provide the (min_shape, opt_shape, max_shape) arguments so that the model can be optimized for
9
-
these range of input shapes. An example usage of static and dynamic shapes is as follows.
9
+
this range of input shapes. An example usage of static and dynamic shapes is as follows.
10
10
11
-
NOTE: The following code uses Dynamo Frontend. Incase of Torchscript Frontend, please swap out ``ir=dynamo`` with ``ir=ts`` and the behavior is exactly the same.
11
+
NOTE: The following code uses Dynamo Frontend. In case of Torchscript Frontend, please swap out ``ir=dynamo`` with ``ir=ts`` and the behavior is exactly the same.
Copy file name to clipboardExpand all lines: docsrc/user_guide/ptq.rst
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,14 +13,14 @@ Users writing TensorRT applications are required to setup a calibrator class whi
13
13
the TensorRT calibrator. With Torch-TensorRT we look to leverage existing infrastructure in PyTorch to make implementing
14
14
calibrators easier.
15
15
16
-
LibTorch provides a ``DataLoader`` and ``Dataset`` API which steamlines preprocessing and batching input data.
16
+
LibTorch provides a ``DataLoader`` and ``Dataset`` API which streamlines preprocessing and batching input data.
17
17
These APIs are exposed via both C++ and Python interface which makes it easier for the end user.
18
18
For C++ interface, we use ``torch::Dataset`` and ``torch::data::make_data_loader`` objects to construct and perform pre-processing on datasets.
19
19
The equivalent functionality in python interface uses ``torch.utils.data.Dataset`` and ``torch.utils.data.DataLoader``.
20
20
This section of the PyTorch documentation has more information https://pytorch.org/tutorials/advanced/cpp_frontend.html#loading-data and https://pytorch.org/tutorials/recipes/recipes/loading_data_recipe.html.
21
21
Torch-TensorRT uses Dataloaders as the base of a generic calibrator implementation. So you will be able to reuse or quickly
22
-
implement a ``torch::Dataset`` for your target domain, place it in a DataLoader and create a INT8 Calibrator
23
-
which you can provide to Torch-TensorRT to run INT8 Calibration during compliation of your module.
22
+
implement a ``torch::Dataset`` for your target domain, place it in a DataLoader and create an INT8 Calibrator
23
+
which you can provide to Torch-TensorRT to run INT8 Calibration during compilation of your module.
24
24
25
25
.. _writing_ptq_cpp:
26
26
@@ -108,7 +108,7 @@ Next we create a calibrator from the ``calibration_dataloader`` using the calibr
108
108
109
109
Here we also define a location to write a calibration cache file to which we can use to reuse the calibration data without needing the dataset and whether or not
110
110
we should use the cache file if it exists. There also exists a ``torch_tensorrt::ptq::make_int8_cache_calibrator`` factory which creates a calibrator that uses the cache
111
-
only for cases where you may do engine building on a machine that has limited storage (i.e. no space for a full dataset) or to have a simpiler deployment application.
111
+
only for cases where you may do engine building on a machine that has limited storage (i.e. no space for a full dataset) or to have a simpler deployment application.
112
112
113
113
The calibrator factories create a calibrator that inherits from a ``nvinfer1::IInt8Calibrator`` virtual class (``nvinfer1::IInt8EntropyCalibrator2`` by default) which
114
114
defines the calibration algorithm used when calibrating. You can explicitly make the selection of calibration algorithm like this:
@@ -118,7 +118,7 @@ defines the calibration algorithm used when calibrating. You can explicitly make
118
118
// MinMax Calibrator is geared more towards NLP tasks
119
119
auto calibrator = torch_tensorrt::ptq::make_int8_calibrator<nvinfer1::IInt8MinMaxCalibrator>(std::move(calibration_dataloader), calibration_cache_file, true);
120
120
121
-
Then all thats required to setup the module for INT8 calibration is to set the following compile settings in the `torch_tensorrt::CompileSpec` struct and compiling the module:
121
+
Then all that's required to setup the module for INT8 calibration is to set the following compile settings in the `torch_tensorrt::CompileSpec` struct and compiling the module:
Copy file name to clipboardExpand all lines: docsrc/user_guide/runtime.rst
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ Deploying Torch-TensorRT Programs
4
4
====================================
5
5
6
6
After compiling and saving Torch-TensorRT programs there is no longer a strict dependency on the full
7
-
Torch-TensorRT library. All that is required to run a compiled program is the runtime. There are therfore a couple
7
+
Torch-TensorRT library. All that is required to run a compiled program is the runtime. There are therefore a couple
8
8
options to deploy your programs other than shipping the full Torch-TensorRT compiler with your applications.
9
9
10
10
Torch-TensorRT package / libtorchtrt.so
@@ -24,7 +24,7 @@ programs just as you would otherwise via PyTorch API.
24
24
25
25
.. note:: If you are using the standard distribution of PyTorch in Python on x86, likely you will need the pre-cxx11-abi variant of ``libtorchtrt_runtime.so``, check :ref:`Installation` documentation for more details.
26
26
27
-
.. note:: If you are linking ``libtorchtrt_runtime.so``, likely using the following flags will help ``-Wl,--no-as-needed -ltorchtrt -Wl,--as-needed`` as theres no direct symbol dependency to anything in the Torch-TensorRT runtime for most Torch-TensorRT runtime applications
27
+
.. note:: If you are linking ``libtorchtrt_runtime.so``, likely using the following flags will help ``-Wl,--no-as-needed -ltorchtrt -Wl,--as-needed`` as there's no direct symbol dependency to anything in the Torch-TensorRT runtime for most Torch-TensorRT runtime applications
28
28
29
29
An example of how to use ``libtorchtrt_runtime.so`` can be found here: https://github.com/pytorch/TensorRT/tree/master/examples/torchtrt_runtime_example
30
30
@@ -33,7 +33,7 @@ Plugin Library
33
33
34
34
In the case you use Torch-TensorRT as a converter to a TensorRT engine and your engine uses plugins provided by Torch-TensorRT, Torch-TensorRT
35
35
ships the library ``libtorchtrt_plugins.so`` which contains the implementation of the TensorRT plugins used by Torch-TensorRT during
36
-
compilation. This library can be ``DL_OPEN`` or ``LD_PRELOAD`` similar to other TensorRT plugin libraries.
36
+
compilation. This library can be ``DL_OPEN`` or ``LD_PRELOAD`` similarly to other TensorRT plugin libraries.
37
37
38
38
Multi Device Safe Mode
39
39
---------------
@@ -60,7 +60,7 @@ doubles as a context manager.
60
60
TensorRT requires that each engine be associated with the CUDA context in the active thread from which it is invoked.
61
61
Therefore, if the device were to change in the active thread, which may be the case when invoking
62
62
engines on multiple GPUs from the same Python process, safe mode will cause Torch-TensorRT to display
63
-
an alert and switch GPUs accordingly. If safe mode were not enabled, there could be a mismatch in the engine
63
+
an alert and switch GPUs accordingly. If safe mode is not enabled, there could be a mismatch in the engine
64
64
device and CUDA context device, which could lead the program to crash.
65
65
66
66
One technique for managing multiple TRT engines on different GPUs while not sacrificing performance for
0 commit comments