Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resnet152 batch inference torch.compile example #3259

Merged
merged 2 commits into from
Jul 19, 2024
Merged

Resnet152 batch inference torch.compile example #3259

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c977aed
Adding an example for TorchServe to use `torch.compile` for a
andrius-meta Jul 19, 2024
f80842d
Merge branch 'master' into resnet152_torch_compile
agunapal Jul 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion examples/image_classifier/resnet_152_batch/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,32 @@
#### TorchServe inference with torch.compile of Resnet152 batch image classifier:
Run the commands given in following steps from the root directory of the repository. For example, if you cloned the repository into /home/my_path/serve, run the steps from /home/my_path/serve

First, we create a configuration file where we enable `torch.compile`. You can find more information about the available configuration options [here](https://pytorch.org/docs/stable/generated/torch.compile.html).

```bash
echo "pt2:
compile:
enable: True" > examples/image_classifier/resnet_152_batch/model-config.yaml
```

Sample commands to create a Resnet152 torch.compile model archive for batch inputs, register it on TorchServe and run image prediction:

```bash
wget https://download.pytorch.org/models/resnet152-394f9c45.pth
torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --config-file examples/image_classifier/resnet_152_batch/model-config.yaml
mkdir model-store
mv resnet-152-batch.mar model-store/
torchserve --start --model-store model-store --disable-token-auth --enable-model-api
curl -X POST "localhost:8081/models?model_name=resnet152&url=resnet-152-batch.mar&batch_size=4&max_batch_delay=5000&initial_workers=3&synchronous=true"
```

And run batched inference:
```bash
curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/croco.jpg &
curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/dog.jpg &
curl http://localhost:8080/predictions/resnet152 -T examples/image_classifier/resnet_152_batch/images/kitten.jpg &
```

#### Sample commands to create a resnet-152 eager mode model archive for batch inputs, register it on TorchServe and run image prediction
Run the commands given in following steps from the root directory of the repository. For example, if you cloned the repository into /home/my_path/serve, run the steps from /home/my_path/serve

Expand All @@ -6,7 +35,7 @@ wget https://download.pytorch.org/models/resnet152-394f9c45.pth
torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json
mkdir model-store
mv resnet-152-batch.mar model-store/
torchserve --start --model-store model-store --disable-token-auth --enable-model-api
torchserve --start --model-store model-store --disable-token-auth --enable-model-api
curl -X POST "localhost:8081/models?model_name=resnet152&url=resnet-152-batch.mar&batch_size=4&max_batch_delay=5000&initial_workers=3&synchronous=true"
```

Expand Down
Loading