Current date and time : 2023-11-15 20:18:40 ## In directory /home/serve | Executing command pip install --force-reinstall . Processing /home/serve Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting Pillow (from torchserve==0.9.0b20231115) Using cached Pillow-10.1.0-cp39-cp39-manylinux_2_28_x86_64.whl.metadata (9.5 kB) Collecting psutil (from torchserve==0.9.0b20231115) Using cached psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB) Collecting packaging (from torchserve==0.9.0b20231115) Using cached packaging-23.2-py3-none-any.whl.metadata (3.2 kB) Collecting wheel (from torchserve==0.9.0b20231115) Using cached wheel-0.41.3-py3-none-any.whl.metadata (2.2 kB) Using cached packaging-23.2-py3-none-any.whl (53 kB) Using cached Pillow-10.1.0-cp39-cp39-manylinux_2_28_x86_64.whl (3.6 MB) Using cached psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB) Using cached wheel-0.41.3-py3-none-any.whl (65 kB) Building wheels for collected packages: torchserve Building wheel for torchserve (setup.py): started Building wheel for torchserve (setup.py): finished with status 'done' Created wheel for torchserve: filename=torchserve-0.9.0b20231115-py3-none-any.whl size=24037463 sha256=8b41d7d4f619d0f4738947f1fbc7a00e32c6f93073156fcc3ddb0f1c7ba4f355 Stored in directory: /tmp/pip-ephem-wheel-cache-petmt8ft/wheels/6d/38/c4/d57c82e47d3f18099ac5fba9f11f3825ce511d9b1f2f8d50c5 Successfully built torchserve Installing collected packages: wheel, psutil, Pillow, packaging, torchserve Attempting uninstall: wheel Found existing installation: wheel 0.41.3 Uninstalling wheel-0.41.3: Successfully uninstalled wheel-0.41.3 Attempting uninstall: psutil Found existing installation: psutil 5.9.6 Uninstalling psutil-5.9.6: Successfully uninstalled psutil-5.9.6 Attempting uninstall: Pillow Found existing installation: Pillow 10.1.0 Uninstalling Pillow-10.1.0: Successfully uninstalled Pillow-10.1.0 Attempting uninstall: packaging Found existing installation: packaging 23.2 Uninstalling packaging-23.2: Successfully uninstalled packaging-23.2 Attempting uninstall: torchserve Found existing installation: torchserve 0.9.0b20231115 Uninstalling torchserve-0.9.0b20231115: Successfully uninstalled torchserve-0.9.0b20231115 Successfully installed Pillow-10.1.0 packaging-23.2 psutil-5.9.6 torchserve-0.9.0b20231115 wheel-0.41.3 ## In directory /home/serve | Executing command pip install --force-reinstall model-archiver/. Processing ./model-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting enum-compat (from torch-model-archiver==0.9.0b20231115) Using cached enum_compat-0.0.3-py3-none-any.whl (1.3 kB) Building wheels for collected packages: torch-model-archiver Building wheel for torch-model-archiver (setup.py): started Building wheel for torch-model-archiver (setup.py): finished with status 'done' Created wheel for torch-model-archiver: filename=torch_model_archiver-0.9.0b20231115-py3-none-any.whl size=15940 sha256=2c26b77242c38528d9cfb9b62550661633513af24ac0eb15240f6a091595af45 Stored in directory: /root/.cache/pip/wheels/85/66/f4/3ead5e817bebb7f6bdb15fb471fd1377af8dd4011c24ac70fb Successfully built torch-model-archiver Installing collected packages: enum-compat, torch-model-archiver Attempting uninstall: enum-compat Found existing installation: enum-compat 0.0.3 Uninstalling enum-compat-0.0.3: Successfully uninstalled enum-compat-0.0.3 Attempting uninstall: torch-model-archiver Found existing installation: torch-model-archiver 0.9.0b20231115 Uninstalling torch-model-archiver-0.9.0b20231115: Successfully uninstalled torch-model-archiver-0.9.0b20231115 Successfully installed enum-compat-0.0.3 torch-model-archiver-0.9.0b20231115 ## In directory /home/serve | Executing command pip install --force-reinstall workflow-archiver/. Processing ./workflow-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Building wheels for collected packages: torch-workflow-archiver Building wheel for torch-workflow-archiver (setup.py): started Building wheel for torch-workflow-archiver (setup.py): finished with status 'done' Created wheel for torch-workflow-archiver: filename=torch_workflow_archiver-0.2.11b20231115-py3-none-any.whl size=12736 sha256=5bbe19b2fedf499201b192d4b6687f6f8c63e4e2c33aa08b15da27fce7e53952 Stored in directory: /root/.cache/pip/wheels/a3/04/9a/c72fed76bb7f18215ba80e9ff64c51fdb3c6ce131a6aeda80d Successfully built torch-workflow-archiver Installing collected packages: torch-workflow-archiver Attempting uninstall: torch-workflow-archiver Found existing installation: torch-workflow-archiver 0.2.11b20231115 Uninstalling torch-workflow-archiver-0.2.11b20231115: Successfully uninstalled torch-workflow-archiver-0.2.11b20231115 Successfully installed torch-workflow-archiver-0.2.11b20231115 ## Starting generate_mars, mar_config:/home/serve/ts_scripts/../ts_scripts/mar_config.json, model_store_dir:/home/serve/ts_scripts/../model_store_gen ## In directory: /home/serve | Executing command: torch-model-archiver --model-name fastrcnn --version 1.0 --model-file examples/object_detector/fast-rcnn/model.py --serialized-file /home/serve/ts_scripts/../model_store_gen/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## fastrcnn.mar is generated. ## In directory: /home/serve | Executing command: torch-model-archiver --model-name alexnet --version 1.0 --model-file examples/image_classifier/alexnet/model.py --serialized-file /home/serve/ts_scripts/../model_store_gen/alexnet-owt-7be5be79.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## alexnet.mar is generated. ## In directory: /home/serve | Executing command: torch-model-archiver --model-name densenet161 --version 1.0 --model-file examples/image_classifier/densenet_161/model.py --serialized-file /home/serve/ts_scripts/../model_store_gen/densenet161-8d451a50.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## densenet161.mar is generated. ## In directory: /home/serve | Executing command: torch-model-archiver --model-name mnist --version 1.0 --model-file examples/image_classifier/mnist/mnist.py --serialized-file examples/image_classifier/mnist/mnist_cnn.pt --handler examples/image_classifier/mnist/mnist_handler.py --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## mnist.mar is generated. ## In directory: /home/serve | Executing command: torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file /home/serve/ts_scripts/../model_store_gen/resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## resnet-152-batch.mar is generated. ## In directory: /home/serve | Executing command: torch-model-archiver --model-name resnet-18 --version 1.0 --model-file examples/image_classifier/resnet_18/model.py --serialized-file /home/serve/ts_scripts/../model_store_gen/resnet18-f37072fd.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## resnet-18.mar is generated. ## In directory: /home/serve | Executing command: torch-model-archiver --model-name squeezenet1_1 --version 1.0 --model-file examples/image_classifier/squeezenet/model.py --serialized-file /home/serve/ts_scripts/../model_store_gen/squeezenet1_1-b8a52dc0.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## squeezenet1_1.mar is generated. ## In directory: /home/serve | Executing command: torch-model-archiver --model-name vgg16 --version 1.0 --model-file examples/image_classifier/vgg_16/model.py --serialized-file /home/serve/ts_scripts/../model_store_gen/vgg16-397923af.pth --handler examples/image_classifier/vgg_16/vgg_handler.py --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## vgg16.mar is generated. ## In directory: /home/serve | Executing command: torch-model-archiver --model-name deeplabv3_resnet_101_eager --version 1.0 --model-file examples/image_segmenter/deeplabv3/model.py --serialized-file /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet101_coco-586e9e4e.pth --handler image_segmenter --extra-files examples/image_segmenter/deeplabv3/deeplabv3.py,examples/image_segmenter/deeplabv3/intermediate_layer_getter.py,examples/image_segmenter/deeplabv3/fcn.py --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## deeplabv3_resnet_101_eager.mar is generated. ## In directory: /home/serve | Executing command: torch-model-archiver --model-name fcn_resnet_101 --version 1.0 --model-file examples/image_segmenter/fcn/model.py --serialized-file /home/serve/ts_scripts/../model_store_gen/fcn_resnet101_coco-7ecb50ca.pth --handler image_segmenter --extra-files examples/image_segmenter/fcn/fcn.py,examples/image_segmenter/fcn/intermediate_layer_getter.py --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## fcn_resnet_101.mar is generated. ## In directory: /home/serve | Executing command: torch-model-archiver --model-name maskrcnn --version 1.0 --model-file examples/object_detector/maskrcnn/model.py --serialized-file /home/serve/ts_scripts/../model_store_gen/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --archive-format zip-store --export-path /home/serve/ts_scripts/../model_store_gen --force ## maskrcnn.mar is generated. ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --ncs ## Successfully started TorchServe newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 184ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 47ms 5ms 174µs 738µs 168ms 8ms 419µs 230ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 39ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 588µs (cache) (cache) 34ms 3ms 158µs 40ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 619ms time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 372µs (cache) (cache) 614ms 3ms 80µs 621ms ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 7ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 270µs (cache) (cache) 3ms 2ms 37µs 8ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 47ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 554µs 24µs 149µs 43ms 1ms 55µs 47ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 4ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 293µs (cache) (cache) 1ms 1ms 32µs 4ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 17ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 428µs 17µs 128µs 14ms 1ms 43µs 18ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 31ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 327µs (cache) (cache) 28ms 1ms 32µs 33ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 7ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 431µs (cache) (cache) 3ms 2ms 29µs 7ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 28ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 220µs (cache) (cache) 25ms 1ms 29µs 28ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 1560ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 219µs (cache) (cache) 1557ms 2ms 49µs 1561ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 19ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 428µs 18µs 164µs 15ms 2ms 45µs 19ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 6ms time ★ 250B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 4ms 1ms 31µs 6ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 4ms time ★ 250B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 250µs (cache) (cache) 1ms 1ms 30µs 4ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 3ms time ★ 246B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 245µs (cache) (cache) 1ms 1ms 30µs 5ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 227µs (cache) (cache) 3ms 1ms 28µs 6ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 5ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 224µs (cache) (cache) 3ms 1ms 27µs 5ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 16ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 400µs 16µs 121µs 13ms 1ms 30µs 16ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 29ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 244µs (cache) (cache) 27ms 1ms 30µs 30ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 5ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 259µs (cache) (cache) 1ms 1ms 39µs 5ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 5ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 339µs (cache) (cache) 2ms 1ms 38µs 8ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 5ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 253µs (cache) (cache) 2ms 1ms 31µs 5ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 4ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 325µs (cache) (cache) 1ms 1ms 38µs 5ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 16ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 563µs 24µs 223µs 12ms 1ms 46µs 17ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 4ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 349µs (cache) (cache) 1ms 1ms 37µs 5ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 4ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 304µs (cache) (cache) 1ms 1ms 35µs 5ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 20ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 529µs 23µs 158µs 16ms 1ms 96µs 20ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 14ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 530µs 22µs 151µs 10ms 1ms 30µs 15ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 29ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 245µs (cache) (cache) 27ms 1ms 32µs 29ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 236µs (cache) (cache) 3ms 1ms 28µs 5ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1901ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 250µs (cache) (cache) 1898ms 2ms 47µs 1901ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 29ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 417µs 16µs 132µs 26ms 1ms 29µs 29ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 260µs (cache) (cache) 2ms 1ms 29µs 5ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 1090ms time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 1088ms 1ms 34µs 1091ms ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 41ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 240µs (cache) (cache) 39ms 1ms 30µs 41ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 6.6s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 269µs (cache) (cache) 6.6s 1ms 35µs 6.6s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 51ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 46ms 3ms 33µs 51ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 6.3s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 250µs (cache) (cache) 6.3s 3ms 100µs 6.3s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 50ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 308µs (cache) (cache) 47ms 1ms 33µs 50ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 29ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 253µs (cache) (cache) 26ms 1ms 28µs 29ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 229µs (cache) (cache) 2ms 1ms 27µs 5ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 222ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 222µs (cache) (cache) 219ms 1ms 32µs 222ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 13ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 241µs (cache) (cache) 10ms 1ms 28µs 13ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 230ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 243µs (cache) (cache) 226ms 2ms 46µs 230ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 12ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 9ms 1ms 28µs 12ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 219ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 234µs (cache) (cache) 216ms 2ms 32µs 219ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 4ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 1ms 1ms 39µs 4ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 3ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 230µs (cache) (cache) 1ms 1ms 26µs 3ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 3ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 225µs (cache) (cache) 822µs 1ms 27µs 3ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 3ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 224µs (cache) (cache) 1ms 996µs 26µs 3ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 220µs (cache) (cache) 1ms 1ms 36µs 3ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 226µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 3ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 3ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 234µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 4ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 226µs (cache) (cache) 1ms 1ms 26µs 4ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 4ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 1ms 1ms 27µs 4ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 4ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 658ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 638µs 23µs 155µs 654ms 2ms 66µs 658ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 5ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 623µs 25µs 181µs 1ms 1ms 40µs 6ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 4ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 539µs 22µs 170µs 1ms 1ms 31µs 5ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 4ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 396µs 16µs 138µs 1ms 1ms 28µs 5ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 4ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 4ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 241µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 4ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 246µs (cache) (cache) 1ms 1ms 57µs 5ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 6ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 232µs (cache) (cache) 1ms 3ms 35µs 6ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 5ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 413µs 16µs 145µs 1ms 1ms 29µs 5ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 4ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 388µs 15µs 135µs 1ms 1ms 27µs 5ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 8ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 230µs (cache) (cache) 5ms 1ms 30µs 9ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 5ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 2ms 1ms 40µs 5ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 3ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 394µs 15µs 138µs 1ms 1ms 27µs 4ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 3ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 444µs 16µs 135µs 1ms 1ms 27µs 3ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 12ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 366µs 15µs 133µs 10ms 971µs 27µs 12ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 3ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 218µs (cache) (cache) 1ms 1ms 37µs 4ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 42ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 369µs 16µs 114µs 40ms 1ms 30µs 42ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 9ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 230µs (cache) (cache) 3ms 4ms 35µs 9ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 48ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 44ms 2ms 39µs 48ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 7ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 333µs (cache) (cache) 4ms 1ms 39µs 8ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 39ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 37ms 1ms 31µs 39ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 5ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 232µs (cache) (cache) 2ms 1ms 29µs 5ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 27ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 983µs 226µs (cache) (cache) 25ms 978µs 31µs 27ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 5ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 224µs (cache) (cache) 2ms 1ms 31µs 5ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 4ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 978µs 210µs (cache) (cache) 1ms 1ms 26µs 4ms ✓ Successful request ┌─────────────────────────┬──────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴──────────────────────┴─────────────────────┤ │ total run duration: 22.2s │ ├──────────────────────────────────────────────────────────────────────┤ │ total data received: 8.02kB (approx) │ ├──────────────────────────────────────────────────────────────────────┤ │ average response time: 250ms [min: 3ms, max: 6.6s, s.d.: 1031ms] │ ├──────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 87µs [min: 15µs, max: 174µs, s.d.: 76µs] │ ├──────────────────────────────────────────────────────────────────────┤ │ average first byte time: 248ms [min: 822µs, max: 6.6s, s.d.: 1031ms] │ └──────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman inference Iteration 1/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_v4.mar&model_name=my_text_classifier&initial_workers=1&synchronous=true 200 OK ★ 10.4s time ★ 388B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"my_text_classifier\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 42ms 5ms 179µs 773µs 10.4s 8ms 416µs 10.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier 200 OK ★ 53ms time ★ 353B↑ 373B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 138B │ { │ "World": 0.02911965735256672, │ "Sports": 2.9431601433316246e-05, │ "Business": 0.9074352383613586, │ "Sci/Tec": 0.06341567635536194 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 40µs 512µs 46ms 3ms 150µs 60ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier 200 OK ★ 86ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"my_text_classifier\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 369µs (cache) (cache) 83ms 2ms 91µs 87ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 20ms time ★ 233B↑ 4.06kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.77kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="7045e2f666c3",} 25.29 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="my_text_classifier",Level="M │ odel",Hostname="7045e2f666c3",} 25.58 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="7045e2f666c3",} 201.2 │ 504119873047 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="7045e2f666c3",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="my_text_clas │ sifier",model_version="default",hostname="7045e2f666c3 │ ",} 148.282 │ # HELP WorkerLoadTime Torchserve prometheus gauge metr │ ic with unit: Milliseconds │ # TYPE WorkerLoadTime gauge │ WorkerLoadTime{WorkerName="W-9000-my_text_classifier_1 │ .0",Level="Host",Hostname="7045e2f666c3",} 5742.0 │ # HELP DiskUtilization Torchserve prometheus gauge met │ ric with unit: Percent │ # │ (showing 2.05kB/3.77kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 458µs 17µs 175µs 17ms 1ms 35µs 21ms ✓ Successful GET request Iteration 2/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_scripted_v3.mar&model_name=my_text_classifier_scripted&initial_workers=1&synchronous=true 200 OK ★ 8.5s time ★ 406B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 103B │ { │ "status": "Model \"my_text_classifier_scripted\" Ver │ sion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 269µs (cache) (cache) 8.5s 3ms 99µs 8.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier_scripted 200 OK ★ 81ms time ★ 362B↑ 372B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 137B │ { │ "World": 0.04559721797704697, │ "Sports": 0.0003771769697777927, │ "Business": 0.08623101562261581, │ "Sci/Tec": 0.8677946329116821 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 449µs (cache) (cache) 78ms 1ms 37µs 83ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier_scripted 200 OK ★ 84ms time ★ 263B↑ 335B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 69B │ { │ "status": "Model \"my_text_classifier_scripted\" unr │ egistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 278µs (cache) (cache) 81ms 1ms 43µs 85ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 4.76kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.47kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 3.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="7045e2f666c3",} 25.29 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="7045e2f666c3",} 73.99 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="my_text_classifier",Level="M │ odel",Hostname="7045e2f666c3",} 25.58 │ PredictionTime{ModelName="my_text_classifier_scripted" │ ,Level="Model",Hostname="7045e2f666c3",} 74.28 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="7045e2f666c3",} 201.2 │ 504119873047 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="7045e2f666c3",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="my_text_clas │ sifier",model_version="d │ (showing 2.05kB/4.47kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 261µs (cache) (cache) 2ms 1ms 37µs 6ms ✓ Successful GET request Iteration 3/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 3.9s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 246µs (cache) (cache) 3.9s 1ms 36µs 3.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1 200 OK ★ 256ms time ★ 111.25kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299155712127686, │ "tiger_cat": 0.24496380984783173, │ "Egyptian_cat": 0.21722552180290222, │ "cougar": 0.0022175442427396774 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 332µs (cache) (cache) 253ms 1ms 38µs 257ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 9ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 262µs (cache) (cache) 7ms 1ms 32µs 9ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 8ms time ★ 233B↑ 5.38kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.09kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 0.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="7045e2f666c3",} 25.29 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="7045e2f666c3",} 73.99 │ HandlerTime{ModelName="squeezenet1_1",Level="Model",Ho │ stname="7045e2f666c3",} 246.17 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="my_text_classifier",Level="M │ odel",Hostname="7045e2f666c3",} 25.58 │ PredictionTime{ModelName="my_text_classifier_scripted" │ ,Level="Model",Hostname="7045e2f666c3",} 74.28 │ PredictionTime{ModelName="squeezenet1_1",Level="Model" │ ,Hostname="7045e2f666c3",} 246.49 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="7045e2f666c3",} 201.2 │ 504119873047 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPU │ (showing 2.05kB/5.09kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 325µs (cache) (cache) 4ms 2ms 53µs 9ms ✓ Successful GET request Iteration 4/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/squeezenet1_1_scripted.mar&model_name=squeezenet1_1_scripted&initial_workers=1&synchronous=true 200 OK ★ 4.1s time ★ 393B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 98B │ { │ "status": "Model \"squeezenet1_1_scripted\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 251µs (cache) (cache) 4.1s 2ms 36µs 4.1s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1_scripted 200 OK ★ 344ms time ★ 111.25kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299155712127686, │ "tiger_cat": 0.24496380984783173, │ "Egyptian_cat": 0.21722552180290222, │ "cougar": 0.0022175442427396774 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 268µs (cache) (cache) 341ms 1ms 38µs 346ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1_scripted 200 OK ★ 10ms time ★ 258B↑ 330B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 64B │ { │ "status": "Model \"squeezenet1_1_scripted\" unregist │ ered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 275µs (cache) (cache) 8ms 1ms 32µs 11ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 6.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.76kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="7045e2f666c3",} 25.29 │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="7045e2f666c3",} 335.52 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="7045e2f666c3",} 73.99 │ HandlerTime{ModelName="squeezenet1_1",Level="Model",Ho │ stname="7045e2f666c3",} 246.17 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="my_text_classifier",Level="M │ odel",Hostname="7045e2f666c3",} 25.58 │ PredictionTime{ModelName="squeezenet1_1_scripted",Leve │ l="Model",Hostname="7045e2f666c3",} 335.83 │ PredictionTime{ModelName="my_text_classifier_scripted" │ ,Level="Model",Hostname="7045e2f666c3",} 74.28 │ PredictionTime{ModelName="squeezenet1_1",Level=" │ (showing 2.05kB/5.76kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 3ms 2ms 53µs 7ms ✓ Successful GET request Iteration 5/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161&initial_workers=1&synchronous=true 200 OK ★ 5.1s time ★ 330B↑ 353B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 87B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 251µs (cache) (cache) 5.1s 1ms 35µs 5.1s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161 200 OK ★ 322ms time ★ 111.24kB↑ 416B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 181B │ { │ "tabby": 0.4666188061237335, │ "tiger_cat": 0.46449077129364014, │ "Egyptian_cat": 0.06614017486572266, │ "lynx": 0.001292433706112206, │ "plastic_bag": 0.00022909622930455953 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 284µs (cache) (cache) 319ms 1ms 39µs 323ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161 200 OK ★ 37ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 258µs (cache) (cache) 34ms 1ms 33µs 37ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 6.66kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.36kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 0.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="7045e2f666c3",} 25.29 │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="7045e2f666c3",} 335.52 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="7045e2f666c3",} 73.99 │ HandlerTime{ModelName="densenet161",Level="Model",Host │ name="7045e2f666c3",} 313.44 │ HandlerTime{ModelName="squeezenet1_1",Level="Model",Ho │ stname="7045e2f666c3",} 246.17 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="my_text_classifier",Level="M │ odel",Hostname="7045e2f666c3",} 25.58 │ PredictionTime{ModelName="squeezenet1_1_scr │ (showing 2.05kB/6.36kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 2ms 1ms 39µs 5ms ✓ Successful GET request Iteration 6/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=alexnet.mar&model_name=alexnet&initial_workers=1&synchronous=true 200 OK ★ 5.7s time ★ 322B↑ 349B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 83B │ { │ "status": "Model \"alexnet\" Version: 1.0 registered │ with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 5.7s 1ms 35µs 5.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet 200 OK ★ 234ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.31847354769706726, │ "tiger_cat": 0.25793972611427307, │ "Egyptian_cat": 0.24254822731018066, │ "lynx": 0.16879358887672424, │ "tiger": 0.006487949751317501 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 412µs (cache) (cache) 231ms 1ms 39µs 236ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet 200 OK ★ 48ms time ★ 243B↑ 315B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 49B │ { │ "status": "Model \"alexnet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 282µs (cache) (cache) 46ms 1ms 36µs 49ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 7.24kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.95kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="7045e2f666c3",} 25.29 │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="7045e2f666c3",} 335.52 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="7045e2f666c3",} 73.99 │ HandlerTime{ModelName="densenet161",Level="Model",Host │ name="7045e2f666c3",} 313.44 │ HandlerTime{ModelName="squeezenet1_1",Level="Model",Ho │ stname="7045e2f666c3",} 246.17 │ HandlerTime{ModelName="alexnet",Level="Model",Hostname │ ="7045e2f666c3",} 225.54 │ # HELP PredictionTime Torchserve promet │ (showing 2.05kB/6.95kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 271µs (cache) (cache) 1ms 1ms 35µs 4ms ✓ Successful GET request Iteration 7/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/alexnet_scripted.mar&model_name=alexnet_scripted&initial_workers=1&synchronous=true 200 OK ★ 8.6s time ★ 381B↑ 358B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 92B │ { │ "status": "Model \"alexnet_scripted\" Version: 1.0 r │ egistered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 290µs (cache) (cache) 8.6s 2ms 71µs 8.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet_scripted 200 OK ★ 309ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.31847354769706726, │ "tiger_cat": 0.25793972611427307, │ "Egyptian_cat": 0.24254822731018066, │ "lynx": 0.16879358887672424, │ "tiger": 0.006487949751317501 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 409µs (cache) (cache) 306ms 1ms 40µs 311ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet_scripted 200 OK ★ 84ms time ★ 252B↑ 324B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58B │ { │ "status": "Model \"alexnet_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 315µs (cache) (cache) 81ms 1ms 37µs 85ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 7.87kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.58kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="7045e2f666c3",} 25.29 │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="7045e2f666c3",} 335.52 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="7045e2f666c3",} 73.99 │ HandlerTime{ModelName="densenet161",Level="Model",Host │ name="7045e2f666c3",} 313.44 │ HandlerTime{ModelName="squeezenet1_1",Level="Model",Ho │ stname="7045e2f666c3",} │ (showing 2.05kB/7.58kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 292µs (cache) (cache) 2ms 1ms 45µs 5ms ✓ Successful GET request Iteration 8/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&initial_workers=1&synchronous=true 200 OK ★ 4.4s time ★ 326B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 282µs (cache) (cache) 4.4s 1ms 36µs 4.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18 200 OK ★ 245ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.4096633195877075, │ "tiger_cat": 0.3467046320438385, │ "Egyptian_cat": 0.13002899289131165, │ "lynx": 0.023919468745589256, │ "bucket": 0.011532159522175789 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 281µs (cache) (cache) 242ms 1ms 42µs 246ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18 200 OK ★ 17ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 275µs (cache) (cache) 15ms 1ms 33µs 18ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 8.47kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.17kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 0.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="7045e2f666c3",} 25.29 │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="7045e2f666c3",} 335.52 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="7045e2f666c3",} 73.99 │ HandlerTime{ModelName="densenet161",Level=" │ (showing 2.05kB/8.17kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 265µs (cache) (cache) 2ms 1ms 34µs 6ms ✓ Successful GET request Iteration 9/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-18_scripted.mar&model_name=resnet-18_scripted&initial_workers=1&synchronous=true 200 OK ★ 5s time ★ 385B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"resnet-18_scripted\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 275µs (cache) (cache) 5s 1ms 42µs 5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18_scripted 200 OK ★ 347ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.4096633195877075, │ "tiger_cat": 0.3467046320438385, │ "Egyptian_cat": 0.13002899289131165, │ "lynx": 0.023919468745589256, │ "bucket": 0.011532159522175789 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 289µs (cache) (cache) 344ms 1ms 34µs 348ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18_scripted 200 OK ★ 24ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"resnet-18_scripted\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 272µs (cache) (cache) 21ms 1ms 322µs 25ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 9.12kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.82kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 3.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="7045e2f666c3",} 25.29 │ HandlerTime{ModelName="resnet-18_scripted",Level="Mode │ l",Hostname="7045e2f666c3",} 338.8 │ HandlerTime{ModelNa │ (showing 2.05kB/8.82kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 2ms 2ms 42µs 6ms ✓ Successful GET request Iteration 10/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=vgg16.mar&model_name=vgg16&initial_workers=1&synchronous=true 200 OK ★ 7.8s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"vgg16\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 7.8s 1ms 38µs 7.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16 200 OK ★ 238ms time ★ 111.24kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tiger_cat": 0.44697248935699463, │ "tabby": 0.4408800005912781, │ "Egyptian_cat": 0.059045590460300446, │ "tiger": 0.020596392452716827, │ "lynx": 0.009934596717357635 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 295µs (cache) (cache) 234ms 2ms 32µs 239ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16 200 OK ★ 100ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"vgg16\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 289µs (cache) (cache) 96ms 1ms 34µs 100ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 9.69kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.39kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="7045e2f666c3",} 23140 │ 4.612 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="7045e2f666c3",} │ (showing 2.05kB/9.39kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 269µs (cache) (cache) 3ms 1ms 48µs 6ms ✓ Successful GET request Iteration 11/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/vgg16_scripted.mar&model_name=vgg16_scripted&initial_workers=1&synchronous=true 200 OK ★ 14.1s time ★ 377B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"vgg16_scripted\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 258µs (cache) (cache) 14.1s 1ms 39µs 14.1s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16_scripted 200 OK ★ 292ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tiger_cat": 0.44697248935699463, │ "tabby": 0.4408800005912781, │ "Egyptian_cat": 0.059045590460300446, │ "tiger": 0.020596392452716827, │ "lynx": 0.009934596717357635 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 273µs (cache) (cache) 289ms 2ms 46µs 294ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16_scripted 200 OK ★ 180ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"vgg16_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 299µs (cache) (cache) 177ms 1ms 37µs 181ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 10.31kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 10.02kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="7045e2f666c3",} 23140 │ 4.612 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 0.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ (showing 2.05kB/10.02kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 285µs (cache) (cache) 2ms 1ms 43µs 6ms ✓ Successful GET request Iteration 12/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_v2.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 4.1s time ★ 362B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 2.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 263µs (cache) (cache) 4.1s 1ms 41µs 4.1s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist 200 OK ★ 197ms time ★ 537B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 309µs (cache) (cache) 194ms 1ms 35µs 198ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 9ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 7ms 1ms 30µs 10ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 10.88kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 10.58kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="7045e2f666c3",} 19353 │ 0.54 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="7045e2f666c3",} 23140 │ 4.612 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime │ (showing 2.05kB/10.58kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 269µs (cache) (cache) 2ms 2ms 57µs 7ms ✓ Successful GET request Iteration 13/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_scripted_v2.mar&model_name=mnist_scripted&initial_workers=1&synchronous=true 200 OK ★ 4s time ★ 380B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"mnist_scripted\" Version: 2.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 4s 1ms 40µs 4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist_scripted 200 OK ★ 266ms time ★ 546B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 274µs (cache) (cache) 264ms 1ms 34µs 267ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist_scripted 200 OK ★ 11ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"mnist_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 8ms 1ms 37µs 11ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 11.5kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 11.21kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="7045e2f666c3",} 19353 │ 0.54 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="7045e2f666c3",} 23140 │ 4.612 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host" │ (showing 2.05kB/11.21kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 314µs (cache) (cache) 2ms 1ms 39µs 6ms ✓ Successful GET request Iteration 14/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fastrcnn.mar&model_name=fastrcnn&initial_workers=1&synchronous=true 200 OK ★ 6.2s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"fastrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 6.2s 1ms 35µs 6.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fastrcnn 200 OK ★ 441ms time ★ 289.21kB↑ 3.01kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.77kB │ [ │ { │ "person": [ │ 167.39581298828125, │ 57.203041076660156, │ 301.3599853515625, │ 436.7974853515625 │ ], │ "score": 0.999519944190979 │ }, │ { │ "person": [ │ 89.54701232910156, │ 64.83567810058594, │ 191.42428588867188, │ 446.7607727050781 │ ], │ "score": 0.9994966983795166 │ }, │ { │ "person": [ │ 362.37823486328125, │ 161.84133911132812, │ 515.5022583007812, │ 385.28985595703125 │ ], │ "score": 0.997706413269043 │ }, │ { │ "handbag": [ │ 67.37623596191406, │ 277.5755615234375, │ 111.67391204833984, │ 400.206787109375 │ ], │ "score": 0.992499053478241 │ }, │ { │ "handbag": [ │ 228.6824951171875, │ 146.00697326660156, │ 303.55120849609375, │ 231.08848571777344 │ ], │ "score": 0.9922404289245605 │ }, │ { │ "handbag": [ │ 379.411376953125, │ 259.957763671875, │ 419.0797424316406, │ 317.9610290527344 │ ], │ "score": 0.9898613691329956 │ }, │ { │ "person": [ │ 518.4950561523438, │ 149.73019409179688, │ 636.6343994140625, │ 365.4129333496094 │ ], │ "score": 0.9821107983589172 │ }, │ { │ "bench": [ │ 269.08184814453125, │ 217.34202575683594, │ 423.77099609375, │ 390.3786315917969 │ ], │ "score": 0.9573412537574768 │ }, │ { │ "person": [ │ 539.6298217773438, │ 157.75868225097656, │ 616.1533813476562, │ 253.1112823486328 │ ], │ "score": 0.8995409607887268 │ }, │ { │ "person": [ │ 477.06231689453125, │ 147.80885314941406, │ 610.777587890625, │ 296.83734130859375 │ ], │ "score": 0.8751544952392578 │ }, │ { │ "bench": [ │ 286.08746337890625, │ 216.53285217285156, │ 550.7698364257812, │ 383.1822509765625 │ ], │ "score": 0.8436442613601685 │ }, │ { │ "person": [ │ 627.4741821289062, │ 177.05838012695312, │ 640.0, │ 247.72152709960938 │ ], │ "score": 0.8257318139076233 │ }, │ { │ "bench": [ │ 88.76377868652344, │ 226.16490173339844, │ 563.614562988 │ (showing 2.05kB/2.77kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 781µs (cache) (cache) 438ms 1ms 40µs 442ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fastrcnn 200 OK ★ 34ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"fastrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 264µs (cache) (cache) 31ms 1ms 33µs 34ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 12.09kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 11.79kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="7045e2f666c3",} 19353 │ 0.54 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="7045e2f666c3",} 23140 │ 4.612 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 1.0 │ # HELP CPUUtil │ (showing 2.05kB/11.79kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 241µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful GET request Iteration 15/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=maskrcnn.mar&model_name=maskrcnn&initial_workers=1&synchronous=true 200 OK ★ 5.5s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"maskrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 303µs (cache) (cache) 5.5s 1ms 39µs 5.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/maskrcnn 200 OK ★ 465ms time ★ 289.21kB↑ 2.99kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.75kB │ [ │ { │ "person": [ │ 169.50636291503906, │ 49.98552322387695, │ 300.8945617675781, │ 442.4545593261719 │ ], │ "score": 0.999543309211731 │ }, │ { │ "person": [ │ 90.4118423461914, │ 66.79048919677734, │ 194.19305419921875, │ 437.2776794433594 │ ], │ "score": 0.9993956089019775 │ }, │ { │ "person": [ │ 362.3172912597656, │ 158.17355346679688, │ 521.2589721679688, │ 385.7226867675781 │ ], │ "score": 0.9952280521392822 │ }, │ { │ "handbag": [ │ 68.57540893554688, │ 279.31817626953125, │ 111.15328979492188, │ 400.9165954589844 │ ], │ "score": 0.9938817024230957 │ }, │ { │ "person": [ │ 474.0157165527344, │ 147.3479461669922, │ 638.1209716796875, │ 364.6508483886719 │ ], │ "score": 0.9897466897964478 │ }, │ { │ "handbag": [ │ 225.59584045410156, │ 142.9000244140625, │ 302.48638916015625, │ 230.3284149169922 │ ], │ "score": 0.9891214966773987 │ }, │ { │ "handbag": [ │ 380.2604675292969, │ 259.2012023925781, │ 419.5366516113281, │ 318.27728271484375 │ ], │ "score": 0.9688038229942322 │ }, │ { │ "bench": [ │ 273.48565673828125, │ 217.48834228515625, │ 441.06536865234375, │ 396.24169921875 │ ], │ "score": 0.961754560470581 │ }, │ { │ "person": [ │ 541.2896728515625, │ 156.66119384765625, │ 619.9386596679688, │ 249.45326232910156 │ ], │ "score": 0.8177024126052856 │ }, │ { │ "person": [ │ 362.9620361328125, │ 163.8992462158203, │ 500.7698059082031, │ 293.9122619628906 │ ], │ "score": 0.8016975522041321 │ }, │ { │ "chair": [ │ 455.20849609375, │ 207.54010009765625, │ 491.08526611328125, │ 274.6475830078125 │ ], │ "score": 0.7758335471153259 │ }, │ { │ "person": [ │ 549.1538696289062, │ 177.42056274414062, │ 640.0, │ 364.5394592285156 │ ], │ "score": 0.7176421880722046 │ }, │ { │ "person": [ │ 626.230712890625, │ 178.6534423828125, │ 640.0, │ 246. │ (showing 2.05kB/2.75kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 279µs (cache) (cache) 462ms 1ms 43µs 466ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/maskrcnn 200 OK ★ 50ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"maskrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 271µs (cache) (cache) 47ms 1ms 34µs 51ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 12.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12.38kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="7045e2f666c3",} 19353 │ 0.54 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="7045e2f666c3",} 23140 │ 4.612 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="7045e2f666c3",} 45 │ 7080.434 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: │ (showing 2.05kB/12.38kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 274µs (cache) (cache) 1ms 1ms 48µs 4ms ✓ Successful GET request Iteration 16/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fcn_resnet_101.mar&model_name=fcn_resnet_101&initial_workers=1&synchronous=true 200 OK ★ 7.2s time ★ 336B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"fcn_resnet_101\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 327µs (cache) (cache) 7.2s 1ms 34µs 7.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101 200 OK ★ 531ms time ★ 289.22kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993864297866821 │ ], │ [ │ 0.0, │ 0.9993854761123657 │ ], │ [ │ 0.0, │ 0.9993811845779419 │ ], │ [ │ 0.0, │ 0.9993742108345032 │ ], │ [ │ 0.0, │ 0.9993641972541809 │ ], │ [ │ 0.0, │ 0.9993515610694885 │ ], │ [ │ 0.0, │ 0.9993364214897156 │ ], │ [ │ 0.0, │ 0.9993187189102173 │ ], │ [ │ 0.0, │ 0.9992934465408325 │ ], │ [ │ 0.0, │ 0.9992607235908508 │ ], │ [ │ 0.0, │ 0.9992249011993408 │ ], │ [ │ 0.0, │ 0.9991866946220398 │ ], │ [ │ 0.0, │ 0.9991452693939209 │ ], │ [ │ 0.0, │ 0.9991005659103394 │ ], │ [ │ 0.0, │ 0.9990523457527161 │ ], │ [ │ 0.0, │ 0.9990004897117615 │ ], │ [ │ 0.0, │ 0.998968243598938 │ ], │ [ │ 0.0, │ 0.9989572763442993 │ ], │ [ │ 0.0, │ 0.9989456534385681 │ ], │ [ │ 0.0, │ 0.9989331364631653 │ ], │ [ │ 0.0, │ 0.9989200830459595 │ ], │ [ │ 0.0, │ 0.9989060163497925 │ ], │ [ │ 0.0, │ 0.998891294002533 │ ], │ [ │ 0.0, │ 0.9988754391670227 │ ], │ [ │ 0.0, │ 0.9988815188407898 │ ], │ [ │ 0.0, │ 0.9989089965820312 │ ], │ [ │ 0.0, │ 0.9989352822303772 │ ], │ [ │ 0.0, │ 0.9989607334136963 │ ], │ [ │ 0.0, │ 0.9989848732948303 │ ], │ [ │ 0.0, │ 0.9990076422691345 │ ], │ [ │ 0.0, │ 0.9990293979644775 │ ], │ [ │ 0.0, │ 0.9990498423576355 │ ], │ [ │ 0.0, │ 0.999051034450531 │ ], │ [ │ 0.0, │ 0.9990313053131104 │ ], │ [ │ 0.0, │ 0.9990096092224121 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.9989587068557739 │ ], │ [ │ 0.0, │ 0.99892979860 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 275µs (cache) (cache) 525ms 4ms 506µs 532ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101 200 OK ★ 46ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"fcn_resnet_101\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 370µs (cache) (cache) 43ms 1ms 68µs 47ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 13.3kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="7045e2f666c3 │ ",} 517639.136 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="7045e2f666c3",} 19353 │ 0.54 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="7045e2f666c3",} 23140 │ 4.612 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_ve │ (showing 2.05kB/13kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 292µs (cache) (cache) 1ms 1ms 41µs 4ms ✓ Successful GET request Iteration 17/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/fcn_resnet_101_scripted.mar&model_name=fcn_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 8.2s time ★ 395B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 99B │ { │ "status": "Model \"fcn_resnet_101_scripted\" Version │ : 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 8.2s 1ms 35µs 8.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101_scripted 200 OK ★ 893ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993864297866821 │ ], │ [ │ 0.0, │ 0.9993854761123657 │ ], │ [ │ 0.0, │ 0.9993811845779419 │ ], │ [ │ 0.0, │ 0.9993742108345032 │ ], │ [ │ 0.0, │ 0.9993641972541809 │ ], │ [ │ 0.0, │ 0.9993515610694885 │ ], │ [ │ 0.0, │ 0.9993364214897156 │ ], │ [ │ 0.0, │ 0.9993187189102173 │ ], │ [ │ 0.0, │ 0.9992934465408325 │ ], │ [ │ 0.0, │ 0.9992607235908508 │ ], │ [ │ 0.0, │ 0.9992249011993408 │ ], │ [ │ 0.0, │ 0.9991866946220398 │ ], │ [ │ 0.0, │ 0.9991452693939209 │ ], │ [ │ 0.0, │ 0.9991005659103394 │ ], │ [ │ 0.0, │ 0.9990523457527161 │ ], │ [ │ 0.0, │ 0.9990004897117615 │ ], │ [ │ 0.0, │ 0.998968243598938 │ ], │ [ │ 0.0, │ 0.9989572763442993 │ ], │ [ │ 0.0, │ 0.9989456534385681 │ ], │ [ │ 0.0, │ 0.9989331364631653 │ ], │ [ │ 0.0, │ 0.9989200830459595 │ ], │ [ │ 0.0, │ 0.9989060163497925 │ ], │ [ │ 0.0, │ 0.998891294002533 │ ], │ [ │ 0.0, │ 0.9988754391670227 │ ], │ [ │ 0.0, │ 0.9988815188407898 │ ], │ [ │ 0.0, │ 0.9989089965820312 │ ], │ [ │ 0.0, │ 0.9989352822303772 │ ], │ [ │ 0.0, │ 0.9989607334136963 │ ], │ [ │ 0.0, │ 0.9989848732948303 │ ], │ [ │ 0.0, │ 0.9990076422691345 │ ], │ [ │ 0.0, │ 0.9990293979644775 │ ], │ [ │ 0.0, │ 0.9990498423576355 │ ], │ [ │ 0.0, │ 0.999051034450531 │ ], │ [ │ 0.0, │ 0.9990313053131104 │ ], │ [ │ 0.0, │ 0.9990096092224121 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.9989587068557739 │ ], │ [ │ 0.0, │ 0.99892979860 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 320µs (cache) (cache) 889ms 2ms 495µs 894ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101_scripted 200 OK ★ 90ms time ★ 259B↑ 331B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 65B │ { │ "status": "Model \"fcn_resnet_101_scripted\" unregis │ tered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 421µs (cache) (cache) 87ms 1ms 40µs 92ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 13.97kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.67kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="7045e2f666c3 │ ",} 517639.136 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="704 │ 5e2f666c3",} 884707.378 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="7045e2f666c3",} 19353 │ 0.54 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ ts_inference_latency_microseconds{model_name=" │ (showing 2.05kB/13.67kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 263µs (cache) (cache) 1ms 1ms 36µs 4ms ✓ Successful GET request Iteration 18/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=deeplabv3_resnet_101_eager.mar&model_name=deeplabv3_resnet_101_eager&initial_workers=1&synchronous=true 200 OK ★ 6.3s time ★ 360B↑ 369B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" Vers │ ion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 285µs (cache) (cache) 6.3s 2ms 60µs 6.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_eager 200 OK ★ 633ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988669157028198 │ ], │ [ │ 0.0, │ 0.998843789100647 │ ], │ [ │ 0.0, │ 0.9988172650337219 │ ], │ [ │ 0.0, │ 0.9987861514091492 │ ], │ [ │ 0.0, │ 0.9987504482269287 │ ], │ [ │ 0.0, │ 0.9987117052078247 │ ], │ [ │ 0.0, │ 0.9986679553985596 │ ], │ [ │ 0.0, │ 0.9986202716827393 │ ], │ [ │ 0.0, │ 0.9985180497169495 │ ], │ [ │ 0.0, │ 0.9983478784561157 │ ], │ [ │ 0.0, │ 0.9981531500816345 │ ], │ [ │ 0.0, │ 0.9979324340820312 │ ], │ [ │ 0.0, │ 0.9976813793182373 │ ], │ [ │ 0.0, │ 0.9973964691162109 │ ], │ [ │ 0.0, │ 0.9970728158950806 │ ], │ [ │ 0.0, │ 0.9967058300971985 │ ], │ [ │ 0.0, │ 0.9964045286178589 │ ], │ [ │ 0.0, │ 0.9961955547332764 │ ], │ [ │ 0.0, │ 0.9959736466407776 │ ], │ [ │ 0.0, │ 0.9957382678985596 │ ], │ [ │ 0.0, │ 0.9954885840415955 │ ], │ [ │ 0.0, │ 0.9952237010002136 │ ], │ [ │ 0.0, │ 0.9949430823326111 │ ], │ [ │ 0.0, │ 0.994645357131958 │ ], │ [ │ 0.0, │ 0.9945149421691895 │ ], │ [ │ 0.0, │ 0.9945647716522217 │ ], │ [ │ 0.0, │ 0.9946145415306091 │ ], │ [ │ 0.0, │ 0.9946632385253906 │ ], │ [ │ 0.0, │ 0.9947116374969482 │ ], │ [ │ 0.0, │ 0.9947587847709656 │ ], │ [ │ 0.0, │ 0.9948055148124695 │ ], │ [ │ 0.0, │ 0.9948518872261047 │ ], │ [ │ 0.0, │ 0.9948965907096863 │ ], │ [ │ 0.0, │ 0.9949400424957275 │ ], │ [ │ 0.0, │ 0.9949839115142822 │ ], │ [ │ 0.0, │ 0.9950267672538757 │ ], │ [ │ 0.0, │ 0.9950693845748901 │ ], │ [ │ 0.0, │ 0.9951115250 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 355µs (cache) (cache) 628ms 3ms 543µs 635ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_eager 200 OK ★ 57ms time ★ 262B↑ 334B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 68B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" unre │ gistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 594µs (cache) (cache) 54ms 2ms 49µs 59ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 14.66kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 14.37kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="7045e2f666c3 │ ",} 517639.136 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="704 │ 5e2f666c3",} 884707.378 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 7045e2f666c3",} 623825.967 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="7045e2f666c3",} 19353 │ 0.54 │ ts_inference_latency_microsec │ (showing 2.05kB/14.37kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 278µs (cache) (cache) 2ms 1ms 35µs 4ms ✓ Successful GET request Iteration 19/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.s3.amazonaws.com/mar_files/deeplabv3_resnet_101_scripted.mar&model_name=deeplabv3_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 16.5s time ★ 412B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 242µs (cache) (cache) 16.5s 2ms 49µs 16.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_scripted 200 OK ★ 797ms time ★ 289.24kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988669157028198 │ ], │ [ │ 0.0, │ 0.998843789100647 │ ], │ [ │ 0.0, │ 0.9988172650337219 │ ], │ [ │ 0.0, │ 0.9987861514091492 │ ], │ [ │ 0.0, │ 0.9987504482269287 │ ], │ [ │ 0.0, │ 0.9987117052078247 │ ], │ [ │ 0.0, │ 0.9986679553985596 │ ], │ [ │ 0.0, │ 0.9986202716827393 │ ], │ [ │ 0.0, │ 0.9985180497169495 │ ], │ [ │ 0.0, │ 0.9983478784561157 │ ], │ [ │ 0.0, │ 0.9981531500816345 │ ], │ [ │ 0.0, │ 0.9979324340820312 │ ], │ [ │ 0.0, │ 0.9976813793182373 │ ], │ [ │ 0.0, │ 0.9973964691162109 │ ], │ [ │ 0.0, │ 0.9970728158950806 │ ], │ [ │ 0.0, │ 0.9967058300971985 │ ], │ [ │ 0.0, │ 0.9964045286178589 │ ], │ [ │ 0.0, │ 0.9961955547332764 │ ], │ [ │ 0.0, │ 0.9959736466407776 │ ], │ [ │ 0.0, │ 0.9957382678985596 │ ], │ [ │ 0.0, │ 0.9954885840415955 │ ], │ [ │ 0.0, │ 0.9952237010002136 │ ], │ [ │ 0.0, │ 0.9949430823326111 │ ], │ [ │ 0.0, │ 0.994645357131958 │ ], │ [ │ 0.0, │ 0.9945149421691895 │ ], │ [ │ 0.0, │ 0.9945647716522217 │ ], │ [ │ 0.0, │ 0.9946145415306091 │ ], │ [ │ 0.0, │ 0.9946632385253906 │ ], │ [ │ 0.0, │ 0.9947116374969482 │ ], │ [ │ 0.0, │ 0.9947587847709656 │ ], │ [ │ 0.0, │ 0.9948055148124695 │ ], │ [ │ 0.0, │ 0.9948518872261047 │ ], │ [ │ 0.0, │ 0.9948965907096863 │ ], │ [ │ 0.0, │ 0.9949400424957275 │ ], │ [ │ 0.0, │ 0.9949839115142822 │ ], │ [ │ 0.0, │ 0.9950267672538757 │ ], │ [ │ 0.0, │ 0.9950693845748901 │ ], │ [ │ 0.0, │ 0.9951115250 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 827µs (cache) (cache) 791ms 3ms 540µs 799ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_scripted 200 OK ★ 85ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 671µs (cache) (cache) 82ms 2ms 79µs 87ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 15.37kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 15.08kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="7045e2f666c3 │ ",} 517639.136 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_scripted",model_version="default",hostnam │ e="7045e2f666c3",} 787278.232 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="704 │ 5e2f666c3",} 884707.378 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 7045e2f666c3",} 623825.967 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="7045e2f666c3",} 227 │ 919.347 │ ts_i │ (showing 2.05kB/15.08kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 272µs (cache) (cache) 1ms 1ms 34µs 5ms ✓ Successful GET request Iteration 20/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&initial_workers=1&synchronous=true 200 OK ★ 6.3s time ★ 333B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 241µs (cache) (cache) 6.3s 1ms 40µs 6.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet152 200 OK ★ 205ms time ★ 111.24kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tiger_cat": 0.5798612833023071, │ "tabby": 0.38344183564186096, │ "Egyptian_cat": 0.0342114083468914, │ "lynx": 0.0005819810903631151, │ "quilt": 0.00027331968885846436 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 323µs (cache) (cache) 202ms 1ms 45µs 206ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet152 200 OK ★ 65ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 63ms 1ms 33µs 66ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 15.97kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 15.67kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="7045e2f666c3 │ ",} 517639.136 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_scripted",model_version="default",hostnam │ e="7045e2f666c3",} 787278.232 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="704 │ 5e2f666c3",} 884707.378 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="resnet15 │ 2",model_version="default",hostname="7045e2f666c3",} 2 │ 00810.321 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 7045e2f666c3",} 623825.967 │ ts │ (showing 2.05kB/15.67kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 240µs (cache) (cache) 1ms 1ms 44µs 4ms ✓ Successful GET request Iteration 21/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-152-batch_scripted.mar&model_name=resnet-152-batch_scripted&initial_workers=1&synchronous=true 200 OK ★ 8.7s time ★ 399B↑ 368B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "status": "Model \"resnet-152-batch_scripted\" Versi │ on: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 230µs (cache) (cache) 8.7s 1ms 40µs 8.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-152-batch_scripted 200 OK ★ 676ms time ★ 111.26kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tiger_cat": 0.5798612833023071, │ "tabby": 0.38344183564186096, │ "Egyptian_cat": 0.0342114083468914, │ "lynx": 0.0005819810903631151, │ "quilt": 0.00027331968885846436 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 354µs (cache) (cache) 671ms 3ms 50µs 677ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-152-batch_scripted 200 OK ★ 88ms time ★ 261B↑ 333B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 67B │ { │ "status": "Model \"resnet-152-batch_scripted\" unreg │ istered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 302µs (cache) (cache) 84ms 2ms 34µs 88ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 16.66kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 16.36kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="7045e2f666c3 │ ",} 517639.136 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_scripted",model_version="default",hostnam │ e="7045e2f666c3",} 787278.232 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="704 │ 5e2f666c3",} 884707.378 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="resnet15 │ 2",model_version="default",hostname="7045e2f666c3",} 2 │ 00810.321 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 7045e2f666c3",} 623825.967 │ ts │ (showing 2.05kB/16.36kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 246µs (cache) (cache) 1ms 1ms 38µs 4ms ✓ Successful GET request Iteration 22/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/distill_bert_qa_eager.mar&model_name=distill_bert_qa_eager&initial_workers=1&synchronous=true 200 OK ★ 11.5s time ★ 391B↑ 363B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "status": "Model \"distill_bert_qa_eager\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 235µs (cache) (cache) 11.5s 3ms 45µs 11.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/distill_bert_qa_eager 200 OK ★ 177ms time ★ 359B↑ 247B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 79B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13B │ a nice puppet └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 400µs (cache) (cache) 173ms 2ms 40µs 179ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/distill_bert_qa_eager 200 OK ★ 108ms time ★ 257B↑ 329B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Model \"distill_bert_qa_eager\" unregiste │ red" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 418µs (cache) (cache) 103ms 3ms 55µs 110ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 17.32kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 17.03kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="7045e2f666c3 │ ",} 517639.136 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_scripted",model_version="default",hostnam │ e="7045e2f666c3",} 787278.232 │ ts_inference_latency_microseconds{model_name="distill_ │ bert_qa_eager",model_version="default",hostname="7045e │ 2f666c3",} 172555.755 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="704 │ 5e2f666c3",} 884707.378 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="resnet15 │ 2",model_version="default",hostname="7045e2f666c3",} 2 │ 00810.321 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_infe │ (showing 2.05kB/17.03kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 363µs (cache) (cache) 2ms 2ms 99µs 7ms ✓ Successful GET request Iteration 23/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_token_classification_no_torchscript.mar&model_name=bert_token_classification_no_torchscript&initial_workers=1&synchronous=true 200 OK ★ 14.5s time ★ 429B↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 116B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" Version: 1.0 registered with 1 initial worke │ rs" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 1ms (cache) (cache) 14.5s 1ms 43µs 14.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_token_classification_no_torchscript 200 OK ★ 158ms time ★ 375B↑ 5.45kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.22kB │ [ │ [ │ "[CLS]", │ "B-LOC" │ ], │ [ │ "bloomberg", │ "O" │ ], │ [ │ "has", │ "I-ORG" │ ], │ [ │ "decided", │ "B-PER" │ ], │ [ │ "to", │ "O" │ ], │ [ │ "publish", │ "I-PER" │ ], │ [ │ "a", │ "I-MISC" │ ], │ [ │ "new", │ "B-PER" │ ], │ [ │ "report", │ "O" │ ], │ [ │ "on", │ "O" │ ], │ [ │ "global", │ "B-PER" │ ], │ [ │ "economic", │ "O" │ ], │ [ │ "situation", │ "B-LOC" │ ], │ [ │ ".", │ "B-PER" │ ], │ [ │ "[SEP]", │ "B-PER" │ ], │ [ │ "[PAD]", │ "I-ORG" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ (showing 2.05kB/5.22kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 353µs (cache) (cache) 155ms 1ms 44µs 160ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_token_classification_no_torchscript 200 OK ★ 138ms time ★ 276B↑ 348B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 82B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 257µs (cache) (cache) 135ms 1ms 37µs 138ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 18.1kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 17.81kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="bert_tok │ en_classification_no_torchscript",model_version="defau │ lt",hostname="7045e2f666c3",} 154570.787 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "7045e2f666c3",} 76323.148 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="7045e2f666c3 │ ",} 517639.136 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7045 │ e2f666c3",} 338254.928 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_scripted",model_version="default",hostnam │ e="7045e2f666c3",} 787278.232 │ ts_inference_latency_microseconds{model_name="distill_ │ bert_qa_eager",model_version="default",hostname="7045e │ 2f666c3",} 172555.755 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="704 │ 5e2f666c3",} 884707.378 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 302362.568 │ ts_inference_latency_microseconds{model_name="resnet15 │ 2",model_version="default",hostname="7045e2f666c3",} 2 │ 00810.321 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ (showing 2.05kB/17.81kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 253µs (cache) (cache) 2ms 1ms 36µs 5ms ✓ Successful GET request Iteration 24/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_seqc_without_torchscript.mar&model_name=bert_seqc_without_torchscript&initial_workers=1&synchronous=true 200 OK ★ 13.4s time ★ 407B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"bert_seqc_without_torchscript\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 236µs (cache) (cache) 13.4s 1ms 43µs 13.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_seqc_without_torchscript 200 OK ★ 145ms time ★ 364B↑ 246B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12B │ Not Accepted └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 283µs (cache) (cache) 142ms 1ms 44µs 146ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_seqc_without_torchscript 200 OK ★ 142ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"bert_seqc_without_torchscript\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 258µs (cache) (cache) 139ms 1ms 38µs 142ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 18.81kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 18.52kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="7045e2f666c3",} 43 │ 2339.611 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="7045e2f6 │ 66c3",} 28492.966 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 262766.655 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="7045e2f666c3 │ ",} 517639.136 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="7045e2f666c3" │ ,} 249137.923 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_scripted",model_version="default",hostnam │ e="7045e2f666c3",} 787278.232 │ ts_inference_latency_microseconds{model_name="distill_ │ bert_qa_eager",model_version="default",hostname="7045e │ 2f666c3",} 172555.755 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="7045e2f666c3",} │ 315744.694 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="704 │ 5e2f666c3",} 884707.378 │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="7045e2f666c3 │ ",} 286859.581 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="7045e2f6 │ 66c3",} 341075.507 │ ts_inference_latency_microseconds{model_name="bert_seq │ c_without_torchscript",model_version="default",hostnam │ e="7045e2f666c3",} 141168.135 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="7045e2f666c3",} 2 │ 39525.834 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="7045e2f666c3",} 23140 │ 4.612 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="7045e2f666c3",} 45 │ 7080.434 │ ts_inference_latency_microsec │ (showing 2.05kB/18.52kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 250µs (cache) (cache) 1ms 1ms 45µs 4ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬───────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ iterations │ 24 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ requests │ 96 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ test-scripts │ 96 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ assertions │ 113 │ 0 │ ├─────────────────────────┴─────────────────────┴───────────────────┤ │ total run duration: 3m 24.4s │ ├───────────────────────────────────────────────────────────────────┤ │ total data received: 10.18MB (approx) │ ├───────────────────────────────────────────────────────────────────┤ │ average response time: 2s [min: 3ms, max: 16.5s, s.d.: 3.8s] │ ├───────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 86µs [min: 17µs, max: 179µs, s.d.: 72µs] │ ├───────────────────────────────────────────────────────────────────┤ │ average first byte time: 2s [min: 1ms, max: 16.5s, s.d.: 3.8s] │ └───────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman inference → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161_scripted.mar&model_name=densenet161_scripted&initial_workers=1&synchronous=true 200 OK ★ 7.9s time ★ 389B↑ 362B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 96B │ { │ "status": "Model \"densenet161_scripted\" Version: 1 │ .0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 45ms 5ms 177µs 554µs 7.9s 12ms 647µs 7.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161_scripted 200 OK ★ 1178ms time ★ 111.25kB↑ 416B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 181B │ { │ "tabby": 0.4666188061237335, │ "tiger_cat": 0.46449077129364014, │ "Egyptian_cat": 0.06614017486572266, │ "lynx": 0.001292433706112206, │ "plastic_bag": 0.00022909622930455953 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 94µs 774µs 1172ms 3ms 354µs 1185ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161_scripted 200 OK ★ 73ms time ★ 256B↑ 328B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 62B │ { │ "status": "Model \"densenet161_scripted\" unregister │ ed" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 362µs (cache) (cache) 70ms 2ms 87µs 75ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 21ms time ★ 233B↑ 4.08kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.79kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="densenet │ 161_scripted",model_version="default",hostname="7045e2 │ f666c3",} 1142183.621 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="densenet161_scripted",Level="Mo │ del",Hostname="7045e2f666c3",} 1135.13 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="densenet161_scripted",Level= │ "Model",Hostname="7045e2f666c3",} 1136.87 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="7045e2f666c3",} 201.5 │ 321044921875 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="7045e2f666c3",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="densenet161_ │ scripted",model_version="default",hostname="7045e2f666 │ c3",} 157.685 │ # HELP WorkerLoadTime Torchserve prometheus gauge metr │ ic with unit: Milliseconds │ # TYPE WorkerLoadTime gauge │ WorkerLoadTime{WorkerName="W-9000-densenet161_scripted │ _1.0",Level="Host",Hostname="7045e2f666c3",} 4526.0 │ # HELP DiskUtilization Torchserve prometheus gauge met │ ric with │ (showing 2.05kB/3.79kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 501µs 20µs 144µs 18ms 1ms 43µs 22ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 9.4s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 4.13kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 2.3s [min: 21ms, max: 7.9s, s.d.: 3.2s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 106µs [min: 20µs, max: 177µs, s.d.: 61µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 2.2s [min: 18ms, max: 7.9s, s.d.: 3.2s] │ └────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config.properties ## Successfully started TorchServe newman https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 221ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 51ms 7ms 208µs 657µs 146ms 53ms 11ms 687µs 272ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 82ms time ★ 230B↑ 60.36kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60.09kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/60.09kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 3ms 1ms 43µs 351µs 41ms 33ms 3ms 170µs 85ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 54ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 735µs 16µs 204µs 44ms 4ms 2ms 50µs 54ms ✓ Status code is 200 → HTTPS Register Model - SqueezeNet POST https://localhost:8444/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 4.1s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 390µs (cache) (cache) (cache) 4.1s 2ms 136µs 4.1s ✓ Successful POST request → HTTPS Get SqueezeNet Model Description GET https://localhost:8444/models/squeezenet1_1 200 OK ★ 44ms time ★ 246B↑ 926B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 659B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-11-15T20:23:56.948Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 1383, │ "gpu": true, │ "gpuUsage": "gpuId::0 utilization.gpu [%]::4 % │ utilization.memory [%]::0 % memory.used [MiB]::127 Mi │ B" │ } │ ], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 401µs (cache) (cache) (cache) 41ms 1ms 51µs 45ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 5ms time ★ 308B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 331µs (cache) (cache) (cache) 3ms 1ms 35µs 6ms ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 9ms time ★ 309B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 316µs (cache) (cache) (cache) 3ms 4ms 34µs 10ms ✓ Successful PUT request → HTTPS - Inference - SqueezeNet POST https://localhost:8443/predictions/squeezenet1_1 200 OK ★ 285ms time ★ 111.25kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299155712127686, │ "tiger_cat": 0.24496380984783173, │ "Egyptian_cat": 0.21722552180290222, │ "cougar": 0.0022175442427396774 │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 6ms 818µs (cache) (cache) (cache) 282ms 1ms 40µs 290ms ✓ Status code is 200 → HTTPS UnRegister Model SqueezeNet DELETE https://localhost:8444/models/squeezenet1_1 200 OK ★ 22ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 317µs (cache) (cache) (cache) 19ms 1ms 39µs 22ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 9 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 5.2s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 87.13kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 539ms [min: 5ms, max: 4.1s, s.d.: 1274ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 89µs [min: 16µs, max: 208µs, s.d.: 84µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 508ms [min: 3ms, max: 4.1s, s.d.: 1283ms] │ └────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 202ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 44ms 5ms 179µs 557µs 187ms 7ms 399µs 246ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 36ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 559µs (cache) (cache) 32ms 2ms 106µs 37ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 560ms time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 290µs (cache) (cache) 556ms 2ms 48µs 561ms ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 7ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 282µs (cache) (cache) 3ms 1ms 33µs 7ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 45ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 582µs 24µs 183µs 42ms 1ms 57µs 47ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 5ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 255µs (cache) (cache) 1ms 2ms 44µs 6ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 19ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 438µs 17µs 134µs 15ms 1ms 47µs 19ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 32ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 339µs (cache) (cache) 29ms 1ms 34µs 33ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 9ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 522µs (cache) (cache) 4ms 3ms 32µs 9ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 30ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 26ms 2ms 45µs 30ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 1600ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 252µs (cache) (cache) 1597ms 2ms 38µs 1600ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 18ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 444µs 17µs 132µs 15ms 1ms 40µs 18ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 7ms time ★ 250B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 246µs (cache) (cache) 4ms 1ms 58µs 7ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 4ms time ★ 250B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 1ms 1ms 30µs 4ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 246B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 254µs (cache) (cache) 1ms 1ms 28µs 5ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 246µs (cache) (cache) 3ms 1ms 28µs 6ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 4ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 250µs (cache) (cache) 1ms 1ms 29µs 4ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 17ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 426µs 16µs 133µs 14ms 1ms 29µs 17ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 29ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 27ms 1ms 33µs 30ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 5ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 240µs (cache) (cache) 2ms 1ms 39µs 5ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 5ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 239µs (cache) (cache) 2ms 1ms 27µs 6ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 5ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 246µs (cache) (cache) 2ms 1ms 30µs 5ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 4ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 255µs (cache) (cache) 2ms 1ms 28µs 4ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 16ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 408µs 15µs 130µs 13ms 1ms 32µs 16ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 5ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 260µs (cache) (cache) 2ms 1ms 32µs 5ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 5ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 251µs (cache) (cache) 2ms 1ms 29µs 5ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 20ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 428µs 16µs 156µs 17ms 1ms 43µs 21ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 19ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 611µs 25µs 731µs 15ms 1ms 32µs 20ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 30ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 28ms 1ms 34µs 30ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 264µs (cache) (cache) 3ms 1ms 31µs 6ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1970ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 1967ms 1ms 42µs 1970ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 29ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 439µs 17µs 140µs 26ms 1ms 32µs 29ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 243µs (cache) (cache) 3ms 1ms 30µs 5ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 1095ms time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 270µs (cache) (cache) 1092ms 1ms 33µs 1095ms ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 43ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 41ms 1ms 31µs 43ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 6.4s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 348µs (cache) (cache) 6.4s 1ms 33µs 6.4s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 52ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 257µs (cache) (cache) 47ms 3ms 34µs 52ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 6.4s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 263µs (cache) (cache) 6.4s 3ms 69µs 6.4s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 63ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 298µs (cache) (cache) 60ms 1ms 34µs 64ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 29ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 271µs (cache) (cache) 26ms 1ms 32µs 29ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 270µs (cache) (cache) 3ms 1ms 27µs 6ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 218ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 244µs (cache) (cache) 216ms 1ms 66µs 219ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 13ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 251µs (cache) (cache) 10ms 1ms 29µs 13ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 226ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 222ms 2ms 44µs 227ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 12ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 259µs (cache) (cache) 9ms 1ms 28µs 12ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 221ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 232µs (cache) (cache) 218ms 1ms 32µs 221ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 4ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 271µs (cache) (cache) 1ms 1ms 32µs 4ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 3ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 1ms 1ms 29µs 3ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 4ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 284µs (cache) (cache) 1ms 2ms 30µs 4ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 4ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 1ms 1ms 37µs 4ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 4ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 4ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 653µs (cache) (cache) 1ms 1ms 28µs 5ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 5ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 253µs (cache) (cache) 1ms 1ms 39µs 5ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 4ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 257µs (cache) (cache) 1ms 1ms 29µs 4ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 4ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 244µs (cache) (cache) 1ms 1ms 41µs 5ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 4ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 2ms 1ms 29µs 4ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 4ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 253µs (cache) (cache) 1ms 1ms 29µs 4ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 649ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 662µs 22µs 160µs 646ms 1ms 52µs 650ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 7ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 442µs 17µs 152µs 4ms 1ms 30µs 7ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 5ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 572µs 24µs 171µs 2ms 1ms 30µs 6ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 4ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 399µs 16µs 155µs 1ms 1ms 52µs 4ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 4ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 261µs (cache) (cache) 2ms 1ms 30µs 4ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 4ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 324µs (cache) (cache) 1ms 1ms 52µs 6ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 4ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 339µs (cache) (cache) 1ms 1ms 29µs 5ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 6ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 242µs (cache) (cache) 1ms 3ms 37µs 6ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 5ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 414µs 17µs 153µs 2ms 1ms 29µs 5ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 4ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 391µs 16µs 145µs 1ms 1ms 28µs 5ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 7ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 235µs (cache) (cache) 5ms 1ms 31µs 8ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 5ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 251µs (cache) (cache) 2ms 1ms 30µs 5ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 3ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 373µs 15µs 119µs 1ms 1ms 27µs 4ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 3ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 496µs 15µs 128µs 1ms 1ms 26µs 4ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 13ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 361µs 15µs 115µs 10ms 1ms 30µs 13ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 3ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 54ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 405µs 16µs 132µs 51ms 2ms 40µs 54ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 12ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 342µs (cache) (cache) 4ms 6ms 41µs 13ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 44ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 381µs (cache) (cache) 41ms 1ms 33µs 46ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 6ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 259µs (cache) (cache) 3ms 1ms 29µs 6ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 29ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 27ms 1ms 35µs 29ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 5ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 244µs (cache) (cache) 3ms 1ms 28µs 6ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 28ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 240µs (cache) (cache) 26ms 1ms 37µs 28ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 5ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 2ms 1ms 29µs 5ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 3ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 715µs (cache) (cache) 1ms 1ms 27µs 4ms ✓ Successful request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 22.2s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 8.02kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 250ms [min: 3ms, max: 6.4s, s.d.: 1026ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 89µs [min: 15µs, max: 179µs, s.d.: 79µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 248ms [min: 1ms, max: 6.4s, s.d.: 1026ms] │ └────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman kf_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 4.2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 43ms 5ms 184µs 552µs 4.2s 7ms 464µs 4.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v1/models/mnist:predict 200 OK ★ 220ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 43µs 549µs 215ms 2ms 83µs 227ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v1/models/mnist:explain 200 OK ★ 58ms time ★ 680B↑ 25.4kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.16kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.0045709484202342545, │ 0.006216969527188252, │ 0.008197564504355558, │ 0.009563574636103758, │ 0.008999273563915732, │ 0.009673474031078457, │ 0.007599905521342725, │ 0.0063613809512706515, │ 0.0057688292873839305, │ 0.00439446596454604, │ 0.004948218040748549, │ 0.00527346076478521, │ 0.005523799543449043, │ 0.007789356530578066, │ 0.008759362944991762, │ 0.004304804422137636, │ 0.010970579496288352, │ 0.003248439108184719, │ 0.005998033215573371, │ 0.0037543660003001404, │ 0.002765290887789118, │ 0.004314086007904382, │ 0.0014008569476638513, │ 0.004841846312960897, │ 0.0006374844970870742, │ 0.0018558538624387638, │ -0.0008280457210026403, │ -0.0 │ ], │ [ │ 0.0016625160972090316, │ 0.004443792128154977, │ 0.012387838952488066, │ 0.009450843236210947, │ 0.016143821475303452, │ 0.007797501928939272, │ 0.013942238574964209, │ 0.007557429184420719, │ 0.005479089762020411, │ 0.009751321389975472, │ 0.0047644084845411185, │ 0.007292148220001189, │ 0.011797998655026407, │ 0.006462684382700551, │ 0.003383213356354625, │ 0.009225058425921365, │ 0.0016750690402313336, │ 0.007362304777231652, │ 0.005270057798930987, │ 0.005453597920636149, │ 0.004342725769625787, │ 0.005856132985561649, │ 0.012168384503340826, │ 0.009902719734876424, │ 0.009813112302984246, │ 0.0034427579317570524, │ 0.0022924286211484157, │ -0.0 │ ], │ [ │ 0.011528253688386956, │ 0.00914698814233881, │ 0.02226542199058181, │ 0.017558218433863886, │ 0.024770555937761955, │ 0.021412694443710016, │ (showing 2.05kB/25.16kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 471µs (cache) (cache) 52ms 4ms 96µs 60ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 24ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 336µs (cache) (cache) 21ms 1ms 50µs 24ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 22ms time ★ 233B↑ 3.99kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.7kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="7045e2f666c3",} 24739 │ 8.648 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="mnist",Level="Model",Hostname=" │ 7045e2f666c3",} 45.42 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="mnist",Level="Model",Hostnam │ e="7045e2f666c3",} 45.66 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="7045e2f666c3",} 201.5 │ 3445053100586 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="7045e2f666c3",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="mnist",model │ _version="default",hostname="7045e2f666c3",} 229.237 │ # HELP WorkerLoadTime Torchserve prometheus gauge metr │ ic with unit: Milliseconds │ # TYPE WorkerLoadTime gauge │ WorkerLoadTime{WorkerName="W-9000-mnist_1.0",Level="Ho │ st",Hostname="7045e2f666c3",} 4078.0 │ # HELP DiskUtilization Torchserve prometheus gauge met │ ric with unit: Percent │ # TYPE DiskUtilization gauge │ DiskUtilization{Level="Host",Hostn │ (showing 2.05kB/3.7kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 490µs 20µs 154µs 19ms 1ms 39µs 23ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 4.8s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 29.02kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 918ms [min: 22ms, max: 4.2s, s.d.: 1676ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 95µs [min: 20µs, max: 184µs, s.d.: 73µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 912ms [min: 19ms, max: 4.2s, s.d.: 1672ms] │ └─────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kf.properties ## Successfully started TorchServe newman kf_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 209ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 49ms 6ms 178µs 781µs 140ms 47ms 12ms 682µs 258ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 76ms time ★ 230B↑ 60.36kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60.09kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/60.09kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 3ms 1ms 39µs 344µs 47ms 23ms 2ms 104µs 79ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 42ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 737µs 16µs 249µs 32ms 6ms 1ms 39µs 42ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 4s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 399µs (cache) (cache) (cache) 4s 2ms 127µs 4s ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 43ms time ★ 238B↑ 910B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 643B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-11-15T20:24:38.776Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 1859, │ "gpu": true, │ "gpuUsage": "gpuId::0 utilization.gpu [%]::3 % │ utilization.memory [%]::0 % memory.used [MiB]::125 Mi │ B" │ } │ ], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 389µs (cache) (cache) (cache) 41ms 1ms 53µs 44ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 6.2s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 315µs (cache) (cache) (cache) 6.2s 1ms 40µs 6.2s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 9ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 316µs (cache) (cache) (cache) 3ms 4ms 44µs 10ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v1/models/mnist:predict 200 OK ★ 214ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 10ms 1ms (cache) (cache) (cache) 210ms 2ms 50µs 224ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v1/models/mnist:explain 200 OK ★ 248ms time ★ 680B↑ 25.38kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.14kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.0045709484202342545, │ 0.006216969527188252, │ 0.008197564504355558, │ 0.009563574636103758, │ 0.008999273563915732, │ 0.009673474031078457, │ 0.007599905521342725, │ 0.0063613809512706515, │ 0.0057688292873839305, │ 0.00439446596454604, │ 0.004948218040748549, │ 0.00527346076478521, │ 0.005523799543449043, │ 0.007789356530578066, │ 0.008759362944991762, │ 0.004304804422137636, │ 0.010970579496288352, │ 0.003248439108184719, │ 0.005998033215573371, │ 0.0037543660003001404, │ 0.002765290887789118, │ 0.004314086007904382, │ 0.0014008569476638513, │ 0.004841846312960897, │ 0.0006374844970870742, │ 0.0018558538624387638, │ -0.0008280457210026403, │ -0.0 │ ], │ [ │ 0.0016625160972090316, │ 0.004443792128154977, │ 0.012387838867815328, │ 0.009450843208410793, │ 0.016143821586504077, │ 0.007797501928939272, │ 0.013942238888548748, │ 0.007557429184420719, │ 0.005479089819898836, │ 0.009751321191989931, │ 0.004764408457787844, │ 0.007292148188114405, │ 0.01179799861156285, │ 0.006462684232265972, │ 0.0033832133094447267, │ 0.009225058125123989, │ 0.0016750689218179835, │ 0.007362304688368002, │ 0.005270057935714648, │ 0.005453597833452519, │ 0.004342725828013315, │ 0.005856132985561649, │ 0.012168384354003959, │ 0.00990271976267658, │ 0.009813112146191977, │ 0.003442757925234222, │ 0.0022924286211484157, │ -0.0 │ ], │ [ │ 0.011528253688386956, │ 0.009146988140040424, │ 0.02226542241881645, │ 0.017558218945622067, │ 0.0247705564993046, │ 0.02141269454470344, │ │ (showing 2.05kB/25.14kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 3ms 582µs (cache) (cache) (cache) 244ms 2ms 57µs 250ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 74ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 332µs (cache) (cache) (cache) 71ms 2ms 50µs 75ms ✓ Successful DELETE request ┌─────────────────────────┬────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴────────────────────┴────────────────────┤ │ total run duration: 11.7s │ ├───────────────────────────────────────────────────────────────────┤ │ total data received: 112.09kB (approx) │ ├───────────────────────────────────────────────────────────────────┤ │ average response time: 1129ms [min: 9ms, max: 6.2s, s.d.: 2s] │ ├───────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 78µs [min: 16µs, max: 178µs, s.d.: 71µs] │ ├───────────────────────────────────────────────────────────────────┤ │ average first byte time: 1102ms [min: 3ms, max: 6.2s, s.d.: 2.1s] │ └───────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman kfv2_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 4.2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 54ms 5ms 175µs 585µs 4.2s 9ms 420µs 4.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v2/models/mnist/infer 200 OK ★ 221ms time ★ 7kB↑ 490B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 255B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "input-0", │ "datatype": "INT64", │ "data": [ │ 1 │ ], │ "shape": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 43µs 554µs 216ms 2ms 87µs 228ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v2/models/mnist/explain 200 OK ★ 55ms time ★ 7kB↑ 14.05kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.81kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "input-0", │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004054752849657972, │ -0.00022612876876764663, │ -0.0001273414558919298, │ 0.005648369070481585, │ 0.00890478412155693, │ 0.002638536372014552, │ │ (showing 2.05kB/13.81kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 462µs (cache) (cache) 51ms 2ms 94µs 57ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 23ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 310µs (cache) (cache) 20ms 1ms 42µs 24ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 21ms time ★ 233B↑ 3.99kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.7kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="7045e2f666c3",} 24591 │ 7.92599999998 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="mnist",Level="Model",Hostname=" │ 7045e2f666c3",} 45.2 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="mnist",Level="Model",Hostnam │ e="7045e2f666c3",} 46.02 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="7045e2f666c3",} 201.5 │ 3519821166992 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="7045e2f666c3",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="mnist",model │ _version="default",hostname="7045e2f666c3",} 244.76 │ # HELP WorkerLoadTime Torchserve prometheus gauge metr │ ic with unit: Milliseconds │ # TYPE WorkerLoadTime gauge │ WorkerLoadTime{WorkerName="W-9000-mnist_1.0",Level="Ho │ st",Hostname="7045e2f666c3",} 4056.0 │ # HELP DiskUtilization Torchserve prometheus gauge met │ ric with unit: Percent │ # TYPE DiskUtilization gauge │ DiskUtilization{Level="Host",H │ (showing 2.05kB/3.7kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 497µs 17µs 160µs 18ms 1ms 37µs 21ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 4.8s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 17.9kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 918ms [min: 21ms, max: 4.2s, s.d.: 1677ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 91µs [min: 17µs, max: 175µs, s.d.: 69µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 912ms [min: 18ms, max: 4.2s, s.d.: 1672ms] │ └─────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kfv2.properties ## Successfully started TorchServe newman kfv2_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 222ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 46ms 6ms 175µs 771µs 148ms 53ms 11ms 415µs 268ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 65ms time ★ 230B↑ 60.36kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60.09kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/60.09kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 1ms 26µs 289µs 34ms 24ms 2ms 112µs 66ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 45ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 763µs 19µs 731µs 35ms 6ms 1ms 36µs 46ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 4.1s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 398µs (cache) (cache) (cache) 4.1s 2ms 128µs 4.1s ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 27ms time ★ 238B↑ 910B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 643B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-11-15T20:25:01.775Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 2185, │ "gpu": true, │ "gpuUsage": "gpuId::0 utilization.gpu [%]::3 % │ utilization.memory [%]::0 % memory.used [MiB]::125 Mi │ B" │ } │ ], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 403µs (cache) (cache) (cache) 24ms 1ms 51µs 28ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 6.2s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 333µs (cache) (cache) (cache) 6.2s 2ms 52µs 6.2s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 11ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 360µs (cache) (cache) (cache) 7ms 3ms 47µs 12ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v2/models/mnist/infer 200 OK ★ 213ms time ★ 7kB↑ 490B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 255B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "input-0", │ "datatype": "INT64", │ "data": [ │ 1 │ ], │ "shape": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 10ms 1ms (cache) (cache) (cache) 209ms 2ms 53µs 223ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v2/models/mnist/explain 200 OK ★ 234ms time ★ 7kB↑ 14.04kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.8kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "input-0", │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004054752788817447, │ -0.00022612877061030135, │ -0.00012734145622798296, │ 0.0056483691320139305, │ 0.008904783962753935, │ 0.0026385364185178045, │ │ (showing 2.05kB/13.8kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 3ms 593µs (cache) (cache) (cache) 230ms 2ms 67µs 236ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 71ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 423µs (cache) (cache) (cache) 68ms 2ms 45µs 72ms ✓ Successful DELETE request ┌─────────────────────────┬────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴────────────────────┴────────────────────┤ │ total run duration: 11.6s │ ├───────────────────────────────────────────────────────────────────┤ │ total data received: 100.97kB (approx) │ ├───────────────────────────────────────────────────────────────────┤ │ average response time: 1126ms [min: 11ms, max: 6.2s, s.d.: 2s] │ ├───────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 73µs [min: 19µs, max: 175µs, s.d.: 71µs] │ ├───────────────────────────────────────────────────────────────────┤ │ average first byte time: 1099ms [min: 6ms, max: 6.2s, s.d.: 2s] │ └───────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman explanation_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 4.2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 49ms 5ms 176µs 554µs 4.2s 8ms 553µs 4.3s ✓ Successful POST request → Model Zoo - Explanations Model POST http://localhost:8080/explanations/mnist 200 OK ★ 255ms time ★ 538B↑ 23.22kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 22.98kB │ [ │ [ │ [ │ -0.00039919451268186876, │ -0.00019002193524758133, │ -0.0008597193324652645, │ -0.0003293672383195613, │ -0.0009114927260373657, │ -0.0001781611341677637, │ -0.0005801030248104138, │ -5.752931319999006e-05, │ -0.00013036399865444518, │ -3.622338996119766e-05, │ 1.2628211584734459e-05, │ -3.151639992838541e-05, │ -7.05836256300227e-05, │ -6.872774034678204e-05, │ -7.877432800814324e-05, │ -0.00013302450116830098, │ -3.925601175888245e-05, │ -0.00022929567937810257, │ 1.7131063481163125e-05, │ -0.0003846355030322186, │ -0.0005095515358523817, │ -0.0003988011607600216, │ -0.0008459620835930776, │ -0.0005034794650467204, │ -0.0006029827263524426, │ -0.00017686124743748917, │ -0.00015337622471496315, │ -0.0 │ ], │ [ │ -0.0004593671225122864, │ -0.0006972021548553824, │ -0.0006564257154960659, │ -0.0006832957249183605, │ -0.0004380072359887562, │ -0.0006188235577606596, │ 8.717785341467914e-06, │ -0.0003930583431214599, │ -0.00010256520931429771, │ -0.00018236534850001436, │ -0.0005491942472500042, │ 4.519847826440966e-05, │ -0.00043977587249738255, │ -0.0003280013039996539, │ -0.0003846253179060722, │ -0.0007280318057663455, │ -0.0003807703086303859, │ -0.0006797485867827073, │ -0.000941059115008411, │ -0.0005321378635100774, │ -0.0010959787694681464, │ -0.000923937345027808, │ -0.0007882728604240228, │ -0.0007635524916401824, │ -0.0003335175603278816, │ 5.101232211791817e-05, │ 7.699819760493559e-05, │ -0.0 │ ], │ [ │ -0.001316505916852844, │ -0.0009519238281164964, │ -0.0021000631325486054, │ -0.0007986435816339172, │ -0.0015434157507984848, │ -0.0008677100671443621, │ -0.0006676297827679016, │ -0.0010158372444342817, │ -0.0006892680738638284, │ -0.0010906414600866602, │ -0.0009184822839527609, │ -0.0010186687562550908, │ │ (showing 2.05kB/22.98kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 42µs 688µs 250ms 2ms 89µs 262ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 22ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 387µs (cache) (cache) 19ms 1ms 80µs 23ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 21ms time ★ 233B↑ 3.99kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.7kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="7045e2f666c3",} 23020 │ 1.689 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="7045e2f666c3", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="7045e2f666c3",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="7045e2f666c3",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="mnist",Level="Model",Hostname=" │ 7045e2f666c3",} 224.47 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="mnist",Level="Model",Hostnam │ e="7045e2f666c3",} 224.78 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="7045e2f666c3",} 201.5 │ 3595352172852 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="7045e2f666c3",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="mnist",model │ _version="default",hostname="7045e2f666c3",} 164.267 │ # HELP WorkerLoadTime Torchserve prometheus gauge metr │ ic with unit: Milliseconds │ # TYPE WorkerLoadTime gauge │ WorkerLoadTime{WorkerName="W-9000-mnist_1.0",Level="Ho │ st",Hostname="7045e2f666c3",} 4055.0 │ # HELP DiskUtilization Torchserve prometheus gauge met │ ric with unit: Percent │ # TYPE DiskUtilization gauge │ DiskUtilization{Level="Host",Hos │ (showing 2.05kB/3.7kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 500µs 20µs 139µs 18ms 1ms 38µs 22ms ✓ Successful GET request ┌─────────────────────────┬──────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴──────────────────────┴─────────────────────┤ │ total run duration: 4.7s │ ├──────────────────────────────────────────────────────────────────────┤ │ total data received: 26.81kB (approx) │ ├──────────────────────────────────────────────────────────────────────┤ │ average response time: 1140ms [min: 21ms, max: 4.2s, s.d.: 1805ms] │ ├──────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 87µs [min: 20µs, max: 176µs, s.d.: 69µs] │ ├──────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1134ms [min: 18ms, max: 4.2s, s.d.: 1800ms] │ └──────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs ## Successfully started TorchServe newman management_api_collection Iteration 1/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 200 OK ★ 8.2s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 47ms 5ms 180µs 742µs 8.2s 8ms 402µs 8.3s ✓ Successful request Iteration 2/11 → workflow management request GET http://localhost:8081/workflows 200 OK ★ 8ms time ★ 235B↑ 423B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 156B │ { │ "workflows": [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws. │ com/war_files/densenet_wf.war" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 537µs (cache) (cache) 4ms 2ms 95µs 9ms ✓ Successful request Iteration 3/11 → workflow management request GET http://localhost:8081/workflows/densenet 200 OK ★ 6ms time ★ 244B↑ 559B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 292B │ [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws.co │ m/war_files/densenet_wf.war", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 50, │ "workflowDag": "{pre_processing=[densenet], densen │ et=[post_processing]}" │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 258µs (cache) (cache) 3ms 1ms 40µs 7ms ✓ Successful request Iteration 4/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 500 Internal Server Error ★ 7ms time ★ 321B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "code": 500, │ "type": "FileAlreadyExistsException", │ "message": "densenet_wf.war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 274µs (cache) (cache) 3ms 1ms 38µs 7ms ✓ Successful request Iteration 5/11 → workflow management request DELETE http://localhost:8081/workflows/densenet 200 OK ★ 111ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 607µs 25µs 200µs 108ms 1ms 58µs 111ms ✓ Successful request Iteration 6/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/does_not_exist.war 400 Bad Request ★ 278ms time ★ 324B↑ 441B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 170B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.s3.amazonaws.com/war_files/does_not_exist. │ war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 260µs (cache) (cache) 276ms 1ms 37µs 279ms ✓ Successful request Iteration 7/11 → workflow management request GET http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 15ms time ★ 250B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 456µs 17µs 139µs 12ms 1ms 34µs 15ms ✓ Successful request Iteration 8/11 → workflow management request DELETE http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 21ms time ★ 253B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 471µs 17µs 174µs 18ms 2ms 36µs 22ms ✓ Successful request Iteration 9/11 → workflow management request POST http://localhost:8081/workflows?url=malformed_url,? 404 Not Found ★ 18ms time ★ 275B↑ 396B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 127B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found in workflow store: ma │ lformed_url,?" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 434µs 16µs 161µs 13ms 2ms 33µs 19ms ✓ Successful request Iteration 10/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/custom_python_dep.war 500 Internal Server Error ★ 1978ms time ★ 327B↑ 531B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 250B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow custom_python_dep has failed to │ register. Failures: [Workflow Node custom_python_dep_ │ _custom_python_dep failed to register. Details: Model │ not found at: custom_python_dep.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 432µs 16µs 174µs 1974ms 1ms 35µs 1978ms ✓ Successful request Iteration 11/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/loading-memory-error.war 500 Internal Server Error ★ 1767ms time ★ 330B↑ 543B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 262B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow loading-memory-error has failed │ to register. Failures: [Workflow Node loading-memory- │ error__loading-memory-error failed to register. Detail │ s: Model not found at: loading-memory-error.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 440µs 18µs 125µs 1764ms 2ms 48µs 1768ms ✓ Successful request ┌─────────────────────────┬────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ iterations │ 11 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ requests │ 11 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ test-scripts │ 11 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ assertions │ 11 │ 0 │ ├─────────────────────────┴────────────────────┴────────────────────┤ │ total run duration: 12.8s │ ├───────────────────────────────────────────────────────────────────┤ │ total data received: 1.7kB (approx) │ ├───────────────────────────────────────────────────────────────────┤ │ average response time: 1135ms [min: 6ms, max: 8.2s, s.d.: 2.3s] │ ├───────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 89µs [min: 16µs, max: 180µs, s.d.: 80µs] │ ├───────────────────────────────────────────────────────────────────┤ │ average first byte time: 1130ms [min: 3ms, max: 8.2s, s.d.: 2.3s] │ └───────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Started regression tests ## Started densenet mar creation ## In directory: /tmp/workspace/model_store | Executing command: torch-model-archiver --model-name densenet161_v1 --version 1.1 --model-file /home/serve/ts_scripts/../examples/image_classifier/densenet_161/model.py --serialized-file /tmp/workspace/model_store/densenet161-8d451a50.pth --extra-files /home/serve/ts_scripts/../examples/image_classifier/index_to_name.json --handler image_classifier --force ## Started regression pytests /home/venv/lib/python3.9/site-packages/grpc_tools/protoc.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources ## In directory: /home/serve/test/pytest | Executing command: python -m pytest -v ./ ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-7.3.1, pluggy-1.3.0 -- /home/venv/bin/python cachedir: .pytest_cache rootdir: /home/serve plugins: mock-3.12.0, cov-4.1.0 collecting ... collected 117 items test_auto_recover.py::test_tp_inference 2023-11-15T20:26:05,490 [INFO ] W-9000-tp_model_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery succeeded, reset recoveryStartTS 2023-11-15T20:26:05,490 [INFO ] W-9000-tp_model_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:2.0|#Level:Host|#hostname:7045e2f666c3,timestamp:1700079965 PASSED [ 0%] test_continuous_batching.py::test_echo_stream_inference PASSED [ 1%] test_continuous_batching.py::test_decoding_stage PASSED [ 2%] test_continuous_batching.py::test_closed_connection PASSED [ 3%] test_distributed_inference_handler.py::test_large_model_inference SKIPPED [ 4%] test_example_dcgan.py::test_model_archive_creation PASSED [ 5%] test_example_dcgan.py::test_model_register_unregister PASSED [ 5%] test_example_dcgan.py::test_image_generation_without_any_input_constraints PASSED [ 6%] test_example_dcgan.py::test_image_generation_with_input_constraints PASSED [ 7%] test_example_intel_extension_for_pytorch.py::test_single_worker_affinity SKIPPED [ 8%] test_example_intel_extension_for_pytorch.py::test_multi_worker_affinity SKIPPED [ 9%] test_example_intel_extension_for_pytorch.py::test_worker_scale_up_affinity SKIPPED [ 10%] test_example_intel_extension_for_pytorch.py::test_worker_scale_down_affinity SKIPPED [ 11%] test_example_micro_batching.py::test_single_example_inference[yaml_config] PASSED [ 11%] test_example_micro_batching.py::test_multi_example_inference[4-yaml_config] PASSED [ 12%] test_example_micro_batching.py::test_multi_example_inference[4-no_config] 2023-11-15T20:30:41,205 [INFO ] W-9001-image_classifier_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:2.0|#Level:Host|#hostname:7045e2f666c3,timestamp:1700080241 PASSED [ 13%] test_example_micro_batching.py::test_single_example_inference[no_config] PASSED [ 14%] test_example_micro_batching.py::test_multi_example_inference[16-no_config] PASSED [ 15%] test_example_micro_batching.py::test_multi_example_inference[16-yaml_config] PASSED [ 16%] test_example_scriptable_tokenzier.py::test_handler PASSED [ 17%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_sample_text PASSED [ 17%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_empty_string 2023-11-15T20:32:27,519 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 ACCESS_LOG - /127.0.0.1:49778 "POST /predictions/scriptable_tokenizer_untrained HTTP/1.1" 200 497 2023-11-15T20:32:27,519 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - result=[METRICS]PredictionTime.Milliseconds:493.73|#ModelName:scriptable_tokenizer_untrained,Level:Model|#hostname:7045e2f666c3,1700080347,0180765d-1a70-4526-91ef-7c97f52f5eff, pattern=[METRICS] 2023-11-15T20:32:27,519 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 TS_METRICS - Requests2XX.Count:1.0|#Level:Host|#hostname:7045e2f666c3,timestamp:1700080347 2023-11-15T20:32:27,519 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0-stdout MODEL_METRICS - PredictionTime.ms:493.73|#ModelName:scriptable_tokenizer_untrained,Level:Model|#hostname:7045e2f666c3,requestID:0180765d-1a70-4526-91ef-7c97f52f5eff,timestamp:1700080347 2023-11-15T20:32:27,519 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 TS_METRICS - ts_inference_latency_microseconds.Microseconds:495644.65|#model_name:scriptable_tokenizer_untrained,model_version:default|#hostname:7045e2f666c3,timestamp:1700080347 2023-11-15T20:32:27,520 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 TS_METRICS - ts_queue_latency_microseconds.Microseconds:92.384|#model_name:scriptable_tokenizer_untrained,model_version:default|#hostname:7045e2f666c3,timestamp:1700080347 2023-11-15T20:32:27,520 [DEBUG] W-9001-scriptable_tokenizer_untrained_1.0 org.pytorch.serve.job.RestJob - Waiting time ns: 92384, Backend time ns: 496560363PASSED 2023-11-15T20:32:27,520 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 TS_METRICS - QueueTime.Milliseconds:0.0|#Level:Host|#hostname:7045e2f666c3,timestamp:1700080347 2023-11-15T20:32:27,520 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 495 2023-11-15T20:32:27,520 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:2.0|#Level:Host|#hostname:7045e2f666c3,timestamp:1700080347 [ 18%] test_example_scriptable_tokenzier.py::test_inference_with_pretrained_model PASSED [ 19%] test_example_torch_tensorrt.py::test_model_archive_creation PASSED [ 20%] test_example_torch_tensorrt.py::test_model_register_unregister PASSED [ 21%] test_example_torch_tensorrt.py::test_run_inference_torch_tensorrt PASSED [ 22%] test_gRPC_inference_api.py::test_inference_apis PASSED [ 23%] test_gRPC_inference_api.py::test_inference_stream_apis PASSED [ 23%] test_gRPC_inference_api.py::test_inference_stream2_apis PASSED [ 24%] test_gRPC_management_apis.py::test_management_apis PASSED [ 25%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model PASSED [ 26%] test_handler.py::test_mnist_model_register_using_non_existent_handler_with_nonzero_workers PASSED [ 27%] test_handler.py::test_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 28%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 29%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model PASSED [ 29%] test_handler.py::test_kserve_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 30%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 31%] test_handler.py::test_huggingface_bert_batch_inference PASSED [ 32%] test_handler.py::test_MMF_activity_recognition_model_register_and_inference_on_valid_model SKIPPED [ 33%] test_handler.py::test_huggingface_bert_model_parallel_inference PASSED [ 34%] test_metrics.py::test_logs_created PASSED [ 35%] test_metrics.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 35%] test_metrics.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 36%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 37%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 38%] test_metrics.py::test_log_location_var_snapshot_disabled PASSED [ 39%] test_metrics.py::test_log_location_var_snapshot_enabled PASSED [ 40%] test_metrics.py::test_async_logging PASSED [ 41%] test_metrics.py::test_async_logging_non_boolean PASSED [ 41%] test_metrics.py::test_metrics_location_var_snapshot_disabled PASSED [ 42%] test_metrics.py::test_metrics_location_var_snapshot_enabled PASSED [ 43%] test_metrics.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 44%] test_metrics.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 45%] test_metrics.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 46%] test_metrics.py::test_metrics_log_mode PASSED [ 47%] test_metrics.py::test_metrics_prometheus_mode PASSED [ 47%] test_metrics.py::test_collect_system_metrics_when_not_disabled PASSED [ 48%] test_metrics.py::test_disable_system_metrics_using_config_properties PASSED [ 49%] test_metrics.py::test_disable_system_metrics_using_environment_variable PASSED [ 50%] test_metrics_kf.py::test_logs_created PASSED [ 51%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 52%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 52%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 53%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 54%] test_metrics_kf.py::test_log_location_var_snapshot_disabled PASSED [ 55%] test_metrics_kf.py::test_log_location_var_snapshot_enabled PASSED [ 56%] test_metrics_kf.py::test_async_logging PASSED [ 57%] test_metrics_kf.py::test_async_logging_non_boolean PASSED [ 58%] test_metrics_kf.py::test_metrics_location_var_snapshot_disabled PASSED [ 58%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled PASSED [ 59%] test_metrics_kf.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 60%] test_metrics_kf.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 61%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 62%] test_model_archiver.py::test_multiple_model_versions_registration PASSED [ 63%] test_model_archiver.py::test_duplicate_model_registration_using_local_url_followed_by_http_url PASSED [ 64%] test_model_archiver.py::test_duplicate_model_registration_using_http_url_followed_by_local_url PASSED [ 64%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_without_force PASSED [ 65%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_with_force PASSED [ 66%] test_model_archiver.py::test_model_archiver_without_handler_flag PASSED [ 67%] test_model_archiver.py::test_model_archiver_without_model_name_flag PASSED [ 68%] test_model_archiver.py::test_model_archiver_without_model_file_flag PASSED [ 69%] test_model_archiver.py::test_model_archiver_without_serialized_flag PASSED [ 70%] test_onnx.py::test_convert_to_onnx PASSED [ 70%] test_onnx.py::test_model_packaging_and_start PASSED [ 71%] test_onnx.py::test_model_start PASSED [ 72%] test_onnx.py::test_inference PASSED [ 73%] test_onnx.py::test_stop PASSED [ 74%] test_parallelism.py::test_tp_inference PASSED [ 75%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[/home/serve/test/pytest/profiler_utils/resnet_custom.py] PASSED [ 76%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[image_classifier] PASSED [ 76%] test_pytorch_profiler.py::test_profiler_arguments_override[/home/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 77%] test_pytorch_profiler.py::test_batch_input[/home/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 78%] test_sm_mme_requirements.py::test_no_model_loaded PASSED [ 79%] test_sm_mme_requirements.py::test_oom_on_model_load SKIPPED (Logic n...) [ 80%] test_sm_mme_requirements.py::test_oom_on_invoke SKIPPED (Logic needs...) [ 81%] test_snapshot.py::test_snapshot_created_on_start_and_stop PASSED [ 82%] test_snapshot.py::test_snapshot_created_on_management_api_invoke PASSED [ 82%] test_snapshot.py::test_start_from_snapshot PASSED [ 83%] test_snapshot.py::test_start_from_latest PASSED [ 84%] test_snapshot.py::test_start_from_read_only_snapshot PASSED [ 85%] test_snapshot.py::test_no_config_snapshots_cli_option PASSED [ 86%] test_snapshot.py::test_start_from_default PASSED [ 87%] test_snapshot.py::test_start_from_non_existing_snapshot PASSED [ 88%] test_snapshot.py::test_torchserve_init_with_non_existent_model_store PASSED [ 88%] test_snapshot.py::test_restart_torchserve_with_last_snapshot_with_model_mar_removed PASSED [ 89%] test_snapshot.py::test_replace_mar_file_with_dummy PASSED [ 90%] test_snapshot.py::test_restart_torchserve_with_one_of_model_mar_removed PASSED [ 91%] test_torch_compile.py::TestTorchCompile::test_archive_model_artifacts PASSED [ 92%] test_torch_compile.py::TestTorchCompile::test_start_torchserve PASSED [ 93%] test_torch_compile.py::TestTorchCompile::test_server_status PASSED [ 94%] test_torch_compile.py::TestTorchCompile::test_registered_model PASSED [ 94%] test_torch_compile.py::TestTorchCompile::test_serve_inference SKIPPED [ 95%] test_torch_xla.py::TestTorchXLA::test_archive_model_artifacts SKIPPED [ 96%] test_torch_xla.py::TestTorchXLA::test_start_torchserve SKIPPED (PyTo...) [ 97%] test_torch_xla.py::TestTorchXLA::test_server_status SKIPPED (PyTorch...) [ 98%] test_torch_xla.py::TestTorchXLA::test_registered_model SKIPPED (PyTo...) [ 99%] test_torch_xla.py::TestTorchXLA::test_serve_inference SKIPPED (PyTor...) [100%] =============================== warnings summary =============================== ../../../venv/lib/python3.9/site-packages/ts/torch_handler/base_handler.py:13 /home/venv/lib/python3.9/site-packages/ts/torch_handler/base_handler.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html from pkg_resources import packaging ../../../venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 /home/venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../../venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 /home/venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../../venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2350 ../../../venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2350 /home/venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2350: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(parent) ../../../venv/lib/python3.9/site-packages/google/rpc/__init__.py:20 /home/venv/lib/python3.9/site-packages/google/rpc/__init__.py:20: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.rpc')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages pkg_resources.declare_namespace(__name__) test/pytest/test_example_scriptable_tokenzier.py::test_handler /home/venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1527: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at ../aten/src/ATen/NestedTensorImpl.cpp:178.) return forward_call(*args, **kwargs) test/pytest/test_example_scriptable_tokenzier.py::test_handler /home/serve/test/pytest/../../examples/text_classification_with_scriptable_tokenizer/handler.py:97: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. data = F.softmax(data) test/pytest/test_gRPC_inference_api.py::test_inference_stream2_apis /home/venv/lib/python3.9/site-packages/_pytest/threadexception.py:73: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-19 Traceback (most recent call last): File "/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner self.run() File "/usr/lib/python3.9/threading.py", line 917, in run self._target(*self._args, **self._kwargs) File "/home/serve/test/pytest/test_gRPC_inference_api.py", line 174, in __infer_stream2 inference_pb2.PredictionsRequest( File "/usr/lib/python3.9/_collections_abc.py", line 941, in update self[key] = other[key] TypeError: expected bytes, int found warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========== 103 passed, 14 skipped, 9 warnings in 5348.92s (1:29:08) =========== Removing file : management_pb2_grpc.py Removing file : inference_pb2_grpc.py Removing file : management_pb2.py Removing file : inference_pb2.py ## Deleting model_store_gen_dir: /home/serve/ts_scripts/../model_store_gen