Processing /home/ubuntu/pytorch/namannan-serve Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting Pillow Using cached Pillow-9.4.0-cp38-cp38-manylinux_2_28_x86_64.whl (3.4 MB) Collecting psutil Using cached psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB) Collecting packaging Using cached packaging-23.0-py3-none-any.whl (42 kB) Collecting wheel Using cached wheel-0.38.4-py3-none-any.whl (36 kB) Building wheels for collected packages: torchserve Building wheel for torchserve (setup.py): started Building wheel for torchserve (setup.py): finished with status 'done' Created wheel for torchserve: filename=torchserve-0.7.1b20230313-py3-none-any.whl size=22719819 sha256=9b6689ddd4675195c0152da455ddc225fcc17e3cacc34b2f161d908f18a4395f Stored in directory: /home/ubuntu/.cache/pip/wheels/cf/00/23/0d309e496cae557c52ac7b04deca6656120111799ceb91076e Successfully built torchserve Installing collected packages: wheel, psutil, Pillow, packaging, torchserve Attempting uninstall: wheel Found existing installation: wheel 0.38.4 Uninstalling wheel-0.38.4: Successfully uninstalled wheel-0.38.4 Attempting uninstall: psutil Found existing installation: psutil 5.9.4 Uninstalling psutil-5.9.4: Successfully uninstalled psutil-5.9.4 Attempting uninstall: Pillow Found existing installation: Pillow 9.3.0 Uninstalling Pillow-9.3.0: Successfully uninstalled Pillow-9.3.0 Attempting uninstall: packaging Found existing installation: packaging 23.0 Uninstalling packaging-23.0: Successfully uninstalled packaging-23.0 Attempting uninstall: torchserve Found existing installation: torchserve 0.7.0b20230203 Uninstalling torchserve-0.7.0b20230203: Successfully uninstalled torchserve-0.7.0b20230203 Successfully installed Pillow-9.4.0 packaging-23.0 psutil-5.9.4 torchserve-0.7.1b20230313 wheel-0.38.4 Processing ./model-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting enum-compat Using cached enum_compat-0.0.3-py3-none-any.whl (1.3 kB) Building wheels for collected packages: torch-model-archiver Building wheel for torch-model-archiver (setup.py): started Building wheel for torch-model-archiver (setup.py): finished with status 'done' Created wheel for torch-model-archiver: filename=torch_model_archiver-0.7.1b20230313-py3-none-any.whl size=14614 sha256=21d2261a15335619a2c8dcb6c568680478022712d092ca98582066a08596b4ef Stored in directory: /home/ubuntu/.cache/pip/wheels/ef/8f/fc/d2cd0fd603478b8451795e030dad6257044d1c83a05f58c32c Successfully built torch-model-archiver Installing collected packages: enum-compat, torch-model-archiver Attempting uninstall: enum-compat Found existing installation: enum-compat 0.0.3 Uninstalling enum-compat-0.0.3: Successfully uninstalled enum-compat-0.0.3 Attempting uninstall: torch-model-archiver Found existing installation: torch-model-archiver 0.7.0b20230203 Uninstalling torch-model-archiver-0.7.0b20230203: Successfully uninstalled torch-model-archiver-0.7.0b20230203 Successfully installed enum-compat-0.0.3 torch-model-archiver-0.7.1b20230313 Processing ./workflow-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Building wheels for collected packages: torch-workflow-archiver Building wheel for torch-workflow-archiver (setup.py): started Building wheel for torch-workflow-archiver (setup.py): finished with status 'done' Created wheel for torch-workflow-archiver: filename=torch_workflow_archiver-0.2.7b20230313-py3-none-any.whl size=12701 sha256=fb2c880e8825f7f7ac7cb10ee199abfde73b77981c05a76e451fceb55126c158 Stored in directory: /home/ubuntu/.cache/pip/wheels/cc/e2/11/58183da397d2d2b105f82c2f527ed7a1766d6f82c35240455e Successfully built torch-workflow-archiver Installing collected packages: torch-workflow-archiver Attempting uninstall: torch-workflow-archiver Found existing installation: torch-workflow-archiver 0.2.6b20230203 Uninstalling torch-workflow-archiver-0.2.6b20230203: Successfully uninstalled torch-workflow-archiver-0.2.6b20230203 Successfully installed torch-workflow-archiver-0.2.7b20230313 Current date and time : 2023-03-13 06:24:25 ------------------------------------------------------------------------------------------ Environment headers ------------------------------------------------------------------------------------------ Torchserve branch: naman-neuron-benchmark torchserve==0.7.1 torch-model-archiver==0.7.1 Python version: 3.8 (64-bit runtime) Python executable: /opt/conda/envs/ts_gpu_benchmark/bin/python Versions of relevant python libraries: captum==0.6.0 intel-extension-for-pytorch==1.13.100 numpy==1.24.2 nvgpu==0.9.0 psutil==5.9.4 pygit2==1.11.1 pylint==2.6.0 pytest==7.2.2 pytest-cov==4.0.0 pytest-mock==3.10.0 requests==2.28.2 requests-toolbelt==0.10.1 torch==1.13.1+cu117 torch-model-archiver==0.7.0b20230203 torch-workflow-archiver==0.2.6b20230203 torchaudio==0.13.1+cu117 torchserve==0.7.0b20230203 torchtext==0.14.1 torchvision==0.14.1+cu117 transformers==4.25.1 wheel==0.38.4 torch==1.13.1+cu117 torchtext==0.14.1 torchvision==0.14.1+cu117 torchaudio==0.13.1+cu117 Java Version: OS: Ubuntu 20.04.5 LTS GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 Clang version: N/A CMake version: version 3.16.3 Is CUDA available: Yes CUDA runtime version: 11.7.99 GPU models and configuration: GPU 0: Tesla T4 GPU 1: Tesla T4 GPU 2: Tesla T4 GPU 3: Tesla T4 Nvidia driver version: 515.65.01 cuDNN version: None ## In directory /home/ubuntu/pytorch/namannan-serve | Executing command pip install --force-reinstall . ## In directory /home/ubuntu/pytorch/namannan-serve | Executing command pip install --force-reinstall model-archiver/. ## In directory /home/ubuntu/pytorch/namannan-serve | Executing command pip install --force-reinstall workflow-archiver/. ## Starting generate_mars, mar_config:/home/ubuntu/pytorch/namannan-serve/ts_scripts/../ts_scripts/mar_config.json, model_store_dir:/home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name fastrcnn --version 1.0 --model-file examples/object_detector/fast-rcnn/model.py --serialized-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## fastrcnn.mar is generated. ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name alexnet --version 1.0 --model-file examples/image_classifier/alexnet/model.py --serialized-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet-owt-7be5be79.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## alexnet.mar is generated. ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name densenet161 --version 1.0 --model-file examples/image_classifier/densenet_161/model.py --serialized-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161-8d451a50.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## densenet161.mar is generated. ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name mnist --version 1.0 --model-file examples/image_classifier/mnist/mnist.py --serialized-file examples/image_classifier/mnist/mnist_cnn.pt --handler examples/image_classifier/mnist/mnist_handler.py --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## mnist.mar is generated. ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## resnet-152-batch.mar is generated. ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name resnet-18 --version 1.0 --model-file examples/image_classifier/resnet_18/model.py --serialized-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet18-f37072fd.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## resnet-18.mar is generated. ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name squeezenet1_1 --version 1.0 --model-file examples/image_classifier/squeezenet/model.py --serialized-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1-b8a52dc0.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## squeezenet1_1.mar is generated. ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name vgg16 --version 1.0 --model-file examples/image_classifier/vgg_16/model.py --serialized-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16-397923af.pth --handler examples/image_classifier/vgg_16/vgg_handler.py --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## vgg16.mar is generated. ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name deeplabv3_resnet_101_eager --version 1.0 --model-file examples/image_segmenter/deeplabv3/model.py --serialized-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet101_coco-586e9e4e.pth --handler image_segmenter --extra-files examples/image_segmenter/deeplabv3/deeplabv3.py,examples/image_segmenter/deeplabv3/intermediate_layer_getter.py,examples/image_segmenter/deeplabv3/fcn.py --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## deeplabv3_resnet_101_eager.mar is generated. ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name fcn_resnet_101 --version 1.0 --model-file examples/image_segmenter/fcn/model.py --serialized-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet101_coco-7ecb50ca.pth --handler image_segmenter --extra-files examples/image_segmenter/fcn/fcn.py,examples/image_segmenter/fcn/intermediate_layer_getter.py --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## fcn_resnet_101.mar is generated. ## In directory: /home/ubuntu/pytorch/namannan-serve | Executing command: torch-model-archiver --model-name maskrcnn --version 1.0 --model-file examples/object_detector/maskrcnn/model.py --serialized-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --export-path /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen --force ## maskrcnn.mar is generated. ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 197ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 5ms 1ms 245µs 182ms 7ms 374µs 225ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 84ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 641µs (cache) (cache) 80ms 2ms 74µs 84ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 1699ms time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 209µs (cache) (cache) 1696ms 1ms 77µs 1700ms ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 4ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 220µs (cache) (cache) 2ms 1ms 53µs 5ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 30ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 490µs 136µs 90µs 27ms 1ms 58µs 30ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 3ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 178µs (cache) (cache) 1ms 1ms 41µs 3ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 15ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 997µs 455µs 135µs 91µs 12ms 1ms 36µs 15ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 76ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 73ms 1ms 37µs 76ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 4ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 892µs 223µs (cache) (cache) 2ms 1ms 29µs 4ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 75ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 161µs (cache) (cache) 72ms 1ms 30µs 75ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 1001ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 998ms 1ms 75µs 1001ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 15ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 419µs 145µs 111µs 12ms 1ms 34µs 16ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 5ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 3ms 1ms 33µs 5ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 3ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 167µs (cache) (cache) 966µs 994µs 29µs 3ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 3ms time ★ 246B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 923µs 168µs (cache) (cache) 880µs 1ms 29µs 3ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 900µs 164µs (cache) (cache) 2ms 957µs 27µs 4ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 3ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 151µs (cache) (cache) 1ms 995µs 28µs 3ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 14ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 950µs 385µs 133µs 103µs 11ms 1ms 28µs 14ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 78ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 891µs 195µs (cache) (cache) 75ms 1ms 34µs 77ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 3ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 927µs 154µs (cache) (cache) 1ms 916µs 26µs 3ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 3ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 904µs 158µs (cache) (cache) 1ms 932µs 47µs 3ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 3ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 923µs 160µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 3ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 989µs 950µs 26µs 3ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 14ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 896µs 356µs 119µs 107µs 11ms 1ms 29µs 13ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 4ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 927µs 163µs (cache) (cache) 1ms 967µs 29µs 3ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 3ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 873µs 176µs (cache) (cache) 1ms 881µs 30µs 3ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 14ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 842µs 412µs 130µs 93µs 11ms 1ms 31µs 14ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 22ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 880µs 397µs 1ms 100µs 18ms 1ms 30µs 22ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 77ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 163µs (cache) (cache) 75ms 929µs 28µs 77ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 167µs (cache) (cache) 2ms 926µs 24µs 4ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1126ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 966µs 152µs (cache) (cache) 1123ms 1ms 35µs 1126ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 89ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 421µs 140µs 96µs 86ms 1ms 40µs 89ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 944µs 186µs (cache) (cache) 3ms 915µs 29µs 5ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 3.4s time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 157µs (cache) (cache) 3.4s 1ms 33µs 3.4s ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 35ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 987µs 177µs (cache) (cache) 32ms 944µs 29µs 35ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 10.2s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 914µs 185µs (cache) (cache) 10.2s 3ms 60µs 10.2s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 49ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 331µs (cache) (cache) 47ms 1ms 33µs 50ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 7.9s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 7.9s 2ms 37µs 7.9s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 52ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 204µs (cache) (cache) 50ms 1ms 32µs 53ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 80ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 188µs (cache) (cache) 77ms 1ms 32µs 80ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 180µs (cache) (cache) 2ms 1ms 31µs 5ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 674ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 183µs (cache) (cache) 671ms 1ms 34µs 674ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 9ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 185µs (cache) (cache) 7ms 899µs 58µs 10ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 677ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 143µs (cache) (cache) 675ms 1ms 39µs 677ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 10ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 159µs (cache) (cache) 7ms 905µs 27µs 10ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 676ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 182µs (cache) (cache) 673ms 1ms 33µs 676ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 2ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 169µs (cache) (cache) 930µs 864µs 26µs 2ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 2ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 941µs 157µs (cache) (cache) 736µs 844µs 25µs 2ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 2ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 906µs 137µs (cache) (cache) 655µs 830µs 22µs 2ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 3ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 132µs (cache) (cache) 698µs 1ms 26µs 3ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 2ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 150µs (cache) (cache) 915µs 824µs 25µs 3ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 2ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 150µs (cache) (cache) 883µs 831µs 25µs 3ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 2ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 826µs 136µs (cache) (cache) 907µs 867µs 25µs 2ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 2ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 858µs 136µs (cache) (cache) 861µs 839µs 26µs 2ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 2ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 796µs 149µs (cache) (cache) 875µs 839µs 27µs 2ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 2ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 803µs 148µs (cache) (cache) 894µs 854µs 26µs 2ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 2ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 864µs 155µs (cache) (cache) 805µs 846µs 26µs 2ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 602ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 950µs 448µs 165µs 106µs 598ms 2ms 43µs 602ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 14ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 409µs 124µs 94µs 11ms 1ms 30µs 14ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 14ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 366µs 124µs 87µs 11ms 1ms 28µs 14ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 13ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 348µs 145µs 99µs 10ms 1ms 28µs 13ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 3ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 886µs 171µs (cache) (cache) 1ms 869µs 29µs 3ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 2ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 175µs (cache) (cache) 783µs 1ms 28µs 3ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 3ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 806µs 157µs (cache) (cache) 1ms 957µs 28µs 3ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 2ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 782µs 174µs (cache) (cache) 799µs 825µs 25µs 2ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 13ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 343µs 147µs 88µs 11ms 990µs 32µs 14ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 14ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 897µs 343µs 821µs 104µs 11ms 1ms 29µs 14ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 8ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 164µs (cache) (cache) 6ms 895µs 28µs 8ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 3ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 152µs (cache) (cache) 1ms 874µs 28µs 4ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 13ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 354µs 124µs 87µs 10ms 909µs 28µs 13ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 14ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 348µs 115µs 111µs 10ms 2ms 27µs 15ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 20ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 811µs 361µs 106µs 78µs 18ms 854µs 27µs 20ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 2ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 802µs 160µs (cache) (cache) 797µs 810µs 27µs 2ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 94ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 883µs 343µs 115µs 92µs 90ms 1ms 31µs 93ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 4ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 191µs (cache) (cache) 2ms 1ms 31µs 4ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 81ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 989µs 174µs (cache) (cache) 78ms 1ms 36µs 81ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 4ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 171µs (cache) (cache) 2ms 980µs 30µs 4ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 75ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 883µs 164µs (cache) (cache) 73ms 892µs 34µs 75ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 3ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 147µs (cache) (cache) 1ms 838µs 30µs 3ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 75ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 998µs 146µs (cache) (cache) 73ms 835µs 30µs 75ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 3ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 836µs 156µs (cache) (cache) 1ms 781µs 27µs 3ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 2ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 963µs 137µs (cache) (cache) 755µs 818µs 26µs 2ms ✓ Successful request ┌─────────────────────────┬──────────────────────┬──────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼──────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼──────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼──────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼──────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼──────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴──────────────────────┴──────────────────────┤ │ total run duration: 31.1s │ ├───────────────────────────────────────────────────────────────────────┤ │ total data received: 7.75kB (approx) │ ├───────────────────────────────────────────────────────────────────────┤ │ average response time: 362ms [min: 2ms, max: 10.2s, s.d.: 1470ms] │ ├───────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 752µs [min: 106µs, max: 1ms, s.d.: 644µs] │ ├───────────────────────────────────────────────────────────────────────┤ │ average first byte time: 360ms [min: 655µs, max: 10.2s, s.d.: 1470ms] │ └───────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman inference Iteration 1/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_v4.mar&model_name=my_text_classifier&initial_workers=1&synchronous=true 200 OK ★ 9.7s time ★ 388B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"my_text_classifier\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 5ms 1ms 251µs 9.7s 6ms 379µs 9.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier 200 OK ★ 2.6s time ★ 353B↑ 373B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 138B │ { │ "World": 0.02911965735256672, │ "Sports": 2.9431601433316246e-05, │ "Business": 0.9074352383613586, │ "Sci/Tec": 0.06341567635536194 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 929µs 243µs 336µs 2.6s 1ms 65µs 2.6s ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier 200 OK ★ 69ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"my_text_classifier\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 390µs (cache) (cache) 66ms 1ms 81µs 69ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 19ms time ★ 233B↑ 1.09kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 796B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="my_text_classifier │ ",model_version="default",} 2600656.396 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 474µs 171µs 99µs 15ms 1ms 73µs 19ms ✓ Successful GET request Iteration 2/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_scripted_v3.mar&model_name=my_text_classifier_scripted&initial_workers=1&synchronous=true 200 OK ★ 7.7s time ★ 406B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 103B │ { │ "status": "Model \"my_text_classifier_scripted\" Ver │ sion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 224µs (cache) (cache) 7.7s 3ms 88µs 7.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier_scripted 200 OK ★ 721ms time ★ 362B↑ 372B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 137B │ { │ "World": 0.04559721797704697, │ "Sports": 0.0003771769697777927, │ "Business": 0.08623101562261581, │ "Sci/Tec": 0.8677946329116821 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 287µs (cache) (cache) 717ms 2ms 59µs 722ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier_scripted 200 OK ★ 74ms time ★ 263B↑ 335B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 69B │ { │ "status": "Model \"my_text_classifier_scripted\" unr │ egistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 214µs (cache) (cache) 72ms 1ms 52µs 75ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 1.53kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1.24kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="my_text_classifier │ ",model_version="default",} 2600656.396 │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="my_text_classifier │ _scripted",model_version="default",} 715958.995 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 224µs (cache) (cache) 1ms 1ms 35µs 4ms ✓ Successful GET request Iteration 3/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 196µs (cache) (cache) 3.3s 1ms 47µs 3.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1 200 OK ★ 3.9s time ★ 111.25kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299155712127686, │ "tiger_cat": 0.24496380984783173, │ "Egyptian_cat": 0.21722552180290222, │ "cougar": 0.0022175442427396774 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 320µs (cache) (cache) 3.9s 1ms 41µs 3.9s ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 11ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 201µs (cache) (cache) 8ms 1ms 36µs 11ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 1.94kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1.65kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="squeezenet1_1",mod │ el_version="default",} 3951365.696 │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="my_text_classifier │ ",model_version="default",} 2600656.396 │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="my_text_classifier │ _scripted",model_version="default",} 715958.995 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 215µs (cache) (cache) 1ms 1ms 31µs 4ms ✓ Successful GET request Iteration 4/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/squeezenet1_1_scripted.mar&model_name=squeezenet1_1_scripted&initial_workers=1&synchronous=true 200 OK ★ 3.6s time ★ 393B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 98B │ { │ "status": "Model \"squeezenet1_1_scripted\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 190µs (cache) (cache) 3.5s 1ms 39µs 3.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1_scripted 200 OK ★ 1602ms time ★ 111.25kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299155712127686, │ "tiger_cat": 0.24496380984783173, │ "Egyptian_cat": 0.21722552180290222, │ "cougar": 0.0022175442427396774 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 202µs (cache) (cache) 1600ms 1ms 39µs 1604ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1_scripted 200 OK ★ 11ms time ★ 258B↑ 330B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 64B │ { │ "status": "Model \"squeezenet1_1_scripted\" unregist │ ered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 196µs (cache) (cache) 9ms 1ms 35µs 12ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 2.37kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.08kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="squeezenet1_1",mod │ el_version="default",} 3951365.696 │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="my_text_classifier │ ",model_version="default",} 2600656.396 │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="my_text_classifier │ _scripted",model_version="default",} 715958.995 │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scri │ pted",mode │ (showing 2.05kB/2.08kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 191µs (cache) (cache) 1ms 1ms 39µs 4ms ✓ Successful GET request Iteration 5/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161&initial_workers=1&synchronous=true 200 OK ★ 5.6s time ★ 330B↑ 353B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 87B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 204µs (cache) (cache) 5.6s 1ms 41µs 5.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161 200 OK ★ 1514ms time ★ 111.24kB↑ 416B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 181B │ { │ "tabby": 0.4666188061237335, │ "tiger_cat": 0.46449077129364014, │ "Egyptian_cat": 0.06614017486572266, │ "lynx": 0.001292433706112206, │ "plastic_bag": 0.00022909622930455953 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 209µs (cache) (cache) 1512ms 1ms 42µs 1516ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161 200 OK ★ 31ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 208µs (cache) (cache) 28ms 1ms 36µs 32ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 2.78kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.48kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="densenet161",model_versi │ on="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="squeezenet1_1",mod │ el_version="default",} 3951365.696 │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ 4b1c-bdd7-85a3d1c7eadc",model_name="my_text_classifier │ ",model_version="default",} 2600656.396 │ ts_inference_l │ (showing 2.05kB/2.48kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 206µs (cache) (cache) 1ms 1ms 36µs 4ms ✓ Successful GET request Iteration 6/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=alexnet.mar&model_name=alexnet&initial_workers=1&synchronous=true 200 OK ★ 7.5s time ★ 322B↑ 349B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 83B │ { │ "status": "Model \"alexnet\" Version: 1.0 registered │ with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 181µs (cache) (cache) 7.5s 1ms 39µs 7.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet 200 OK ★ 1509ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.31847354769706726, │ "tiger_cat": 0.25793972611427307, │ "Egyptian_cat": 0.24254822731018066, │ "lynx": 0.16879358887672424, │ "tiger": 0.006487949751317501 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 209µs (cache) (cache) 1506ms 1ms 35µs 1511ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet 200 OK ★ 74ms time ★ 243B↑ 315B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 49B │ { │ "status": "Model \"alexnet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 219µs (cache) (cache) 69ms 3ms 63µs 75ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 8ms time ★ 233B↑ 3.16kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.87kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="densenet161",model_versi │ on="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="121f83e3-3582- │ (showing 2.05kB/2.87kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 2ms (cache) (cache) 2ms 2ms 66µs 11ms ✓ Successful GET request Iteration 7/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/alexnet_scripted.mar&model_name=alexnet_scripted&initial_workers=1&synchronous=true 200 OK ★ 9.2s time ★ 381B↑ 358B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 92B │ { │ "status": "Model \"alexnet_scripted\" Version: 1.0 r │ egistered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 202µs (cache) (cache) 9.2s 1ms 41µs 9.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet_scripted 200 OK ★ 1585ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.31847354769706726, │ "tiger_cat": 0.25793972611427307, │ "Egyptian_cat": 0.24254822731018066, │ "lynx": 0.16879358887672424, │ "tiger": 0.006487949751317501 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 198µs (cache) (cache) 1582ms 1ms 37µs 1586ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet_scripted 200 OK ★ 91ms time ★ 252B↑ 324B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58B │ { │ "status": "Model \"alexnet_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 204µs (cache) (cache) 88ms 1ms 37µs 91ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 3.58kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.29kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet_scripted",mode │ l_version="default",} 67.98 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4 │ (showing 2.05kB/3.29kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 1ms 1ms 46µs 4ms ✓ Successful GET request Iteration 8/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&initial_workers=1&synchronous=true 200 OK ★ 4.2s time ★ 326B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 207µs (cache) (cache) 4.2s 1ms 40µs 4.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18 200 OK ★ 1509ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.4096633195877075, │ "tiger_cat": 0.3467046320438385, │ "Egyptian_cat": 0.13002899289131165, │ "lynx": 0.023919468745589256, │ "bucket": 0.011532159522175789 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 205µs (cache) (cache) 1506ms 1ms 37µs 1510ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18 200 OK ★ 18ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 213µs (cache) (cache) 15ms 1ms 34µs 19ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 3.97kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.68kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet_scripted",mode │ l_version="default",} 67.98 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18",model_versi │ on="default",} 123.187 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b │ (showing 2.05kB/3.68kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 916µs 176µs (cache) (cache) 1ms 941µs 35µs 3ms ✓ Successful GET request Iteration 9/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-18_scripted.mar&model_name=resnet-18_scripted&initial_workers=1&synchronous=true 200 OK ★ 4.4s time ★ 385B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"resnet-18_scripted\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 189µs (cache) (cache) 4.4s 1ms 51µs 4.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18_scripted 200 OK ★ 1622ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.4096633195877075, │ "tiger_cat": 0.3467046320438385, │ "Egyptian_cat": 0.13002899289131165, │ "lynx": 0.023919468745589256, │ "bucket": 0.011532159522175789 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 189µs (cache) (cache) 1619ms 1ms 39µs 1622ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18_scripted 200 OK ★ 27ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"resnet-18_scripted\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 217µs (cache) (cache) 24ms 1ms 33µs 28ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 4.39kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.1kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet_scripted",mode │ l_version="default",} 67.98 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18_scripted",mo │ del_version="default",} 62.769 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18",model_versi │ on="default",} 123.187 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4 │ (showing 2.05kB/4.1kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 204µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful GET request Iteration 10/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=vgg16.mar&model_name=vgg16&initial_workers=1&synchronous=true 200 OK ★ 12.7s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"vgg16\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 180µs (cache) (cache) 12.7s 1ms 37µs 12.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16 200 OK ★ 1533ms time ★ 111.24kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tiger_cat": 0.44697248935699463, │ "tabby": 0.4408800005912781, │ "Egyptian_cat": 0.059045590460300446, │ "tiger": 0.020596392452716827, │ "lynx": 0.009934596717357635 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 187µs (cache) (cache) 1530ms 1ms 37µs 1534ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16 200 OK ★ 91ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"vgg16\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 201µs (cache) (cache) 88ms 1ms 41µs 91ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 4.77kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.48kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet_scripted",mode │ l_version="default",} 67.98 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18_scripted",mo │ del_version="default",} 62.769 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18",model_versi │ on="default",} 123.187 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7 │ (showing 2.05kB/4.48kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 205µs (cache) (cache) 1ms 3ms 35µs 6ms ✓ Successful GET request Iteration 11/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/vgg16_scripted.mar&model_name=vgg16_scripted&initial_workers=1&synchronous=true 200 OK ★ 13.7s time ★ 377B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"vgg16_scripted\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 206µs (cache) (cache) 13.7s 1ms 39µs 13.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16_scripted 200 OK ★ 1576ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tiger_cat": 0.44697248935699463, │ "tabby": 0.4408800005912781, │ "Egyptian_cat": 0.059045590460300446, │ "tiger": 0.020596392452716827, │ "lynx": 0.009934596717357635 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 207µs (cache) (cache) 1574ms 1ms 39µs 1578ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16_scripted 200 OK ★ 171ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"vgg16_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 210µs (cache) (cache) 168ms 1ms 34µs 171ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 5.18kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.89kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet_scripted",mode │ l_version="default",} 67.98 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18_scripted",mo │ del_version="default",} 62.769 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18",model_versi │ on="default",} 123.187 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d │ (showing 2.05kB/4.89kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 200µs (cache) (cache) 1ms 1ms 41µs 3ms ✓ Successful GET request Iteration 12/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_v2.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.5s time ★ 362B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 2.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 217µs (cache) (cache) 3.5s 1ms 39µs 3.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist 200 OK ★ 1510ms time ★ 537B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 198µs (cache) (cache) 1508ms 1ms 39µs 1511ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 10ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 200µs (cache) (cache) 8ms 1ms 34µs 11ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 5.57kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.27kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet_scripted",mode │ l_version="default",} 67.98 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18_scripted",mo │ del_version="default",} 62.769 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18",model_versi │ on="default",} 123.187 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7eadc",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ dd7-85a3d1c7e │ (showing 2.05kB/5.27kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 881µs 174µs (cache) (cache) 1ms 837µs 42µs 3ms ✓ Successful GET request Iteration 13/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_scripted_v2.mar&model_name=mnist_scripted&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 380B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"mnist_scripted\" Version: 2.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 178µs (cache) (cache) 3.3s 1ms 40µs 3.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist_scripted 200 OK ★ 1577ms time ★ 546B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 172µs (cache) (cache) 1575ms 1ms 39µs 1578ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist_scripted 200 OK ★ 10ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"mnist_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 192µs (cache) (cache) 8ms 967µs 31µs 11ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 5.98kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.68kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist_scripted",model_ │ version="default",} 61.715 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet_scripted",mode │ l_version="default",} 67.98 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18_scripted",mo │ del_version="default",} 62.769 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18",model_versi │ on="default",} 123.187 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="121f83e3-3582-4b1c-b │ d │ (showing 2.05kB/5.68kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 903µs 172µs (cache) (cache) 1ms 844µs 31µs 3ms ✓ Successful GET request Iteration 14/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fastrcnn.mar&model_name=fastrcnn&initial_workers=1&synchronous=true 200 OK ★ 6.9s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"fastrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 881µs 181µs (cache) (cache) 6.9s 1ms 38µs 6.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fastrcnn 200 OK ★ 1640ms time ★ 289.21kB↑ 3.01kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.77kB │ [ │ { │ "person": [ │ 167.39581298828125, │ 57.203041076660156, │ 301.3599853515625, │ 436.7974853515625 │ ], │ "score": 0.999519944190979 │ }, │ { │ "person": [ │ 89.54701232910156, │ 64.83567810058594, │ 191.42428588867188, │ 446.7607727050781 │ ], │ "score": 0.9994966983795166 │ }, │ { │ "person": [ │ 362.37823486328125, │ 161.84133911132812, │ 515.5022583007812, │ 385.28985595703125 │ ], │ "score": 0.997706413269043 │ }, │ { │ "handbag": [ │ 67.37623596191406, │ 277.5755615234375, │ 111.67391204833984, │ 400.206787109375 │ ], │ "score": 0.992499053478241 │ }, │ { │ "handbag": [ │ 228.6824951171875, │ 146.00697326660156, │ 303.55120849609375, │ 231.08848571777344 │ ], │ "score": 0.9922404289245605 │ }, │ { │ "handbag": [ │ 379.411376953125, │ 259.957763671875, │ 419.0797424316406, │ 317.9610290527344 │ ], │ "score": 0.9898613691329956 │ }, │ { │ "person": [ │ 518.4950561523438, │ 149.73019409179688, │ 636.6343994140625, │ 365.4129333496094 │ ], │ "score": 0.9821107983589172 │ }, │ { │ "bench": [ │ 269.08184814453125, │ 217.34202575683594, │ 423.77099609375, │ 390.3786315917969 │ ], │ "score": 0.9573412537574768 │ }, │ { │ "person": [ │ 539.6298217773438, │ 157.75868225097656, │ 616.1533813476562, │ 253.1112823486328 │ ], │ "score": 0.8995409607887268 │ }, │ { │ "person": [ │ 477.06231689453125, │ 147.80885314941406, │ 610.777587890625, │ 296.83734130859375 │ ], │ "score": 0.8751544952392578 │ }, │ { │ "bench": [ │ 286.08746337890625, │ 216.53285217285156, │ 550.7698364257812, │ 383.1822509765625 │ ], │ "score": 0.8436442613601685 │ }, │ { │ "person": [ │ 627.4741821289062, │ 177.05838012695312, │ 640.0, │ 247.72152709960938 │ ], │ "score": 0.8257318139076233 │ }, │ { │ "bench": [ │ 88.76377868652344, │ 226.16490173339844, │ 563.614562988 │ (showing 2.05kB/2.77kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 542µs (cache) (cache) 1636ms 2ms 41µs 1643ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fastrcnn 200 OK ★ 40ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"fastrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 213µs (cache) (cache) 38ms 1ms 32µs 41ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 6.37kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.07kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist_scripted",model_ │ version="default",} 61.715 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet_scripted",mode │ l_version="default",} 67.98 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18_scripted",mo │ del_version="default",} 62.769 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18",model_versi │ on="default",} 123.187 │ # HELP ts_inference_requests_total Total │ (showing 2.05kB/6.07kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 1ms 1ms 34µs 4ms ✓ Successful GET request Iteration 15/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=maskrcnn.mar&model_name=maskrcnn&initial_workers=1&synchronous=true 200 OK ★ 6.5s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"maskrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 899µs 179µs (cache) (cache) 6.5s 1ms 37µs 6.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/maskrcnn 200 OK ★ 1624ms time ★ 289.21kB↑ 2.99kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.75kB │ [ │ { │ "person": [ │ 169.50636291503906, │ 49.98552322387695, │ 300.8945617675781, │ 442.4545593261719 │ ], │ "score": 0.999543309211731 │ }, │ { │ "person": [ │ 90.4118423461914, │ 66.79048919677734, │ 194.19305419921875, │ 437.2776794433594 │ ], │ "score": 0.9993956089019775 │ }, │ { │ "person": [ │ 362.3172912597656, │ 158.17355346679688, │ 521.2589721679688, │ 385.7226867675781 │ ], │ "score": 0.9952280521392822 │ }, │ { │ "handbag": [ │ 68.57540893554688, │ 279.31817626953125, │ 111.15328979492188, │ 400.9165954589844 │ ], │ "score": 0.9938817024230957 │ }, │ { │ "person": [ │ 474.0157165527344, │ 147.3479461669922, │ 638.1209716796875, │ 364.6508483886719 │ ], │ "score": 0.9897466897964478 │ }, │ { │ "handbag": [ │ 225.59584045410156, │ 142.9000244140625, │ 302.48638916015625, │ 230.3284149169922 │ ], │ "score": 0.9891214966773987 │ }, │ { │ "handbag": [ │ 380.2604675292969, │ 259.2012023925781, │ 419.5366516113281, │ 318.27728271484375 │ ], │ "score": 0.9688038229942322 │ }, │ { │ "bench": [ │ 273.48565673828125, │ 217.48834228515625, │ 441.06536865234375, │ 396.24169921875 │ ], │ "score": 0.961754560470581 │ }, │ { │ "person": [ │ 541.2896728515625, │ 156.66119384765625, │ 619.9386596679688, │ 249.45326232910156 │ ], │ "score": 0.8177024126052856 │ }, │ { │ "person": [ │ 362.9620361328125, │ 163.8992462158203, │ 500.7698059082031, │ 293.9122619628906 │ ], │ "score": 0.8016975522041321 │ }, │ { │ "chair": [ │ 455.20849609375, │ 207.54010009765625, │ 491.08526611328125, │ 274.6475830078125 │ ], │ "score": 0.7758335471153259 │ }, │ { │ "person": [ │ 549.1538696289062, │ 177.42056274414062, │ 640.0, │ 364.5394592285156 │ ], │ "score": 0.7176421880722046 │ }, │ { │ "person": [ │ 626.230712890625, │ 178.6534423828125, │ 640.0, │ 246. │ (showing 2.05kB/2.75kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 195µs (cache) (cache) 1621ms 1ms 42µs 1625ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/maskrcnn 200 OK ★ 38ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"maskrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 219µs (cache) (cache) 35ms 1ms 34µs 38ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 6.76kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.46kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="maskrcnn",model_versio │ n="default",} 63.023 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist_scripted",model_ │ version="default",} 61.715 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet_scripted",mode │ l_version="default",} 67.98 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18_scripted",mo │ del_version="default",} 62.769 │ ts_queue_latency_microseconds{uuid="121f83e │ (showing 2.05kB/6.46kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 235µs (cache) (cache) 1ms 1ms 47µs 4ms ✓ Successful GET request Iteration 16/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fcn_resnet_101.mar&model_name=fcn_resnet_101&initial_workers=1&synchronous=true 200 OK ★ 8.2s time ★ 336B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"fcn_resnet_101\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 868µs 185µs (cache) (cache) 8.2s 1ms 37µs 8.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101 200 OK ★ 1693ms time ★ 289.22kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993864297866821 │ ], │ [ │ 0.0, │ 0.9993854761123657 │ ], │ [ │ 0.0, │ 0.9993811845779419 │ ], │ [ │ 0.0, │ 0.9993742108345032 │ ], │ [ │ 0.0, │ 0.9993641972541809 │ ], │ [ │ 0.0, │ 0.9993515610694885 │ ], │ [ │ 0.0, │ 0.9993364214897156 │ ], │ [ │ 0.0, │ 0.9993187189102173 │ ], │ [ │ 0.0, │ 0.9992934465408325 │ ], │ [ │ 0.0, │ 0.9992607235908508 │ ], │ [ │ 0.0, │ 0.9992249011993408 │ ], │ [ │ 0.0, │ 0.9991866946220398 │ ], │ [ │ 0.0, │ 0.9991452693939209 │ ], │ [ │ 0.0, │ 0.9991005659103394 │ ], │ [ │ 0.0, │ 0.9990523457527161 │ ], │ [ │ 0.0, │ 0.9990004897117615 │ ], │ [ │ 0.0, │ 0.998968243598938 │ ], │ [ │ 0.0, │ 0.9989572763442993 │ ], │ [ │ 0.0, │ 0.9989456534385681 │ ], │ [ │ 0.0, │ 0.9989331364631653 │ ], │ [ │ 0.0, │ 0.9989200830459595 │ ], │ [ │ 0.0, │ 0.9989060163497925 │ ], │ [ │ 0.0, │ 0.998891294002533 │ ], │ [ │ 0.0, │ 0.9988754391670227 │ ], │ [ │ 0.0, │ 0.9988815188407898 │ ], │ [ │ 0.0, │ 0.9989089965820312 │ ], │ [ │ 0.0, │ 0.9989352822303772 │ ], │ [ │ 0.0, │ 0.9989607334136963 │ ], │ [ │ 0.0, │ 0.9989848732948303 │ ], │ [ │ 0.0, │ 0.9990076422691345 │ ], │ [ │ 0.0, │ 0.9990293979644775 │ ], │ [ │ 0.0, │ 0.9990498423576355 │ ], │ [ │ 0.0, │ 0.999051034450531 │ ], │ [ │ 0.0, │ 0.9990313053131104 │ ], │ [ │ 0.0, │ 0.9990096092224121 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.9989587068557739 │ ], │ [ │ 0.0, │ 0.99892979860 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 647µs (cache) (cache) 1687ms 3ms 1ms 1697ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101 200 OK ★ 50ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"fcn_resnet_101\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 284µs (cache) (cache) 47ms 1ms 61µs 52ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 7.17kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.87kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101",model_ │ version="default",} 69.005 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="maskrcnn",model_versio │ n="default",} 63.023 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist_scripted",model_ │ version="default",} 61.715 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet_scripted",mode │ l_version="default",} 67.98 │ ts_queue_latency_microseconds{uuid="121f83e3-35 │ (showing 2.05kB/6.87kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 262µs (cache) (cache) 1ms 1ms 36µs 3ms ✓ Successful GET request Iteration 17/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/fcn_resnet_101_scripted.mar&model_name=fcn_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 8.4s time ★ 395B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 99B │ { │ "status": "Model \"fcn_resnet_101_scripted\" Version │ : 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 192µs (cache) (cache) 8.4s 2ms 61µs 8.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101_scripted 200 OK ★ 2s time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993864297866821 │ ], │ [ │ 0.0, │ 0.9993854761123657 │ ], │ [ │ 0.0, │ 0.9993811845779419 │ ], │ [ │ 0.0, │ 0.9993742108345032 │ ], │ [ │ 0.0, │ 0.9993641972541809 │ ], │ [ │ 0.0, │ 0.9993515610694885 │ ], │ [ │ 0.0, │ 0.9993364214897156 │ ], │ [ │ 0.0, │ 0.9993187189102173 │ ], │ [ │ 0.0, │ 0.9992934465408325 │ ], │ [ │ 0.0, │ 0.9992607235908508 │ ], │ [ │ 0.0, │ 0.9992249011993408 │ ], │ [ │ 0.0, │ 0.9991866946220398 │ ], │ [ │ 0.0, │ 0.9991452693939209 │ ], │ [ │ 0.0, │ 0.9991005659103394 │ ], │ [ │ 0.0, │ 0.9990523457527161 │ ], │ [ │ 0.0, │ 0.9990004897117615 │ ], │ [ │ 0.0, │ 0.998968243598938 │ ], │ [ │ 0.0, │ 0.9989572763442993 │ ], │ [ │ 0.0, │ 0.9989456534385681 │ ], │ [ │ 0.0, │ 0.9989331364631653 │ ], │ [ │ 0.0, │ 0.9989200830459595 │ ], │ [ │ 0.0, │ 0.9989060163497925 │ ], │ [ │ 0.0, │ 0.998891294002533 │ ], │ [ │ 0.0, │ 0.9988754391670227 │ ], │ [ │ 0.0, │ 0.9988815188407898 │ ], │ [ │ 0.0, │ 0.9989089965820312 │ ], │ [ │ 0.0, │ 0.9989352822303772 │ ], │ [ │ 0.0, │ 0.9989607334136963 │ ], │ [ │ 0.0, │ 0.9989848732948303 │ ], │ [ │ 0.0, │ 0.9990076422691345 │ ], │ [ │ 0.0, │ 0.9990293979644775 │ ], │ [ │ 0.0, │ 0.9990498423576355 │ ], │ [ │ 0.0, │ 0.999051034450531 │ ], │ [ │ 0.0, │ 0.9990313053131104 │ ], │ [ │ 0.0, │ 0.9990096092224121 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.9989587068557739 │ ], │ [ │ 0.0, │ 0.99892979860 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 213µs (cache) (cache) 2s 4ms 1ms 2s ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101_scripted 200 OK ★ 84ms time ★ 259B↑ 331B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 65B │ { │ "status": "Model \"fcn_resnet_101_scripted\" unregis │ tered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 306µs (cache) (cache) 81ms 1ms 61µs 85ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 7.6kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.31kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 69.072 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101",model_ │ version="default",} 69.005 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="maskrcnn",model_versio │ n="default",} 63.023 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist_scripted",model_ │ version="default",} 61.715 │ ts_queue_latency_microseconds{uuid="121 │ (showing 2.05kB/7.31kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 1ms 1ms 34µs 3ms ✓ Successful GET request Iteration 18/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=deeplabv3_resnet_101_eager.mar&model_name=deeplabv3_resnet_101_eager&initial_workers=1&synchronous=true 200 OK ★ 8s time ★ 360B↑ 369B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" Vers │ ion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 251µs (cache) (cache) 7.9s 2ms 58µs 8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_eager 200 OK ★ 1692ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988669157028198 │ ], │ [ │ 0.0, │ 0.998843789100647 │ ], │ [ │ 0.0, │ 0.9988172650337219 │ ], │ [ │ 0.0, │ 0.9987861514091492 │ ], │ [ │ 0.0, │ 0.9987504482269287 │ ], │ [ │ 0.0, │ 0.9987117052078247 │ ], │ [ │ 0.0, │ 0.9986679553985596 │ ], │ [ │ 0.0, │ 0.9986202716827393 │ ], │ [ │ 0.0, │ 0.9985180497169495 │ ], │ [ │ 0.0, │ 0.9983478784561157 │ ], │ [ │ 0.0, │ 0.9981531500816345 │ ], │ [ │ 0.0, │ 0.9979324340820312 │ ], │ [ │ 0.0, │ 0.9976813793182373 │ ], │ [ │ 0.0, │ 0.9973964691162109 │ ], │ [ │ 0.0, │ 0.9970728158950806 │ ], │ [ │ 0.0, │ 0.9967058300971985 │ ], │ [ │ 0.0, │ 0.9964045286178589 │ ], │ [ │ 0.0, │ 0.9961955547332764 │ ], │ [ │ 0.0, │ 0.9959736466407776 │ ], │ [ │ 0.0, │ 0.9957382678985596 │ ], │ [ │ 0.0, │ 0.9954885840415955 │ ], │ [ │ 0.0, │ 0.9952237010002136 │ ], │ [ │ 0.0, │ 0.9949430823326111 │ ], │ [ │ 0.0, │ 0.994645357131958 │ ], │ [ │ 0.0, │ 0.9945149421691895 │ ], │ [ │ 0.0, │ 0.9945647716522217 │ ], │ [ │ 0.0, │ 0.9946145415306091 │ ], │ [ │ 0.0, │ 0.9946632385253906 │ ], │ [ │ 0.0, │ 0.9947116374969482 │ ], │ [ │ 0.0, │ 0.9947587847709656 │ ], │ [ │ 0.0, │ 0.9948055148124695 │ ], │ [ │ 0.0, │ 0.9948518872261047 │ ], │ [ │ 0.0, │ 0.9948965907096863 │ ], │ [ │ 0.0, │ 0.9949400424957275 │ ], │ [ │ 0.0, │ 0.9949839115142822 │ ], │ [ │ 0.0, │ 0.9950267672538757 │ ], │ [ │ 0.0, │ 0.9950693845748901 │ ], │ [ │ 0.0, │ 0.9951115250 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 189µs (cache) (cache) 1686ms 4ms 533µs 1693ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_eager 200 OK ★ 55ms time ★ 262B↑ 334B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 68B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" unre │ gistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 251µs (cache) (cache) 52ms 1ms 59µs 57ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 8.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.75kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 69.072 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101",model_ │ version="default",} 69.005 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_e │ ager",model_version="default",} 71.262 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="maskrcnn",model_versio │ n="default",} 63.023 │ ts_queue_latency_microsecon │ (showing 2.05kB/7.75kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 210µs (cache) (cache) 1ms 1ms 36µs 4ms ✓ Successful GET request Iteration 19/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.s3.amazonaws.com/mar_files/deeplabv3_resnet_101_scripted.mar&model_name=deeplabv3_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 14.3s time ★ 412B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 183µs (cache) (cache) 14.3s 2ms 67µs 14.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_scripted 200 OK ★ 1919ms time ★ 289.24kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988669157028198 │ ], │ [ │ 0.0, │ 0.998843789100647 │ ], │ [ │ 0.0, │ 0.9988172650337219 │ ], │ [ │ 0.0, │ 0.9987861514091492 │ ], │ [ │ 0.0, │ 0.9987504482269287 │ ], │ [ │ 0.0, │ 0.9987117052078247 │ ], │ [ │ 0.0, │ 0.9986679553985596 │ ], │ [ │ 0.0, │ 0.9986202716827393 │ ], │ [ │ 0.0, │ 0.9985180497169495 │ ], │ [ │ 0.0, │ 0.9983478784561157 │ ], │ [ │ 0.0, │ 0.9981531500816345 │ ], │ [ │ 0.0, │ 0.9979324340820312 │ ], │ [ │ 0.0, │ 0.9976813793182373 │ ], │ [ │ 0.0, │ 0.9973964691162109 │ ], │ [ │ 0.0, │ 0.9970728158950806 │ ], │ [ │ 0.0, │ 0.9967058300971985 │ ], │ [ │ 0.0, │ 0.9964045286178589 │ ], │ [ │ 0.0, │ 0.9961955547332764 │ ], │ [ │ 0.0, │ 0.9959736466407776 │ ], │ [ │ 0.0, │ 0.9957382678985596 │ ], │ [ │ 0.0, │ 0.9954885840415955 │ ], │ [ │ 0.0, │ 0.9952237010002136 │ ], │ [ │ 0.0, │ 0.9949430823326111 │ ], │ [ │ 0.0, │ 0.994645357131958 │ ], │ [ │ 0.0, │ 0.9945149421691895 │ ], │ [ │ 0.0, │ 0.9945647716522217 │ ], │ [ │ 0.0, │ 0.9946145415306091 │ ], │ [ │ 0.0, │ 0.9946632385253906 │ ], │ [ │ 0.0, │ 0.9947116374969482 │ ], │ [ │ 0.0, │ 0.9947587847709656 │ ], │ [ │ 0.0, │ 0.9948055148124695 │ ], │ [ │ 0.0, │ 0.9948518872261047 │ ], │ [ │ 0.0, │ 0.9948965907096863 │ ], │ [ │ 0.0, │ 0.9949400424957275 │ ], │ [ │ 0.0, │ 0.9949839115142822 │ ], │ [ │ 0.0, │ 0.9950267672538757 │ ], │ [ │ 0.0, │ 0.9950693845748901 │ ], │ [ │ 0.0, │ 0.9951115250 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 6ms 442µs (cache) (cache) 1915ms 3ms 1ms 1926ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_scripted 200 OK ★ 82ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 316µs (cache) (cache) 80ms 1ms 59µs 84ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 8.5kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.21kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 69.072 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_s │ cripted",model_version="default",} 67.131 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101",model_ │ version="default",} 69.005 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_e │ ager",model_version="default",} 71.262 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_que │ (showing 2.05kB/8.21kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 211µs (cache) (cache) 1ms 1ms 42µs 4ms ✓ Successful GET request Iteration 20/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&initial_workers=1&synchronous=true 200 OK ★ 8s time ★ 333B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 200µs (cache) (cache) 8s 2ms 64µs 8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet152 200 OK ★ 1455ms time ★ 111.24kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tiger_cat": 0.5798612833023071, │ "tabby": 0.38344183564186096, │ "Egyptian_cat": 0.0342114083468914, │ "lynx": 0.0005819810903631151, │ "quilt": 0.00027331968885846436 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 196µs (cache) (cache) 1451ms 2ms 62µs 1456ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet152 200 OK ★ 55ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 182µs (cache) (cache) 52ms 1ms 41µs 55ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 8.89kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.6kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 69.072 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_s │ cripted",model_version="default",} 67.131 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101",model_ │ version="default",} 69.005 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_e │ ager",model_version="default",} 71.262 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_que │ (showing 2.05kB/8.6kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 904µs 189µs (cache) (cache) 2ms 1ms 41µs 4ms ✓ Successful GET request Iteration 21/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-152-batch_scripted.mar&model_name=resnet-152-batch_scripted&initial_workers=1&synchronous=true 200 OK ★ 9.3s time ★ 399B↑ 368B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "status": "Model \"resnet-152-batch_scripted\" Versi │ on: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 186µs (cache) (cache) 9.3s 1ms 38µs 9.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-152-batch_scripted 200 OK ★ 1870ms time ★ 111.26kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tiger_cat": 0.5798612833023071, │ "tabby": 0.38344183564186096, │ "Egyptian_cat": 0.0342114083468914, │ "lynx": 0.0005819810903631151, │ "quilt": 0.00027331968885846436 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 231µs (cache) (cache) 1867ms 1ms 42µs 1871ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-152-batch_scripted 200 OK ★ 93ms time ★ 261B↑ 333B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 67B │ { │ "status": "Model \"resnet-152-batch_scripted\" unreg │ istered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 202µs (cache) (cache) 90ms 1ms 38µs 94ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 9.34kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.04kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 69.072 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_s │ cripted",model_version="default",} 67.131 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101",model_ │ version="default",} 69.005 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_e │ ager",model_version="default",} 71.262 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_que │ (showing 2.05kB/9.04kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 217µs (cache) (cache) 1ms 1ms 44µs 4ms ✓ Successful GET request Iteration 22/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/distill_bert_qa_eager.mar&model_name=distill_bert_qa_eager&initial_workers=1&synchronous=true 200 OK ★ 11.5s time ★ 391B↑ 363B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "status": "Model \"distill_bert_qa_eager\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 201µs (cache) (cache) 11.5s 2ms 37µs 11.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/distill_bert_qa_eager 200 OK ★ 695ms time ★ 359B↑ 247B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 79B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13B │ a nice puppet └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 437µs (cache) (cache) 692ms 1ms 36µs 697ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/distill_bert_qa_eager 200 OK ★ 85ms time ★ 257B↑ 329B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Model \"distill_bert_qa_eager\" unregiste │ red" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 203µs (cache) (cache) 82ms 1ms 37µs 85ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 9.76kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.47kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 69.072 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_s │ cripted",model_version="default",} 67.131 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101",model_ │ version="default",} 69.005 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_e │ ager",model_version="default",} 71.262 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_que │ (showing 2.05kB/9.47kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 1ms 1ms 35µs 3ms ✓ Successful GET request Iteration 23/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_token_classification_no_torchscript.mar&model_name=bert_token_classification_no_torchscript&initial_workers=1&synchronous=true 200 OK ★ 14.6s time ★ 429B↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 116B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" Version: 1.0 registered with 1 initial worke │ rs" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 192µs (cache) (cache) 14.6s 2ms 39µs 14.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_token_classification_no_torchscript 200 OK ★ 706ms time ★ 375B↑ 5.45kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.22kB │ [ │ [ │ "[CLS]", │ "B-LOC" │ ], │ [ │ "bloomberg", │ "O" │ ], │ [ │ "has", │ "I-ORG" │ ], │ [ │ "decided", │ "B-PER" │ ], │ [ │ "to", │ "O" │ ], │ [ │ "publish", │ "I-PER" │ ], │ [ │ "a", │ "I-MISC" │ ], │ [ │ "new", │ "B-PER" │ ], │ [ │ "report", │ "O" │ ], │ [ │ "on", │ "O" │ ], │ [ │ "global", │ "B-PER" │ ], │ [ │ "economic", │ "O" │ ], │ [ │ "situation", │ "B-LOC" │ ], │ [ │ ".", │ "B-PER" │ ], │ [ │ "[SEP]", │ "B-PER" │ ], │ [ │ "[PAD]", │ "I-ORG" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ (showing 2.05kB/5.22kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 186µs (cache) (cache) 703ms 1ms 41µs 707ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_token_classification_no_torchscript 200 OK ★ 139ms time ★ 276B↑ 348B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 82B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 206µs (cache) (cache) 136ms 1ms 41µs 140ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 10.25kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.96kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="alexnet",model_version │ ="default",} 66.985 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 69.072 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier",mo │ del_version="default",} 180.802 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_s │ cripted",model_version="default",} 67.131 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fcn_resnet_101",model_ │ version="default",} 69.005 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="bert_token_classificat │ ion_no_torchscript",model_version="default",} 120.258 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_e │ ager",model_version="default",} 71.262 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version │ (showing 2.05kB/9.96kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 201µs (cache) (cache) 1ms 1ms 37µs 4ms ✓ Successful GET request Iteration 24/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_seqc_without_torchscript.mar&model_name=bert_seqc_without_torchscript&initial_workers=1&synchronous=true 200 OK ★ 16.2s time ★ 407B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"bert_seqc_without_torchscript\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 194µs (cache) (cache) 16.2s 1ms 39µs 16.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_seqc_without_torchscript 200 OK ★ 715ms time ★ 364B↑ 246B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12B │ Not Accepted └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 199µs (cache) (cache) 712ms 1ms 43µs 717ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_seqc_without_torchscript 200 OK ★ 354ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"bert_seqc_without_torchscript\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 214µs (cache) (cache) 351ms 1ms 37µs 355ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 10.7kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 10.41kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1",model_v │ ersion="default",} 177.31 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="my_text_classifier_scr │ ipted",model_version="default",} 59.84 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="fastrcnn",model_versio │ n="default",} 67.72 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="squeezenet1_1_scripted │ ",model_version="default",} 143.567 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_s │ cripted",model_version="default",} 67.131 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16",model_version=" │ default",} 47.366 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="densenet161",model_ver │ sion="default",} 143.295 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="bert_token_classificat │ ion_no_torchscript",model_version="default",} 120.258 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="deeplabv3_resnet_101_e │ ager",model_version="default",} 71.262 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="bert_seqc_without_torc │ hscript",model_version="default",} 58.467 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="mnist",model_version=" │ default",} 70.863 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="vgg16_scripted",model_ │ version="default",} 120.564 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet152",model_versi │ on="default",} 58.36 │ ts_queue_latency_microseconds{uuid="121f83e3-3582-4b1c │ -bdd7-85a3d1c7eadc",model_name="resnet-18_scripted",mo │ del_ │ (showing 2.05kB/10.41kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 192µs (cache) (cache) 1ms 1ms 49µs 3ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 24 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 96 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 96 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 113 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 4m 4.8s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 10.05MB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 2.5s [min: 3ms, max: 16.2s, s.d.: 3.9s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 627µs [min: 171µs, max: 1ms, s.d.: 533µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 2.5s [min: 1ms, max: 16.2s, s.d.: 3.9s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman inference → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161_scripted.mar&model_name=densenet161_scripted&initial_workers=1&synchronous=true 200 OK ★ 6.9s time ★ 389B↑ 362B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 96B │ { │ "status": "Model \"densenet161_scripted\" Version: 1 │ .0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 5ms 1ms 259µs 6.9s 6ms 386µs 6.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161_scripted 200 OK ★ 2.2s time ★ 111.25kB↑ 416B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 181B │ { │ "tabby": 0.4666188061237335, │ "tiger_cat": 0.46449077129364014, │ "Egyptian_cat": 0.06614017486572266, │ "lynx": 0.001292433706112206, │ "plastic_bag": 0.00022909622930455953 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 1ms 417µs 156µs 2.2s 1ms 66µs 2.2s ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161_scripted 200 OK ★ 49ms time ★ 256B↑ 328B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 62B │ { │ "status": "Model \"densenet161_scripted\" unregister │ ed" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 387µs (cache) (cache) 46ms 1ms 80µs 49ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 19ms time ★ 233B↑ 1.09kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 800B │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="78609246-723d-467c-a │ e19-43dae20b2b9e",model_name="densenet161_scripted",mo │ del_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="78609246-723d- │ 467c-ae19-43dae20b2b9e",model_name="densenet161_script │ ed",model_version="default",} 2234006.107 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="78609246-723d-467c │ -ae19-43dae20b2b9e",model_name="densenet161_scripted", │ model_version="default",} 132.5 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 474µs 167µs 98µs 16ms 1ms 72µs 19ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 9.4s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 1.14kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 2.3s [min: 19ms, max: 6.9s, s.d.: 2.8s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 722µs [min: 167µs, max: 1ms, s.d.: 531µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 2.3s [min: 16ms, max: 6.9s, s.d.: 2.8s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 173ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 27ms 6ms 1ms 424µs 123ms 34ms 6ms 388µs 201ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 62ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 947µs 165µs 269µs 31ms 26ms 1ms 71µs 62ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 34ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 863µs 155µs 214µs 28ms 2ms 1ms 69µs 34ms ✓ Status code is 200 → HTTPS Register Model - SqueezeNet POST https://localhost:8444/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 3.5s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 357µs (cache) (cache) (cache) 3.5s 1ms 83µs 3.5s ✓ Successful POST request → HTTPS Get SqueezeNet Model Description GET https://localhost:8444/models/squeezenet1_1 200 OK ★ 27ms time ★ 246B↑ 837B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 570B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-03-13T06:33:57.250Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 13662, │ "gpu": true, │ "gpuUsage": "gpuId::1 utilization.gpu [%]::6 % │ utilization.memory [%]::0 % memory.used [MiB]::589 Mi │ B" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 256µs (cache) (cache) (cache) 24ms 1ms 50µs 27ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 4ms time ★ 308B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 241µs (cache) (cache) (cache) 1ms 1ms 53µs 4ms ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 3ms time ★ 309B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 231µs (cache) (cache) (cache) 1ms 1ms 35µs 4ms ✓ Successful PUT request → HTTPS - Inference - SqueezeNet POST https://localhost:8443/predictions/squeezenet1_1 200 OK ★ 1537ms time ★ 111.25kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299155712127686, │ "tiger_cat": 0.24496380984783173, │ "Egyptian_cat": 0.21722552180290222, │ "cougar": 0.0022175442427396774 │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 4ms 597µs (cache) (cache) (cache) 1534ms 1ms 52µs 1541ms ✓ Status code is 200 → HTTPS UnRegister Model SqueezeNet DELETE https://localhost:8444/models/squeezenet1_1 200 OK ★ 12ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 211µs (cache) (cache) (cache) 9ms 1ms 38µs 12ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 9 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 5.6s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 85.53kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 596ms [min: 3ms, max: 3.5s, s.d.: 1132ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 534µs [min: 155µs, max: 1ms, s.d.: 528µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 571ms [min: 1ms, max: 3.5s, s.d.: 1142ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 212ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 5ms 1ms 262µs 197ms 7ms 368µs 239ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 86ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 659µs (cache) (cache) 82ms 2ms 84µs 87ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 1699ms time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 1696ms 1ms 70µs 1700ms ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 4ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 221µs (cache) (cache) 2ms 1ms 51µs 5ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 31ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 443µs 140µs 100µs 28ms 1ms 56µs 31ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 3ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 176µs (cache) (cache) 1ms 1ms 39µs 3ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 15ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 455µs 149µs 125µs 12ms 1ms 39µs 15ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 75ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 202µs (cache) (cache) 73ms 1ms 35µs 75ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 4ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 200µs (cache) (cache) 2ms 1ms 29µs 4ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 74ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 162µs (cache) (cache) 72ms 1ms 41µs 74ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 975ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 167µs (cache) (cache) 972ms 1ms 38µs 975ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 16ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 470µs 163µs 107µs 13ms 1ms 33µs 16ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 5ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 994µs 184µs (cache) (cache) 3ms 1ms 32µs 5ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 3ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 156µs (cache) (cache) 1ms 954µs 28µs 3ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 246B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 945µs 170µs (cache) (cache) 1ms 1ms 37µs 4ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 869µs 169µs (cache) (cache) 2ms 966µs 27µs 4ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 3ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 997µs 169µs (cache) (cache) 1ms 987µs 27µs 3ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 14ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 945µs 353µs 121µs 94µs 11ms 1ms 29µs 14ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 85ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 897µs 208µs (cache) (cache) 82ms 1ms 38µs 85ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 3ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 184µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 3ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 180µs (cache) (cache) 1ms 1ms 32µs 3ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 4ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 186µs (cache) (cache) 1ms 1ms 29µs 5ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 3ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 186µs (cache) (cache) 1ms 1ms 28µs 3ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 14ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 976µs 415µs 143µs 93µs 11ms 1ms 31µs 14ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 3ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 974µs 176µs (cache) (cache) 1ms 959µs 28µs 3ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 4ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 865µs 172µs (cache) (cache) 2ms 961µs 28µs 4ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 14ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 894µs 386µs 128µs 103µs 11ms 1ms 29µs 14ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 20ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 865µs 381µs 154µs 104µs 17ms 1ms 31µs 20ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 77ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 938µs 179µs (cache) (cache) 75ms 968µs 28µs 77ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 872µs 171µs (cache) (cache) 2ms 862µs 26µs 4ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1136ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 984µs 152µs (cache) (cache) 1133ms 1ms 38µs 1136ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 90ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 427µs 140µs 103µs 87ms 1ms 29µs 90ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 963µs 167µs (cache) (cache) 2ms 924µs 27µs 4ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 3.4s time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 173µs (cache) (cache) 3.4s 1ms 35µs 3.4s ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 35ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 183µs (cache) (cache) 33ms 932µs 28µs 35ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 7.9s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 899µs 173µs (cache) (cache) 7.9s 3ms 58µs 7.9s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 50ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 308µs (cache) (cache) 47ms 1ms 53µs 51ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 7.9s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 220µs (cache) (cache) 7.9s 3ms 61µs 7.9s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 53ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 296µs (cache) (cache) 50ms 1ms 29µs 54ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 80ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 208µs (cache) (cache) 77ms 1ms 30µs 80ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 165µs (cache) (cache) 2ms 1ms 29µs 4ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 675ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 155µs (cache) (cache) 672ms 1ms 41µs 675ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 9ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 171µs (cache) (cache) 7ms 885µs 24µs 9ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 675ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 924µs 187µs (cache) (cache) 672ms 1ms 40µs 675ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 9ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 168µs (cache) (cache) 7ms 876µs 26µs 9ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 676ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 157µs (cache) (cache) 673ms 1ms 40µs 676ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 2ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 170µs (cache) (cache) 861µs 887µs 28µs 3ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 2ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 879µs 164µs (cache) (cache) 760µs 851µs 24µs 2ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 2ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 987µs 150µs (cache) (cache) 568µs 1ms 24µs 2ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 3ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 905µs 148µs (cache) (cache) 620µs 1ms 25µs 2ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 2ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 150µs (cache) (cache) 908µs 855µs 25µs 3ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 2ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 843µs 148µs (cache) (cache) 809µs 875µs 24µs 2ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 2ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 827µs 138µs (cache) (cache) 893µs 870µs 24µs 2ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 2ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 856µs 138µs (cache) (cache) 857µs 844µs 25µs 2ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 2ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 808µs 168µs (cache) (cache) 836µs 857µs 25µs 2ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 2ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 815µs 137µs (cache) (cache) 938µs 854µs 25µs 2ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 2ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 833µs 159µs (cache) (cache) 782µs 834µs 44µs 2ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 604ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 936µs 426µs 195µs 96µs 600ms 1ms 43µs 604ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 14ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 377µs 117µs 126µs 11ms 1ms 29µs 14ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 13ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 888µs 354µs 119µs 86µs 11ms 969µs 28µs 13ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 13ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 352µs 113µs 120µs 10ms 1ms 29µs 13ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 3ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 866µs 157µs (cache) (cache) 1ms 891µs 27µs 3ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 3ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 170µs (cache) (cache) 796µs 1ms 29µs 3ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 2ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 781µs 149µs (cache) (cache) 992µs 830µs 26µs 2ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 2ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 802µs 164µs (cache) (cache) 873µs 860µs 25µs 2ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 14ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 356µs 120µs 83µs 11ms 985µs 30µs 15ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 13ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 876µs 347µs 117µs 91µs 11ms 933µs 26µs 13ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 8ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 158µs (cache) (cache) 6ms 884µs 28µs 8ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 3ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 913µs 165µs (cache) (cache) 1ms 878µs 27µs 3ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 13ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 828µs 359µs 120µs 86µs 10ms 890µs 26µs 13ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 13ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 786µs 330µs 106µs 78µs 10ms 868µs 25µs 12ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 20ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 354µs 110µs 81µs 17ms 861µs 26µs 20ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 2ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 830µs 159µs (cache) (cache) 705µs 843µs 25µs 2ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 87ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 966µs 330µs 128µs 76µs 85ms 940µs 28µs 87ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 3ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 963µs 165µs (cache) (cache) 1ms 808µs 35µs 3ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 75ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 807µs 143µs (cache) (cache) 73ms 887µs 27µs 75ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 3ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 168µs (cache) (cache) 1ms 836µs 25µs 4ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 76ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 825µs 156µs (cache) (cache) 74ms 880µs 25µs 76ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 3ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 155µs (cache) (cache) 1ms 843µs 25µs 3ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 75ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 942µs 164µs (cache) (cache) 73ms 895µs 29µs 75ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 5ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 977µs 160µs (cache) (cache) 1ms 2ms 25µs 5ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 2ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 148µs (cache) (cache) 749µs 825µs 25µs 2ms ✓ Successful request ┌─────────────────────────┬──────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴──────────────────────┴─────────────────────┤ │ total run duration: 28.9s │ ├──────────────────────────────────────────────────────────────────────┤ │ total data received: 7.75kB (approx) │ ├──────────────────────────────────────────────────────────────────────┤ │ average response time: 335ms [min: 2ms, max: 7.9s, s.d.: 1292ms] │ ├──────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 712µs [min: 106µs, max: 1ms, s.d.: 648µs] │ ├──────────────────────────────────────────────────────────────────────┤ │ average first byte time: 332ms [min: 568µs, max: 7.9s, s.d.: 1292ms] │ └──────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman kf_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.5s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 5ms 1ms 267µs 3.4s 6ms 383µs 3.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v1/models/mnist:predict 200 OK ★ 1534ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 927µs 461µs 171µs 1530ms 1ms 63µs 1538ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v1/models/mnist:explain 200 OK ★ 27ms time ★ 680B↑ 25.4kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.16kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.0045709484202342545, │ 0.006216969527188252, │ 0.008197564504355558, │ 0.009563574636103758, │ 0.008999273563915732, │ 0.009673474031078457, │ 0.007599905521342725, │ 0.0063613809512706515, │ 0.0057688292873839305, │ 0.00439446596454604, │ 0.004948218040748549, │ 0.00527346076478521, │ 0.005523799543449043, │ 0.007789356530578066, │ 0.008759362944991762, │ 0.004304804422137636, │ 0.010970579496288352, │ 0.003248439108184719, │ 0.005998033215573371, │ 0.0037543660003001404, │ 0.002765290887789118, │ 0.004314086007904382, │ 0.0014008569476638513, │ 0.004841846312960897, │ 0.0006374844970870742, │ 0.0018558538624387638, │ -0.0008280457210026403, │ -0.0 │ ], │ [ │ 0.0016625160972090316, │ 0.004443792128154977, │ 0.012387838867815328, │ 0.009450843180610638, │ 0.016143821586504077, │ 0.007797501928939272, │ 0.013942238477764502, │ 0.007557429185743731, │ 0.005479089675943439, │ 0.009751321226117562, │ 0.004764408511815325, │ 0.007292148062964123, │ 0.011797998495449305, │ 0.006462684375807869, │ 0.003383213379053915, │ 0.009225058353702676, │ 0.001675069212766654, │ 0.007362304512999291, │ 0.0052700578015770095, │ 0.005453597816894837, │ 0.004342725744464275, │ 0.005856132994265271, │ 0.012168384298940963, │ 0.009902719753086821, │ 0.00981311216079365, │ 0.003442758076136457, │ 0.0022924286211484157, │ -0.0 │ ], │ [ │ 0.011528253688386956, │ 0.009146988140040424, │ 0.02226542212072167, │ 0.017558218362664812, │ 0.024770555247723827, │ 0.021412695215025923, │ │ (showing 2.05kB/25.16kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 361µs (cache) (cache) 24ms 1ms 78µs 28ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 11ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 277µs (cache) (cache) 9ms 1ms 50µs 11ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 19ms time ★ 233B↑ 1.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 757B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="9fd84a03-a417-4ea3 │ -b89f-77ea40e60b3c",model_name="mnist",model_version=" │ default",} 323.597 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="9fd84a03-a417-4ea3-b │ 89f-77ea40e60b3c",model_name="mnist",model_version="de │ fault",} 2.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="9fd84a03-a417- │ 4ea3-b89f-77ea40e60b3c",model_name="mnist",model_versi │ on="default",} 1532319.693 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 517µs 154µs 113µs 16ms 1ms 59µs 19ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 5.3s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 26.08kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 1019ms [min: 11ms, max: 3.5s, s.d.: 1374ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 772µs [min: 154µs, max: 1ms, s.d.: 518µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1014ms [min: 9ms, max: 3.4s, s.d.: 1370ms] │ └─────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman kf_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 174ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 28ms 6ms 1ms 464µs 123ms 34ms 6ms 383µs 202ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 51ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 1ms 182µs 246µs 29ms 17ms 1ms 85µs 51ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 37ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 821µs 2ms 266µs 27ms 3ms 1ms 68µs 37ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.4s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 327µs (cache) (cache) (cache) 3.4s 1ms 92µs 3.4s ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 27ms time ★ 238B↑ 821B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 554B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-03-13T06:35:40.212Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 15225, │ "gpu": true, │ "gpuUsage": "gpuId::1 utilization.gpu [%]::9 % │ utilization.memory [%]::0 % memory.used [MiB]::587 Mi │ B" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 258µs (cache) (cache) (cache) 25ms 1ms 57µs 27ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 4.1s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 239µs (cache) (cache) (cache) 4.1s 3ms 78µs 4.1s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 5ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 280µs (cache) (cache) (cache) 2ms 1ms 42µs 6ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v1/models/mnist:predict 200 OK ★ 1605ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 5ms 558µs (cache) (cache) (cache) 1600ms 3ms 97µs 1610ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v1/models/mnist:explain 200 OK ★ 1584ms time ★ 680B↑ 25.36kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.12kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.0045709484202342545, │ 0.006216969527188252, │ 0.008197564504355558, │ 0.009563574636103758, │ 0.008999273563915732, │ 0.009673474031078457, │ 0.007599905521342725, │ 0.0063613809512706515, │ 0.0057688292873839305, │ 0.00439446596454604, │ 0.004948218040748549, │ 0.00527346076478521, │ 0.005523799543449043, │ 0.007789356530578066, │ 0.008759362944991762, │ 0.004304804422137636, │ 0.010970579496288352, │ 0.003248439108184719, │ 0.005998033215573371, │ 0.0037543660003001404, │ 0.002765290887789118, │ 0.004314086007904382, │ 0.0014008569476638513, │ 0.004841846312960897, │ 0.0006374844970870742, │ 0.0018558538624387638, │ -0.0008280457210026403, │ -0.0 │ ], │ [ │ 0.0016625160972090316, │ 0.004443792128154977, │ 0.012387838887823936, │ 0.009450843180610638, │ 0.016143821586504077, │ 0.007797501797560293, │ 0.013942238477764502, │ 0.007557429135145573, │ 0.005479089815248496, │ 0.009751321195093603, │ 0.004764408456673867, │ 0.007292148383022155, │ 0.011797998520862593, │ 0.00646268421291213, │ 0.0033832133390367, │ 0.00922505853792274, │ 0.00167506898287736, │ 0.007362304867213889, │ 0.005270057781568403, │ 0.005453597816894837, │ 0.004342725724455668, │ 0.005856132994265271, │ 0.012168384435724624, │ 0.009902719733946967, │ 0.00981311212169668, │ 0.003442757869545909, │ 0.0022924286211484157, │ -0.0 │ ], │ [ │ 0.011528253688386956, │ 0.009146988130036119, │ 0.022265421846946565, │ 0.017558218422281058, │ 0.024770556463347786, │ 0.021412694466207218, │ │ (showing 2.05kB/25.12kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 355µs (cache) (cache) (cache) 1581ms 2ms 46µs 1586ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 42ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 233µs (cache) (cache) (cache) 39ms 1ms 39µs 42ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 11.4s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 110.47kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 1113ms [min: 5ms, max: 4.1s, s.d.: 1480ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 1ms [min: 182µs, max: 2ms, s.d.: 956µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1091ms [min: 2ms, max: 4.1s, s.d.: 1494ms] │ └─────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman kfv2_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.5s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 5ms 1ms 273µs 3.4s 6ms 339µs 3.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v2/models/mnist/infer 200 OK ★ 1527ms time ★ 7kB↑ 473B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 238B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "predict", │ "shape": [], │ "datatype": "INT64", │ "data": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 862µs 275µs 338µs 1522ms 2ms 56µs 1531ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v2/models/mnist/explain 200 OK ★ 14ms time ★ 7kB↑ 14.06kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.83kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "explain", │ "shape": [ │ 1, │ 28, │ 28 │ ], │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004054752792208984, │ -0.00022612876545230096, │ -0.00012734148135577474, │ 0.00564836905052733, │ │ (showing 2.05kB/13.83kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 392µs (cache) (cache) 12ms 1ms 80µs 15ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 11ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 283µs (cache) (cache) 8ms 1ms 49µs 11ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 19ms time ★ 233B↑ 1.06kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 764B │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="b4dc971a-7746- │ 40ac-9e9f-861823a9cbc9",model_name="mnist",model_versi │ on="default",} 1512510.6279999998 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="b4dc971a-7746-40ac-9 │ e9f-861823a9cbc9",model_name="mnist",model_version="de │ fault",} 2.0 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="b4dc971a-7746-40ac │ -9e9f-861823a9cbc9",model_name="mnist",model_version=" │ default",} 335.736 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 548µs 166µs 102µs 16ms 1ms 68µs 19ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 5.2s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 14.96kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 1015ms [min: 11ms, max: 3.5s, s.d.: 1376ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 709µs [min: 166µs, max: 1ms, s.d.: 577µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1010ms [min: 8ms, max: 3.4s, s.d.: 1372ms] │ └─────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kf.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. newman kfv2_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 173ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 28ms 6ms 1ms 487µs 123ms 34ms 6ms 405µs 201ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 49ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 1ms 182µs 238µs 28ms 17ms 1ms 71µs 50ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 35ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 825µs 162µs 228µs 29ms 2ms 1ms 70µs 36ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.4s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 311µs (cache) (cache) (cache) 3.4s 1ms 82µs 3.4s ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 27ms time ★ 238B↑ 821B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 554B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-03-13T06:36:39.050Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 16275, │ "gpu": true, │ "gpuUsage": "gpuId::1 utilization.gpu [%]::7 % │ utilization.memory [%]::0 % memory.used [MiB]::587 Mi │ B" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 255µs (cache) (cache) (cache) 24ms 1ms 59µs 27ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 4.1s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 220µs (cache) (cache) (cache) 4.1s 3ms 86µs 4.1s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 4ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 258µs (cache) (cache) (cache) 2ms 1ms 50µs 5ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v2/models/mnist/infer 200 OK ★ 1597ms time ★ 7kB↑ 473B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 238B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "predict", │ "shape": [], │ "datatype": "INT64", │ "data": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 5ms 555µs (cache) (cache) (cache) 1592ms 3ms 75µs 1602ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v2/models/mnist/explain 200 OK ★ 1544ms time ★ 7kB↑ 14.06kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.83kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "explain", │ "shape": [ │ 1, │ 28, │ 28 │ ], │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004054752695155431, │ -0.0002261287695611153, │ -0.00012734148539185055, │ 0.005648369114658842, │ │ (showing 2.05kB/13.83kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 352µs (cache) (cache) (cache) 1540ms 2ms 57µs 1545ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 42ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 241µs (cache) (cache) (cache) 39ms 1ms 38µs 42ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 11.4s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 99.38kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 1106ms [min: 4ms, max: 4.1s, s.d.: 1476ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 503µs [min: 162µs, max: 1ms, s.d.: 468µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1084ms [min: 2ms, max: 4.1s, s.d.: 1489ms] │ └─────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman explanation_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.5s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 5ms 1ms 262µs 3.5s 6ms 386µs 3.5s ✓ Successful POST request → Model Zoo - Explanations Model POST http://localhost:8080/explanations/mnist 200 OK ★ 1539ms time ★ 538B↑ 23.21kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 22.97kB │ [ │ [ │ [ │ -0.00039919451268186876, │ -0.00019002193524758133, │ -0.0008597193324652645, │ -0.0003293672383195613, │ -0.0009114927260373657, │ -0.0001781611341677637, │ -0.0005801030248104138, │ -5.752931319999006e-05, │ -0.00013036399865444518, │ -3.622338996119766e-05, │ 1.2628211584734459e-05, │ -3.151639992838541e-05, │ -7.05836256300227e-05, │ -6.872774034678204e-05, │ -7.877432800814324e-05, │ -0.00013302450116830098, │ -3.925601175888245e-05, │ -0.00022929567937810257, │ 1.7131063481163125e-05, │ -0.0003846355030322186, │ -0.0005095515358523817, │ -0.0003988011607600216, │ -0.0008459620835930776, │ -0.0005034794650467204, │ -0.0006029827263524426, │ -0.00017686124743748917, │ -0.00015337622471496315, │ -0.0 │ ], │ [ │ -0.0004593671225122864, │ -0.0006972021548553824, │ -0.0006564257204982178, │ -0.0006832957249183601, │ -0.0004380072359887562, │ -0.0006188235580668948, │ 8.717785654102366e-06, │ -0.00039305834148913873, │ -0.00010256521066618115, │ -0.00018236535079864998, │ -0.0005491942472500042, │ 4.519847826428437e-05, │ -0.000439775875972402, │ -0.000328001299898675, │ -0.0003846253179061976, │ -0.0007280318037091852, │ -0.0003807703109285203, │ -0.000679748583054692, │ -0.0009410591150086615, │ -0.0005321378635100774, │ -0.001095978768954696, │ -0.000923937345590328, │ -0.0007882728604240227, │ -0.0007635524922102895, │ -0.0003335175782412505, │ 5.101232263111722e-05, │ 7.699819760493559e-05, │ -0.0 │ ], │ [ │ -0.001316505916852844, │ -0.0009519238281162459, │ -0.0021000631378496307, │ -0.0007986435829569287, │ -0.0015434157599830472, │ -0.0008677100729058861, │ -0.0006676297752182214, │ -0.00101583725027751, │ -0.0006892680761622132, │ -0.0010906414436619807, │ -0.000918482285471507, │ -0.0010186687668344435, │ -0.000 │ (showing 2.05kB/22.97kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 930µs 468µs 172µs 1534ms 2ms 73µs 1543ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 12ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 377µs (cache) (cache) 9ms 1ms 90µs 12ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 19ms time ★ 233B↑ 1.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 755B │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="96055dec-2792- │ 4c9d-9477-639f7453e593",model_name="mnist",model_versi │ on="default",} 1514098.884 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="96055dec-2792-4c9d-9 │ 477-639f7453e593",model_name="mnist",model_version="de │ fault",} 1.0 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="96055dec-2792-4c9d │ -9477-639f7453e593",model_name="mnist",model_version=" │ default",} 186.5 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 476µs 158µs 104µs 16ms 1ms 61µs 19ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 5.2s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 23.85kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 1273ms [min: 12ms, max: 3.5s, s.d.: 1440ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 734µs [min: 158µs, max: 1ms, s.d.: 517µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1267ms [min: 9ms, max: 3.5s, s.d.: 1435ms] │ └─────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman management_api_collection Iteration 1/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 200 OK ★ 13.2s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 5ms 1ms 250µs 13.2s 7ms 358µs 13.2s ✓ Successful request Iteration 2/11 → workflow management request GET http://localhost:8081/workflows 200 OK ★ 7ms time ★ 235B↑ 423B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 156B │ { │ "workflows": [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws. │ com/war_files/densenet_wf.war" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 552µs (cache) (cache) 4ms 1ms 65µs 8ms ✓ Successful request Iteration 3/11 → workflow management request GET http://localhost:8081/workflows/densenet 200 OK ★ 4ms time ★ 244B↑ 559B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 292B │ [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws.co │ m/war_files/densenet_wf.war", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 50, │ "workflowDag": "{pre_processing=[densenet], densen │ et=[post_processing]}" │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 214µs (cache) (cache) 2ms 1ms 59µs 4ms ✓ Successful request Iteration 4/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 500 Internal Server Error ★ 4ms time ★ 321B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "code": 500, │ "type": "FileAlreadyExistsException", │ "message": "densenet_wf.war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 186µs (cache) (cache) 1ms 1ms 50µs 4ms ✓ Successful request Iteration 5/11 → workflow management request DELETE http://localhost:8081/workflows/densenet 200 OK ★ 63ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 394µs 143µs 89µs 60ms 1ms 69µs 64ms ✓ Successful request Iteration 6/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/does_not_exist.war 400 Bad Request ★ 290ms time ★ 324B↑ 441B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 170B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.s3.amazonaws.com/war_files/does_not_exist. │ war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 220µs (cache) (cache) 287ms 1ms 44µs 290ms ✓ Successful request Iteration 7/11 → workflow management request GET http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 15ms time ★ 250B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 449µs 157µs 136µs 12ms 1ms 37µs 15ms ✓ Successful request Iteration 8/11 → workflow management request DELETE http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 13ms time ★ 253B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 499µs 136µs 93µs 10ms 1ms 36µs 13ms ✓ Successful request Iteration 9/11 → workflow management request POST http://localhost:8081/workflows?url=malformed_url,? 404 Not Found ★ 14ms time ★ 275B↑ 396B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 127B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found in workflow store: ma │ lformed_url,?" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 418µs 139µs 99µs 11ms 1ms 34µs 14ms ✓ Successful request Iteration 10/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/custom_python_dep.war 500 Internal Server Error ★ 1325ms time ★ 327B↑ 531B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 250B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow custom_python_dep has failed to │ register. Failures: [Workflow Node custom_python_dep_ │ _custom_python_dep failed to register. Details: Model │ not found at: custom_python_dep.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 411µs 111µs 76µs 1321ms 1ms 37µs 1325ms ✓ Successful request Iteration 11/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/loading-memory-error.war 500 Internal Server Error ★ 1119ms time ★ 330B↑ 543B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 262B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow loading-memory-error has failed │ to register. Failures: [Workflow Node loading-memory- │ error__loading-memory-error failed to register. Detail │ s: Model not found at: loading-memory-error.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 438µs 137µs 98µs 1114ms 3ms 67µs 1120ms ✓ Successful request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 11 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 16.4s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 1.7kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 1465ms [min: 4ms, max: 13.2s, s.d.: 3.7s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 671µs [min: 111µs, max: 1ms, s.d.: 606µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1461ms [min: 1ms, max: 13.2s, s.d.: 3.7s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman workflow_inference Iteration 1/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 200 OK ★ 8.2s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 28ms 5ms 1ms 268µs 8.2s 7ms 362µs 8.3s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/densenet 200 OK ★ 3.3s time ★ 111.24kB↑ 448B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 181B │ { │ "tabby": 0.4666188061237335, │ "tiger_cat": 0.46449077129364014, │ "Egyptian_cat": 0.06614017486572266, │ "lynx": 0.001292433706112206, │ "plastic_bag": 0.00022909622930455953 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 1ms 425µs 128µs 3.3s 2ms 68µs 3.3s ✓ Successful request ✓ Test expected JSON response → Workflow - Unregister DELETE http://localhost:8081/workflows/densenet 200 OK ★ 65ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 389µs (cache) (cache) 62ms 1ms 86µs 66ms ✓ Successful DELETE request Iteration 2/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/timeout-retry-test.war 200 OK ★ 7.5s time ★ 328B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 203µs (cache) (cache) 7.5s 3ms 79µs 7.5s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/densenet 500 Internal Server Error ★ 12s time ★ 111.24kB↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Error executing pre_processing" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 283µs (cache) (cache) 12s 2ms 89µs 12s ✓ Successful request → Workflow - Unregister DELETE http://localhost:8081/workflows/densenet 200 OK ★ 58ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 253µs (cache) (cache) 55ms 1ms 52µs 59ms ✓ Successful DELETE request Iteration 3/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/dog_breed_wf.war 200 OK ★ 8.9s time ★ 322B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Workflow dog_breed_wf has been registered │ and scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 263µs (cache) (cache) 8.9s 2ms 48µs 8.9s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/dog_breed_wf 200 OK ★ 3.2s time ★ 77.75kB↑ 472B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 77.47kB │ (showing 2.05kB/77.47kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 205B │ { │ "Cardigan_welsh_corgi": 0.6281965970993042, │ "Briard": 0.2746279835700989, │ "Icelandic_sheepdog": 0.07453587651252747, │ "Tibetan_mastiff": 0.015545734204351902, │ "Chow_chow": 0.002062039216980338 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 1ms 271µs 115µs 3.2s 1ms 72µs 3.2s ✓ Successful request ✓ Test expected JSON response → Workflow - Unregister DELETE http://localhost:8081/workflows/dog_breed_wf 200 OK ★ 58ms time ★ 251B↑ 323B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 57B │ { │ "status": "Workflow \"dog_breed_wf\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 250µs (cache) (cache) 55ms 1ms 58µs 59ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 3 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 11 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 43.8s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 898B (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 4.8s [min: 58ms, max: 12s, s.d.: 4.2s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 821µs [min: 271µs, max: 1ms, s.d.: 489µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 4.8s [min: 55ms, max: 12s, s.d.: 4.2s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. ============================= test session starts ============================== platform linux -- Python 3.8.15, pytest-7.2.2, pluggy-1.0.0 -- /opt/conda/envs/ts_gpu_benchmark/bin/python cachedir: .pytest_cache rootdir: /home/ubuntu/pytorch/namannan-serve plugins: mock-3.10.0, cov-4.0.0 collecting ... collected 85 items test_example_dcgan.py::test_model_archive_creation PASSED [ 1%] test_example_dcgan.py::test_model_register_unregister PASSED [ 2%] test_example_dcgan.py::test_image_generation_without_any_input_constraints PASSED [ 3%] test_example_dcgan.py::test_image_generation_with_input_constraints PASSED [ 4%] test_example_intel_extension_for_pytorch.py::test_single_worker_affinity SKIPPED [ 5%] test_example_intel_extension_for_pytorch.py::test_multi_worker_affinity SKIPPED [ 7%] test_example_intel_extension_for_pytorch.py::test_worker_scale_up_affinity SKIPPED [ 8%] test_example_intel_extension_for_pytorch.py::test_worker_scale_down_affinity SKIPPED [ 9%] test_example_scriptable_tokenzier.py::test_handler PASSED [ 10%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_sample_text PASSED [ 11%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_empty_string PASSED [ 12%] test_example_scriptable_tokenzier.py::test_inference_with_pretrained_model PASSED [ 14%] test_gRPC_inference_api.py::test_inference_apis PASSED [ 15%] test_gRPC_management_apis.py::test_management_apis PASSED [ 16%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model PASSED [ 17%] test_handler.py::test_mnist_model_register_using_non_existent_handler_with_nonzero_workers PASSED [ 18%] test_handler.py::test_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 20%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 21%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model PASSED [ 22%] test_handler.py::test_kserve_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 23%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 24%] test_handler.py::test_huggingface_bert_batch_inference PASSED [ 25%] test_handler.py::test_MMF_activity_recognition_model_register_and_inference_on_valid_model SKIPPED [ 27%] test_handler.py::test_huggingface_bert_model_parallel_inference PASSED [ 28%] test_metrics.py::test_logs_created PASSED [ 29%] test_metrics.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 30%] test_metrics.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 31%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 32%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 34%] test_metrics.py::test_log_location_var_snapshot_disabled PASSED [ 35%] test_metrics.py::test_log_location_var_snapshot_enabled PASSED [ 36%] test_metrics.py::test_async_logging PASSED [ 37%] test_metrics.py::test_async_logging_non_boolean PASSED [ 38%] test_metrics.py::test_metrics_location_var_snapshot_disabled PASSED [ 40%] test_metrics.py::test_metrics_location_var_snapshot_enabled PASSED [ 41%] test_metrics.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 42%] test_metrics.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 43%] test_metrics.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 44%] test_metrics.py::test_collect_system_metrics_when_not_disabled PASSED [ 45%] test_metrics.py::test_disable_system_metrics_using_config_properties PASSED [ 47%] test_metrics.py::test_disable_system_metrics_using_environment_variable PASSED [ 48%] test_metrics_kf.py::test_logs_created PASSED [ 49%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 50%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 51%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 52%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 54%] test_metrics_kf.py::test_log_location_var_snapshot_disabled PASSED [ 55%] test_metrics_kf.py::test_log_location_var_snapshot_enabled PASSED [ 56%] test_metrics_kf.py::test_async_logging PASSED [ 57%] test_metrics_kf.py::test_async_logging_non_boolean PASSED [ 58%] test_metrics_kf.py::test_metrics_location_var_snapshot_disabled PASSED [ 60%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled PASSED [ 61%] test_metrics_kf.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 62%] test_metrics_kf.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 63%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 64%] test_model_archiver.py::test_multiple_model_versions_registration PASSED [ 65%] test_model_archiver.py::test_duplicate_model_registration_using_local_url_followed_by_http_url PASSED [ 67%] test_model_archiver.py::test_duplicate_model_registration_using_http_url_followed_by_local_url PASSED [ 68%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_without_force PASSED [ 69%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_with_force PASSED [ 70%] test_model_archiver.py::test_model_archiver_without_handler_flag PASSED [ 71%] test_model_archiver.py::test_model_archiver_without_model_name_flag PASSED [ 72%] test_model_archiver.py::test_model_archiver_without_model_file_flag PASSED [ 74%] test_model_archiver.py::test_model_archiver_without_serialized_flag PASSED [ 75%] test_onnx.py::test_convert_to_onnx SKIPPED (ONNX is not installed) [ 76%] test_onnx.py::test_model_packaging_and_start SKIPPED (ONNX is not in...) [ 77%] test_onnx.py::test_model_start SKIPPED (ONNX is not installed) [ 78%] test_onnx.py::test_inference SKIPPED (ONNX is not installed) [ 80%] test_onnx.py::test_stop SKIPPED (ONNX is not installed) [ 81%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[/home/ubuntu/pytorch/namannan-serve/test/pytest/profiler_utils/resnet_custom.py] PASSED [ 82%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[image_classifier] PASSED [ 83%] test_pytorch_profiler.py::test_profiler_arguments_override[/home/ubuntu/pytorch/namannan-serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 84%] test_pytorch_profiler.py::test_batch_input[/home/ubuntu/pytorch/namannan-serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 85%] test_snapshot.py::test_snapshot_created_on_start_and_stop PASSED [ 87%] test_snapshot.py::test_snapshot_created_on_management_api_invoke PASSED [ 88%] test_snapshot.py::test_start_from_snapshot PASSED [ 89%] test_snapshot.py::test_start_from_latest PASSED [ 90%] test_snapshot.py::test_start_from_read_only_snapshot PASSED [ 91%] test_snapshot.py::test_no_config_snapshots_cli_option PASSED [ 92%] test_snapshot.py::test_start_from_default PASSED [ 94%] test_snapshot.py::test_start_from_non_existing_snapshot PASSED [ 95%] test_snapshot.py::test_torchserve_init_with_non_existent_model_store PASSED [ 96%] test_snapshot.py::test_restart_torchserve_with_last_snapshot_with_model_mar_removed PASSED [ 97%] test_snapshot.py::test_replace_mar_file_with_dummy PASSED [ 98%] test_snapshot.py::test_restart_torchserve_with_one_of_model_mar_removed PASSED [100%] =============================== warnings summary =============================== test/pytest/test_example_scriptable_tokenzier.py::test_handler /opt/conda/envs/ts_gpu_benchmark/lib/python3.8/site-packages/torch/jit/_recursive.py:262: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/pytest/test_example_scriptable_tokenzier.py::test_handler /opt/conda/envs/ts_gpu_benchmark/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning) test/pytest/test_example_scriptable_tokenzier.py::test_handler /opt/conda/envs/ts_gpu_benchmark/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) test/pytest/test_example_scriptable_tokenzier.py::test_handler test/pytest/test_example_scriptable_tokenzier.py::test_handler /opt/conda/envs/ts_gpu_benchmark/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) test/pytest/test_example_scriptable_tokenzier.py::test_handler /opt/conda/envs/ts_gpu_benchmark/lib/python3.8/site-packages/torch/nn/modules/module.py:1194: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at ../aten/src/ATen/NestedTensorImpl.cpp:175.) return forward_call(*input, **kwargs) test/pytest/test_example_scriptable_tokenzier.py::test_handler /home/ubuntu/pytorch/namannan-serve/test/pytest/../../examples/text_classification_with_scriptable_tokenizer/handler.py:97: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. data = F.softmax(data) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========== 75 passed, 10 skipped, 7 warnings in 3207.37s (0:53:27) ============ ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kfv2.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/pytorch/namannan-serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Started regression tests ## Started densenet mar creation ## In directory: /tmp/workspace/model_store | Executing command: torch-model-archiver --model-name densenet161_v1 --version 1.1 --model-file /home/ubuntu/pytorch/namannan-serve/ts_scripts/../examples/image_classifier/densenet_161/model.py --serialized-file /tmp/workspace/model_store/densenet161-8d451a50.pth --extra-files /home/ubuntu/pytorch/namannan-serve/ts_scripts/../examples/image_classifier/index_to_name.json --handler image_classifier --force ## Started regression pytests ## In directory: /home/ubuntu/pytorch/namannan-serve/test/pytest | Executing command: python -m pytest -v ./ Removing file : management_pb2_grpc.py Removing file : inference_pb2_grpc.py Removing file : management_pb2.py Removing file : inference_pb2.py ## Deleting model_store_gen_dir: /home/ubuntu/pytorch/namannan-serve/ts_scripts/../model_store_gen