Processing /home/ubuntu/serve Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting Pillow Using cached Pillow-9.5.0-cp38-cp38-manylinux_2_28_x86_64.whl (3.4 MB) Collecting psutil Using cached psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB) Collecting packaging Using cached packaging-23.0-py3-none-any.whl (42 kB) Collecting wheel Using cached wheel-0.40.0-py3-none-any.whl (64 kB) Building wheels for collected packages: torchserve Building wheel for torchserve (setup.py): started Building wheel for torchserve (setup.py): finished with status 'done' Created wheel for torchserve: filename=torchserve-0.7.1b20230408-py3-none-any.whl size=23699256 sha256=2ec5d7e7bd75e8f3db623575b6b21daa53bf35a442f9ec617359ce73bbd6d709 Stored in directory: /tmp/pip-ephem-wheel-cache-jkg_duzg/wheels/31/36/53/4ebf241bef6982efd88c312f3be33096274ef0422259656506 Successfully built torchserve Installing collected packages: wheel, psutil, Pillow, packaging, torchserve Attempting uninstall: wheel Found existing installation: wheel 0.40.0 Uninstalling wheel-0.40.0: Successfully uninstalled wheel-0.40.0 Attempting uninstall: psutil Found existing installation: psutil 5.9.4 Uninstalling psutil-5.9.4: Successfully uninstalled psutil-5.9.4 Attempting uninstall: Pillow Found existing installation: Pillow 9.3.0 Uninstalling Pillow-9.3.0: Successfully uninstalled Pillow-9.3.0 Attempting uninstall: packaging Found existing installation: packaging 23.0 Uninstalling packaging-23.0: Successfully uninstalled packaging-23.0 Attempting uninstall: torchserve Found existing installation: torchserve 0.7.1b20230408 Uninstalling torchserve-0.7.1b20230408: Successfully uninstalled torchserve-0.7.1b20230408 Successfully installed Pillow-9.5.0 packaging-23.0 psutil-5.9.4 torchserve-0.7.1b20230408 wheel-0.40.0 Processing ./model-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting enum-compat Using cached enum_compat-0.0.3-py3-none-any.whl (1.3 kB) Building wheels for collected packages: torch-model-archiver Building wheel for torch-model-archiver (setup.py): started Building wheel for torch-model-archiver (setup.py): finished with status 'done' Created wheel for torch-model-archiver: filename=torch_model_archiver-0.7.1b20230408-py3-none-any.whl size=14736 sha256=471939ff72cea325af1fdc8fa304f90478efce0773148aa3e65db7a92be5d66c Stored in directory: /home/ubuntu/.cache/pip/wheels/e5/cb/30/a0997a79fe89919cd4f84623edf89126a2973d8956e8f223e6 Successfully built torch-model-archiver Installing collected packages: enum-compat, torch-model-archiver Attempting uninstall: enum-compat Found existing installation: enum-compat 0.0.3 Uninstalling enum-compat-0.0.3: Successfully uninstalled enum-compat-0.0.3 Attempting uninstall: torch-model-archiver Found existing installation: torch-model-archiver 0.7.1b20230408 Uninstalling torch-model-archiver-0.7.1b20230408: Successfully uninstalled torch-model-archiver-0.7.1b20230408 Successfully installed enum-compat-0.0.3 torch-model-archiver-0.7.1b20230408 Processing ./workflow-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Building wheels for collected packages: torch-workflow-archiver Building wheel for torch-workflow-archiver (setup.py): started Building wheel for torch-workflow-archiver (setup.py): finished with status 'done' Created wheel for torch-workflow-archiver: filename=torch_workflow_archiver-0.2.7b20230408-py3-none-any.whl size=12700 sha256=4062f1d71dd58ed24f42451df26c1cca95887fd3057efb9cbadde04ed6d9c9d6 Stored in directory: /home/ubuntu/.cache/pip/wheels/c2/0a/ec/3d00816a4b8f0fd84fd68c65eca314fc30cc1a41a0b90ad7c1 Successfully built torch-workflow-archiver Installing collected packages: torch-workflow-archiver Attempting uninstall: torch-workflow-archiver Found existing installation: torch-workflow-archiver 0.2.7b20230408 Uninstalling torch-workflow-archiver-0.2.7b20230408: Successfully uninstalled torch-workflow-archiver-0.2.7b20230408 Successfully installed torch-workflow-archiver-0.2.7b20230408 Current date and time : 2023-04-08 22:10:55 ------------------------------------------------------------------------------------------ Environment headers ------------------------------------------------------------------------------------------ Torchserve branch: feat/ds torchserve==0.7.1 torch-model-archiver==0.7.1 Python version: 3.8 (64-bit runtime) Python executable: /opt/conda/envs/py38/bin/python Versions of relevant python libraries: captum==0.6.0 intel-extension-for-pytorch==2.0.0 numpy==1.24.2 nvgpu==0.10.0 psutil==5.9.4 pygit2==1.12.0 pylint==2.6.0 pytest==7.3.0 pytest-cov==4.0.0 pytest-mock==3.10.0 pytorch-triton==2.1.0+2c32f43999 requests==2.28.2 requests-toolbelt==0.10.1 torch==2.0.0+cu117 torch-model-archiver==0.7.1b20230408 torch-workflow-archiver==0.2.7b20230408 torchaudio==2.0.1+cu117 torchdata==0.6.0 torchserve==0.7.1b20230408 torchtext==0.15.1+cpu torchvision==0.15.1+cu117 transformers==4.25.1 wheel==0.40.0 torch==2.0.0+cu117 torchtext==0.15.1+cpu torchvision==0.15.1+cu117 torchaudio==2.0.1+cu117 Java Version: OS: Ubuntu 20.04.5 LTS GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 Clang version: N/A CMake version: version 3.25.0 Is CUDA available: Yes CUDA runtime version: 11.7.99 GPU models and configuration: GPU 0: NVIDIA A10G GPU 1: NVIDIA A10G GPU 2: NVIDIA A10G GPU 3: NVIDIA A10G Nvidia driver version: 515.65.01 cuDNN version: None ## In directory /home/ubuntu/serve | Executing command pip install --force-reinstall . ## In directory /home/ubuntu/serve | Executing command pip install --force-reinstall model-archiver/. ## In directory /home/ubuntu/serve | Executing command pip install --force-reinstall workflow-archiver/. ## Starting generate_mars, mar_config:/home/ubuntu/serve/ts_scripts/../ts_scripts/mar_config.json, model_store_dir:/home/ubuntu/serve/ts_scripts/../model_store_gen ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name fastrcnn --version 1.0 --model-file examples/object_detector/fast-rcnn/model.py --serialized-file /home/ubuntu/serve/ts_scripts/../model_store_gen/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## fastrcnn.mar is generated. ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name alexnet --version 1.0 --model-file examples/image_classifier/alexnet/model.py --serialized-file /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet-owt-7be5be79.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## alexnet.mar is generated. ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name densenet161 --version 1.0 --model-file examples/image_classifier/densenet_161/model.py --serialized-file /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161-8d451a50.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## densenet161.mar is generated. ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name mnist --version 1.0 --model-file examples/image_classifier/mnist/mnist.py --serialized-file examples/image_classifier/mnist/mnist_cnn.pt --handler examples/image_classifier/mnist/mnist_handler.py --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## mnist.mar is generated. ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## resnet-152-batch.mar is generated. ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name resnet-18 --version 1.0 --model-file examples/image_classifier/resnet_18/model.py --serialized-file /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet18-f37072fd.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## resnet-18.mar is generated. ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name squeezenet1_1 --version 1.0 --model-file examples/image_classifier/squeezenet/model.py --serialized-file /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1-b8a52dc0.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## squeezenet1_1.mar is generated. ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name vgg16 --version 1.0 --model-file examples/image_classifier/vgg_16/model.py --serialized-file /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16-397923af.pth --handler examples/image_classifier/vgg_16/vgg_handler.py --extra-files examples/image_classifier/index_to_name.json --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## vgg16.mar is generated. ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name deeplabv3_resnet_101_eager --version 1.0 --model-file examples/image_segmenter/deeplabv3/model.py --serialized-file /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet101_coco-586e9e4e.pth --handler image_segmenter --extra-files examples/image_segmenter/deeplabv3/deeplabv3.py,examples/image_segmenter/deeplabv3/intermediate_layer_getter.py,examples/image_segmenter/deeplabv3/fcn.py --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## deeplabv3_resnet_101_eager.mar is generated. ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name fcn_resnet_101 --version 1.0 --model-file examples/image_segmenter/fcn/model.py --serialized-file /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet101_coco-7ecb50ca.pth --handler image_segmenter --extra-files examples/image_segmenter/fcn/fcn.py,examples/image_segmenter/fcn/intermediate_layer_getter.py --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## fcn_resnet_101.mar is generated. ## In directory: /home/ubuntu/serve | Executing command: torch-model-archiver --model-name maskrcnn --version 1.0 --model-file examples/object_detector/maskrcnn/model.py --serialized-file /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --export-path /home/ubuntu/serve/ts_scripts/../model_store_gen --force ## maskrcnn.mar is generated. ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 184ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 35ms 4ms 1ms 246µs 168ms 8ms 498µs 219ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 63ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 473µs (cache) (cache) 60ms 1ms 60µs 63ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 1234ms time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 198µs (cache) (cache) 1231ms 1ms 56µs 1234ms ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 4ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 173µs (cache) (cache) 2ms 1ms 42µs 4ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 27ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 452µs 149µs 124µs 25ms 999µs 49µs 28ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 3ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 968µs 154µs (cache) (cache) 1ms 1ms 31µs 3ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 12ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 987µs 371µs 138µs 130µs 9ms 970µs 31µs 12ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 55ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 176µs (cache) (cache) 53ms 969µs 30µs 55ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 4ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 179µs (cache) (cache) 2ms 907µs 25µs 4ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 55ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 142µs (cache) (cache) 53ms 1ms 25µs 55ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 1232ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 169µs (cache) (cache) 1229ms 1ms 28µs 1232ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 12ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 394µs 140µs 106µs 9ms 879µs 27µs 12ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 5ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 915µs 144µs (cache) (cache) 3ms 862µs 25µs 5ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 3ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 940µs 164µs (cache) (cache) 1ms 838µs 29µs 3ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 3ms time ★ 246B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 937µs 144µs (cache) (cache) 1ms 1ms 24µs 3ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 870µs 139µs (cache) (cache) 2ms 809µs 25µs 4ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 3ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 143µs (cache) (cache) 1ms 846µs 24µs 3ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 11ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 980µs 366µs 169µs 112µs 9ms 836µs 23µs 11ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 57ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 978µs 148µs (cache) (cache) 55ms 861µs 27µs 57ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 3ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 908µs 135µs (cache) (cache) 1ms 818µs 22µs 3ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 3ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 164µs (cache) (cache) 1ms 861µs 30µs 3ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 3ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 915µs 135µs (cache) (cache) 1ms 1ms 23µs 3ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 3ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 168µs (cache) (cache) 963µs 884µs 24µs 3ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 10ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 897µs 335µs 114µs 88µs 7ms 829µs 22µs 9ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 3ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 852µs 140µs (cache) (cache) 2ms 814µs 27µs 3ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 3ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 832µs 143µs (cache) (cache) 1ms 813µs 23µs 3ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 10ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 883µs 355µs 158µs 100µs 7ms 837µs 23µs 10ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 14ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 825µs 329µs 110µs 85µs 12ms 815µs 25µs 14ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 57ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 973µs 149µs (cache) (cache) 55ms 861µs 23µs 57ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 3ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 760µs 136µs (cache) (cache) 1ms 748µs 21µs 3ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1255ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 876µs 138µs (cache) (cache) 1252ms 1ms 26µs 1255ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 64ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 431µs 125µs 94µs 61ms 913µs 23µs 64ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 3ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 848µs 137µs (cache) (cache) 1ms 772µs 28µs 3ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 2.5s time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 134µs (cache) (cache) 2.5s 1ms 23µs 2.5s ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 34ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 858µs 142µs (cache) (cache) 32ms 782µs 22µs 34ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 6.7s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 863µs 158µs (cache) (cache) 6.7s 1ms 24µs 6.7s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 45ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 152µs (cache) (cache) 43ms 715µs 22µs 45ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 6.6s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 889µs 131µs (cache) (cache) 6.6s 2ms 59µs 6.6s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 46ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 313µs (cache) (cache) 44ms 1ms 34µs 48ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 57ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 178µs (cache) (cache) 55ms 970µs 24µs 57ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 980µs 143µs (cache) (cache) 1ms 1ms 24µs 4ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 499ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 151µs (cache) (cache) 497ms 933µs 22µs 499ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 9ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 143µs (cache) (cache) 7ms 735µs 22µs 9ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 505ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 137µs (cache) (cache) 502ms 1ms 25µs 505ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 10ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 906µs 148µs (cache) (cache) 8ms 1ms 23µs 10ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 501ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 995µs 159µs (cache) (cache) 498ms 1ms 33µs 501ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 2ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 882µs 133µs (cache) (cache) 799µs 748µs 26µs 2ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 2ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 827µs 121µs (cache) (cache) 604µs 794µs 21µs 2ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 2ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 948µs 132µs (cache) (cache) 633µs 768µs 25µs 2ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 2ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 871µs 143µs (cache) (cache) 824µs 892µs 22µs 2ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 2ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 966µs 158µs (cache) (cache) 877µs 817µs 22µs 2ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 2ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 808µs 125µs (cache) (cache) 832µs 786µs 20µs 2ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 2ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 901µs 133µs (cache) (cache) 796µs 824µs 24µs 2ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 2ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 916µs 137µs (cache) (cache) 645µs 781µs 22µs 2ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 2ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 852µs 130µs (cache) (cache) 773µs 799µs 20µs 2ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 3ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 803µs 129µs (cache) (cache) 1ms 770µs 21µs 3ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 2ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 799µs 150µs (cache) (cache) 910µs 828µs 21µs 2ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 592ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 414µs 265µs 96µs 589ms 1ms 30µs 592ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 9ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 944µs 334µs 129µs 100µs 7ms 769µs 24µs 9ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 9ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 854µs 314µs 106µs 90µs 7ms 773µs 23µs 9ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 10ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 855µs 293µs 97µs 96µs 7ms 1ms 23µs 10ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 3ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 861µs 1ms (cache) (cache) 918µs 764µs 21µs 3ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 2ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 897µs 137µs (cache) (cache) 790µs 845µs 22µs 2ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 2ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 141µs (cache) (cache) 967µs 815µs 24µs 3ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 2ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 915µs 134µs (cache) (cache) 631µs 855µs 22µs 2ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 8ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 315µs 111µs 94µs 6ms 833µs 31µs 10ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 9ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 785µs 309µs 95µs 78µs 7ms 773µs 22µs 9ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 5ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 827µs 126µs (cache) (cache) 4ms 731µs 21µs 5ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 3ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 138µs (cache) (cache) 2ms 762µs 25µs 4ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 9ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 826µs 308µs 104µs 88µs 7ms 742µs 21µs 9ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 8ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 845µs 302µs 106µs 86µs 6ms 758µs 22µs 8ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 16ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 869µs 345µs 102µs 101µs 14ms 799µs 21µs 16ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 2ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 825µs 128µs (cache) (cache) 738µs 755µs 22µs 2ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 87ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 878µs 307µs 107µs 89µs 84ms 1ms 30µs 87ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 3ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 152µs (cache) (cache) 1ms 773µs 23µs 3ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 57ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 157µs (cache) (cache) 54ms 1ms 27µs 57ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 3ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 157µs (cache) (cache) 1ms 983µs 24µs 3ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 57ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 150µs (cache) (cache) 54ms 1ms 31µs 57ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 3ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 818µs 128µs (cache) (cache) 1ms 760µs 22µs 3ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 56ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 835µs 126µs (cache) (cache) 55ms 806µs 22µs 56ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 5ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 131µs (cache) (cache) 1ms 2ms 21µs 6ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 2ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 886µs 132µs (cache) (cache) 930µs 722µs 25µs 2ms ✓ Successful request ┌─────────────────────────┬──────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴──────────────────────┴─────────────────────┤ │ total run duration: 24.4s │ ├──────────────────────────────────────────────────────────────────────┤ │ total data received: 7.75kB (approx) │ ├──────────────────────────────────────────────────────────────────────┤ │ average response time: 281ms [min: 2ms, max: 6.7s, s.d.: 1080ms] │ ├──────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 656µs [min: 95µs, max: 1ms, s.d.: 588µs] │ ├──────────────────────────────────────────────────────────────────────┤ │ average first byte time: 279ms [min: 604µs, max: 6.7s, s.d.: 1080ms] │ └──────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman inference Iteration 1/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_v4.mar&model_name=my_text_classifier&initial_workers=1&synchronous=true 200 OK ★ 8.1s time ★ 388B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"my_text_classifier\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 34ms 4ms 1ms 249µs 8.1s 6ms 338µs 8.1s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier 200 OK ★ 703ms time ★ 353B↑ 374B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 139B │ { │ "World": 0.029119636863470078, │ "Sports": 2.9431575967464596e-05, │ "Business": 0.9074352383613586, │ "Sci/Tec": 0.06341564655303955 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 947µs 251µs 344µs 698ms 1ms 66µs 707ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier 200 OK ★ 66ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"my_text_classifier\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 363µs (cache) (cache) 64ms 1ms 69µs 67ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 16ms time ★ 233B↑ 1.09kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 795B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ ",model_version="default",} 682104.857 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="my_text_classifier",mode │ l_version="default",} 1.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 400µs 137µs 109µs 12ms 1ms 52µs 16ms ✓ Successful GET request Iteration 2/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_scripted_v3.mar&model_name=my_text_classifier_scripted&initial_workers=1&synchronous=true 200 OK ★ 7s time ★ 406B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 103B │ { │ "status": "Model \"my_text_classifier_scripted\" Ver │ sion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 196µs (cache) (cache) 7s 1ms 45µs 7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier_scripted 200 OK ★ 743ms time ★ 362B↑ 373B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 138B │ { │ "World": 0.04559723287820816, │ "Sports": 0.00037717679515480995, │ "Business": 0.08623101562261581, │ "Sci/Tec": 0.8677946329116821 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 219µs (cache) (cache) 737ms 4ms 94µs 745ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier_scripted 200 OK ★ 67ms time ★ 263B↑ 335B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 69B │ { │ "status": "Model \"my_text_classifier_scripted\" unr │ egistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 207µs (cache) (cache) 64ms 1ms 35µs 68ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 1.53kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1.24kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ ",model_version="default",} 682104.857 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ _scripted",model_version="default",} 735645.09 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 202µs (cache) (cache) 1ms 1ms 38µs 4ms ✓ Successful GET request Iteration 3/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 3.2s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 193µs (cache) (cache) 3.2s 1ms 64µs 3.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1 200 OK ★ 1441ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.27828869223594666, │ "lynx": 0.25356364250183105, │ "tiger_cat": 0.2448405772447586, │ "Egyptian_cat": 0.21699286997318268, │ "cougar": 0.002220059046521783 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 308µs (cache) (cache) 1439ms 1ms 27µs 1442ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 9ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 7ms 885µs 27µs 9ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 1.94kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1.65kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squeezenet1_1",mod │ el_version="default",} 1434612.313 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ ",model_version="default",} 682104.857 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ _scripted",model_version="default",} 735645.09 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 970µs 156µs (cache) (cache) 1ms 861µs 25µs 3ms ✓ Successful GET request Iteration 4/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/squeezenet1_1_scripted.mar&model_name=squeezenet1_1_scripted&initial_workers=1&synchronous=true 200 OK ★ 3.4s time ★ 393B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 98B │ { │ "status": "Model \"squeezenet1_1_scripted\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 154µs (cache) (cache) 3.4s 1ms 28µs 3.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1_scripted 200 OK ★ 1533ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.27828869223594666, │ "lynx": 0.25356364250183105, │ "tiger_cat": 0.2448405772447586, │ "Egyptian_cat": 0.21699286997318268, │ "cougar": 0.002220059046521783 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 170µs (cache) (cache) 1531ms 1ms 27µs 1534ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1_scripted 200 OK ★ 9ms time ★ 258B↑ 330B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 64B │ { │ "status": "Model \"squeezenet1_1_scripted\" unregist │ ered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 978µs 147µs (cache) (cache) 7ms 987µs 26µs 9ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 2.37kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.08kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squeezenet1_1",mod │ el_version="default",} 1434612.313 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ ",model_version="default",} 682104.857 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ _scripted",model_version="default",} 735645.09 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squeezenet1_1_scri │ pted",model_version="default",} 1527322.375 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="squeezenet1_1_script │ (showing 2.05kB/2.08kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 165µs (cache) (cache) 1ms 1ms 40µs 4ms ✓ Successful GET request Iteration 5/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161&initial_workers=1&synchronous=true 200 OK ★ 4.9s time ★ 330B↑ 353B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 87B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 173µs (cache) (cache) 4.9s 1ms 26µs 4.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161 200 OK ★ 1604ms time ★ 111.24kB↑ 416B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 181B │ { │ "tabby": 0.4659089148044586, │ "tiger_cat": 0.4652513265609741, │ "Egyptian_cat": 0.06609020382165909, │ "lynx": 0.0012922680471092463, │ "plastic_bag": 0.00022853966220282018 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 308µs (cache) (cache) 1602ms 988µs 23µs 1605ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161 200 OK ★ 27ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 890µs 149µs (cache) (cache) 25ms 803µs 24µs 27ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 2.77kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.48kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squeezenet1_1",mod │ el_version="default",} 1434612.313 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ ",model_version="default",} 682104.857 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ _scripted",model_version="default",} 735645.09 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squeezenet1_1_scri │ pted",model_version="default",} 1527322.375 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="densenet161",model │ _version="default",} 1597527.33 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="c1c72680-9135-4be8-9 │ 840-46423a7b5429",model_name="my_text_classifier",mode │ (showing 2.05kB/2.48kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 153µs (cache) (cache) 1ms 1ms 26µs 4ms ✓ Successful GET request Iteration 6/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=alexnet.mar&model_name=alexnet&initial_workers=1&synchronous=true 200 OK ★ 6.3s time ★ 322B↑ 349B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 83B │ { │ "status": "Model \"alexnet\" Version: 1.0 registered │ with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 151µs (cache) (cache) 6.3s 1ms 25µs 6.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet 200 OK ★ 1394ms time ★ 111.24kB↑ 406B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 171B │ { │ "tabby": 0.3188331425189972, │ "tiger_cat": 0.25798729062080383, │ "Egyptian_cat": 0.24234458804130554, │ "lynx": 0.1685788482427597, │ "tiger": 0.006501472555100918 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 162µs (cache) (cache) 1391ms 2ms 26µs 1395ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet 200 OK ★ 45ms time ★ 243B↑ 315B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 49B │ { │ "status": "Model \"alexnet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 156µs (cache) (cache) 43ms 950µs 27µs 45ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 3.16kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.87kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet",model_ver │ sion="default",} 1387731.715 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squeezenet1_1",mod │ el_version="default",} 1434612.313 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ ",model_version="default",} 682104.857 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ _scripted",model_version="default",} 735645.09 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squeezenet1_1_scri │ pted",model_version="default",} 1527322.375 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="densenet161",model │ _version="default",} 1597527.33 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_infe │ (showing 2.05kB/2.87kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 161µs (cache) (cache) 1ms 1ms 25µs 4ms ✓ Successful GET request Iteration 7/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/alexnet_scripted.mar&model_name=alexnet_scripted&initial_workers=1&synchronous=true 200 OK ★ 7.9s time ★ 381B↑ 358B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 92B │ { │ "status": "Model \"alexnet_scripted\" Version: 1.0 r │ egistered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 152µs (cache) (cache) 7.9s 1ms 30µs 7.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet_scripted 200 OK ★ 2.2s time ★ 111.25kB↑ 404B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 169B │ { │ "tabby": 0.3188335597515106, │ "tiger_cat": 0.2579871118068695, │ "Egyptian_cat": 0.242344468832016, │ "lynx": 0.16857875883579254, │ "tiger": 0.006501468829810619 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 170µs (cache) (cache) 2.2s 2ms 28µs 2.2s ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet_scripted 200 OK ★ 81ms time ★ 252B↑ 324B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58B │ { │ "status": "Model \"alexnet_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 169µs (cache) (cache) 79ms 1ms 24µs 82ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 3.58kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.29kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet_scripted", │ model_version="default",} 2276314.455 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet",model_ver │ sion="default",} 1387731.715 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squeezenet1_1",mod │ el_version="default",} 1434612.313 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ ",model_version="default",} 682104.857 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ _scripted",model_version="default",} 735645.09 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squ │ (showing 2.05kB/3.29kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 159µs (cache) (cache) 2ms 897µs 28µs 4ms ✓ Successful GET request Iteration 8/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&initial_workers=1&synchronous=true 200 OK ★ 3.8s time ★ 326B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 170µs (cache) (cache) 3.8s 1ms 24µs 3.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18 200 OK ★ 1411ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.4098131060600281, │ "tiger_cat": 0.3465622365474701, │ "Egyptian_cat": 0.12998847663402557, │ "lynx": 0.023901348933577538, │ "bucket": 0.011543930508196354 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 158µs (cache) (cache) 1408ms 1ms 26µs 1411ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18 200 OK ★ 16ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 565µs (cache) (cache) 13ms 1ms 25µs 16ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 2ms time ★ 233B↑ 3.97kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.68kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet_scripted", │ model_version="default",} 2276314.455 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet",model_ver │ sion="default",} 1387731.715 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squeezenet1_1",mod │ el_version="default",} 1434612.313 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ ",model_version="default",} 682104.857 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="my_text_classifier │ _scripted │ (showing 2.05kB/3.68kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 150µs (cache) (cache) 1ms 750µs 22µs 3ms ✓ Successful GET request Iteration 9/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-18_scripted.mar&model_name=resnet-18_scripted&initial_workers=1&synchronous=true 200 OK ★ 4.5s time ★ 385B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"resnet-18_scripted\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 151µs (cache) (cache) 4.5s 1ms 28µs 4.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18_scripted 200 OK ★ 1530ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.40981295704841614, │ "tiger_cat": 0.3465624451637268, │ "Egyptian_cat": 0.12998832762241364, │ "lynx": 0.023901384323835373, │ "bucket": 0.01154392585158348 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 160µs (cache) (cache) 1527ms 1ms 23µs 1530ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18_scripted 200 OK ★ 21ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"resnet-18_scripted\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 898µs 149µs (cache) (cache) 19ms 1ms 27µs 21ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 4.39kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.1kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet_scripted", │ model_version="default",} 2276314.455 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="resnet-18_scripted │ ",model_version="default",} 1524608.67 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet",model_ver │ sion="default",} 1387731.715 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="squeezenet1_1",mod │ el_version=" │ (showing 2.05kB/4.1kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful GET request Iteration 10/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=vgg16.mar&model_name=vgg16&initial_workers=1&synchronous=true 200 OK ★ 10.1s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"vgg16\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 909µs 147µs (cache) (cache) 10.1s 1ms 30µs 10.1s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16 200 OK ★ 1430ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tiger_cat": 0.44678616523742676, │ "tabby": 0.4409930408000946, │ "Egyptian_cat": 0.05908344313502312, │ "tiger": 0.020602263510227203, │ "lynx": 0.009946360252797604 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 188µs (cache) (cache) 1426ms 2ms 30µs 1430ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16 200 OK ★ 94ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"vgg16\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 954µs 173µs (cache) (cache) 92ms 956µs 26µs 94ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 4.78kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.49kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet_scripted", │ model_version="default",} 2276314.455 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="resnet-18_scripted │ ",model_version="default",} 1524608.67 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet",model_ver │ sion="default",} 138773 │ (showing 2.05kB/4.49kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 996µs 157µs (cache) (cache) 1ms 1ms 40µs 3ms ✓ Successful GET request Iteration 11/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/vgg16_scripted.mar&model_name=vgg16_scripted&initial_workers=1&synchronous=true 200 OK ★ 11.3s time ★ 377B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"vgg16_scripted\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 194µs (cache) (cache) 11.3s 1ms 28µs 11.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16_scripted 200 OK ★ 1478ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tiger_cat": 0.44678613543510437, │ "tabby": 0.44099298119544983, │ "Egyptian_cat": 0.0590834878385067, │ "tiger": 0.020602280274033546, │ "lynx": 0.009946359321475029 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 161µs (cache) (cache) 1475ms 1ms 29µs 1479ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16_scripted 200 OK ★ 157ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"vgg16_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 171µs (cache) (cache) 155ms 873µs 27µs 158ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 5.19kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.89kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet_scripted", │ model_version="default",} 2276314.455 │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="resnet-18_scripted │ ",model_version="default",} 152460 │ (showing 2.05kB/4.89kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 165µs (cache) (cache) 1ms 868µs 28µs 3ms ✓ Successful GET request Iteration 12/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_v2.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 362B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 2.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 164µs (cache) (cache) 3.3s 1ms 33µs 3.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist 200 OK ★ 1380ms time ★ 537B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 150µs (cache) (cache) 1378ms 1ms 28µs 1380ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 9ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 907µs 447µs (cache) (cache) 7ms 751µs 24µs 9ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 5.57kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.28kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c1c72680-9135- │ 4be8-9840-46423a7b5429",model_name="alexnet_scripted", │ model_version="default",} 2276314.455 │ ts_inference_la │ (showing 2.05kB/5.28kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 155µs (cache) (cache) 1ms 999µs 32µs 3ms ✓ Successful GET request Iteration 13/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_scripted_v2.mar&model_name=mnist_scripted&initial_workers=1&synchronous=true 200 OK ★ 3.6s time ★ 380B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"mnist_scripted\" Version: 2.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 926µs 155µs (cache) (cache) 3.6s 1ms 26µs 3.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist_scripted 200 OK ★ 1460ms time ★ 546B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 160µs (cache) (cache) 1456ms 3ms 29µs 1461ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist_scripted 200 OK ★ 8ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"mnist_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 997µs 162µs (cache) (cache) 6ms 1ms 23µs 8ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 5.98kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.69kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_micro │ (showing 2.05kB/5.69kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 855µs 144µs (cache) (cache) 1ms 1ms 29µs 3ms ✓ Successful GET request Iteration 14/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fastrcnn.mar&model_name=fastrcnn&initial_workers=1&synchronous=true 200 OK ★ 5.5s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"fastrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 797µs 142µs (cache) (cache) 5.5s 892µs 26µs 5.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fastrcnn 200 OK ★ 1509ms time ★ 289.21kB↑ 2.99kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.75kB │ [ │ { │ "person": [ │ 167.39218139648438, │ 57.20952224731445, │ 301.36273193359375, │ 436.7870788574219 │ ], │ "score": 0.9995195865631104 │ }, │ { │ "person": [ │ 89.5367202758789, │ 64.83736419677734, │ 191.42674255371094, │ 446.7557373046875 │ ], │ "score": 0.9994959831237793 │ }, │ { │ "person": [ │ 362.38818359375, │ 161.84059143066406, │ 515.5030517578125, │ 385.28466796875 │ ], │ "score": 0.9976996779441833 │ }, │ { │ "handbag": [ │ 67.37800598144531, │ 277.58001708984375, │ 111.67491912841797, │ 400.20526123046875 │ ], │ "score": 0.9925121068954468 │ }, │ { │ "handbag": [ │ 228.68167114257812, │ 146.003173828125, │ 303.550537109375, │ 231.08546447753906 │ ], │ "score": 0.992243230342865 │ }, │ { │ "handbag": [ │ 379.4122619628906, │ 259.9644775390625, │ 419.0796813964844, │ 317.9609375 │ ], │ "score": 0.9898591637611389 │ }, │ { │ "person": [ │ 518.5001831054688, │ 149.73573303222656, │ 636.6318359375, │ 365.4100036621094 │ ], │ "score": 0.9820877909660339 │ }, │ { │ "bench": [ │ 269.0777893066406, │ 217.3325653076172, │ 423.795166015625, │ 390.3846130371094 │ ], │ "score": 0.9573010802268982 │ }, │ { │ "person": [ │ 539.6248779296875, │ 157.75697326660156, │ 616.1575927734375, │ 253.11447143554688 │ ], │ "score": 0.8995442390441895 │ }, │ { │ "person": [ │ 477.05487060546875, │ 147.80801391601562, │ 610.78662109375, │ 296.78521728515625 │ ], │ "score": 0.8751821517944336 │ }, │ { │ "bench": [ │ 286.079833984375, │ 216.5288848876953, │ 550.7716064453125, │ 383.19091796875 │ ], │ "score": 0.8439943790435791 │ }, │ { │ "person": [ │ 627.474365234375, │ 177.0586395263672, │ 640.0, │ 247.72096252441406 │ ], │ "score": 0.8255492448806763 │ }, │ { │ "bench": [ │ 88.77957916259766, │ 226.16111755371094, │ 563.5453491210938, │ 421.6773 │ (showing 2.05kB/2.75kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 210µs (cache) (cache) 1506ms 1ms 33µs 1509ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fastrcnn 200 OK ★ 35ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"fastrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 185µs (cache) (cache) 33ms 814µs 27µs 35ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 6.37kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.08kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_versio │ n="default",} 115.451 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ # HELP ts_inference_latency_micr │ (showing 2.05kB/6.08kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 893µs 154µs (cache) (cache) 1ms 849µs 24µs 3ms ✓ Successful GET request Iteration 15/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=maskrcnn.mar&model_name=maskrcnn&initial_workers=1&synchronous=true 200 OK ★ 6.4s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"maskrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 176µs (cache) (cache) 6.4s 1ms 24µs 6.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/maskrcnn 200 OK ★ 1533ms time ★ 289.21kB↑ 2.99kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.76kB │ [ │ { │ "person": [ │ 169.51451110839844, │ 49.975894927978516, │ 300.8909606933594, │ 442.4670104980469 │ ], │ "score": 0.9995430707931519 │ }, │ { │ "person": [ │ 90.41272735595703, │ 66.78936004638672, │ 194.19186401367188, │ 437.27520751953125 │ ], │ "score": 0.9993950128555298 │ }, │ { │ "person": [ │ 362.3165283203125, │ 158.17308044433594, │ 521.2577514648438, │ 385.7251281738281 │ ], │ "score": 0.9952288866043091 │ }, │ { │ "handbag": [ │ 68.57566833496094, │ 279.3168029785156, │ 111.15178680419922, │ 400.9194641113281 │ ], │ "score": 0.9938732385635376 │ }, │ { │ "person": [ │ 474.0313415527344, │ 147.354736328125, │ 638.128662109375, │ 364.6588134765625 │ ], │ "score": 0.9897533655166626 │ }, │ { │ "handbag": [ │ 225.59214782714844, │ 142.90037536621094, │ 302.4877624511719, │ 230.3253936767578 │ ], │ "score": 0.9891176819801331 │ }, │ { │ "handbag": [ │ 380.2597961425781, │ 259.201904296875, │ 419.536865234375, │ 318.2766418457031 │ ], │ "score": 0.9687877297401428 │ }, │ { │ "bench": [ │ 273.49591064453125, │ 217.4812774658203, │ 441.1485595703125, │ 396.2586975097656 │ ], │ "score": 0.9617788195610046 │ }, │ { │ "person": [ │ 541.2871704101562, │ 156.66062927246094, │ 619.9363403320312, │ 249.44639587402344 │ ], │ "score": 0.8176164031028748 │ }, │ { │ "person": [ │ 362.96575927734375, │ 163.89447021484375, │ 500.7733154296875, │ 293.93084716796875 │ ], │ "score": 0.8017446994781494 │ }, │ { │ "chair": [ │ 455.20831298828125, │ 207.53904724121094, │ 491.0845642089844, │ 274.6470947265625 │ ], │ "score": 0.7759630680084229 │ }, │ { │ "person": [ │ 549.1516723632812, │ 177.4287872314453, │ 640.0, │ 364.5427551269531 │ ], │ "score": 0.7178217768669128 │ }, │ { │ "person": [ │ 626.22998046875, │ 178.65225219726562, │ 640.0, │ 2 │ (showing 2.05kB/2.76kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 165µs (cache) (cache) 1531ms 978µs 35µs 1534ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/maskrcnn 200 OK ★ 38ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"maskrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 919µs 161µs (cache) (cache) 36ms 1ms 28µs 38ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 6.76kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.47kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_versio │ n="default",} 115.451 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="densenet161",model_ver │ sion="default",} 152.792 │ ts_queue_latency_microseconds{uu │ (showing 2.05kB/6.47kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 896µs 516µs (cache) (cache) 1ms 799µs 25µs 3ms ✓ Successful GET request Iteration 16/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fcn_resnet_101.mar&model_name=fcn_resnet_101&initial_workers=1&synchronous=true 200 OK ★ 6.3s time ★ 336B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"fcn_resnet_101\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 869µs 147µs (cache) (cache) 6.3s 1ms 37µs 6.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101 200 OK ★ 1715ms time ★ 289.22kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993860721588135 │ ], │ [ │ 0.0, │ 0.9993860721588135 │ ], │ [ │ 0.0, │ 0.9993860721588135 │ ], │ [ │ 0.0, │ 0.9993860721588135 │ ], │ [ │ 0.0, │ 0.9993869066238403 │ ], │ [ │ 0.0, │ 0.9993859529495239 │ ], │ [ │ 0.0, │ 0.9993817806243896 │ ], │ [ │ 0.0, │ 0.9993745684623718 │ ], │ [ │ 0.0, │ 0.9993647933006287 │ ], │ [ │ 0.0, │ 0.9993521571159363 │ ], │ [ │ 0.0, │ 0.9993371367454529 │ ], │ [ │ 0.0, │ 0.9993196725845337 │ ], │ [ │ 0.0, │ 0.9992942810058594 │ ], │ [ │ 0.0, │ 0.9992616772651672 │ ], │ [ │ 0.0, │ 0.9992257356643677 │ ], │ [ │ 0.0, │ 0.9991874098777771 │ ], │ [ │ 0.0, │ 0.9991459846496582 │ ], │ [ │ 0.0, │ 0.9991014003753662 │ ], │ [ │ 0.0, │ 0.9990529417991638 │ ], │ [ │ 0.0, │ 0.9990009665489197 │ ], │ [ │ 0.0, │ 0.9989688396453857 │ ], │ [ │ 0.0, │ 0.9989578723907471 │ ], │ [ │ 0.0, │ 0.9989460110664368 │ ], │ [ │ 0.0, │ 0.998933732509613 │ ], │ [ │ 0.0, │ 0.9989204406738281 │ ], │ [ │ 0.0, │ 0.9989066123962402 │ ], │ [ │ 0.0, │ 0.9988916516304016 │ ], │ [ │ 0.0, │ 0.9988756775856018 │ ], │ [ │ 0.0, │ 0.9988818764686584 │ ], │ [ │ 0.0, │ 0.9989092350006104 │ ], │ [ │ 0.0, │ 0.9989354014396667 │ ], │ [ │ 0.0, │ 0.9989608526229858 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.999007523059845 │ ], │ [ │ 0.0, │ 0.999029278755188 │ ], │ [ │ 0.0, │ 0.999049723148346 │ ], │ [ │ 0.0, │ 0.9990509152412415 │ ], │ [ │ 0.0, │ 0.9990314245223999 │ ], │ [ │ 0.0, │ 0.9990097284317017 │ ], │ [ │ 0.0, │ 0.9989853501319885 │ ], │ [ │ 0.0, │ 0.9989590644836426 │ ], │ [ │ 0.0, │ 0.998930275440 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 169µs (cache) (cache) 1710ms 3ms 736µs 1717ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101 200 OK ★ 44ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"fcn_resnet_101\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 197µs (cache) (cache) 42ms 1ms 27µs 46ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 7.17kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.88kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_versio │ n="default",} 115.451 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fcn_resnet_101",model_ │ version="default",} 114.732 │ ts_queue_latency_microseconds │ (showing 2.05kB/6.88kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 923µs 154µs (cache) (cache) 1ms 873µs 26µs 3ms ✓ Successful GET request Iteration 17/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/fcn_resnet_101_scripted.mar&model_name=fcn_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 7.4s time ★ 395B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 99B │ { │ "status": "Model \"fcn_resnet_101_scripted\" Version │ : 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 180µs (cache) (cache) 7.4s 1ms 30µs 7.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101_scripted 200 OK ★ 1922ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993860721588135 │ ], │ [ │ 0.0, │ 0.9993860721588135 │ ], │ [ │ 0.0, │ 0.9993860721588135 │ ], │ [ │ 0.0, │ 0.9993860721588135 │ ], │ [ │ 0.0, │ 0.9993869066238403 │ ], │ [ │ 0.0, │ 0.9993859529495239 │ ], │ [ │ 0.0, │ 0.9993817806243896 │ ], │ [ │ 0.0, │ 0.9993745684623718 │ ], │ [ │ 0.0, │ 0.9993647933006287 │ ], │ [ │ 0.0, │ 0.9993521571159363 │ ], │ [ │ 0.0, │ 0.9993371367454529 │ ], │ [ │ 0.0, │ 0.9993196725845337 │ ], │ [ │ 0.0, │ 0.9992942810058594 │ ], │ [ │ 0.0, │ 0.9992616772651672 │ ], │ [ │ 0.0, │ 0.9992257356643677 │ ], │ [ │ 0.0, │ 0.9991874098777771 │ ], │ [ │ 0.0, │ 0.9991459846496582 │ ], │ [ │ 0.0, │ 0.9991014003753662 │ ], │ [ │ 0.0, │ 0.9990529417991638 │ ], │ [ │ 0.0, │ 0.9990009665489197 │ ], │ [ │ 0.0, │ 0.9989688396453857 │ ], │ [ │ 0.0, │ 0.9989578723907471 │ ], │ [ │ 0.0, │ 0.9989460110664368 │ ], │ [ │ 0.0, │ 0.998933732509613 │ ], │ [ │ 0.0, │ 0.9989204406738281 │ ], │ [ │ 0.0, │ 0.9989066123962402 │ ], │ [ │ 0.0, │ 0.9988916516304016 │ ], │ [ │ 0.0, │ 0.9988756775856018 │ ], │ [ │ 0.0, │ 0.9988818764686584 │ ], │ [ │ 0.0, │ 0.9989092350006104 │ ], │ [ │ 0.0, │ 0.9989354014396667 │ ], │ [ │ 0.0, │ 0.9989608526229858 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.999007523059845 │ ], │ [ │ 0.0, │ 0.999029278755188 │ ], │ [ │ 0.0, │ 0.999049723148346 │ ], │ [ │ 0.0, │ 0.9990509152412415 │ ], │ [ │ 0.0, │ 0.9990314245223999 │ ], │ [ │ 0.0, │ 0.9990097284317017 │ ], │ [ │ 0.0, │ 0.9989853501319885 │ ], │ [ │ 0.0, │ 0.9989590644836426 │ ], │ [ │ 0.0, │ 0.998930275440 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 151µs (cache) (cache) 1916ms 4ms 301µs 1923ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101_scripted 200 OK ★ 74ms time ★ 259B↑ 331B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 65B │ { │ "status": "Model \"fcn_resnet_101_scripted\" unregis │ tered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 234µs (cache) (cache) 72ms 1ms 45µs 75ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 7.61kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.32kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 130.722 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_versio │ n="default",} 115.451 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_mic │ (showing 2.05kB/7.32kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 939µs 151µs (cache) (cache) 1ms 875µs 26µs 3ms ✓ Successful GET request Iteration 18/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=deeplabv3_resnet_101_eager.mar&model_name=deeplabv3_resnet_101_eager&initial_workers=1&synchronous=true 200 OK ★ 6.6s time ★ 360B↑ 369B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" Vers │ ion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 170µs (cache) (cache) 6.6s 1ms 31µs 6.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_eager 200 OK ★ 1698ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988777041435242 │ ], │ [ │ 0.0, │ 0.9988777041435242 │ ], │ [ │ 0.0, │ 0.9988777041435242 │ ], │ [ │ 0.0, │ 0.9988777041435242 │ ], │ [ │ 0.0, │ 0.9988682270050049 │ ], │ [ │ 0.0, │ 0.9988452196121216 │ ], │ [ │ 0.0, │ 0.998818576335907 │ ], │ [ │ 0.0, │ 0.9987878203392029 │ ], │ [ │ 0.0, │ 0.9987521171569824 │ ], │ [ │ 0.0, │ 0.9987133741378784 │ ], │ [ │ 0.0, │ 0.9986698627471924 │ ], │ [ │ 0.0, │ 0.9986224174499512 │ ], │ [ │ 0.0, │ 0.9985201954841614 │ ], │ [ │ 0.0, │ 0.9983503818511963 │ ], │ [ │ 0.0, │ 0.9981562495231628 │ ], │ [ │ 0.0, │ 0.9979358911514282 │ ], │ [ │ 0.0, │ 0.9976853132247925 │ ], │ [ │ 0.0, │ 0.9974009990692139 │ ], │ [ │ 0.0, │ 0.9970781803131104 │ ], │ [ │ 0.0, │ 0.9967118501663208 │ ], │ [ │ 0.0, │ 0.9964112639427185 │ ], │ [ │ 0.0, │ 0.9962021708488464 │ ], │ [ │ 0.0, │ 0.9959802627563477 │ ], │ [ │ 0.0, │ 0.9957451224327087 │ ], │ [ │ 0.0, │ 0.9954954385757446 │ ], │ [ │ 0.0, │ 0.9952303767204285 │ ], │ [ │ 0.0, │ 0.9949499368667603 │ ], │ [ │ 0.0, │ 0.9946521520614624 │ ], │ [ │ 0.0, │ 0.9945217370986938 │ ], │ [ │ 0.0, │ 0.9945715069770813 │ ], │ [ │ 0.0, │ 0.9946213960647583 │ ], │ [ │ 0.0, │ 0.9946700930595398 │ ], │ [ │ 0.0, │ 0.9947183728218079 │ ], │ [ │ 0.0, │ 0.9947657585144043 │ ], │ [ │ 0.0, │ 0.9948124885559082 │ ], │ [ │ 0.0, │ 0.994858980178833 │ ], │ [ │ 0.0, │ 0.9949036836624146 │ ], │ [ │ 0.0, │ 0.9949473142623901 │ ], │ [ │ 0.0, │ 0.9949913620948792 │ ], │ [ │ 0.0, │ 0.9950345754623413 │ ], │ [ │ 0.0, │ 0.9950774312019348 │ ], │ [ │ 0.0, │ 0.9951196908 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 156µs (cache) (cache) 1692ms 4ms 259µs 1699ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_eager 200 OK ★ 49ms time ★ 262B↑ 334B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 68B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" unre │ gistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 195µs (cache) (cache) 47ms 998µs 44µs 50ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 8.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.76kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 130.722 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_versio │ n="default",} 115.451 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_mic │ (showing 2.05kB/7.76kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 1ms 1ms 26µs 3ms ✓ Successful GET request Iteration 19/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.s3.amazonaws.com/mar_files/deeplabv3_resnet_101_scripted.mar&model_name=deeplabv3_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 13.5s time ★ 412B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 171µs (cache) (cache) 13.5s 2ms 56µs 13.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_scripted 200 OK ★ 1909ms time ★ 289.24kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988777041435242 │ ], │ [ │ 0.0, │ 0.9988777041435242 │ ], │ [ │ 0.0, │ 0.9988777041435242 │ ], │ [ │ 0.0, │ 0.9988777041435242 │ ], │ [ │ 0.0, │ 0.9988682270050049 │ ], │ [ │ 0.0, │ 0.9988452196121216 │ ], │ [ │ 0.0, │ 0.998818576335907 │ ], │ [ │ 0.0, │ 0.9987878203392029 │ ], │ [ │ 0.0, │ 0.9987521171569824 │ ], │ [ │ 0.0, │ 0.9987133741378784 │ ], │ [ │ 0.0, │ 0.9986698627471924 │ ], │ [ │ 0.0, │ 0.9986224174499512 │ ], │ [ │ 0.0, │ 0.9985201954841614 │ ], │ [ │ 0.0, │ 0.9983503818511963 │ ], │ [ │ 0.0, │ 0.9981562495231628 │ ], │ [ │ 0.0, │ 0.9979358911514282 │ ], │ [ │ 0.0, │ 0.9976853132247925 │ ], │ [ │ 0.0, │ 0.9974009990692139 │ ], │ [ │ 0.0, │ 0.9970781803131104 │ ], │ [ │ 0.0, │ 0.9967118501663208 │ ], │ [ │ 0.0, │ 0.9964112639427185 │ ], │ [ │ 0.0, │ 0.9962021708488464 │ ], │ [ │ 0.0, │ 0.9959802627563477 │ ], │ [ │ 0.0, │ 0.9957451224327087 │ ], │ [ │ 0.0, │ 0.9954954385757446 │ ], │ [ │ 0.0, │ 0.9952303767204285 │ ], │ [ │ 0.0, │ 0.9949499368667603 │ ], │ [ │ 0.0, │ 0.9946521520614624 │ ], │ [ │ 0.0, │ 0.9945217370986938 │ ], │ [ │ 0.0, │ 0.9945715069770813 │ ], │ [ │ 0.0, │ 0.9946213960647583 │ ], │ [ │ 0.0, │ 0.9946700930595398 │ ], │ [ │ 0.0, │ 0.9947183728218079 │ ], │ [ │ 0.0, │ 0.9947657585144043 │ ], │ [ │ 0.0, │ 0.9948124885559082 │ ], │ [ │ 0.0, │ 0.994858980178833 │ ], │ [ │ 0.0, │ 0.9949036836624146 │ ], │ [ │ 0.0, │ 0.9949473142623901 │ ], │ [ │ 0.0, │ 0.9949913620948792 │ ], │ [ │ 0.0, │ 0.9950345754623413 │ ], │ [ │ 0.0, │ 0.9950774312019348 │ ], │ [ │ 0.0, │ 0.9951196908 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 383µs (cache) (cache) 1906ms 2ms 778µs 1915ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_scripted 200 OK ★ 84ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 211µs (cache) (cache) 81ms 965µs 43µs 85ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 8.51kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.22kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 130.722 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_versio │ n="default",} 115.451 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier_scr │ ipted",model_version="default",} 70.691 │ ts_queue_latency_mic │ (showing 2.05kB/8.22kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 895µs 145µs (cache) (cache) 1ms 852µs 27µs 3ms ✓ Successful GET request Iteration 20/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&initial_workers=1&synchronous=true 200 OK ★ 6.7s time ★ 333B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 169µs (cache) (cache) 6.7s 1ms 28µs 6.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet152 200 OK ★ 1414ms time ★ 111.24kB↑ 410B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 175B │ { │ "tiger_cat": 0.5799298286437988, │ "tabby": 0.3833445906639099, │ "Egyptian_cat": 0.034234222024679184, │ "lynx": 0.0005803013336844742, │ "quilt": 0.0002746647223830223 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 171µs (cache) (cache) 1411ms 1ms 126µs 1414ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet152 200 OK ★ 47ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 888µs 149µs (cache) (cache) 45ms 812µs 26µs 47ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 8.9kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.61kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet152",model_versi │ on="default",} 364.445 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 130.722 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_versio │ n="default",} 115.451 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1",model_v │ ersion="default",} 125.962 │ ts_queue_latency_microseconds{uuid="c │ (showing 2.05kB/8.61kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 806µs 154µs (cache) (cache) 2ms 716µs 26µs 3ms ✓ Successful GET request Iteration 21/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-152-batch_scripted.mar&model_name=resnet-152-batch_scripted&initial_workers=1&synchronous=true 200 OK ★ 8.2s time ★ 399B↑ 368B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "status": "Model \"resnet-152-batch_scripted\" Versi │ on: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 148µs (cache) (cache) 8.2s 2ms 48µs 8.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-152-batch_scripted 200 OK ★ 1813ms time ★ 111.26kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tiger_cat": 0.579930305480957, │ "tabby": 0.3833441436290741, │ "Egyptian_cat": 0.03423421457409859, │ "lynx": 0.0005803006351925433, │ "quilt": 0.0002746645186562091 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 1810ms 2ms 57µs 1814ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-152-batch_scripted 200 OK ★ 80ms time ★ 261B↑ 333B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 67B │ { │ "status": "Model \"resnet-152-batch_scripted\" unreg │ istered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 973µs 149µs (cache) (cache) 78ms 1ms 26µs 80ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 9.35kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.05kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet152",model_versi │ on="default",} 364.445 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-152-batch_scrip │ ted",model_version="default",} 120.621 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 130.722 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_versio │ n="default",} 115.451 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microsec │ (showing 2.05kB/9.05kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 923µs 156µs (cache) (cache) 1ms 790µs 29µs 3ms ✓ Successful GET request Iteration 22/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/distill_bert_qa_eager.mar&model_name=distill_bert_qa_eager&initial_workers=1&synchronous=true 200 OK ★ 8.5s time ★ 391B↑ 363B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "status": "Model \"distill_bert_qa_eager\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 994µs 166µs (cache) (cache) 8.5s 1ms 32µs 8.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/distill_bert_qa_eager 200 OK ★ 719ms time ★ 359B↑ 247B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 79B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13B │ a nice puppet └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 158µs (cache) (cache) 716ms 1ms 25µs 720ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/distill_bert_qa_eager 200 OK ★ 98ms time ★ 257B↑ 329B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Model \"distill_bert_qa_eager\" unregiste │ red" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 205µs (cache) (cache) 95ms 1ms 31µs 98ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 9.78kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.48kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet152",model_versi │ on="default",} 364.445 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-152-batch_scrip │ ted",model_version="default",} 120.621 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="distill_bert_qa_eager" │ ,model_version="default",} 136.502 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 130.722 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_versio │ n="default",} 115.451 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16",model_version=" │ default",} 88.342 │ ts_queue_la │ (showing 2.05kB/9.48kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 173µs (cache) (cache) 1ms 947µs 31µs 3ms ✓ Successful GET request Iteration 23/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_token_classification_no_torchscript.mar&model_name=bert_token_classification_no_torchscript&initial_workers=1&synchronous=true 200 OK ★ 11.6s time ★ 429B↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 116B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" Version: 1.0 registered with 1 initial worke │ rs" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 183µs (cache) (cache) 11.6s 1ms 26µs 11.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_token_classification_no_torchscript 200 OK ★ 730ms time ★ 375B↑ 5.45kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.22kB │ [ │ [ │ "[CLS]", │ "B-LOC" │ ], │ [ │ "bloomberg", │ "O" │ ], │ [ │ "has", │ "I-ORG" │ ], │ [ │ "decided", │ "B-PER" │ ], │ [ │ "to", │ "O" │ ], │ [ │ "publish", │ "I-PER" │ ], │ [ │ "a", │ "I-MISC" │ ], │ [ │ "new", │ "B-PER" │ ], │ [ │ "report", │ "O" │ ], │ [ │ "on", │ "O" │ ], │ [ │ "global", │ "B-PER" │ ], │ [ │ "economic", │ "O" │ ], │ [ │ "situation", │ "B-LOC" │ ], │ [ │ ".", │ "B-PER" │ ], │ [ │ "[SEP]", │ "B-PER" │ ], │ [ │ "[PAD]", │ "I-ORG" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ (showing 2.05kB/5.22kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 162µs (cache) (cache) 728ms 1ms 30µs 731ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_token_classification_no_torchscript 200 OK ★ 132ms time ★ 276B↑ 348B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 82B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 159µs (cache) (cache) 129ms 1ms 29µs 132ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 10.26kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.97kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet_scripted",mode │ l_version="default",} 126.462 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="bert_token_classificat │ ion_no_torchscript",model_version="default",} 127.722 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet152",model_versi │ on="default",} 364.445 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-152-batch_scrip │ ted",model_version="default",} 120.621 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="my_text_classifier",mo │ del_version="default",} 715.909 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="distill_bert_qa_eager" │ ,model_version="default",} 136.502 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fcn_resnet_101_scripte │ d",model_version="default",} 130.722 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_ver │ (showing 2.05kB/9.97kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 162µs (cache) (cache) 1ms 828µs 27µs 3ms ✓ Successful GET request Iteration 24/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_seqc_without_torchscript.mar&model_name=bert_seqc_without_torchscript&initial_workers=1&synchronous=true 200 OK ★ 11s time ★ 407B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"bert_seqc_without_torchscript\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 172µs (cache) (cache) 11s 1ms 29µs 11s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_seqc_without_torchscript 200 OK ★ 731ms time ★ 364B↑ 246B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12B │ Not Accepted └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 240µs (cache) (cache) 729ms 1ms 31µs 734ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_seqc_without_torchscript 200 OK ★ 133ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"bert_seqc_without_torchscript\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 890µs 168µs (cache) (cache) 131ms 1ms 27µs 133ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 10.72kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 10.42kB │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18_scripted",mo │ del_version="default",} 120.052 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist",model_version=" │ default",} 121.251 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="vgg16_scripted",model_ │ version="default",} 120.041 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="mnist_scripted",model_ │ version="default",} 92.802 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="bert_token_classificat │ ion_no_torchscript",model_version="default",} 127.722 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet152",model_versi │ on="default",} 364.445 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-152-batch_scrip │ ted",model_version="default",} 120.621 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="squeezenet1_1_scripted │ ",model_version="default",} 110.822 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="distill_bert_qa_eager" │ ,model_version="default",} 136.502 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="resnet-18",model_versi │ on="default",} 123.252 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="fastrcnn",model_versio │ n="default",} 115.451 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="bert_seqc_without_torc │ hscript",model_version="default",} 105.691 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="alexnet",model_version │ ="default",} 112.912 │ ts_queue_latency_microseconds{uuid="c1c72680-9135-4be8 │ -9840-46423a7b5429",model_name="deeplabv3_resnet_101_e │ ager",mode │ (showing 2.05kB/10.42kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 955µs 159µs (cache) (cache) 1ms 853µs 28µs 3ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 24 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 96 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 96 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 113 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 3m 28.4s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 10.05MB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 2.1s [min: 2ms, max: 13.5s, s.d.: 3.2s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 617µs [min: 137µs, max: 1ms, s.d.: 532µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 2.1s [min: 1ms, max: 13.5s, s.d.: 3.2s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman inference → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161_scripted.mar&model_name=densenet161_scripted&initial_workers=1&synchronous=true 200 OK ★ 6.5s time ★ 389B↑ 362B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 96B │ { │ "status": "Model \"densenet161_scripted\" Version: 1 │ .0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 34ms 4ms 1ms 276µs 6.5s 6ms 361µs 6.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161_scripted 200 OK ★ 2.3s time ★ 111.25kB↑ 416B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 181B │ { │ "tabby": 0.4659089148044586, │ "tiger_cat": 0.4652513265609741, │ "Egyptian_cat": 0.06609020382165909, │ "lynx": 0.0012922680471092463, │ "plastic_bag": 0.00022853966220282018 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 4ms 954µs 227µs 376µs 2.3s 2ms 46µs 2.3s ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161_scripted 200 OK ★ 47ms time ★ 256B↑ 328B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 62B │ { │ "status": "Model \"densenet161_scripted\" unregister │ ed" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 320µs (cache) (cache) 45ms 1ms 75µs 48ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 16ms time ★ 233B↑ 1.09kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 802B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="5838d102-cd26-4b97 │ -afd2-6699f5b4737c",model_name="densenet161_scripted", │ model_version="default",} 143.642 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="5838d102-cd26-4b97-a │ fd2-6699f5b4737c",model_name="densenet161_scripted",mo │ del_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="5838d102-cd26- │ 4b97-afd2-6699f5b4737c",model_name="densenet161_script │ ed",model_version="default",} 2292168.008 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 428µs 155µs 119µs 13ms 1ms 55µs 17ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 9.1s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 1.14kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 2.2s [min: 16ms, max: 6.5s, s.d.: 2.6s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 602µs [min: 155µs, max: 1ms, s.d.: 521µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 2.2s [min: 13ms, max: 6.5s, s.d.: 2.6s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 156ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 34ms 6ms 1ms 436µs 109ms 32ms 6ms 344µs 190ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 40ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 934µs 164µs 259µs 23ms 13ms 1ms 58µs 41ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 26ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 726µs 140µs 217µs 19ms 3ms 1ms 55µs 27ms ✓ Status code is 200 → HTTPS Register Model - SqueezeNet POST https://localhost:8444/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 298µs (cache) (cache) (cache) 3.3s 1ms 71µs 3.3s ✓ Successful POST request → HTTPS Get SqueezeNet Model Description GET https://localhost:8444/models/squeezenet1_1 200 OK ★ 22ms time ★ 246B↑ 838B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 571B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-04-08T22:18:16.378Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 542378, │ "gpu": true, │ "gpuUsage": "gpuId::1 utilization.gpu [%]::1 % │ utilization.memory [%]::0 % memory.used [MiB]::807 Mi │ B" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 214µs (cache) (cache) (cache) 19ms 985µs 39µs 22ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 4ms time ★ 308B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 181µs (cache) (cache) (cache) 2ms 1ms 36µs 4ms ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 4ms time ★ 309B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 190µs (cache) (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful PUT request → HTTPS - Inference - SqueezeNet POST https://localhost:8443/predictions/squeezenet1_1 200 OK ★ 1441ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.27828869223594666, │ "lynx": 0.25356364250183105, │ "tiger_cat": 0.2448405772447586, │ "Egyptian_cat": 0.21699286997318268, │ "cougar": 0.002220059046521783 │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 4ms 568µs (cache) (cache) (cache) 1438ms 1ms 33µs 1444ms ✓ Status code is 200 → HTTPS UnRegister Model SqueezeNet DELETE https://localhost:8444/models/squeezenet1_1 200 OK ★ 11ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 169µs (cache) (cache) (cache) 8ms 1ms 25µs 11ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 9 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 5.3s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 85.53kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 558ms [min: 4ms, max: 3.3s, s.d.: 1070ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 462µs [min: 140µs, max: 1ms, s.d.: 438µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 537ms [min: 1ms, max: 3.3s, s.d.: 1078ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 178ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 34ms 4ms 1ms 260µs 164ms 6ms 345µs 212ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 64ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 499µs (cache) (cache) 61ms 1ms 60µs 65ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 1248ms time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 181µs (cache) (cache) 1246ms 1ms 56µs 1248ms ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 4ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 206µs (cache) (cache) 2ms 1ms 42µs 5ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 28ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 490µs 148µs 115µs 25ms 1ms 50µs 28ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 3ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 992µs 161µs (cache) (cache) 1ms 1ms 34µs 3ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 12ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 955µs 383µs 137µs 102µs 9ms 989µs 31µs 12ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 56ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 192µs (cache) (cache) 53ms 1ms 31µs 56ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 4ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 178µs (cache) (cache) 2ms 946µs 26µs 4ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 55ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 143µs (cache) (cache) 53ms 978µs 25µs 55ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 1244ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 152µs (cache) (cache) 1241ms 1ms 28µs 1244ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 12ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 386µs 131µs 96µs 10ms 895µs 27µs 12ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 5ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 970µs 144µs (cache) (cache) 3ms 899µs 27µs 5ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 3ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 940µs 144µs (cache) (cache) 1ms 846µs 31µs 3ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 3ms time ★ 246B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 969µs 145µs (cache) (cache) 1ms 1ms 25µs 3ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 156µs (cache) (cache) 2ms 835µs 25µs 4ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 3ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 139µs (cache) (cache) 1ms 852µs 24µs 3ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 11ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 364µs 120µs 110µs 8ms 847µs 23µs 11ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 57ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 150µs (cache) (cache) 55ms 880µs 25µs 57ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 3ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 929µs 136µs (cache) (cache) 1ms 874µs 22µs 3ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 3ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 924µs 133µs (cache) (cache) 1ms 840µs 21µs 3ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 3ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 148µs (cache) (cache) 1ms 1ms 24µs 4ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 3ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 157µs (cache) (cache) 1ms 893µs 28µs 3ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 10ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 904µs 335µs 113µs 120µs 7ms 851µs 21µs 10ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 3ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 924µs 141µs (cache) (cache) 1ms 803µs 22µs 3ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 3ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 937µs 144µs (cache) (cache) 1ms 804µs 22µs 3ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 10ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 909µs 351µs 114µs 95µs 8ms 805µs 22µs 10ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 14ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 911µs 343µs 115µs 94µs 12ms 788µs 23µs 14ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 56ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 981µs 154µs (cache) (cache) 54ms 840µs 22µs 56ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 3ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 853µs 136µs (cache) (cache) 1ms 741µs 20µs 3ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1268ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 906µs 137µs (cache) (cache) 1266ms 1ms 25µs 1268ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 64ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 368µs 132µs 100µs 62ms 779µs 23µs 64ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 3ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 788µs 132µs (cache) (cache) 1ms 727µs 30µs 3ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 2.5s time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 143µs (cache) (cache) 2.5s 1ms 24µs 2.5s ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 35ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 900µs 154µs (cache) (cache) 33ms 736µs 26µs 35ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 6.7s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 892µs 142µs (cache) (cache) 6.7s 1ms 26µs 6.7s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 44ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 154µs (cache) (cache) 42ms 856µs 23µs 44ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 6.7s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 947µs 150µs (cache) (cache) 6.6s 3ms 51µs 6.7s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 48ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 326µs (cache) (cache) 45ms 1ms 38µs 49ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 57ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 212µs (cache) (cache) 55ms 1ms 30µs 58ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 151µs (cache) (cache) 1ms 1ms 27µs 4ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 498ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 145µs (cache) (cache) 496ms 1ms 22µs 498ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 9ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 919µs 134µs (cache) (cache) 7ms 736µs 21µs 9ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 513ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 984µs 146µs (cache) (cache) 510ms 1ms 32µs 513ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 10ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 130µs (cache) (cache) 8ms 764µs 21µs 10ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 499ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 135µs (cache) (cache) 497ms 1ms 28µs 500ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 2ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 924µs 135µs (cache) (cache) 867µs 758µs 22µs 2ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 2ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 806µs 122µs (cache) (cache) 663µs 699µs 20µs 2ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 2ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 862µs 125µs (cache) (cache) 557µs 744µs 26µs 2ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 2ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 870µs 127µs (cache) (cache) 678µs 758µs 25µs 2ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 2ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 921µs 137µs (cache) (cache) 785µs 990µs 24µs 2ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 2ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 816µs 127µs (cache) (cache) 667µs 760µs 25µs 2ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 2ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 858µs 140µs (cache) (cache) 746µs 747µs 20µs 2ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 2ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 941µs 134µs (cache) (cache) 639µs 1ms 22µs 2ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 3ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 855µs 546µs (cache) (cache) 713µs 746µs 21µs 2ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 2ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 139µs (cache) (cache) 730µs 737µs 20µs 2ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 2ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 809µs 128µs (cache) (cache) 662µs 729µs 20µs 2ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 605ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 969µs 411µs 159µs 99µs 602ms 1ms 29µs 605ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 9ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 865µs 309µs 107µs 107µs 6ms 740µs 21µs 9ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 8ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 849µs 296µs 105µs 89µs 6ms 742µs 21µs 8ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 9ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 768µs 277µs 97µs 88µs 7ms 696µs 20µs 9ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 3ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 818µs 129µs (cache) (cache) 779µs 1ms 22µs 3ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 2ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 860µs 133µs (cache) (cache) 759µs 800µs 23µs 2ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 3ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 844µs 689µs (cache) (cache) 934µs 795µs 21µs 3ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 2ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 824µs 132µs (cache) (cache) 657µs 820µs 21µs 2ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 14ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 310µs 108µs 93µs 12ms 799µs 21µs 16ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 8ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 780µs 284µs 97µs 83µs 6ms 718µs 21µs 8ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 5ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 791µs 146µs (cache) (cache) 4ms 695µs 23µs 5ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 3ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 815µs 129µs (cache) (cache) 1ms 757µs 21µs 3ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 9ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 876µs 307µs 105µs 88µs 7ms 725µs 21µs 9ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 8ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 748µs 277µs 99µs 81µs 6ms 726µs 20µs 8ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 16ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 797µs 316µs 105µs 78µs 13ms 730µs 20µs 15ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 2ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 883µs 161µs (cache) (cache) 765µs 738µs 22µs 2ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 91ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 799µs 303µs 104µs 89µs 88ms 1ms 27µs 91ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 3ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 160µs (cache) (cache) 1ms 929µs 25µs 3ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 57ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 159µs (cache) (cache) 55ms 1ms 27µs 57ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 3ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 871µs 153µs (cache) (cache) 1ms 753µs 21µs 3ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 55ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 791µs 123µs (cache) (cache) 53ms 723µs 20µs 55ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 3ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 783µs 118µs (cache) (cache) 1ms 700µs 21µs 3ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 56ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 831µs 127µs (cache) (cache) 54ms 730µs 23µs 55ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 5ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 801µs 134µs (cache) (cache) 1ms 2ms 22µs 5ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 2ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 802µs 125µs (cache) (cache) 642µs 687µs 20µs 2ms ✓ Successful request ┌─────────────────────────┬──────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴──────────────────────┴─────────────────────┤ │ total run duration: 24.5s │ ├──────────────────────────────────────────────────────────────────────┤ │ total data received: 7.75kB (approx) │ ├──────────────────────────────────────────────────────────────────────┤ │ average response time: 282ms [min: 2ms, max: 6.7s, s.d.: 1083ms] │ ├──────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 644µs [min: 97µs, max: 1ms, s.d.: 586µs] │ ├──────────────────────────────────────────────────────────────────────┤ │ average first byte time: 280ms [min: 557µs, max: 6.7s, s.d.: 1083ms] │ └──────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. newman kf_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 33ms 4ms 1ms 248µs 3.3s 6ms 341µs 3.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v1/models/mnist:predict 200 OK ★ 1415ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 4ms 850µs 446µs 173µs 1411ms 1ms 53µs 1419ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v1/models/mnist:explain 200 OK ★ 26ms time ★ 680B↑ 25.38kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.14kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.004570419130158362, │ 0.006216780615071686, │ 0.008196404206855249, │ 0.009563578320155966, │ 0.008998479032243894, │ 0.00967338117423429, │ 0.007599394053029064, │ 0.0063618084308192215, │ 0.005766252899684984, │ 0.004395602568065742, │ 0.004947496413012757, │ 0.00527219806774403, │ 0.005524940336685202, │ 0.0077881446250853215, │ 0.008759199464115405, │ 0.004304839044419332, │ 0.010970190416928156, │ 0.003249420324929019, │ 0.0059979099904359885, │ 0.003753624604982373, │ 0.002764799745252722, │ 0.004313224317694643, │ 0.0014007422387716276, │ 0.004841376472574037, │ 0.0006375842180235781, │ 0.0018559948237162954, │ -0.0008284260212862827, │ -0.0 │ ], │ [ │ 0.0016625761608440215, │ 0.0044438755909099945, │ 0.012388350904779515, │ 0.009449814532132424, │ 0.016143537886730314, │ 0.007797077730342893, │ 0.013943222894357057, │ 0.007555637697001038, │ 0.005477962658619858, │ 0.009750243081394357, │ 0.004767057641995692, │ 0.007289728591435181, │ 0.011794867286205343, │ 0.00646216622595877, │ 0.0033806750517516113, │ 0.0092231585183747, │ 0.0016746844630073662, │ 0.0073633934516542715, │ 0.005263437134210535, │ 0.005455041444154606, │ 0.0043371227203374, │ 0.005861930400923258, │ 0.012167524980509683, │ 0.009902460388844854, │ 0.009812529272275475, │ 0.0034418936980847707, │ 0.0022930087164541296, │ -0.0 │ ], │ [ │ 0.011527708055682444, │ 0.009147803403534266, │ 0.022263724457513977, │ 0.017559268568186133, │ 0.024770601433977135, │ 0.021413098406409974, │ │ (showing 2.05kB/25.14kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 314µs (cache) (cache) 23ms 1ms 88µs 27ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 9ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 7ms 1ms 42µs 9ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 15ms time ★ 233B↑ 1.07kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 775B │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="d07f589f-b746- │ 4e98-b84b-281facc7a7e1",model_name="mnist",model_versi │ on="default",} 1417651.8669999999 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="d07f589f-b746-4e98 │ -b84b-281facc7a7e1",model_name="mnist",model_version=" │ default",} 200.18200000000002 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="d07f589f-b746-4e98-b │ 84b-281facc7a7e1",model_name="mnist",model_version="de │ fault",} 2.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 445µs 143µs 110µs 13ms 1ms 51µs 15ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 5s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 26.08kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 970ms [min: 9ms, max: 3.3s, s.d.: 1324ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 725µs [min: 143µs, max: 1ms, s.d.: 478µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 966ms [min: 7ms, max: 3.3s, s.d.: 1320ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman kf_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 155ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 33ms 6ms 1ms 418µs 108ms 31ms 6ms 393µs 188ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 40ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 974µs 164µs 245µs 22ms 13ms 1ms 60µs 41ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 26ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 748µs 144µs 225µs 19ms 3ms 1ms 57µs 26ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 294µs (cache) (cache) (cache) 3.2s 1ms 71µs 3.2s ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 22ms time ★ 238B↑ 822B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 555B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-04-08T22:19:54.422Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 543369, │ "gpu": true, │ "gpuUsage": "gpuId::1 utilization.gpu [%]::2 % │ utilization.memory [%]::0 % memory.used [MiB]::805 Mi │ B" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 219µs (cache) (cache) (cache) 19ms 1ms 47µs 22ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 3.6s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 231µs (cache) (cache) (cache) 3.6s 1ms 40µs 3.6s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 4ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 206µs (cache) (cache) (cache) 2ms 1ms 37µs 5ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v1/models/mnist:predict 200 OK ★ 1400ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 4ms 489µs (cache) (cache) (cache) 1397ms 1ms 34µs 1404ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v1/models/mnist:explain 200 OK ★ 1444ms time ★ 680B↑ 25.39kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.15kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.004570419130158362, │ 0.006216780615071686, │ 0.008196404206855249, │ 0.009563578320155966, │ 0.008998479032243894, │ 0.00967338117423429, │ 0.007599394053029064, │ 0.0063618084308192215, │ 0.005766252899684984, │ 0.004395602568065742, │ 0.004947496413012757, │ 0.00527219806774403, │ 0.005524940336685202, │ 0.0077881446250853215, │ 0.008759199464115405, │ 0.004304839044419332, │ 0.010970190416928156, │ 0.003249420324929019, │ 0.0059979099904359885, │ 0.003753624604982373, │ 0.002764799745252722, │ 0.004313224317694643, │ 0.0014007422387716276, │ 0.004841376472574037, │ 0.0006375842180235781, │ 0.0018559948237162954, │ -0.0008284260212862827, │ -0.0 │ ], │ [ │ 0.0016625761608440215, │ 0.0044438755909099945, │ 0.012388350904779515, │ 0.009449814532132424, │ 0.016143537886730314, │ 0.007797077730342893, │ 0.013943222894357057, │ 0.007555637697001038, │ 0.005477962567054438, │ 0.009750242953196795, │ 0.004767057715544014, │ 0.0072897288564200665, │ 0.01179486700772972, │ 0.006462166512591492, │ 0.0033806750517516113, │ 0.009223158516076312, │ 0.001674684422786864, │ 0.007363393211610795, │ 0.005263437134210535, │ 0.005455041444154606, │ 0.004337122780952846, │ 0.005861930402646429, │ 0.01216752493190983, │ 0.009902460549904842, │ 0.009812529418631278, │ 0.003441893697216016, │ 0.0022930087164541296, │ -0.0 │ ], │ [ │ 0.011527708055682444, │ 0.009147803403534266, │ 0.022263724460989003, │ 0.017559268599352324, │ 0.024770601531770072, │ 0.02141309806292099, │ │ (showing 2.05kB/25.15kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 202µs (cache) (cache) (cache) 1441ms 1ms 32µs 1445ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 32ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 187µs (cache) (cache) (cache) 30ms 1ms 27µs 32ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 10.3s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 110.5kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 1004ms [min: 4ms, max: 3.6s, s.d.: 1342ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 478µs [min: 144µs, max: 1ms, s.d.: 458µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 986ms [min: 2ms, max: 3.6s, s.d.: 1353ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman kfv2_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 33ms 4ms 1ms 264µs 3.3s 6ms 336µs 3.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v2/models/mnist/infer 200 OK ★ 1409ms time ★ 7kB↑ 473B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 238B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "predict", │ "shape": [], │ "datatype": "INT64", │ "data": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 844µs 457µs 172µs 1405ms 1ms 53µs 1414ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v2/models/mnist/explain 200 OK ★ 25ms time ★ 7kB↑ 14.05kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.82kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "explain", │ "shape": [ │ 1, │ 28, │ 28 │ ], │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004049064320929253, │ -0.0002261480408859747, │ -0.00012589426107441167, │ 0.005639241449476159, │ │ (showing 2.05kB/13.82kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 324µs (cache) (cache) 23ms 1ms 81µs 26ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 10ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 237µs (cache) (cache) 8ms 1ms 38µs 10ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 17ms time ★ 233B↑ 1.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 757B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="16c1d4e7-61c0-4403 │ -82fe-e61eeabf765b",model_name="mnist",model_version=" │ default",} 339.895 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="16c1d4e7-61c0- │ 4403-82fe-e61eeabf765b",model_name="mnist",model_versi │ on="default",} 1411189.164 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="16c1d4e7-61c0-4403-8 │ 2fe-e61eeabf765b",model_name="mnist",model_version="de │ fault",} 2.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 415µs 146µs 111µs 14ms 969µs 47µs 17ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 5s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 14.94kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 971ms [min: 10ms, max: 3.3s, s.d.: 1325ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 726µs [min: 146µs, max: 1ms, s.d.: 469µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 966ms [min: 8ms, max: 3.3s, s.d.: 1321ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman kfv2_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 154ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 34ms 6ms 1ms 450µs 106ms 32ms 6ms 363µs 188ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 40ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 979µs 165µs 229µs 22ms 13ms 1ms 58µs 40ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 27ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.1" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 782µs 142µs 223µs 20ms 3ms 1ms 57µs 28ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 312µs (cache) (cache) (cache) 3.2s 2ms 104µs 3.2s ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 22ms time ★ 238B↑ 822B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 555B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-04-08T22:20:51.875Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 543992, │ "gpu": true, │ "gpuUsage": "gpuId::1 utilization.gpu [%]::4 % │ utilization.memory [%]::0 % memory.used [MiB]::805 Mi │ B" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 247µs (cache) (cache) (cache) 19ms 1ms 41µs 22ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 3.6s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 190µs (cache) (cache) (cache) 3.6s 1ms 40µs 3.6s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 5ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 222µs (cache) (cache) (cache) 3ms 939µs 30µs 6ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v2/models/mnist/infer 200 OK ★ 1397ms time ★ 7kB↑ 473B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 238B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "predict", │ "shape": [], │ "datatype": "INT64", │ "data": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 4ms 492µs (cache) (cache) (cache) 1393ms 3ms 66µs 1401ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v2/models/mnist/explain 200 OK ★ 1422ms time ★ 7kB↑ 14.07kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.83kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "explain", │ "shape": [ │ 1, │ 28, │ 28 │ ], │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004049064267555364, │ -0.00022614803670985772, │ -0.000125894311062721, │ 0.005639241449476159, │ │ (showing 2.05kB/13.83kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 313µs (cache) (cache) (cache) 1418ms 3ms 59µs 1424ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 36ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 199µs (cache) (cache) (cache) 34ms 1ms 31µs 37ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 10.3s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 99.38kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 1001ms [min: 5ms, max: 3.6s, s.d.: 1338ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 464µs [min: 142µs, max: 1ms, s.d.: 439µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 982ms [min: 3ms, max: 3.6s, s.d.: 1349ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kf.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kfv2.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe newman explanation_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 34ms 4ms 1ms 250µs 3.3s 6ms 336µs 3.4s ✓ Successful POST request → Model Zoo - Explanations Model POST http://localhost:8080/explanations/mnist 200 OK ★ 1420ms time ★ 538B↑ 23.22kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 22.98kB │ [ │ [ │ [ │ -0.0003991785063704808, │ -0.00019005183649499594, │ -0.0008596008383216306, │ -0.00032935710164163295, │ -0.0009113792305722923, │ -0.00017813154469996625, │ -0.0005800273908895458, │ -5.752862264050362e-05, │ -0.00013031554195310177, │ -3.6185186686619105e-05, │ 9.859345235645867e-06, │ -3.1322990007005414e-05, │ -7.260528366964536e-05, │ -6.938369459889667e-05, │ -7.912179499369329e-05, │ -0.0001327206915316399, │ -3.935731983227288e-05, │ -0.0002292446729944469, │ 1.714529421385004e-05, │ -0.00038464913679301467, │ -0.0005093983196437234, │ -0.00039886684858325314, │ -0.0008457953571782339, │ -0.0005034886785064641, │ -0.0006029295784119773, │ -0.00017683063708337623, │ -0.0001533743208473118, │ -0.0 │ ], │ [ │ -0.000459345527882033, │ -0.0006971699571051699, │ -0.0006564659070067933, │ -0.0006830931626540645, │ -0.0004380970660165447, │ -0.0006187712079583349, │ 8.688311655085769e-06, │ -0.000393034352301958, │ -0.00010260755677233997, │ -0.00018228958683106633, │ -0.0005478749752902416, │ 4.2834343703625867e-05, │ -0.0004340522576048089, │ -0.00032755057068926713, │ -0.00038378314053139317, │ -0.000728927640255037, │ -0.00038079880061969436, │ -0.0006797716575494392, │ -0.0009409081957694532, │ -0.0005321481266328637, │ -0.0010959409145787084, │ -0.0009239120658899724, │ -0.0007883704046475438, │ -0.0007633460024542947, │ -0.0003336904940238334, │ 5.105810925255858e-05, │ 7.698870752143505e-05, │ -0.0 │ ], │ [ │ -0.0013163357735564549, │ -0.0009519457941349145, │ -0.0020998483598478716, │ -0.0007985290053798306, │ -0.0015433587493700013, │ -0.000868489678573524, │ -0.00066677869462317, │ -0.001015588196581527, │ -0.0006894860757221408, │ -0.0010897637345519537, │ -0.0009198684732811271, │ -0.0010171450952950508, │ │ (showing 2.05kB/22.98kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 4ms 874µs 230µs 130µs 1415ms 2ms 56µs 1423ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 9ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 376µs (cache) (cache) 7ms 1ms 65µs 10ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 15ms time ★ 233B↑ 1.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 757B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="c855e4b0-fd1c-42fd │ -af1e-6f423be1194b",model_name="mnist",model_version=" │ default",} 140.381 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="c855e4b0-fd1c- │ 42fd-af1e-6f423be1194b",model_name="mnist",model_versi │ on="default",} 1399328.432 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="c855e4b0-fd1c-42fd-a │ f1e-6f423be1194b",model_name="mnist",model_version="de │ fault",} 1.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 395µs 140µs 109µs 12ms 1ms 49µs 16ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 5s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 23.87kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 1208ms [min: 9ms, max: 3.3s, s.d.: 1384ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 599µs [min: 140µs, max: 1ms, s.d.: 524µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1202ms [min: 7ms, max: 3.3s, s.d.: 1379ms] │ └─────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman management_api_collection Iteration 1/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 200 OK ★ 7.2s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 33ms 4ms 1ms 244µs 7.2s 6ms 350µs 7.3s ✓ Successful request Iteration 2/11 → workflow management request GET http://localhost:8081/workflows 200 OK ★ 7ms time ★ 235B↑ 423B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 156B │ { │ "workflows": [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws. │ com/war_files/densenet_wf.war" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 465µs (cache) (cache) 4ms 1ms 55µs 7ms ✓ Successful request Iteration 3/11 → workflow management request GET http://localhost:8081/workflows/densenet 200 OK ★ 4ms time ★ 244B↑ 559B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 292B │ [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws.co │ m/war_files/densenet_wf.war", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 50, │ "workflowDag": "{pre_processing=[densenet], densen │ et=[post_processing]}" │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 176µs (cache) (cache) 2ms 1ms 53µs 5ms ✓ Successful request Iteration 4/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 500 Internal Server Error ★ 5ms time ★ 321B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "code": 500, │ "type": "FileAlreadyExistsException", │ "message": "densenet_wf.war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 168µs (cache) (cache) 2ms 1ms 40µs 5ms ✓ Successful request Iteration 5/11 → workflow management request DELETE http://localhost:8081/workflows/densenet 200 OK ★ 55ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 428µs 146µs 106µs 52ms 1ms 54µs 58ms ✓ Successful request Iteration 6/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/does_not_exist.war 400 Bad Request ★ 274ms time ★ 324B↑ 441B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 170B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.s3.amazonaws.com/war_files/does_not_exist. │ war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 172µs (cache) (cache) 272ms 1ms 35µs 275ms ✓ Successful request Iteration 7/11 → workflow management request GET http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 11ms time ★ 250B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 394µs 138µs 104µs 8ms 981µs 36µs 11ms ✓ Successful request Iteration 8/11 → workflow management request DELETE http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 10ms time ★ 253B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 969µs 382µs 117µs 102µs 8ms 931µs 27µs 10ms ✓ Successful request Iteration 9/11 → workflow management request POST http://localhost:8081/workflows?url=malformed_url,? 404 Not Found ★ 10ms time ★ 275B↑ 396B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 127B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found in workflow store: ma │ lformed_url,?" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 986µs 340µs 119µs 100µs 7ms 930µs 27µs 10ms ✓ Successful request Iteration 10/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/custom_python_dep.war 500 Internal Server Error ★ 1510ms time ★ 327B↑ 531B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 250B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow custom_python_dep has failed to │ register. Failures: [Workflow Node custom_python_dep_ │ _custom_python_dep failed to register. Details: Model │ not found at: custom_python_dep.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 336µs 109µs 88µs 1507ms 1ms 28µs 1510ms ✓ Successful request Iteration 11/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/loading-memory-error.war 500 Internal Server Error ★ 1288ms time ★ 330B↑ 543B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 262B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow loading-memory-error has failed │ to register. Failures: [Workflow Node loading-memory- │ error__loading-memory-error failed to register. Detail │ s: Model not found at: loading-memory-error.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 441µs 125µs 97µs 1285ms 1ms 42µs 1288ms ✓ Successful request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 11 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 10.7s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 1.7kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 949ms [min: 4ms, max: 7.2s, s.d.: 2s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 620µs [min: 109µs, max: 1ms, s.d.: 561µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 946ms [min: 2ms, max: 7.2s, s.d.: 2s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman workflow_inference Iteration 1/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 200 OK ★ 6.9s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 33ms 4ms 1ms 250µs 6.9s 6ms 363µs 6.9s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/densenet 200 OK ★ 2.9s time ★ 111.24kB↑ 448B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 181B │ { │ "tabby": 0.4659089148044586, │ "tiger_cat": 0.4652513265609741, │ "Egyptian_cat": 0.06609020382165909, │ "lynx": 0.0012922680471092463, │ "plastic_bag": 0.00022853966220282018 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 4ms 919µs 225µs 302µs 2.9s 1ms 53µs 2.9s ✓ Successful request ✓ Test expected JSON response → Workflow - Unregister DELETE http://localhost:8081/workflows/densenet 200 OK ★ 59ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 345µs (cache) (cache) 56ms 1ms 85µs 59ms ✓ Successful DELETE request Iteration 2/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/timeout-retry-test.war 200 OK ★ 6.3s time ★ 328B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 156µs (cache) (cache) 6.3s 1ms 40µs 6.3s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/densenet 500 Internal Server Error ★ 12s time ★ 111.24kB↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Error executing pre_processing" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 264µs (cache) (cache) 12s 3ms 74µs 12s ✓ Successful request → Workflow - Unregister DELETE http://localhost:8081/workflows/densenet 200 OK ★ 50ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 202µs (cache) (cache) 46ms 2ms 41µs 50ms ✓ Successful DELETE request Iteration 3/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/dog_breed_wf.war 200 OK ★ 8.2s time ★ 322B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Workflow dog_breed_wf has been registered │ and scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 211µs (cache) (cache) 8.2s 1ms 35µs 8.2s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/dog_breed_wf 200 OK ★ 3s time ★ 77.75kB↑ 472B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 77.47kB │ (showing 2.05kB/77.47kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 205B │ { │ "Cardigan_welsh_corgi": 0.6288245916366577, │ "Briard": 0.2740449011325836, │ "Icelandic_sheepdog": 0.07451803982257843, │ "Tibetan_mastiff": 0.015528509393334389, │ "Chow_chow": 0.002058096695691347 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 523µs 249µs 112µs 3s 2ms 54µs 3s ✓ Successful request ✓ Test expected JSON response → Workflow - Unregister DELETE http://localhost:8081/workflows/dog_breed_wf 200 OK ★ 51ms time ★ 251B↑ 323B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 57B │ { │ "status": "Workflow \"dog_breed_wf\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 195µs (cache) (cache) 49ms 1ms 34µs 52ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 3 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 11 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 39.9s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 898B (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 4.4s [min: 50ms, max: 12s, s.d.: 3.9s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 694µs [min: 225µs, max: 1ms, s.d.: 526µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 4.4s [min: 46ms, max: 12s, s.d.: 3.9s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. ============================= test session starts ============================== platform linux -- Python 3.8.16, pytest-7.3.0, pluggy-1.0.0 -- /opt/conda/envs/py38/bin/python cachedir: .pytest_cache rootdir: /home/ubuntu/serve plugins: mock-3.10.0, cov-4.0.0 collecting ... collected 90 items test_example_dcgan.py::test_model_archive_creation PASSED [ 1%] test_example_dcgan.py::test_model_register_unregister PASSED [ 2%] test_example_dcgan.py::test_image_generation_without_any_input_constraints PASSED [ 3%] test_example_dcgan.py::test_image_generation_with_input_constraints PASSED [ 4%] test_example_intel_extension_for_pytorch.py::test_single_worker_affinity SKIPPED [ 5%] test_example_intel_extension_for_pytorch.py::test_multi_worker_affinity SKIPPED [ 6%] test_example_intel_extension_for_pytorch.py::test_worker_scale_up_affinity SKIPPED [ 7%] test_example_intel_extension_for_pytorch.py::test_worker_scale_down_affinity SKIPPED [ 8%] test_example_scriptable_tokenzier.py::test_handler PASSED [ 10%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_sample_text PASSED [ 11%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_empty_string PASSED [ 12%] test_example_scriptable_tokenzier.py::test_inference_with_pretrained_model FAILED [ 13%] test_gRPC_inference_api.py::test_inference_apis PASSED [ 14%] test_gRPC_inference_api.py::test_inference_stream_apis PASSED [ 15%] test_gRPC_management_apis.py::test_management_apis PASSED [ 16%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model PASSED [ 17%] test_handler.py::test_mnist_model_register_using_non_existent_handler_with_nonzero_workers PASSED [ 18%] test_handler.py::test_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 20%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 21%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model PASSED [ 22%] test_handler.py::test_kserve_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 23%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 24%] test_handler.py::test_huggingface_bert_batch_inference PASSED [ 25%] test_handler.py::test_MMF_activity_recognition_model_register_and_inference_on_valid_model SKIPPED [ 26%] test_handler.py::test_huggingface_bert_model_parallel_inference PASSED [ 27%] test_handler.py::test_huggingface_opt_distributed_inference_deepspeed FAILED [ 28%] test_metrics.py::test_logs_created PASSED [ 30%] test_metrics.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 31%] test_metrics.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 32%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 33%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 34%] test_metrics.py::test_log_location_var_snapshot_disabled PASSED [ 35%] test_metrics.py::test_log_location_var_snapshot_enabled PASSED [ 36%] test_metrics.py::test_async_logging PASSED [ 37%] test_metrics.py::test_async_logging_non_boolean PASSED [ 38%] test_metrics.py::test_metrics_location_var_snapshot_disabled PASSED [ 40%] test_metrics.py::test_metrics_location_var_snapshot_enabled PASSED [ 41%] test_metrics.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 42%] test_metrics.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 43%] test_metrics.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 44%] test_metrics.py::test_collect_system_metrics_when_not_disabled PASSED [ 45%] test_metrics.py::test_disable_system_metrics_using_config_properties PASSED [ 46%] test_metrics.py::test_disable_system_metrics_using_environment_variable PASSED [ 47%] test_metrics_kf.py::test_logs_created PASSED [ 48%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 50%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 51%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 52%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 53%] test_metrics_kf.py::test_log_location_var_snapshot_disabled PASSED [ 54%] test_metrics_kf.py::test_log_location_var_snapshot_enabled PASSED [ 55%] test_metrics_kf.py::test_async_logging PASSED [ 56%] test_metrics_kf.py::test_async_logging_non_boolean PASSED [ 57%] test_metrics_kf.py::test_metrics_location_var_snapshot_disabled PASSED [ 58%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled PASSED [ 60%] test_metrics_kf.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 61%] test_metrics_kf.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 62%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 63%] test_model_archiver.py::test_multiple_model_versions_registration PASSED [ 64%] test_model_archiver.py::test_duplicate_model_registration_using_local_url_followed_by_http_url PASSED [ 65%] test_model_archiver.py::test_duplicate_model_registration_using_http_url_followed_by_local_url PASSED [ 66%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_without_force PASSED [ 67%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_with_force PASSED [ 68%] test_model_archiver.py::test_model_archiver_without_handler_flag PASSED [ 70%] test_model_archiver.py::test_model_archiver_without_model_name_flag PASSED [ 71%] test_model_archiver.py::test_model_archiver_without_model_file_flag PASSED [ 72%] test_model_archiver.py::test_model_archiver_without_serialized_flag PASSED [ 73%] test_onnx.py::test_convert_to_onnx SKIPPED (ONNX is not installed) [ 74%] test_onnx.py::test_model_packaging_and_start SKIPPED (ONNX is not in...) [ 75%] test_onnx.py::test_model_start SKIPPED (ONNX is not installed) [ 76%] test_onnx.py::test_inference SKIPPED (ONNX is not installed) [ 77%] test_onnx.py::test_stop SKIPPED (ONNX is not installed) [ 78%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[/home/ubuntu/serve/test/pytest/profiler_utils/resnet_custom.py] PASSED [ 80%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[image_classifier] PASSED [ 81%] test_pytorch_profiler.py::test_profiler_arguments_override[/home/ubuntu/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 82%] test_pytorch_profiler.py::test_batch_input[/home/ubuntu/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 83%] test_sm_mme_requirements.py::test_no_model_loaded PASSED [ 84%] test_sm_mme_requirements.py::test_oom_on_model_load FAILED [ 85%] test_sm_mme_requirements.py::test_oom_on_invoke FAILED [ 86%] test_snapshot.py::test_snapshot_created_on_start_and_stop PASSED [ 87%] test_snapshot.py::test_snapshot_created_on_management_api_invoke PASSED [ 88%] test_snapshot.py::test_start_from_snapshot PASSED [ 90%] test_snapshot.py::test_start_from_latest PASSED [ 91%] test_snapshot.py::test_start_from_read_only_snapshot PASSED [ 92%] test_snapshot.py::test_no_config_snapshots_cli_option PASSED [ 93%] test_snapshot.py::test_start_from_default PASSED [ 94%] test_snapshot.py::test_start_from_non_existing_snapshot PASSED [ 95%] test_snapshot.py::test_torchserve_init_with_non_existent_model_store PASSED [ 96%] test_snapshot.py::test_restart_torchserve_with_last_snapshot_with_model_mar_removed PASSED [ 97%] test_snapshot.py::test_replace_mar_file_with_dummy PASSED [ 98%] test_snapshot.py::test_restart_torchserve_with_one_of_model_mar_removed PASSED [100%] =================================== FAILURES =================================== _____________________ test_inference_with_pretrained_model _____________________ model_store = '/tmp/pytest-of-ubuntu/pytest-0/work_dir1/model_store' test_file = '/home/ubuntu/serve/test/pytest/../../examples/text_classification_with_scriptable_tokenizer/sample_text.txt' torchserve = None def test_inference_with_pretrained_model(model_store, test_file, torchserve): model_name = "scriptable_tokenizer" params = ( ("model_name", model_name), ( "url", "https://bert-mar-file.s3.us-west-2.amazonaws.com/text_classification_with_scriptable_tokenizer/scriptable_tokenizer.mar", ), ("initial_workers", "1"), ) test_utils.reg_resp = test_utils.register_model_with_params(params) # Give test some time for model to be downloaded from S3 bucket for sleep_time in [2, 4, 8, 16, 32, 64]: with open(test_file, "rb") as f: response = requests.post( url=f"http://localhost:8080/predictions/{model_name}", data=f ) if response.status_code == 200: break time.sleep(sleep_time) assert response.status_code == 200 result_entries = json.loads(response.text) assert "Negative" in result_entries assert "Positive" in result_entries > assert float(result_entries["Negative"]) == pytest.approx( 0.0001851904089562595, 1e-3 ) E assert 0.00018543108308222145 == 0.00018519040...2595 ± 1.9e-07 E comparison failed E Obtained: 0.00018543108308222145 E Expected: 0.0001851904089562595 ± 1.9e-07 test_example_scriptable_tokenzier.py:320: AssertionError ----------------------------- Captured stdout call ----------------------------- 2023-04-08T22:27:47,992 [DEBUG] epollEventLoopGroup-3-7 org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model scriptable_tokenizer 2023-04-08T22:27:47,993 [DEBUG] epollEventLoopGroup-3-7 org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model scriptable_tokenizer 2023-04-08T22:27:47,993 [INFO ] epollEventLoopGroup-3-7 org.pytorch.serve.wlm.ModelManager - Model scriptable_tokenizer loaded. 2023-04-08T22:27:47,993 [DEBUG] epollEventLoopGroup-3-7 org.pytorch.serve.wlm.ModelManager - updateModel: scriptable_tokenizer, count: 1 2023-04-08T22:27:47,994 [DEBUG] W-9002-scriptable_tokenizer_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9002, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T22:27:49,159 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9002, pid=545877 2023-04-08T22:27:49,159 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9002 2023-04-08T22:27:49,168 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T22:27:49,168 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_LOG - [PID]545877 2023-04-08T22:27:49,168 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T22:27:49,168 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T22:27:49,169 [DEBUG] W-9002-scriptable_tokenizer_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-scriptable_tokenizer_1.0 State change null -> WORKER_STARTED 2023-04-08T22:27:49,169 [INFO ] W-9002-scriptable_tokenizer_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9002 2023-04-08T22:27:49,170 [INFO ] W-9002-scriptable_tokenizer_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680992869170 2023-04-08T22:27:49,170 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9002. 2023-04-08T22:27:49,181 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_LOG - model_name: scriptable_tokenizer, batchSize: 1 2023-04-08T22:27:51,408 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_LOG - /tmp/models/3bcc5c690935455f8e1120d2eabe6fbf/compile.json is missing. PT 2.0 will not be used 2023-04-08T22:27:51,593 [INFO ] W-9002-scriptable_tokenizer_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2413 2023-04-08T22:27:51,594 [DEBUG] W-9002-scriptable_tokenizer_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-scriptable_tokenizer_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T22:27:51,594 [INFO ] W-9002-scriptable_tokenizer_1.0 TS_METRICS - W-9002-scriptable_tokenizer_1.0.ms:3601|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992871 2023-04-08T22:27:51,594 [INFO ] W-9002-scriptable_tokenizer_1.0 TS_METRICS - WorkerThreadTime.ms:11|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992871 2023-04-08T22:27:51,594 [INFO ] epollEventLoopGroup-3-7 ACCESS_LOG - /127.0.0.1:54864 "POST /models?model_name=scriptable_tokenizer&url=https%3A%2F%2Fbert-mar-file.s3.us-west-2.amazonaws.com%2Ftext_classification_with_scriptable_tokenizer%2Fscriptable_tokenizer.mar&initial_workers=1 HTTP/1.1" 200 33629 2023-04-08T22:27:51,594 [INFO ] epollEventLoopGroup-3-7 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992820 2023-04-08T22:27:51,603 [INFO ] W-9002-scriptable_tokenizer_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd PREDICT to backend at: 1680992871603 2023-04-08T22:27:51,604 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_LOG - Backend received inference at: 1680992871 2023-04-08T22:27:52,332 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,332 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:844.9758834838867|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,332 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:147.6138687133789|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,332 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:14.9|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,332 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:8.342018412367553|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,332 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:1921|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,333 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,333 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,333 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,333 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,333 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:6.379190550634011|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,333 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:1469|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,333 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,333 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,334 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,334 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,334 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:182028.234375|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,334 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:7128.06640625|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,334 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:4.8|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,641 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - result=[METRICS]HandlerTime.Milliseconds:1036.29|#ModelName:scriptable_tokenizer,Level:Model|#hostname:ip-172-31-45-70,1680992872,6dc814aa-a4d0-4917-98f6-095181e7070f, pattern=[METRICS] 2023-04-08T22:27:52,641 [INFO ] W-9002-scriptable_tokenizer_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1037 2023-04-08T22:27:52,641 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:1036.29|#ModelName:scriptable_tokenizer,Level:Model|#hostname:ip-172-31-45-70,requestID:6dc814aa-a4d0-4917-98f6-095181e7070f,timestamp:1680992872 2023-04-08T22:27:52,642 [INFO ] W-9002-scriptable_tokenizer_1.0 ACCESS_LOG - /127.0.0.1:47092 "POST /predictions/scriptable_tokenizer HTTP/1.1" 200 1039 2023-04-08T22:27:52,642 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - result=[METRICS]PredictionTime.Milliseconds:1036.49|#ModelName:scriptable_tokenizer,Level:Model|#hostname:ip-172-31-45-70,1680992872,6dc814aa-a4d0-4917-98f6-095181e7070f, pattern=[METRICS] 2023-04-08T22:27:52,642 [INFO ] W-9002-scriptable_tokenizer_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:1036.49|#ModelName:scriptable_tokenizer,Level:Model|#hostname:ip-172-31-45-70,requestID:6dc814aa-a4d0-4917-98f6-095181e7070f,timestamp:1680992872 2023-04-08T22:27:52,642 [INFO ] W-9002-scriptable_tokenizer_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992820 2023-04-08T22:27:52,642 [DEBUG] W-9002-scriptable_tokenizer_1.0 org.pytorch.serve.job.Job - Waiting time ns: 78131, Backend time ns: 1039291262 2023-04-08T22:27:52,642 [INFO ] W-9002-scriptable_tokenizer_1.0 TS_METRICS - QueueTime.ms:0|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992872 2023-04-08T22:27:52,642 [INFO ] W-9002-scriptable_tokenizer_1.0 TS_METRICS - WorkerThreadTime.ms:2|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680992872 --------------------------- Captured stdout teardown --------------------------- TorchServe has stopped. _____________ test_huggingface_opt_distributed_inference_deepspeed _____________ def test_huggingface_opt_distributed_inference_deepspeed(): TORCHSERVE_URL = "https://torchserve.s3.amazonaws.com/mar_files/opt-ds.tar.gz" BATCH_SIZE = 1 BATCH_DELAY = 10000 # 10 seconds INPUT_TEXT = os.path.join( REPO_ROOT, "examples", "large_models", "deepspeed", "opt", "sample_text.txt" ) number_of_gpus = torch.cuda.device_count() logger.info(f"Number of available GPUs on this machine: {number_of_gpus}") if number_of_gpus > 1: try: with os.popen(f"curl http://localhost:8081/models") as check: logger.debug( f"Check if any model is already registered: {check.read()}" ) params = ( ("model_name", "opt"), ("url", TORCHSERVE_URL), ("initial_workers", "1"), ("batch_size", str(BATCH_SIZE)), ("max_batch_delay", str(BATCH_DELAY)), ) test_utils.start_torchserve(no_config_snapshots=True) test_utils.register_model_with_params(params) with os.popen( f"curl http://127.0.0.1:8080/predictions/opt -T {INPUT_TEXT}" ) as response: response_text = response.read() > assert ( "Today the weather is really nice and I am planning on\n\n\nI am planning on the next day.\n\nI am planning on the next day.\n\nI am planning on the next day.\nI am planning on the next" in response_text ), "Incorrect response from model" E AssertionError: Incorrect response from model E assert 'Today the weather is really nice and I am planning on\n\n\nI am planning on the next day.\n\nI am planning on the next day.\n\nI am planning on the next day.\nI am planning on the next' in '{\n "code": 404,\n "type": "ModelNotFoundException",\n "message": "Model not found: opt"\n}\n' test_handler.py:423: AssertionError ----------------------------- Captured stdout call ----------------------------- 2023-04-08T22:35:39,528 [INFO ] W-9000-Textgeneration_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-Textgeneration_1.0-stderr 2023-04-08T22:35:39,576 [INFO ] epollEventLoopGroup-3-4 ACCESS_LOG - /127.0.0.1:51390 "GET /models HTTP/1.1" 200 1 2023-04-08T22:35:39,577 [INFO ] epollEventLoopGroup-3-4 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993314 TorchServe has stopped. ## Starting gen_mar: /tmp/workspace/model_store/ ## Create symlink for mar files ## /tmp/workspace/model_store//fastrcnn.mar already exists. ## /tmp/workspace/model_store//deeplabv3_resnet_101_eager.mar already exists. ## /tmp/workspace/model_store//alexnet.mar already exists. ## /tmp/workspace/model_store//mnist.mar already exists. ## /tmp/workspace/model_store//resnet-152-batch.mar already exists. ## /tmp/workspace/model_store//densenet161.mar already exists. ## /tmp/workspace/model_store//squeezenet1_1.mar already exists. ## /tmp/workspace/model_store//vgg16.mar already exists. ## /tmp/workspace/model_store//maskrcnn.mar already exists. ## /tmp/workspace/model_store//resnet-18.mar already exists. ## /tmp/workspace/model_store//fcn_resnet_101.mar already exists. ['torchserve', '--start', '--model-store', '/tmp/workspace/model_store/', '--no-config-snapshots'] WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. 2023-04-08T22:35:50,395 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2023-04-08T22:35:50,542 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.7.1 TS Home: /opt/conda/envs/py38/lib/python3.8/site-packages Current directory: /home/ubuntu/serve/test/pytest Temp directory: /tmp Metrics config path: /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml Number of GPUs: 4 Number of CPUs: 48 Max heap size: 30688 M Python executable: /opt/conda/envs/py38/bin/python Config file: N/A Inference address: http://127.0.0.1:8080 Management address: http://127.0.0.1:8081 Metrics address: http://127.0.0.1:8082 Model Store: /tmp/workspace/model_store Initial Models: N/A Log dir: /home/ubuntu/serve/test/pytest/logs Metrics dir: /home/ubuntu/serve/test/pytest/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 4 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Metrics report format: prometheus Enable metrics API: true Disable system metrics: false Workflow Store: /tmp/workspace/model_store Model config: N/A 2023-04-08T22:35:50,547 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2023-04-08T22:35:50,566 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2023-04-08T22:35:50,615 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8080 2023-04-08T22:35:50,615 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2023-04-08T22:35:50,616 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8081 2023-04-08T22:35:50,616 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2023-04-08T22:35:50,617 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082 Model server started. 2023-04-08T22:35:52,124 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,125 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:849.7603454589844|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,125 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:142.82940673828125|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,125 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:14.4|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,126 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:8.342018412367553|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,126 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:1921|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,126 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,126 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,126 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,127 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,127 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,127 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,127 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,128 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,128 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,128 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,128 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:183665.69140625|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,128 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5505.1328125|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:35:52,129 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:3.9|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993352 2023-04-08T22:36:52,098 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,098 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:848.0728340148926|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,099 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:144.51691818237305|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,099 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:14.6|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,099 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:8.342018412367553|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,099 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:1921|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,099 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,100 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,100 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,100 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,100 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,100 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,100 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,101 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,101 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,101 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,101 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:183573.203125|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,101 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5596.3671875|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:36:52,102 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:4.0|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993412 2023-04-08T22:37:04,319 [DEBUG] epollEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model opt 2023-04-08T22:37:04,319 [DEBUG] epollEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model opt 2023-04-08T22:37:04,319 [INFO ] epollEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelManager - Model opt loaded. 2023-04-08T22:37:04,320 [INFO ] epollEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelManager - model opt set minWorkers: 1, maxWorkers: 1 for parallelLevel: 2 2023-04-08T22:37:04,320 [DEBUG] epollEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelManager - updateModel: opt, count: 1 2023-04-08T22:37:04,325 [DEBUG] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [torchrun, --nnodes, 1, --nproc_per_node, 2, --max_restarts, 3, --log_dir, /home/ubuntu/serve/test/pytest/logs/torchelastic_ts, --rdzv_backend, c10d, --rdzv_endpoint, localhost:29500, --rdzv_id, opt_29500, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.29500, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T22:37:05,782 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified. 2023-04-08T22:37:05,783 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs: 2023-04-08T22:37:05,783 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - entrypoint : /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py 2023-04-08T22:37:05,784 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - min_nodes : 1 2023-04-08T22:37:05,784 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - max_nodes : 1 2023-04-08T22:37:05,784 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - nproc_per_node : 2 2023-04-08T22:37:05,784 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - run_id : opt_29500 2023-04-08T22:37:05,784 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - rdzv_backend : c10d 2023-04-08T22:37:05,784 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - rdzv_endpoint : localhost:29500 2023-04-08T22:37:05,784 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - rdzv_configs : {'timeout': 900} 2023-04-08T22:37:05,785 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - max_restarts : 3 2023-04-08T22:37:05,785 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - monitor_interval : 5 2023-04-08T22:37:05,785 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - log_dir : /home/ubuntu/serve/test/pytest/logs/torchelastic_ts 2023-04-08T22:37:05,785 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - metrics_cfg : {} 2023-04-08T22:37:05,792 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - 2023-04-08T22:37:05,792 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /home/ubuntu/serve/test/pytest/logs/torchelastic_ts/opt_29500_6r7qa03n 2023-04-08T22:37:05,793 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python 2023-04-08T22:37:05,793 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group 2023-04-08T22:37:05,820 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: 2023-04-08T22:37:05,821 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - restart_count=0 2023-04-08T22:37:05,821 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - master_addr=ip-172-31-45-70.us-west-2.compute.internal 2023-04-08T22:37:05,821 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - master_port=51423 2023-04-08T22:37:05,821 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - group_rank=0 2023-04-08T22:37:05,821 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - group_world_size=1 2023-04-08T22:37:05,821 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - local_ranks=[0, 1] 2023-04-08T22:37:05,821 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - role_ranks=[0, 1] 2023-04-08T22:37:05,822 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - global_ranks=[0, 1] 2023-04-08T22:37:05,822 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - role_world_sizes=[2, 2] 2023-04-08T22:37:05,822 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - global_world_sizes=[2, 2] 2023-04-08T22:37:05,822 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - 2023-04-08T22:37:05,822 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group 2023-04-08T22:37:05,822 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - INFO:torch.distributed.elastic.agent.server.local_elastic_agent:Environment variable 'TORCHELASTIC_ENABLE_FILE_TIMER' not found. Do not start FileTimerServer. 2023-04-08T22:37:05,822 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /home/ubuntu/serve/test/pytest/logs/torchelastic_ts/opt_29500_6r7qa03n/attempt_0/0/error.json 2023-04-08T22:37:05,823 [WARN ] W-29500-opt_1.0-stderr MODEL_LOG - INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /home/ubuntu/serve/test/pytest/logs/torchelastic_ts/opt_29500_6r7qa03n/attempt_0/1/error.json 2023-04-08T22:37:07,001 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=29500, pid=551886 2023-04-08T22:37:07,002 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.29500 2023-04-08T22:37:07,005 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=29500, pid=551885 2023-04-08T22:37:07,006 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.29500 2023-04-08T22:37:07,012 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T22:37:07,013 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - [PID]551886 2023-04-08T22:37:07,013 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T22:37:07,013 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T22:37:07,014 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T22:37:07,015 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - [PID]551885 2023-04-08T22:37:07,015 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T22:37:07,015 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T22:37:07,016 [DEBUG] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerThread - W-29500-opt_1.0 State change null -> WORKER_STARTED 2023-04-08T22:37:07,018 [INFO ] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.29500 2023-04-08T22:37:07,023 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.29500. 2023-04-08T22:37:07,024 [INFO ] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.29501 2023-04-08T22:37:07,025 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.29501. 2023-04-08T22:37:07,026 [INFO ] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680993427026 2023-04-08T22:37:07,039 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - model_name: opt, batchSize: 1 2023-04-08T22:37:07,049 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - model_name: opt, batchSize: 1 2023-04-08T22:37:07,247 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Backend worker process died. 2023-04-08T22:37:07,247 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2023-04-08T22:37:07,248 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_loader.py", line 100, in load 2023-04-08T22:37:07,248 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2023-04-08T22:37:07,248 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_loader.py", line 145, in _load_handler_file 2023-04-08T22:37:07,248 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2023-04-08T22:37:07,248 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/importlib/__init__.py", line 127, in import_module 2023-04-08T22:37:07,248 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2023-04-08T22:37:07,248 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 29500 Worker disconnected. WORKER_STARTED 2023-04-08T22:37:07,248 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 1014, in _gcd_import 2023-04-08T22:37:07,248 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 991, in _find_and_load 2023-04-08T22:37:07,248 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 975, in _find_and_load_unlocked 2023-04-08T22:37:07,249 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 671, in _load_unlocked 2023-04-08T22:37:07,249 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 843, in exec_module 2023-04-08T22:37:07,249 [DEBUG] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2023-04-08T22:37:07,249 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 219, in _call_with_frames_removed 2023-04-08T22:37:07,249 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/tmp/models/e0dc9df2fe7c481884a53fc788ace69e/custom_handler.py", line 9, in 2023-04-08T22:37:07,249 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - from ts.torch_handler.distributed.base_deepspeed import BaseDeepSpeedHandler 2023-04-08T22:37:07,249 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/site-packages/ts/torch_handler/distributed/base_deepspeed.py", line 5, in 2023-04-08T22:37:07,249 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - from ts.handler_utils.distributed.deepspeed import get_ds_engine 2023-04-08T22:37:07,249 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/site-packages/ts/handler_utils/distributed/deepspeed.py", line 4, in 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - import deepspeed 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - ModuleNotFoundError: No module named 'deepspeed' 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - During handling of the above exception, another exception occurred: 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py", line 244, in 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - worker.run_server() 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py", line 212, in run_server 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2023-04-08T22:37:07,250 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py", line 175, in handle_connection 2023-04-08T22:37:07,251 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2023-04-08T22:37:07,251 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py", line 131, in load_model 2023-04-08T22:37:07,251 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - service = model_loader.load( 2023-04-08T22:37:07,251 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_loader.py", line 102, in load 2023-04-08T22:37:07,251 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - module = self._load_default_handler(handler) 2023-04-08T22:37:07,251 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_loader.py", line 151, in _load_default_handler 2023-04-08T22:37:07,251 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name, "ts.torch_handler") 2023-04-08T22:37:07,251 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "/opt/conda/envs/py38/lib/python3.8/importlib/__init__.py", line 127, in import_module 2023-04-08T22:37:07,251 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2023-04-08T22:37:07,252 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 1014, in _gcd_import 2023-04-08T22:37:07,252 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 991, in _find_and_load 2023-04-08T22:37:07,252 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 961, in _find_and_load_unlocked 2023-04-08T22:37:07,252 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 219, in _call_with_frames_removed 2023-04-08T22:37:07,252 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 1014, in _gcd_import 2023-04-08T22:37:07,252 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 991, in _find_and_load 2023-04-08T22:37:07,252 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - File "", line 973, in _find_and_load_unlocked 2023-04-08T22:37:07,252 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - ModuleNotFoundError: No module named 'ts.torch_handler.custom_handler' 2023-04-08T22:37:07,249 [DEBUG] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died. java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1679) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:435) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:208) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:833) [?:?] 2023-04-08T22:37:07,253 [WARN ] W-29500-opt_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: opt, error: Worker died. 2023-04-08T22:37:07,253 [DEBUG] W-29500-opt_1.0 org.pytorch.serve.wlm.ModelVersionedRefs - Removed model: opt version: 1.0 2023-04-08T22:37:07,253 [DEBUG] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerThread - W-29500-opt_1.0 State change WORKER_STARTED -> WORKER_SCALED_DOWN 2023-04-08T22:37:07,253 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 29500 Worker disconnected. WORKER_STARTED 2023-04-08T22:37:07,253 [WARN ] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-29500-opt_1.0-stderr 2023-04-08T22:37:07,253 [WARN ] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-29500-opt_1.0-stdout 2023-04-08T22:37:07,259 [INFO ] W-29500-opt_1.0-stdout MODEL_LOG - Backend worker process died. 2023-04-08T22:37:07,259 [INFO ] W-29500-opt_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-29500-opt_1.0-stdout 2023-04-08T22:37:07,487 [INFO ] W-29500-opt_1.0 org.pytorch.serve.wlm.ModelManager - Model opt unregistered. 2023-04-08T22:37:07,519 [INFO ] W-29500-opt_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-29500-opt_1.0-stderr 2023-04-08T22:37:07,686 [DEBUG] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerThread - W-29500-opt_1.0 State change WORKER_SCALED_DOWN -> WORKER_STOPPED 2023-04-08T22:37:07,686 [WARN ] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-29500-opt_1.0-stderr 2023-04-08T22:37:07,686 [WARN ] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-29500-opt_1.0-stdout 2023-04-08T22:37:07,686 [DEBUG] W-29500-opt_1.0 org.pytorch.serve.wlm.WorkerThread - Worker terminated due to scale-down call. 2023-04-08T22:37:07,690 [INFO ] epollEventLoopGroup-3-1 ACCESS_LOG - /127.0.0.1:36676 "POST /models?model_name=opt&url=https%3A%2F%2Ftorchserve.s3.amazonaws.com%2Fmar_files%2Fopt-ds.tar.gz&initial_workers=1&batch_size=1&max_batch_delay=10000 HTTP/1.1" 500 67765 2023-04-08T22:37:07,691 [INFO ] epollEventLoopGroup-3-1 TS_METRICS - Requests5XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993359 2023-04-08T22:37:07,807 [INFO ] epollEventLoopGroup-3-2 ACCESS_LOG - /127.0.0.1:51766 "PUT /predictions/opt HTTP/1.1" 404 4 2023-04-08T22:37:07,807 [INFO ] epollEventLoopGroup-3-2 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993359 2023-04-08T22:37:07,819 [WARN ] epollEventLoopGroup-3-3 org.pytorch.serve.wlm.ModelManager - Model not found: opt 2023-04-08T22:37:07,820 [INFO ] epollEventLoopGroup-3-3 ACCESS_LOG - /127.0.0.1:33672 "DELETE /models/opt HTTP/1.1" 404 1 2023-04-08T22:37:07,820 [INFO ] epollEventLoopGroup-3-3 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680993359 ----------------------------- Captured stderr call ----------------------------- % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 19 100 19 0 0 1900 0 --:--:-- --:--:-- --:--:-- 1900 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 145 100 91 100 54 5687 3375 --:--:-- --:--:-- --:--:-- 9062 --------------------------- Captured stdout teardown --------------------------- TorchServe has stopped. ____________________________ test_oom_on_model_load ____________________________ @pytest.mark.skipif( not ((torch.cuda.device_count() > 0) and torch.cuda.is_available()), reason="Test to be run on GPU only", ) def test_oom_on_model_load(): """ Validates that TorchServe returns reponse code 507 if there is OOM on model loading. """ # Create model store directory pathlib.Path(test_utils.MODEL_STORE).mkdir(parents=True, exist_ok=True) # Start TorchServe test_utils.start_torchserve(no_config_snapshots=True) # Register model params = { "model_name": "BERTSeqClassification", "url": "https://torchserve.pytorch.org/mar_files/BERTSeqClassification.mar", "batch_size": 1, "initial_workers": 16, } response = test_utils.register_model_with_params(params) > assert response.status_code == 507, "OOM Error expected" E AssertionError: OOM Error expected E assert 200 == 507 E + where 200 = .status_code test_sm_mme_requirements.py:57: AssertionError ----------------------------- Captured stdout call ----------------------------- TorchServe has stopped. ## Starting gen_mar: /tmp/workspace/model_store/ ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, /tmp/workspace/model_store//fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, /tmp/workspace/model_store//deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, /tmp/workspace/model_store//alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, /tmp/workspace/model_store//mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, /tmp/workspace/model_store//resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, /tmp/workspace/model_store//densenet161.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, /tmp/workspace/model_store//squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, /tmp/workspace/model_store//vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, /tmp/workspace/model_store//maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, /tmp/workspace/model_store//resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, /tmp/workspace/model_store//fcn_resnet_101.mar successfully. ['torchserve', '--start', '--model-store', '/tmp/workspace/model_store/', '--no-config-snapshots'] WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. 2023-04-08T23:01:50,935 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2023-04-08T23:01:51,074 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.7.1 TS Home: /opt/conda/envs/py38/lib/python3.8/site-packages Current directory: /home/ubuntu/serve/test/pytest Temp directory: /tmp Metrics config path: /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml Number of GPUs: 4 Number of CPUs: 48 Max heap size: 30688 M Python executable: /opt/conda/envs/py38/bin/python Config file: N/A Inference address: http://127.0.0.1:8080 Management address: http://127.0.0.1:8081 Metrics address: http://127.0.0.1:8082 Model Store: /tmp/workspace/model_store Initial Models: N/A Log dir: /home/ubuntu/serve/test/pytest/logs Metrics dir: /home/ubuntu/serve/test/pytest/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 4 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Metrics report format: prometheus Enable metrics API: true Disable system metrics: false Workflow Store: /tmp/workspace/model_store Model config: N/A 2023-04-08T23:01:51,079 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2023-04-08T23:01:51,098 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2023-04-08T23:01:51,147 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8080 2023-04-08T23:01:51,148 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2023-04-08T23:01:51,149 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8081 2023-04-08T23:01:51,149 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2023-04-08T23:01:51,149 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082 Model server started. 2023-04-08T23:01:52,630 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,631 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:849.3932456970215|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,631 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:143.19650650024414|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,632 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:14.4|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,633 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:8.342018412367553|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,633 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:1921|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,633 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,633 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,634 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,634 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,634 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,634 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,634 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,635 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,635 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,635 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,635 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:183689.52734375|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,635 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5480.2109375|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:01:52,636 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:3.9|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994912 2023-04-08T23:02:08,060 [DEBUG] epollEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model BERTSeqClassification 2023-04-08T23:02:08,060 [DEBUG] epollEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model BERTSeqClassification 2023-04-08T23:02:08,061 [INFO ] epollEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelManager - Model BERTSeqClassification loaded. 2023-04-08T23:02:08,061 [DEBUG] epollEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelManager - updateModel: BERTSeqClassification, count: 16 2023-04-08T23:02:08,066 [DEBUG] W-9000-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,066 [DEBUG] W-9002-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9002, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,066 [DEBUG] W-9003-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9003, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,066 [DEBUG] W-9004-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9004, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,066 [DEBUG] W-9001-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9001, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,066 [DEBUG] W-9005-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9005, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,067 [DEBUG] W-9007-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9007, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,067 [DEBUG] W-9006-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9006, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,068 [DEBUG] W-9008-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9008, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,068 [DEBUG] W-9009-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9009, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,069 [DEBUG] W-9010-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9010, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,070 [DEBUG] W-9011-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9011, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,071 [DEBUG] W-9012-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9012, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,074 [DEBUG] W-9013-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9013, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,078 [DEBUG] W-9014-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9014, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:08,081 [DEBUG] W-9015-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/envs/py38/bin/python, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9015, --metrics-config, /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml] 2023-04-08T23:02:09,228 [INFO ] W-9012-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9012, pid=561699 2023-04-08T23:02:09,229 [INFO ] W-9012-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9012 2023-04-08T23:02:09,236 [INFO ] W-9012-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,237 [INFO ] W-9012-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561699 2023-04-08T23:02:09,237 [INFO ] W-9012-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,237 [DEBUG] W-9012-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9012-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,237 [INFO ] W-9012-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,238 [INFO ] W-9014-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9014, pid=561726 2023-04-08T23:02:09,239 [INFO ] W-9014-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9014 2023-04-08T23:02:09,240 [INFO ] W-9012-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9012 2023-04-08T23:02:09,239 [INFO ] W-9000-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=561674 2023-04-08T23:02:09,240 [INFO ] W-9000-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2023-04-08T23:02:09,244 [INFO ] W-9012-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9012. 2023-04-08T23:02:09,247 [INFO ] W-9012-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929246 2023-04-08T23:02:09,247 [INFO ] W-9014-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,247 [INFO ] W-9014-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561726 2023-04-08T23:02:09,248 [INFO ] W-9014-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,248 [DEBUG] W-9014-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9014-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,248 [INFO ] W-9014-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9014 2023-04-08T23:02:09,248 [INFO ] W-9014-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,248 [INFO ] W-9000-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,249 [INFO ] W-9000-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561674 2023-04-08T23:02:09,249 [INFO ] W-9000-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,249 [INFO ] W-9000-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,249 [DEBUG] W-9000-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,250 [INFO ] W-9014-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9014. 2023-04-08T23:02:09,250 [INFO ] W-9000-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2023-04-08T23:02:09,250 [INFO ] W-9014-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929249 2023-04-08T23:02:09,251 [INFO ] W-9008-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9008, pid=561687 2023-04-08T23:02:09,251 [INFO ] W-9000-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929251 2023-04-08T23:02:09,251 [INFO ] W-9000-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2023-04-08T23:02:09,251 [INFO ] W-9008-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9008 2023-04-08T23:02:09,261 [INFO ] W-9008-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,261 [INFO ] W-9015-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9015, pid=561740 2023-04-08T23:02:09,261 [INFO ] W-9008-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561687 2023-04-08T23:02:09,261 [INFO ] W-9015-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9015 2023-04-08T23:02:09,261 [INFO ] W-9008-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,262 [DEBUG] W-9008-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9008-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,262 [INFO ] W-9008-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,262 [INFO ] W-9008-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9008 2023-04-08T23:02:09,264 [INFO ] W-9008-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9008. 2023-04-08T23:02:09,264 [INFO ] W-9008-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929264 2023-04-08T23:02:09,264 [INFO ] W-9002-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9002, pid=561675 2023-04-08T23:02:09,265 [INFO ] W-9002-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9002 2023-04-08T23:02:09,269 [INFO ] W-9012-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,269 [INFO ] W-9011-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9011, pid=561694 2023-04-08T23:02:09,269 [INFO ] W-9014-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,269 [INFO ] W-9011-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9011 2023-04-08T23:02:09,269 [INFO ] W-9001-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9001, pid=561680 2023-04-08T23:02:09,270 [INFO ] W-9001-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9001 2023-04-08T23:02:09,271 [INFO ] W-9015-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,271 [INFO ] W-9015-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561740 2023-04-08T23:02:09,271 [INFO ] W-9015-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,271 [DEBUG] W-9015-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9015-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,271 [INFO ] W-9015-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,272 [INFO ] W-9015-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9015 2023-04-08T23:02:09,273 [INFO ] W-9015-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9015. 2023-04-08T23:02:09,273 [INFO ] W-9015-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929273 2023-04-08T23:02:09,273 [INFO ] W-9002-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,274 [INFO ] W-9000-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,274 [INFO ] W-9002-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561675 2023-04-08T23:02:09,274 [INFO ] W-9002-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,274 [INFO ] W-9002-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,274 [DEBUG] W-9002-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,282 [INFO ] W-9011-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,282 [INFO ] W-9001-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,282 [INFO ] W-9011-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561694 2023-04-08T23:02:09,282 [INFO ] W-9001-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561680 2023-04-08T23:02:09,282 [INFO ] W-9002-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9002 2023-04-08T23:02:09,282 [INFO ] W-9003-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9003, pid=561679 2023-04-08T23:02:09,282 [INFO ] W-9011-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,282 [INFO ] W-9008-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,282 [DEBUG] W-9011-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9011-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,282 [DEBUG] W-9001-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,282 [INFO ] W-9001-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,283 [INFO ] W-9001-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9001 2023-04-08T23:02:09,283 [INFO ] W-9011-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9011 2023-04-08T23:02:09,283 [INFO ] W-9011-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,283 [INFO ] W-9003-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9003 2023-04-08T23:02:09,283 [INFO ] W-9001-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,284 [INFO ] W-9002-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9002. 2023-04-08T23:02:09,284 [INFO ] W-9001-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9001. 2023-04-08T23:02:09,284 [INFO ] W-9002-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929284 2023-04-08T23:02:09,284 [INFO ] W-9001-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929284 2023-04-08T23:02:09,284 [INFO ] W-9011-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9011. 2023-04-08T23:02:09,284 [INFO ] W-9011-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929284 2023-04-08T23:02:09,288 [INFO ] W-9003-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,288 [INFO ] W-9003-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561679 2023-04-08T23:02:09,288 [INFO ] W-9003-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,288 [INFO ] W-9003-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,288 [DEBUG] W-9003-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,292 [INFO ] W-9003-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9003 2023-04-08T23:02:09,293 [INFO ] W-9015-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,295 [INFO ] W-9003-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929295 2023-04-08T23:02:09,295 [INFO ] W-9003-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9003. 2023-04-08T23:02:09,305 [INFO ] W-9011-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,305 [INFO ] W-9002-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,305 [INFO ] W-9001-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,313 [INFO ] W-9006-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9006, pid=561685 2023-04-08T23:02:09,314 [INFO ] W-9006-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9006 2023-04-08T23:02:09,314 [INFO ] W-9009-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9009, pid=561688 2023-04-08T23:02:09,315 [INFO ] W-9009-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9009 2023-04-08T23:02:09,315 [INFO ] W-9005-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9005, pid=561682 2023-04-08T23:02:09,315 [INFO ] W-9003-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,315 [INFO ] W-9005-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9005 2023-04-08T23:02:09,322 [INFO ] W-9004-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9004, pid=561678 2023-04-08T23:02:09,323 [INFO ] W-9004-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9004 2023-04-08T23:02:09,323 [INFO ] W-9006-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,323 [INFO ] W-9006-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561685 2023-04-08T23:02:09,323 [INFO ] W-9006-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,323 [INFO ] W-9006-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,323 [DEBUG] W-9006-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9006-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,324 [INFO ] W-9006-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9006 2023-04-08T23:02:09,324 [INFO ] W-9009-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,324 [INFO ] W-9010-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9010, pid=561690 2023-04-08T23:02:09,324 [INFO ] W-9010-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9010 2023-04-08T23:02:09,324 [INFO ] W-9009-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561688 2023-04-08T23:02:09,324 [INFO ] W-9009-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,324 [INFO ] W-9009-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,324 [DEBUG] W-9009-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9009-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,324 [INFO ] W-9005-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,324 [INFO ] W-9009-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9009 2023-04-08T23:02:09,325 [INFO ] W-9005-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561682 2023-04-08T23:02:09,325 [INFO ] W-9005-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,325 [INFO ] W-9006-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929325 2023-04-08T23:02:09,325 [DEBUG] W-9005-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9005-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,325 [INFO ] W-9006-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9006. 2023-04-08T23:02:09,325 [INFO ] W-9005-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,325 [INFO ] W-9005-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9005 2023-04-08T23:02:09,326 [INFO ] W-9009-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929326 2023-04-08T23:02:09,326 [INFO ] W-9009-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9009. 2023-04-08T23:02:09,327 [INFO ] W-9005-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9005. 2023-04-08T23:02:09,327 [INFO ] W-9005-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929327 2023-04-08T23:02:09,333 [INFO ] W-9004-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,334 [INFO ] W-9004-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561678 2023-04-08T23:02:09,334 [INFO ] W-9010-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,334 [INFO ] W-9004-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,334 [INFO ] W-9004-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,334 [INFO ] W-9010-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561690 2023-04-08T23:02:09,334 [DEBUG] W-9004-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9004-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,334 [INFO ] W-9013-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9013, pid=561710 2023-04-08T23:02:09,334 [INFO ] W-9010-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,334 [INFO ] W-9004-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9004 2023-04-08T23:02:09,334 [DEBUG] W-9010-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9010-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,334 [INFO ] W-9013-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9013 2023-04-08T23:02:09,334 [INFO ] W-9010-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9010 2023-04-08T23:02:09,334 [INFO ] W-9010-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,336 [INFO ] W-9004-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929336 2023-04-08T23:02:09,336 [INFO ] W-9004-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9004. 2023-04-08T23:02:09,336 [INFO ] W-9010-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9010. 2023-04-08T23:02:09,336 [INFO ] W-9010-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929336 2023-04-08T23:02:09,346 [INFO ] W-9013-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,346 [INFO ] W-9007-BERTSeqClassification_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9007, pid=561683 2023-04-08T23:02:09,346 [INFO ] W-9013-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561710 2023-04-08T23:02:09,347 [INFO ] W-9013-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,347 [INFO ] W-9007-BERTSeqClassification_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9007 2023-04-08T23:02:09,347 [INFO ] W-9013-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,347 [INFO ] W-9006-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,347 [INFO ] W-9009-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,347 [DEBUG] W-9013-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9013-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,347 [INFO ] W-9013-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9013 2023-04-08T23:02:09,349 [INFO ] W-9013-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929349 2023-04-08T23:02:09,349 [INFO ] W-9013-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9013. 2023-04-08T23:02:09,350 [INFO ] W-9007-BERTSeqClassification_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml. 2023-04-08T23:02:09,350 [INFO ] W-9007-BERTSeqClassification_1.0-stdout MODEL_LOG - [PID]561683 2023-04-08T23:02:09,350 [INFO ] W-9007-BERTSeqClassification_1.0-stdout MODEL_LOG - Torch worker started. 2023-04-08T23:02:09,350 [INFO ] W-9007-BERTSeqClassification_1.0-stdout MODEL_LOG - Python runtime: 3.8.16 2023-04-08T23:02:09,350 [DEBUG] W-9007-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9007-BERTSeqClassification_1.0 State change null -> WORKER_STARTED 2023-04-08T23:02:09,350 [INFO ] W-9007-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9007 2023-04-08T23:02:09,352 [INFO ] W-9007-BERTSeqClassification_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9007. 2023-04-08T23:02:09,353 [INFO ] W-9007-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1680994929353 2023-04-08T23:02:09,354 [INFO ] W-9005-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,369 [INFO ] W-9013-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,369 [INFO ] W-9010-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,369 [INFO ] W-9004-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,369 [INFO ] W-9007-BERTSeqClassification_1.0-stdout MODEL_LOG - model_name: BERTSeqClassification, batchSize: 1 2023-04-08T23:02:09,863 [INFO ] W-9000-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,863 [INFO ] W-9012-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,876 [INFO ] W-9014-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,891 [INFO ] W-9015-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,894 [INFO ] W-9008-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,898 [INFO ] W-9001-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,907 [INFO ] W-9003-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,919 [INFO ] W-9011-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,934 [INFO ] W-9002-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,934 [INFO ] W-9009-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,936 [INFO ] W-9006-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,940 [INFO ] W-9005-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,954 [INFO ] W-9007-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,959 [INFO ] W-9004-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,969 [INFO ] W-9010-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:09,982 [INFO ] W-9013-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformers version 4.25.1 2023-04-08T23:02:12,816 [INFO ] W-9012-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:12,818 [INFO ] W-9012-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 3549 2023-04-08T23:02:12,818 [DEBUG] W-9012-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9012-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:12,818 [INFO ] W-9012-BERTSeqClassification_1.0 TS_METRICS - W-9012-BERTSeqClassification_1.0.ms:4749|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994932 2023-04-08T23:02:12,818 [INFO ] W-9012-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:23|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994932 2023-04-08T23:02:13,005 [INFO ] W-9000-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,005 [INFO ] W-9000-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 3732 2023-04-08T23:02:13,006 [DEBUG] W-9000-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,006 [INFO ] W-9000-BERTSeqClassification_1.0 TS_METRICS - W-9000-BERTSeqClassification_1.0.ms:4942|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,006 [INFO ] W-9000-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:23|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,420 [INFO ] W-9001-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,421 [INFO ] W-9001-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4115 2023-04-08T23:02:13,421 [DEBUG] W-9001-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,421 [INFO ] W-9001-BERTSeqClassification_1.0 TS_METRICS - W-9001-BERTSeqClassification_1.0.ms:5356|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,421 [INFO ] W-9001-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:22|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,450 [INFO ] W-9008-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,451 [INFO ] W-9008-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4169 2023-04-08T23:02:13,451 [DEBUG] W-9008-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9008-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,451 [INFO ] W-9008-BERTSeqClassification_1.0 TS_METRICS - W-9008-BERTSeqClassification_1.0.ms:5385|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,452 [INFO ] W-9008-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:19|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,792 [INFO ] W-9003-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,793 [INFO ] W-9003-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4478 2023-04-08T23:02:13,793 [DEBUG] W-9003-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,793 [INFO ] W-9003-BERTSeqClassification_1.0 TS_METRICS - W-9003-BERTSeqClassification_1.0.ms:5728|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,793 [INFO ] W-9003-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:20|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,817 [INFO ] W-9005-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,818 [INFO ] W-9005-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4464 2023-04-08T23:02:13,818 [DEBUG] W-9005-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9005-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,818 [INFO ] W-9005-BERTSeqClassification_1.0 TS_METRICS - W-9005-BERTSeqClassification_1.0.ms:5752|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,819 [INFO ] W-9005-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:28|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,893 [INFO ] W-9004-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,894 [INFO ] W-9004-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4525 2023-04-08T23:02:13,894 [DEBUG] W-9004-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9004-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,894 [INFO ] W-9004-BERTSeqClassification_1.0 TS_METRICS - W-9004-BERTSeqClassification_1.0.ms:5829|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,894 [INFO ] W-9004-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:33|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,938 [INFO ] W-9002-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,938 [INFO ] W-9002-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4633 2023-04-08T23:02:13,938 [DEBUG] W-9002-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,938 [INFO ] W-9002-BERTSeqClassification_1.0 TS_METRICS - W-9002-BERTSeqClassification_1.0.ms:5873|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,938 [INFO ] W-9002-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:21|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,942 [INFO ] W-9007-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,943 [INFO ] W-9007-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4574 2023-04-08T23:02:13,943 [DEBUG] W-9007-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9007-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,943 [INFO ] W-9007-BERTSeqClassification_1.0 TS_METRICS - W-9007-BERTSeqClassification_1.0.ms:5877|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,943 [INFO ] W-9007-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:16|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,976 [INFO ] W-9013-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,977 [INFO ] W-9013-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4607 2023-04-08T23:02:13,977 [DEBUG] W-9013-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9013-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,977 [INFO ] W-9013-BERTSeqClassification_1.0 TS_METRICS - W-9013-BERTSeqClassification_1.0.ms:5906|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,977 [INFO ] W-9013-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:21|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,981 [INFO ] W-9009-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,982 [INFO ] W-9009-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4636 2023-04-08T23:02:13,982 [DEBUG] W-9009-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9009-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,982 [INFO ] W-9009-BERTSeqClassification_1.0 TS_METRICS - W-9009-BERTSeqClassification_1.0.ms:5916|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,982 [INFO ] W-9009-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:20|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,985 [INFO ] W-9011-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,985 [INFO ] W-9011-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4680 2023-04-08T23:02:13,986 [DEBUG] W-9011-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9011-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,986 [INFO ] W-9011-BERTSeqClassification_1.0 TS_METRICS - W-9011-BERTSeqClassification_1.0.ms:5919|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,986 [INFO ] W-9011-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:22|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,997 [INFO ] W-9015-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:13,997 [INFO ] W-9015-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4705 2023-04-08T23:02:13,997 [DEBUG] W-9015-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9015-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:13,997 [INFO ] W-9015-BERTSeqClassification_1.0 TS_METRICS - W-9015-BERTSeqClassification_1.0.ms:5920|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:13,997 [INFO ] W-9015-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:19|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994933 2023-04-08T23:02:14,008 [INFO ] W-9010-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:14,008 [INFO ] W-9010-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4639 2023-04-08T23:02:14,008 [DEBUG] W-9010-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9010-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:14,008 [INFO ] W-9010-BERTSeqClassification_1.0 TS_METRICS - W-9010-BERTSeqClassification_1.0.ms:5941|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994934 2023-04-08T23:02:14,008 [INFO ] W-9010-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:33|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994934 2023-04-08T23:02:14,016 [INFO ] W-9006-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:14,017 [INFO ] W-9006-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4671 2023-04-08T23:02:14,017 [DEBUG] W-9006-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9006-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:14,017 [INFO ] W-9006-BERTSeqClassification_1.0 TS_METRICS - W-9006-BERTSeqClassification_1.0.ms:5951|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994934 2023-04-08T23:02:14,017 [INFO ] W-9006-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:21|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994934 2023-04-08T23:02:14,021 [INFO ] W-9014-BERTSeqClassification_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/9baa35a1ccac4977a818192b92baee67 loaded successfully 2023-04-08T23:02:14,022 [INFO ] W-9014-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4753 2023-04-08T23:02:14,022 [DEBUG] W-9014-BERTSeqClassification_1.0 org.pytorch.serve.wlm.WorkerThread - W-9014-BERTSeqClassification_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-04-08T23:02:14,022 [INFO ] W-9014-BERTSeqClassification_1.0 TS_METRICS - W-9014-BERTSeqClassification_1.0.ms:5948|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994934 2023-04-08T23:02:14,022 [INFO ] W-9014-BERTSeqClassification_1.0 TS_METRICS - WorkerThreadTime.ms:20|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994934 2023-04-08T23:02:14,027 [INFO ] epollEventLoopGroup-3-1 ACCESS_LOG - /127.0.0.1:45490 "POST /models?model_name=BERTSeqClassification&url=https%3A%2F%2Ftorchserve.pytorch.org%2Fmar_files%2FBERTSeqClassification.mar&batch_size=1&initial_workers=16 HTTP/1.1" 200 13552 2023-04-08T23:02:14,027 [INFO ] epollEventLoopGroup-3-1 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994920 ______________________________ test_oom_on_invoke ______________________________ @pytest.mark.skipif( not ((torch.cuda.device_count() > 0) and torch.cuda.is_available()), reason="Test to be run on GPU only", ) def test_oom_on_invoke(): # Create model store directory pathlib.Path(test_utils.MODEL_STORE).mkdir(parents=True, exist_ok=True) # Start TorchServe test_utils.start_torchserve(no_config_snapshots=True) # Register model params = { "model_name": "BERTSeqClassification", "url": "https://torchserve.pytorch.org/mar_files/BERTSeqClassification.mar", "batch_size": 8, "initial_workers": 12, } response = test_utils.register_model_with_params(params) input_text = os.path.join( REPO_ROOT, "examples", "Huggingface_Transformers", "Seq_classification_artifacts", "sample_text_captum_input.txt", ) # Make 8 curl requests in parallel with & # Send multiple requests to make sure to hit OOM for i in range(10): response = os.popen( f"curl http://127.0.0.1:8080/models/BERTSeqClassification/invoke -T {input_text} && " f"curl http://127.0.0.1:8080/models/BERTSeqClassification/invoke -T {input_text} && " f"curl http://127.0.0.1:8080/models/BERTSeqClassification/invoke -T {input_text} && " f"curl http://127.0.0.1:8080/models/BERTSeqClassification/invoke -T {input_text} && " f"curl http://127.0.0.1:8080/models/BERTSeqClassification/invoke -T {input_text} && " f"curl http://127.0.0.1:8080/models/BERTSeqClassification/invoke -T {input_text} && " f"curl http://127.0.0.1:8080/models/BERTSeqClassification/invoke -T {input_text} && " f"curl http://127.0.0.1:8080/models/BERTSeqClassification/invoke -T {input_text} " ) response = response.read() # If OOM is hit, we expect code 507 to be present in the response string lines = response.split("\n") output = "" for line in lines: if "code" in line: line = line.strip() output = line break > assert output == '"code": 507,', "OOM Error expected" E AssertionError: OOM Error expected E assert '"code": 404,' == '"code": 507,' E - "code": 507, E ? ^ ^ E + "code": 404, E ? ^ ^ test_sm_mme_requirements.py:113: AssertionError ----------------------------- Captured stdout call ----------------------------- TorchServe has stopped. ## Starting gen_mar: /tmp/workspace/model_store/ ## Create symlink for mar files ## /tmp/workspace/model_store//fastrcnn.mar already exists. ## /tmp/workspace/model_store//deeplabv3_resnet_101_eager.mar already exists. ## /tmp/workspace/model_store//alexnet.mar already exists. ## /tmp/workspace/model_store//mnist.mar already exists. ## /tmp/workspace/model_store//resnet-152-batch.mar already exists. ## /tmp/workspace/model_store//densenet161.mar already exists. ## /tmp/workspace/model_store//squeezenet1_1.mar already exists. ## /tmp/workspace/model_store//vgg16.mar already exists. ## /tmp/workspace/model_store//maskrcnn.mar already exists. ## /tmp/workspace/model_store//resnet-18.mar already exists. ## /tmp/workspace/model_store//fcn_resnet_101.mar already exists. ['torchserve', '--start', '--model-store', '/tmp/workspace/model_store/', '--no-config-snapshots'] WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. 2023-04-08T23:02:24,855 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2023-04-08T23:02:25,010 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.7.1 TS Home: /opt/conda/envs/py38/lib/python3.8/site-packages Current directory: /home/ubuntu/serve/test/pytest Temp directory: /tmp Metrics config path: /opt/conda/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml Number of GPUs: 4 Number of CPUs: 48 Max heap size: 30688 M Python executable: /opt/conda/envs/py38/bin/python Config file: N/A Inference address: http://127.0.0.1:8080 Management address: http://127.0.0.1:8081 Metrics address: http://127.0.0.1:8082 Model Store: /tmp/workspace/model_store Initial Models: N/A Log dir: /home/ubuntu/serve/test/pytest/logs Metrics dir: /home/ubuntu/serve/test/pytest/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 4 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Metrics report format: prometheus Enable metrics API: true Disable system metrics: false Workflow Store: /tmp/workspace/model_store Model config: N/A 2023-04-08T23:02:25,016 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2023-04-08T23:02:25,035 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2023-04-08T23:02:25,083 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8080 2023-04-08T23:02:25,084 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2023-04-08T23:02:25,085 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8081 2023-04-08T23:02:25,085 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2023-04-08T23:02:25,085 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082 Model server started. 2023-04-08T23:02:26,592 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,593 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:849.0153961181641|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,593 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:143.57435607910156|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,593 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:14.5|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,594 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:8.342018412367553|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,594 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:1921|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,594 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,594 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,595 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,595 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,595 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.00868507903421921|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,595 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:2|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,596 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:0|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,596 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:1|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,596 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:2|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,596 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:3|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,596 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:183683.828125|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,597 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5485.7578125|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:26,597 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:3.9|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994946 2023-04-08T23:02:34,396 [INFO ] epollEventLoopGroup-3-1 ACCESS_LOG - /127.0.0.1:55700 "POST /models?model_name=BERTSeqClassification&url=https%3A%2F%2Ftorchserve.pytorch.org%2Fmar_files%2FBERTSeqClassification.mar&batch_size=8&initial_workers=12 HTTP/1.1" 500 9 2023-04-08T23:02:34,397 [INFO ] epollEventLoopGroup-3-1 TS_METRICS - Requests5XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,515 [INFO ] epollEventLoopGroup-3-2 ACCESS_LOG - /127.0.0.1:52392 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 5 2023-04-08T23:02:34,516 [INFO ] epollEventLoopGroup-3-2 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,532 [INFO ] epollEventLoopGroup-3-3 ACCESS_LOG - /127.0.0.1:52402 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,533 [INFO ] epollEventLoopGroup-3-3 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,549 [INFO ] epollEventLoopGroup-3-4 ACCESS_LOG - /127.0.0.1:52418 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,550 [INFO ] epollEventLoopGroup-3-4 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,566 [INFO ] epollEventLoopGroup-3-5 ACCESS_LOG - /127.0.0.1:52428 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,567 [INFO ] epollEventLoopGroup-3-5 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,581 [INFO ] epollEventLoopGroup-3-6 ACCESS_LOG - /127.0.0.1:52434 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,582 [INFO ] epollEventLoopGroup-3-6 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,607 [INFO ] epollEventLoopGroup-3-7 ACCESS_LOG - /127.0.0.1:52444 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:34,607 [INFO ] epollEventLoopGroup-3-7 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,622 [INFO ] epollEventLoopGroup-3-8 ACCESS_LOG - /127.0.0.1:52458 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,623 [INFO ] epollEventLoopGroup-3-8 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,638 [INFO ] epollEventLoopGroup-3-9 ACCESS_LOG - /127.0.0.1:52472 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:34,638 [INFO ] epollEventLoopGroup-3-9 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,746 [INFO ] epollEventLoopGroup-3-10 ACCESS_LOG - /127.0.0.1:52482 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:34,746 [INFO ] epollEventLoopGroup-3-10 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,762 [INFO ] epollEventLoopGroup-3-11 ACCESS_LOG - /127.0.0.1:52486 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:34,762 [INFO ] epollEventLoopGroup-3-11 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,776 [INFO ] epollEventLoopGroup-3-12 ACCESS_LOG - /127.0.0.1:52488 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,777 [INFO ] epollEventLoopGroup-3-12 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,791 [INFO ] epollEventLoopGroup-3-13 ACCESS_LOG - /127.0.0.1:52504 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,791 [INFO ] epollEventLoopGroup-3-13 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,806 [INFO ] epollEventLoopGroup-3-14 ACCESS_LOG - /127.0.0.1:52510 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,806 [INFO ] epollEventLoopGroup-3-14 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,821 [INFO ] epollEventLoopGroup-3-15 ACCESS_LOG - /127.0.0.1:52518 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,822 [INFO ] epollEventLoopGroup-3-15 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,835 [INFO ] epollEventLoopGroup-3-16 ACCESS_LOG - /127.0.0.1:52534 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,836 [INFO ] epollEventLoopGroup-3-16 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,851 [INFO ] epollEventLoopGroup-3-17 ACCESS_LOG - /127.0.0.1:52546 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:34,851 [INFO ] epollEventLoopGroup-3-17 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,958 [INFO ] epollEventLoopGroup-3-18 ACCESS_LOG - /127.0.0.1:52556 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,959 [INFO ] epollEventLoopGroup-3-18 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,973 [INFO ] epollEventLoopGroup-3-19 ACCESS_LOG - /127.0.0.1:52560 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,974 [INFO ] epollEventLoopGroup-3-19 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:34,988 [INFO ] epollEventLoopGroup-3-20 ACCESS_LOG - /127.0.0.1:52566 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:34,989 [INFO ] epollEventLoopGroup-3-20 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,004 [INFO ] epollEventLoopGroup-3-21 ACCESS_LOG - /127.0.0.1:52570 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,004 [INFO ] epollEventLoopGroup-3-21 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,018 [INFO ] epollEventLoopGroup-3-22 ACCESS_LOG - /127.0.0.1:52576 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,019 [INFO ] epollEventLoopGroup-3-22 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,034 [INFO ] epollEventLoopGroup-3-23 ACCESS_LOG - /127.0.0.1:52578 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,034 [INFO ] epollEventLoopGroup-3-23 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,050 [INFO ] epollEventLoopGroup-3-24 ACCESS_LOG - /127.0.0.1:52590 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,050 [INFO ] epollEventLoopGroup-3-24 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,065 [INFO ] epollEventLoopGroup-3-25 ACCESS_LOG - /127.0.0.1:52600 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,065 [INFO ] epollEventLoopGroup-3-25 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,174 [INFO ] epollEventLoopGroup-3-26 ACCESS_LOG - /127.0.0.1:52614 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,175 [INFO ] epollEventLoopGroup-3-26 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,190 [INFO ] epollEventLoopGroup-3-27 ACCESS_LOG - /127.0.0.1:52622 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,190 [INFO ] epollEventLoopGroup-3-27 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,205 [INFO ] epollEventLoopGroup-3-28 ACCESS_LOG - /127.0.0.1:52630 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,205 [INFO ] epollEventLoopGroup-3-28 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,220 [INFO ] epollEventLoopGroup-3-29 ACCESS_LOG - /127.0.0.1:52634 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,221 [INFO ] epollEventLoopGroup-3-29 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,235 [INFO ] epollEventLoopGroup-3-30 ACCESS_LOG - /127.0.0.1:52638 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,236 [INFO ] epollEventLoopGroup-3-30 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,250 [INFO ] epollEventLoopGroup-3-31 ACCESS_LOG - /127.0.0.1:52646 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,250 [INFO ] epollEventLoopGroup-3-31 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,264 [INFO ] epollEventLoopGroup-3-32 ACCESS_LOG - /127.0.0.1:52650 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,264 [INFO ] epollEventLoopGroup-3-32 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,279 [INFO ] epollEventLoopGroup-3-33 ACCESS_LOG - /127.0.0.1:52666 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,279 [INFO ] epollEventLoopGroup-3-33 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,388 [INFO ] epollEventLoopGroup-3-34 ACCESS_LOG - /127.0.0.1:52672 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,389 [INFO ] epollEventLoopGroup-3-34 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,404 [INFO ] epollEventLoopGroup-3-35 ACCESS_LOG - /127.0.0.1:52688 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,404 [INFO ] epollEventLoopGroup-3-35 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,418 [INFO ] epollEventLoopGroup-3-36 ACCESS_LOG - /127.0.0.1:52694 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,418 [INFO ] epollEventLoopGroup-3-36 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,432 [INFO ] epollEventLoopGroup-3-37 ACCESS_LOG - /127.0.0.1:52698 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,432 [INFO ] epollEventLoopGroup-3-37 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,446 [INFO ] epollEventLoopGroup-3-38 ACCESS_LOG - /127.0.0.1:52714 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,447 [INFO ] epollEventLoopGroup-3-38 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,460 [INFO ] epollEventLoopGroup-3-39 ACCESS_LOG - /127.0.0.1:52726 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,461 [INFO ] epollEventLoopGroup-3-39 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,475 [INFO ] epollEventLoopGroup-3-40 ACCESS_LOG - /127.0.0.1:52734 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,476 [INFO ] epollEventLoopGroup-3-40 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,490 [INFO ] epollEventLoopGroup-3-41 ACCESS_LOG - /127.0.0.1:52740 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,490 [INFO ] epollEventLoopGroup-3-41 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,601 [INFO ] epollEventLoopGroup-3-42 ACCESS_LOG - /127.0.0.1:52746 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,601 [INFO ] epollEventLoopGroup-3-42 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,617 [INFO ] epollEventLoopGroup-3-43 ACCESS_LOG - /127.0.0.1:52752 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,617 [INFO ] epollEventLoopGroup-3-43 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,632 [INFO ] epollEventLoopGroup-3-44 ACCESS_LOG - /127.0.0.1:52754 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,632 [INFO ] epollEventLoopGroup-3-44 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,646 [INFO ] epollEventLoopGroup-3-45 ACCESS_LOG - /127.0.0.1:52768 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,646 [INFO ] epollEventLoopGroup-3-45 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,660 [INFO ] epollEventLoopGroup-3-46 ACCESS_LOG - /127.0.0.1:52784 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,661 [INFO ] epollEventLoopGroup-3-46 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,674 [INFO ] epollEventLoopGroup-3-47 ACCESS_LOG - /127.0.0.1:52788 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,675 [INFO ] epollEventLoopGroup-3-47 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,688 [INFO ] epollEventLoopGroup-3-48 ACCESS_LOG - /127.0.0.1:52802 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,688 [INFO ] epollEventLoopGroup-3-48 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,702 [INFO ] epollEventLoopGroup-3-49 ACCESS_LOG - /127.0.0.1:52818 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,703 [INFO ] epollEventLoopGroup-3-49 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,810 [INFO ] epollEventLoopGroup-3-50 ACCESS_LOG - /127.0.0.1:52832 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,810 [INFO ] epollEventLoopGroup-3-50 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,826 [INFO ] epollEventLoopGroup-3-51 ACCESS_LOG - /127.0.0.1:52846 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,827 [INFO ] epollEventLoopGroup-3-51 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,841 [INFO ] epollEventLoopGroup-3-52 ACCESS_LOG - /127.0.0.1:52854 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,842 [INFO ] epollEventLoopGroup-3-52 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,857 [INFO ] epollEventLoopGroup-3-53 ACCESS_LOG - /127.0.0.1:52864 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,857 [INFO ] epollEventLoopGroup-3-53 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,871 [INFO ] epollEventLoopGroup-3-54 ACCESS_LOG - /127.0.0.1:52878 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,872 [INFO ] epollEventLoopGroup-3-54 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,885 [INFO ] epollEventLoopGroup-3-55 ACCESS_LOG - /127.0.0.1:52884 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,886 [INFO ] epollEventLoopGroup-3-55 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,902 [INFO ] epollEventLoopGroup-3-56 ACCESS_LOG - /127.0.0.1:52896 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:35,902 [INFO ] epollEventLoopGroup-3-56 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:35,917 [INFO ] epollEventLoopGroup-3-57 ACCESS_LOG - /127.0.0.1:52912 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:35,918 [INFO ] epollEventLoopGroup-3-57 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,026 [INFO ] epollEventLoopGroup-3-58 ACCESS_LOG - /127.0.0.1:58678 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:36,026 [INFO ] epollEventLoopGroup-3-58 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,040 [INFO ] epollEventLoopGroup-3-59 ACCESS_LOG - /127.0.0.1:58692 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,041 [INFO ] epollEventLoopGroup-3-59 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,054 [INFO ] epollEventLoopGroup-3-60 ACCESS_LOG - /127.0.0.1:58708 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,055 [INFO ] epollEventLoopGroup-3-60 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,068 [INFO ] epollEventLoopGroup-3-61 ACCESS_LOG - /127.0.0.1:58722 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,068 [INFO ] epollEventLoopGroup-3-61 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,083 [INFO ] epollEventLoopGroup-3-62 ACCESS_LOG - /127.0.0.1:58724 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,084 [INFO ] epollEventLoopGroup-3-62 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,108 [INFO ] epollEventLoopGroup-3-63 ACCESS_LOG - /127.0.0.1:58726 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,108 [INFO ] epollEventLoopGroup-3-63 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,123 [INFO ] epollEventLoopGroup-3-64 ACCESS_LOG - /127.0.0.1:58728 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,124 [INFO ] epollEventLoopGroup-3-64 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,139 [INFO ] epollEventLoopGroup-3-65 ACCESS_LOG - /127.0.0.1:58730 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,140 [INFO ] epollEventLoopGroup-3-65 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,247 [INFO ] epollEventLoopGroup-3-66 ACCESS_LOG - /127.0.0.1:58744 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,247 [INFO ] epollEventLoopGroup-3-66 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,262 [INFO ] epollEventLoopGroup-3-67 ACCESS_LOG - /127.0.0.1:58758 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:36,262 [INFO ] epollEventLoopGroup-3-67 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,276 [INFO ] epollEventLoopGroup-3-68 ACCESS_LOG - /127.0.0.1:58766 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,276 [INFO ] epollEventLoopGroup-3-68 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,289 [INFO ] epollEventLoopGroup-3-69 ACCESS_LOG - /127.0.0.1:58774 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,290 [INFO ] epollEventLoopGroup-3-69 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,304 [INFO ] epollEventLoopGroup-3-70 ACCESS_LOG - /127.0.0.1:58776 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:36,304 [INFO ] epollEventLoopGroup-3-70 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,318 [INFO ] epollEventLoopGroup-3-71 ACCESS_LOG - /127.0.0.1:58780 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,318 [INFO ] epollEventLoopGroup-3-71 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,331 [INFO ] epollEventLoopGroup-3-72 ACCESS_LOG - /127.0.0.1:58790 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:36,331 [INFO ] epollEventLoopGroup-3-72 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,345 [INFO ] epollEventLoopGroup-3-73 ACCESS_LOG - /127.0.0.1:58792 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,345 [INFO ] epollEventLoopGroup-3-73 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,453 [INFO ] epollEventLoopGroup-3-74 ACCESS_LOG - /127.0.0.1:58800 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,454 [INFO ] epollEventLoopGroup-3-74 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,468 [INFO ] epollEventLoopGroup-3-75 ACCESS_LOG - /127.0.0.1:58816 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,468 [INFO ] epollEventLoopGroup-3-75 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,482 [INFO ] epollEventLoopGroup-3-76 ACCESS_LOG - /127.0.0.1:58828 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,482 [INFO ] epollEventLoopGroup-3-76 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,496 [INFO ] epollEventLoopGroup-3-77 ACCESS_LOG - /127.0.0.1:58840 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:36,496 [INFO ] epollEventLoopGroup-3-77 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,510 [INFO ] epollEventLoopGroup-3-78 ACCESS_LOG - /127.0.0.1:58850 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:36,510 [INFO ] epollEventLoopGroup-3-78 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,524 [INFO ] epollEventLoopGroup-3-79 ACCESS_LOG - /127.0.0.1:58852 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,524 [INFO ] epollEventLoopGroup-3-79 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,536 [INFO ] epollEventLoopGroup-3-80 ACCESS_LOG - /127.0.0.1:58858 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 0 2023-04-08T23:02:36,537 [INFO ] epollEventLoopGroup-3-80 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 2023-04-08T23:02:36,551 [INFO ] epollEventLoopGroup-3-81 ACCESS_LOG - /127.0.0.1:58874 "PUT /models/BERTSeqClassification/invoke HTTP/1.1" 404 1 2023-04-08T23:02:36,551 [INFO ] epollEventLoopGroup-3-81 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:ip-172-31-45-70,timestamp:1680994954 ----------------------------- Captured stderr call ----------------------------- % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 6812 5750 --:--:-- --:--:-- --:--:-- 12562 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 12111 10222 --:--:-- --:--:-- --:--:-- 22333 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 6055 5111 --:--:-- --:--:-- --:--:-- 11823 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 8384 7076 --:--:-- --:--:-- --:--:-- 15461 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 10900 9200 --:--:-- --:--:-- --:--:-- 20100 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 13625 11500 --:--:-- --:--:-- --:--:-- 25125 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 28714 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 15571 13142 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 201 100 109 100 92 18166 15333 --:--:-- --:--:-- --:--:-- 33500 =============================== warnings summary =============================== test/pytest/test_example_scriptable_tokenzier.py::test_handler /opt/conda/envs/py38/lib/python3.8/site-packages/torch/jit/_recursive.py:266: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/pytest/test_example_scriptable_tokenzier.py::test_handler /opt/conda/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning) test/pytest/test_example_scriptable_tokenzier.py::test_handler /opt/conda/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) test/pytest/test_example_scriptable_tokenzier.py::test_handler test/pytest/test_example_scriptable_tokenzier.py::test_handler /opt/conda/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) test/pytest/test_example_scriptable_tokenzier.py::test_handler /opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py:1501: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at ../aten/src/ATen/NestedTensorImpl.cpp:177.) return forward_call(*args, **kwargs) test/pytest/test_example_scriptable_tokenzier.py::test_handler /home/ubuntu/serve/test/pytest/../../examples/text_classification_with_scriptable_tokenizer/handler.py:97: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. data = F.softmax(data) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================== short test summary info ============================ FAILED test_example_scriptable_tokenzier.py::test_inference_with_pretrained_model FAILED test_handler.py::test_huggingface_opt_distributed_inference_deepspeed FAILED test_sm_mme_requirements.py::test_oom_on_model_load - AssertionError: ... FAILED test_sm_mme_requirements.py::test_oom_on_invoke - AssertionError: OOM ... ====== 4 failed, 76 passed, 10 skipped, 7 warnings in 2933.68s (0:48:53) ======= ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/ubuntu/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Started regression tests ## Started densenet mar creation ## In directory: /tmp/workspace/model_store | Executing command: torch-model-archiver --model-name densenet161_v1 --version 1.1 --model-file /home/ubuntu/serve/ts_scripts/../examples/image_classifier/densenet_161/model.py --serialized-file /tmp/workspace/model_store/densenet161-8d451a50.pth --extra-files /home/ubuntu/serve/ts_scripts/../examples/image_classifier/index_to_name.json --handler image_classifier --force ## Started regression pytests ## In directory: /home/ubuntu/serve/test/pytest | Executing command: python -m pytest -v ./ Removing file : management_pb2_grpc.py Removing file : inference_pb2_grpc.py Removing file : management_pb2.py Removing file : inference_pb2.py ## TorchServe Regression Pytests Failed