Fix launch CLI bug in launch endpoints delete causing the command to fail with a TypeError

[anant-marur]

Summary

modify launch.cli.delete_endpoint() to pass endpoint_name directly into client.delete_model_endpoint() rather than incorrectly passing it within a ModelEndpoint object.

Context

Linear ticket: CLI command launch endpoints delete failing with TypeError.

Failure examples:

$ launch endpoints delete alpha-camera-tampering-sync-service
ValueError: Invalid type passed in got input=ModelEndpoint(name='alpha-camera-tampering-sync-service', bundle_name='None', status='None', resource_state='null', deployment_state='null', endpoint_type='None', metadata='None') type=<class 'launch.model_endpoint.ModelEndpoint'>

$ launch endpoints delete [end_cflv9kr19r300303l9d0](https://dashboard.scale.com/corp/lookup/end_cflv9kr19r300303l9d0)
ValueError: Invalid type passed in got input=ModelEndpoint(name='[end_cflv9kr19r300303l9d0](https://dashboard.scale.com/corp/lookup/end_cflv9kr19r300303l9d0)', bundle_name='None', status='None', resource_state='null', deployment_state='null', endpoint_type='None', metadata='None') type=<class 'launch.model_endpoint.ModelEndpoint'>

Test Plan

Spin up an LLM endpoint and successfully delete using CLI.

Initial state: no endpoints

$ scale-launch endpoints list
Endpoints                                                                                                                        
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━┓
┃             ┃               ┃             ┃        ┃ Endpoint ┃ Min     ┃ Max     ┃ Available ┃ Unavailable ┃ Num  ┃          ┃
┃ Endpoint ID ┃ Endpoint name ┃ Bundle name ┃ Status ┃ type     ┃ Workers ┃ Workers ┃ Workers   ┃ Workers     ┃ GPUs ┃ Metadata ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━┩
└─────────────┴───────────────┴─────────────┴────────┴──────────┴─────────┴─────────┴───────────┴─────────────┴──────┴──────────┘

Create a llama-2-7b endpoint

Issue POST request

$ curl -X POST 'https://model-engine.ml-serving-internal.scale.com/v1/llm/model-endpoints' \                                                          
    -H 'Content-Type: application/json' \                                                                                                                                                                                
    -d '{                                                                                                                                                                                                                
        "name": "llama-2-7b-anantmarur-test",                                                                                                                                                                            
        "model_name": "llama-2-7b",                                                                                                                                                                                      
        "source": "hugging_face",                                                                                                                                                                                        
        "inference_framework": "text_generation_inference",                                                                                                                                                              
        "inference_framework_image_tag": "0.9.3",                                                                                                                                                                        
        "num_shards": 4,                                                                                                                                                                                                 
        "endpoint_type": "streaming",                                                                                                                                                                                    
        "cpus": 32,                                                                                                                                                                                                      
        "gpus": 4,                                                                                                                                                                                                       
        "memory": "40Gi",                                                                                                                                                                                                
        "storage": "40Gi",                                                                                                                                                                                               
        "gpu_type": "nvidia-ampere-a10",                                                                                                                                                                                 
        "min_workers": 1,                                                                                                                                                                                                
        "max_workers": 1,                                                                                                                                                                                                
        "per_worker": 1,                                                                                                                                                                                                 
        "labels": {                                                                                                                                                                                                      
            "team": "gen_ai",                                                                                                                                                                                            
            "product": "inference.chat_task"                                                                                                                                                                             
        },                                                                                                                                                                                                               
        "checkpoint_path": "s3://scale-ml/models/hf-llama/hf-llama-2-7b/",                                                                                                                                               
        "metadata": {}                                                                                                                                                                                                   
    }' \                                                                                                                                                                                                                 
    -u {SCALE_UID}                                                                                                                                                                                          
Enter host password for user '{SCALE_UID}':                                                                                                                                                                 
{"endpoint_creation_task_id":"37d90a77-897b-4dfa-a461-2f4683a20928"}

Check endpoint was created successfully:

$ scale-launch endpoints list                                                                                                                                      
Endpoints                                                                                                                                                                                                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                          ┃                            ┃                            ┃        ┃ Endpoint  ┃ Min     ┃ Max     ┃ Available ┃ Unavailable ┃ Num  ┃                                                        ┃
┃ Endpoint ID              ┃ Endpoint name              ┃ Bundle name                ┃ Status ┃ type      ┃ Workers ┃ Workers ┃ Workers   ┃ Workers     ┃ GPUs ┃ Metadata                                               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ end_cpbcbd5is36gveeeo27g │ llama-2-7b-anantmarur-test │ llama-2-7b-anantmarur-test │ READY  │ streaming │ 1       │ 1       │ None      │ None        │ 4    │ {'_llm': {'source': 'hugging_face', 'quantize': None,  │
│                          │                            │                            │        │           │         │         │           │             │      │ 'model_name': 'llama-2-7b', 'num_shards': 4,           │
│                          │                            │                            │        │           │         │         │           │             │      │ 'checkpoint_path':                                     │
│                          │                            │                            │        │           │         │         │           │             │      │ 's3://scale-ml/models/hf-llama/hf-llama-2-7b/',        │
│                          │                            │                            │        │           │         │         │           │             │      │ 'inference_framework': 'text_generation_inference',    │
│                          │                            │                            │        │           │         │         │           │             │      │ 'inference_framework_image_tag': '0.9.3'}}             │
└──────────────────────────┴────────────────────────────┴────────────────────────────┴────────┴───────────┴─────────┴─────────┴───────────┴─────────────┴──────┴────────────────────────────────────────────────────────┘

Deletion fails before rebuilding with fix

$ scale-launch endpoints delete llama-2-7b-anantmarur-test
Traceback (most recent call last):
  File "/opt/conda/envs/launch/bin/scale-launch", line 8, in <module>
    sys.exit(entry_point())
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/launch/cli/endpoints.py", line 108, in delete_endpoint
    res = client.delete_model_endpoint(endpoint)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/launch/client.py", line 1922, in delete_model_endpoint
    endpoint = self.get_model_endpoint(model_endpoint_name)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/launch/client.py", line 1789, in get_model_endpoint
    response = api_instance.list_model_endpoints_v1_model_endpoints_get(  # type: ignore
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/launch/api_client/paths/v1_model_endpoints/get.py", line 249, in list_model_endpoints_v1_model_endpoints_get
    return self._list_model_endpoints_v1_model_endpoints_get_oapg(
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/launch/api_client/paths/v1_model_endpoints/get.py", line 171, in _list_model_endpoints_v1_model_endpoints_get_oapg
    serialized_data = parameter.serialize(parameter_data, prefix_separator_iterator)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/launch/api_client/api_client.py", line 605, in serialize
    cast_in_data = self.schema(in_data)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/launch/api_client/schemas.py", line 2514, in __new__
    return super().__new__(cls, _arg, **kwargs)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/launch/api_client/schemas.py", line 583, in __new__
    __arg = cast_to_allowed_types(__arg, __from_server, __validated_path_to_schemas)
  File "/opt/conda/envs/launch/lib/python3.8/site-packages/launch/api_client/schemas.py", line 2104, in cast_to_allowed_types
    raise ValueError("Invalid type passed in got input={} type={}".format(arg, type(arg)))
ValueError: Invalid type passed in got input=ModelEndpoint(name='llama-2-7b-anantmarur-test', bundle_name='None', status='None', resource_state='null', deployment_state='null', endpoint_type='None', metadata='None') t
ype=<class 'launch.model_endpoint.ModelEndpoint'>

Deletion succeeds after rebuilding with fix

Trying deletion:

$ scale-launch endpoints delete llama-2-7b-anantmarur-test
True

Checking to ensure deletion completed successfully:

$ scale-launch endpoints list
Endpoints                                                                                                                        
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━┓
┃             ┃               ┃             ┃        ┃ Endpoint ┃ Min     ┃ Max     ┃ Available ┃ Unavailable ┃ Num  ┃          ┃
┃ Endpoint ID ┃ Endpoint name ┃ Bundle name ┃ Status ┃ type     ┃ Workers ┃ Workers ┃ Workers   ┃ Workers     ┃ GPUs ┃ Metadata ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━┩
└─────────────┴───────────────┴─────────────┴────────┴──────────┴─────────┴─────────┴───────────┴─────────────┴──────┴──────────┘

[edgan8]

Great find! Did you test that this works?

[anant-marur]

Great find! Did you test that this works?

Yep! Added the testing plan to the PR description.