Implement API for Inference Endpoints

[Wauplin]

Implement #1541 (+fix #1605).

Ping @philschmid @jeffboudier Feedback is very welcomed if you see anything that can be improved product-wise :)

EDIT / TL;DR: here is the guide written in this PR.

---

This PR adds support for Inference Endpoints, following the Swagger API docs.

• HfApi methods:

  • list_inference_endpoints
  • create_inference_endpoint
  • get_inference_endpoint
  • update_inference_endpoint
  • delete_inference_endpoint
  • pause_inference_endpoint
  • resume_inference_endpoint
  • scale_to_zero_inference_endpoint

• InferenceEndpoint object + methods:

  • client / async_client => return InferenceClient object
  • wait() => wait until fully deployed
  • update() => alias for update_inference_endpoint
  • resume() => alias for resume_inference_endpoint
  • pause() => alias for pause_inference_endpoint
  • scale_to_zero() => alias for scale_to_zero_inference_endpoint
  • delete() => alias for delete_inference_endpoint

I intentionally did not implement metrics and logs endpoints yet as I don't think that should be a priority.

Listing/getting inference endpoints information is quite straightforward, given the user namespace or endpoint name. Same goes for resume/pause/scale_to_zero feature which AFAIK would be the most useful ones in scripts. Creating and updating endpoints is more difficult as the user needs to know exactly the configuration they want to use.

The main object returned by most methods is an InferenceEndpoint dataclass with useful information like name, status, url, model, framework, task, created_at/updated_at,... It also has 2 properties .client and .async_client to run inference.

Regarding testing the API, I am not sure we can and we want to do that in the CI. Like for the Spaces API, it's quite hard (and costly) to do end-to-end tests on production and for little benefit IMO (as we can assume v2 endpoints will not be updated). I ran some tests locally to make sure everything runs as expected though.

---

Example:

>>> from huggingface_hub import create_inference_endpoint

# Create endpoint
>>> endpoint = create_inference_endpoint(
...     "my-endpoint-name6",
...     repository="gpt2",
...     framework="pytorch",
...     task="text-generation",
...     accelerator="cpu",
...     vendor="aws",
...     region="us-east-1",
...     type="protected",
...     instance_size="medium",
...     instance_type="c6i"
... )

# pending creation => no url => cannot get client
>>> endpoint
InferenceEndpoint(name='my-endpoint-name6', namespace='Wauplin', repository='gpt2', status='pending', url=None)
>>> endpoint.client 
*** huggingface_hub._inference_endpoints.InferenceEndpointException: Cannot create a client for this endpoint as it is not yet deployed. Please wait for the endpoint to be deployed and try again.

# ... wait until it's initialized
>>> endpoint.wait()
InferenceEndpoint(name='my-endpoint-name6', namespace='Wauplin', repository='gpt2', status='running', url='https://kqehm5t0lfe628b2.us-east-1.aws.endpoints.huggingface.cloud')

# endpoint running => url => client available
>>> endpoint.client 
<InferenceClient(model='https://kqehm5t0lfe628b2.us-east-1.aws.endpoints.huggingface.cloud', timeout=None)>
>>> endpoint.client.text_generation("I am")
' not a fan of the idea of a "big-budget" movie. I think it\'s a'

# pause endpoint => no more url
>>> endpoint.pause()
InferenceEndpoint(name='my-endpoint-name6', namespace='Wauplin', repository='gpt2', status='paused', url=None)

---

Useful resources:

• Generated documentation from this PR:
  • guide
  • package_reference
• Swagger API docs: https://api.endpoints.huggingface.cloud/
• Inference Endpoints: https://ui.endpoints.huggingface.co/
• Inference Endpoints documentation: https://huggingface.co/docs/inference-endpoints/index

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[stevhliu]

Super nice, the huggingface_hub library is becoming so versatile! 🚀

[julien-c]

I should know this for sure but what's the ≠ between paused & scaled to zero, again?

Another question: do you have an example of how to chain .wait and an actual inference call?

would you do something like this?

create_inference_endpoint(
    "my-endpoint-name6",
    repository="gpt2",
).wait().client.text_generation("I am").scale_to_zero()

[Wauplin]

I should know this for sure but what's the ≠ between paused & scaled to zero, again?

I'll let @philschmid confirm, not sure myself what's the difference.
What I understood is that a paused endpoint must be resumed manually while a scaled to zero endpoint is automatically restarted on next call (with cold start).

would you do something like this?

Not exactly no since InferenceEndpoint != InferenceClient objects.
What you can do:

endpoint = create_inference_endpoint("my-endpoint-name6", repository="gpt2",...).wait()
endpoint.client.text_generation("I am")
endpoint.scale_to_zero()

[Wauplin]

@philschmid @stevhliu I have added a guide about how to manage Inference Endpoints from huggingface_hub. Would you mind having a look at it? 🙏

[stevhliu]

Nice guide, especially like the end-to-end example at the end that puts everything together in context :)

[McPatate]

Few nits, but lgtm overall!

Nice job!

[McPatate]

To deploy any kind of model, I don't think we're limited to the libraries you mention here.
cc @philschmid

[Wauplin]

I took that part from the official documentation there: https://huggingface.co/docs/inference-endpoints/main/en/index

[philschmid]

those models are what we defaultly support without custom handler or container. But not limiting.

[McPatate]

Maybe wording could be improved, when I read this at first it's not clear to me that you can run any model

[Wauplin]

I've changed the wording in 1688851 to be more generic:

Inference Endpoints provides a secure production solution to easily deploy models on a dedicated and autoscaling infrastructure managed by Hugging Face. An Inference Endpoint is built from a model from the Hub. This page is a reference for huggingface_hub's integration with Inference Endpoints. For more information about the Inference Endpoints product, check out its official documentation.

Keeping it vague on purpose. If the user wants more details about supported models, the official Inference Endpoints documentation should be the appropriate location to list them. These docs are referenced twice from the huggingface_hub's doc (from this PR) so should be fine.

[Wauplin]

Thanks @stevhliu and @McPatate for your feedback on the documentation, It helped a lot! I've made the requested changes :)

[LysandreJik]

Looks awesome! Only left a few nits. The API is intuitive.

[Wauplin]

Thanks @LysandreJik for the review! I think we are good to merge when CI is green then :)
Thanks everyone here for the feedback on this PR! 🤗

[Wauplin]

Failing tests are unrelated. Merging this!