Add Prompt Depth Anything Model

[haotongl]

What does this PR do?

This PR adds the Prompt Depth Anything Model. Prompt Depth Anything builds upon Depth Anything V2 and incorporates metric prompt depth to enable accurate and high-resolution metric depth estimation.

The implementation leverages Modular Transformers. The main file can be found here.

Before submitting

• [ N/A] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [ ✅] Did you read the [contributor guideline] (https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
  Pull Request section?
• [ N/A] Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• [ ✅] Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• [ ✅] Did you write any new necessary tests?

[haotongl]

@NielsRogge @qubvel @pcuenca Could you help review this PR when you have some time? Thanks so much in advance! Let me know if you have any questions or suggestions. 😊

[qubvel]

Hi @haotongl! Thanks for working on the model integration to transformers 🤗 I'm on holidays until Jan 3rd, and I'll do a review after that if it's still necessary.

[haotongl]

Hi, @xenova @NielsRogge ! All suggestions have been addressed. Could you please take another look and provide any further suggestions, or go ahead and merge this PR? Thanks!

[qubvel]

Yes, only members can run slow tests

[github-actions]

This comment contains run-slow, running the specified jobs: ['models/prompt_depth_anything'] ...

[qubvel]

run-slow: prompt_depth_anything

[github-actions]

This comment contains run-slow, running the specified jobs: ['models/prompt_depth_anything'] ...

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[haotongl]

run-slow: prompt_depth_anything

Thank you! All checks have passed. @qubvel

[qubvel]

Thanks for fixing the tests! Waiting for the final review form @ArthurZucker

[qubvel]

Friendly ping @ArthurZucker 🤗

[haotongl]

@ArthurZucker Hey, just a friendly ping to see if you’ve had a chance to look at the PR. Appreciate your help when you get a moment!

[ArthurZucker]

Super sorry i will get to this soon 😭

[ArthurZucker]

Kudos it's super clean 🤗
Thanks a lot @haotongl and @qubvel
Mostly missing a licence and good to go!

[ArthurZucker]

should this go in image_processor.visualize_detph_estimation directly? 🤗

[ArthurZucker]

nice thanks! 🤗

[ArthurZucker]

Suggested change

	if is_vision_available():
	pass

[ArthurZucker]

this file is missing a licence! 🤗

[ArthurZucker]

A few small conflicts to fix

[haotongl]

Kudos it's super clean 🤗 Thanks a lot @haotongl and @qubvel Mostly missing a licence and good to go!

Thanks for your reviews! I am working on these suggestions.

[qubvel]

run-slow: prompt_depth_anything

[github-actions]

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/prompt_depth_anything']
quantizations: [] ...

[haotongl]

Thanks for supporting! @qubvel @ArthurZucker

[qubvel]

Thanks for adding the model @haotongl! Can you please update model cards in HF repos to include the transformers code snippet?

[haotongl]

Thanks for adding the model @haotongl! Can you please update model cards in HF repos to include the transformers code snippet?

@qubvel Yeah, could you please provide some example repos for reference?

[qubvel]

Sure, here is an example of the model card for another depth estimation model https://huggingface.co/apple/DepthPro-hf
You can make smth similar to the model doc page - https://huggingface.co/docs/transformers/main/en/model_doc/prompt_depth_anything

[haotongl]

@qubvel Thanks for your help! Should I open a new PR to update the doc?

[qubvel]

I mean for these models:

https://huggingface.co/depth-anything/prompt-depth-anything-vits-hf
https://huggingface.co/depth-anything/prompt-depth-anything-vitl-hf
https://huggingface.co/depth-anything/prompt-depth-anything-vits-transparent-hf

model cards have not a transformers library code snippet, but it should be similar to the one in docs

import torch
import requests
import numpy as np

from PIL import Image
from transformers import AutoImageProcessor, AutoModelForDepthEstimation

url = "https://github.com/DepthAnything/PromptDA/blob/main/assets/example_images/image.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("depth-anything/prompt-depth-anything-vits-hf")
model = AutoModelForDepthEstimation.from_pretrained("depth-anything/prompt-depth-anything-vits-hf")

prompt_depth_url = "https://github.com/DepthAnything/PromptDA/blob/main/assets/example_images/arkit_depth.png?raw=true"
prompt_depth = Image.open(requests.get(prompt_depth_url, stream=True).raw)
# the prompt depth can be None, and the model will output a monocular relative depth.

# prepare image for the model
inputs = image_processor(images=image, return_tensors="pt", prompt_depth=prompt_depth)

with torch.no_grad():
    outputs = model(**inputs)

# interpolate to original size
post_processed_output = image_processor.post_process_depth_estimation(
    outputs,
    target_sizes=[(image.height, image.width)],
)

# visualize the prediction
predicted_depth = post_processed_output[0]["predicted_depth"]
depth = predicted_depth * 1000 
depth = depth.detach().cpu().numpy()
depth = Image.fromarray(depth.astype("uint16")) # mm

[haotongl]

https://huggingface.co/depth-anything/prompt-depth-anything-vits-hf
Hi, @qubvel , should it be like this?

[qubvel]

Yes, this one looks good, thanks!