When and how can the Gemini API be monitored in Datadog?

[Gekko0114]

Seems like Gemini is supported in dd-trace-py starting from version 2.14.
https://github.com/DataDog/dd-trace-py/releases/tag/v2.14.0

However, I can't find Gemini integration dashboard. Also I can't monitor Gemini logs on Datadog LLM observability.
When and how can the Gemini API be monitored in Datadog?

Also, I want dd-trace-py to support the Gemini API via vertexai library.
https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal

[Yun-Kim]

Hi @Gekko0114, thanks for reaching out! We have plans to build and release an out-of-the-box integration dashboard for the Gemini integration (for LLM Observability users) in the next couple months.

To monitor Gemini prompt/completions/traces via Datadog's LLM Observability feature, you can go to LLM Observability --> Traces and select the Model Provider: Google facet filter on the left side of the traces page.

The VertexAI library is currently on our backlog of libraries to integrate with, please look out for it in the next coming months.

[Gekko0114]

Thanks @Yun-Kim,

I have additional questions.
Q1: Do you plan to enable integration dashboard for gemini similar to OpenAI? It would be very useful for me.
https://docs.datadoghq.com/integrations/openai/?tab=python
Q2: Can engineers outside of DataDog like me contribute dd-trace-py? If yes, I might be able to help implementing support of gemini via vertexai library.

[Yun-Kim]

We have plans to build and release an out-of-the-box integration dashboard for the Gemini integration (for LLM Observability users) in the next couple months.
Do you plan to enable integration dashboard for gemini similar to OpenAI? It would be very useful for me.

@Gekko0114 Apologies for the wrong information. Instead of releasing out-of-the-box dashboards for every integration that we release, LLM Observability users can instead go to the LLM Observability Overview Dashboard (link) or LLM Observability --> Dashboards --> LLM Overview.

On the Overview Dashboard, you can set model_provider (global scope variable at the top of the dashboard) to any values of interest that you'd like to filter by, and all relevant metrics and graphs for that model provider will be displayed in the LLM Calls section.

Can engineers outside of DataDog like me contribute dd-trace-py?

Yes, please check out the contributing docs first! Just a note that the vertexAI integration is next on our to-do list of integrations this quarter.

I'll be closing this issue for now, but if you have any further questions feel free to open a new issue.