From 3282d76b8e2509d3aad0a1e1e66910f143853964 Mon Sep 17 00:00:00 2001
From: Maciej Majek <maciej.majek@robotec.ai>
Date: Sun, 29 Sep 2024 18:51:54 +0200
Subject: [PATCH] docs: add tracing documentation
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Apply suggestions from code review

Co-authored-by: Adam Dąbrowski <adam.dabrowski@robotec.ai>

docs: rewrite tracing.md (prioritize open source solution), fix misleading info
---
 README.md               |  1 +
 docs/developer_guide.md |  1 +
 docs/tracing.md         | 68 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 70 insertions(+)
 create mode 100644 docs/tracing.md

diff --git a/README.md b/README.md
index 7b3db826..c6dc764e 100644
--- a/README.md
+++ b/README.md
@@ -60,6 +60,7 @@ The RAI framework aims to:
 - [x] Integration with LangChain to abstract vendors and access convenient AI tools.
 - [x] Tasks in natural language to nav2 goals.
 - [x] NoMaD integration.
+- [x] Tracing.
 - [ ] Grounded SAM 2 integration.
 - [ ] Improved Human-Robot Interaction with voice and text.
 - [ ] SDK for RAI developers.
diff --git a/docs/developer_guide.md b/docs/developer_guide.md
index c1fff3bc..7a38aee2 100644
--- a/docs/developer_guide.md
+++ b/docs/developer_guide.md
@@ -94,6 +94,7 @@ agent.invoke({"messages": ["Please pick up an object and scan it."]})
 
 Additional resources:
 
+- [Tracing](tracing.md) configuration for genAI models and agents.
 - [Beta demos](demos.md).
 - [Multimodal Messages](multimodal_messages.md) definition.
 - Available ROS 2 packages: [ros packages](ros_packages.md).
diff --git a/docs/tracing.md b/docs/tracing.md
new file mode 100644
index 00000000..01ad852c
--- /dev/null
+++ b/docs/tracing.md
@@ -0,0 +1,68 @@
+# Tracing Configuration
+
+RAI supports tracing capabilities to help monitor and analyze the performance of your AI models, at a minor performance cost. By default, tracing is off. This document outlines how to configure tracing for your RAI project.
+
+## Configuration
+
+Tracing configuration is managed through the `config.toml` file. The relevant parameters for tracing are:
+
+### Project Name
+
+The `project` field under the `[tracing]` section sets the name for your tracing project. This name will be used to identify your project in the tracing tools.
+
+> [!NOTE]  
+> Project name is currently only used by LangSmith. Langfuse will upload traces to the default project.
+
+### Langfuse (open-source)
+
+[Langfuse](https://langfuse.com/) is an open-source observability & analytics platform for LLM applications.
+
+To enable Langfuse tracing:
+
+1. Set `use_langfuse = true` in the `config.toml` file.
+2. Set the `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` environment variables with your Langfuse credentials.
+3. Optionally, you can specify a custom Langfuse host by modifying the `host` field under `[tracing.langfuse]`.
+
+### LangSmith (closed-source, paid)
+
+[LangSmith](https://www.langchain.com/langsmith) is a platform for debugging, testing, and monitoring LangChain applications.
+
+To enable LangSmith tracing:
+
+1. Set `use_langsmith = true` in the `config.toml` file.
+2. Set the `LANGCHAIN_API_KEY` environment variable with your LangSmith API key.
+
+## Usage
+
+To enable tracing in your RAI application, you need to import the get_tracing_callbacks() function and add it to the configuration when invoking your agent or model. Here's how to do it:
+
+1. First, import the get_tracing_callbacks() function:
+
+```python
+from rai.utils.model_initialization import get_tracing_callbacks
+```
+
+2. Then, add it to the configuration when invoking your agent or model:
+
+```python
+response = agent.invoke(
+    input_dict,
+    config={"callbacks": get_tracing_callbacks()}
+)
+```
+
+By adding the get_tracing_callbacks() to the config parameter, you enable tracing for that specific invocation. The get_tracing_callbacks() function returns a list of callback handlers based on your configuration in config.toml.
+
+## Troubleshooting
+
+If you encounter issues with tracing:
+
+1. Ensure all required environment variables are set correctly.
+2. Check whether tracing is on by checking whether `use_langsmith` or `use_langfuse` flag is set to `true` in `config.toml`.
+3. Verify that you have the necessary permissions and valid API keys for the tracing services you're using.
+4. Look for any error messages in your application logs related to tracing initialization.
+
+For more detailed information on using these tracing tools, refer to their respective documentation:
+
+- [LangSmith Documentation](https://docs.langchain.com/docs/langsmith)
+- [Langfuse Documentation](https://langfuse.com/docs)