Docs: Adding Metapilot Section (#18213)

openmetadata-docs/content/v1.5.x/Metapilot/how-to-use-metapilot.md

@@ -0,0 +1,110 @@
+---
+title: How to Use MetaPilot
+slug: /metapilot/how-to-use-metapilot
+collate: true
+---
+
+# How to Use MetaPilot
+
+## 1. Setting Up MetaPilot
+- Navigate to **Settings > Applications** in the Collate platform.
+
+{% image
+src="/images/v1.5/metapilot/setting-up-metapilot-1.png"
+alt="setting up metapilot"
+caption="Navigate to Settings > Applications"
+/%}
+
+- Install MetaPilot by following the on-screen instructions.
+
+{% image
+src="/images/v1.5/metapilot/setting-up-metapilot-2.png"
+alt="Install MetaPilot"
+caption="Install MetaPilot"
+/%}
+
+- Select the databases for which you want MetaPilot to automatically generate descriptions. You can configure multiple databases and set a default database for the MetaPilot chatbot to work with.
+
+{% image
+src="/images/v1.5/metapilot/setting-up-metapilot-3.png"
+alt="automatically generate descriptions"
+caption="automatically generate descriptions"
+/%}
+
+- **Scheduling**: Schedule MetaPilot to run regularly, automatically generating metadata at predefined intervals (e.g., weekly).
+
+{% image
+src="/images/v1.5/metapilot/setting-up-metapilot-4.png"
+alt="Schedule MetaPilot"
+caption="Schedule MetaPilot"
+/%}
+
+## 2. Using the MetaPilot Chatbot
+- The MetaPilot chatbot icon appears on every page after installation.
+
+{% image
+src="/images/v1.5/metapilot/using-metapilot-chatbot-1.png"
+alt="chatbot icon"
+caption="chatbot icon"
+/%}
+
+- Interact with the chatbot by typing natural language questions. For example:
+  - “Show me sales data for Q1.”
+  - “What is the average revenue per customer?”
+
+- MetaPilot will generate the corresponding SQL query and provide a detailed explanation of the query logic.
+
+{% image
+src="/images/v1.5/metapilot/using-metapilot-chatbot-2.png"
+alt="natural language questions"
+caption="natural language questions"
+/%}
+
+- Users can refine queries by providing further instructions, and the chatbot will adjust the SQL query accordingly.
+
+## 3. Optimizing and Fixing SQL Queries
+
+- The **Metadata Usage** workflows will ingest the queries being run against the tables. You can see how long each query has been executed in the **Queries** tab.
+
+{% image
+src="/images/v1.5/metapilot/fixing-sql-queries-1.png"
+alt="metadata usage workflows"
+caption="metadata usage workflows"
+/%}
+
+- If a query runs inefficiently, ask the chatbot to optimize it by typing: “Optimize this query.”
+
+{% image
+src="/images/v1.5/metapilot/fixing-sql-queries-3.png"
+alt="Optimize this query"
+caption="Optimize this query"
+/%}
+
+- MetaPilot will return a more efficient version of the SQL query, which you can then copy and execute in your database.
+
+- If the query contains errors or isn’t functioning correctly, ask MetaPilot: “Can you fix this query?” 
+
+- MetaPilot will correct the query and provide a working version.
+
+{% image
+src="/images/v1.5/metapilot/fixing-sql-queries-4.png"
+alt="Can you fix this query"
+caption="Can you fix this query"
+/%}
+
+## 4. Reviewing Generated Metadata
+- Once MetaPilot generates descriptions for tables and columns, navigate to the **database view** to review the metadata.
+- You can accept or reject each suggestion individually or choose to accept all suggestions in bulk.
+
+{% image
+src="/images/v1.5/metapilot/reviewing-generated-metadata.png"
+alt="reviewing Generated Metadata"
+caption="reviewing Generated Metadata"
+/%}
+
+- MetaPilot allows you to document entire datasets in a matter of minutes, significantly reducing the manual effort required to maintain metadata.
+
+## Best Practices
+- **Regular Scheduling**: Schedule MetaPilot to run at regular intervals to ensure your metadata is always up-to-date, especially when dealing with frequently changing datasets.
+- **Leverage the Chatbot for Query Writing**: Encourage both technical and non-technical users to use the MetaPilot chatbot for SQL query generation. It simplifies complex query writing and ensures accurate results.
+- **Optimize Queries Regularly**: Monitor the performance of your queries and use MetaPilot to optimize them, especially when working with large datasets or queries that require complex joins and filters.

openmetadata-docs/content/v1.5.x/Metapilot/index.md

@@ -0,0 +1,85 @@
+---
+title: Metapilot
+slug: /metapilot
+collate: true
+---
+
+# Metapilot Technical Documentation
+
+{%  youtube videoId="6glMYLzxNqk" start="0:00" end="04:20” width="560px" height="315px" /%}
+
+MetaPilot is an AI-powered tool within Collate that simplifies and enhances metadata management. By integrating generative AI, MetaPilot assists users in automating the documentation of data assets, writing and optimizing SQL queries, and interacting with data through natural language. This first-of-its-kind data copilot improves productivity by automating tedious tasks and providing intelligent insights into your data environment.
+
+## Key Features
+
+- **Automated Data Documentation**: Automatically generates metadata descriptions for tables and columns, saving time and effort for data owners and stewards.
+- **Natural Language SQL Query Generation**: Allows users to interact with MetaPilot through a chatbot to generate SQL queries by simply asking questions in plain English.
+- **SQL Query Optimization and Fixing**: Capable of optimizing and troubleshooting SQL queries to improve their performance and efficiency.
+
+## Why MetaPilot is Useful
+
+### Metadata Management Challenges
+
+Managing metadata across multiple data assets can be overwhelming due to the influx of new data and changing team dynamics. MetaPilot addresses these challenges by:
+
+- Automating metadata description generation.
+- Simplifying the creation and optimization of SQL queries.
+- Reducing manual effort and enhancing data quality.
+
+### Time-Saving Features
+
+Documenting thousands of tables manually is tedious and time-consuming. MetaPilot automates metadata generation, allowing data teams to focus on high-value tasks and ensuring that data assets are consistently documented and understood across the organization.
+
+## Use Cases
+
+### 1. Automatic Data Asset Documentation
+
+{% image
+src="/images/v1.5/metapilot/reviewing-generated-metadata.png"
+alt="Automatic Data Asset Documentation"
+caption="Auto Generate data Asset Documentation"
+/%}
+
+- **Problem**: Manually creating metadata descriptions for large datasets is labor-intensive and error-prone.
+- **Solution**: MetaPilot’s generative AI automates the process, analyzing database structures and suggesting accurate descriptions for tables and columns.
+- **How It Works**: After configuring MetaPilot, it scans the database schema and generates metadata descriptions automatically. Users can review these descriptions and approve or reject them in bulk or individually.
+- **Benefit**: Streamlines the documentation process, ensuring consistent and up-to-date metadata across all datasets.
+
+### 2. Natural Language SQL Query Generation
+
+{% image
+src="/images/v1.5/metapilot/using-metapilot-chatbot-2.png"
+alt="Natural Language SQL Query Generation"
+caption="Natural Language SQL Query Generation"
+/%}
+
+- **Problem**: Non-technical users often struggle with writing SQL queries to extract insights from databases.
+- **Solution**: MetaPilot’s chatbot allows users to ask questions in natural language, generating SQL queries and providing explanations to help extract the required data quickly and easily.
+- **How It Works**: Users can click on the MetaPilot chatbot widget and ask questions like "Show me sales data from last quarter." MetaPilot generates the corresponding SQL query and explains its logic.
+- **Benefit**: Democratizes data access by enabling users of all technical levels to interact with data without deep SQL knowledge.
+
+### 3. SQL Query Optimization and Troubleshooting
+
+{% image
+src="/images/v1.5/metapilot/fixing-sql-queries-3.png"
+alt="SQL Query Optimization and Troubleshooting"
+caption="SQL Query Optimization"
+/%}
+
+- **Problem**: SQL queries can become complex and inefficient, leading to performance issues and increased costs.
+- **Solution**: MetaPilot optimizes inefficient queries to improve performance, saving both time and resources.
+- **How It Works**: If a query runs too long, users can request MetaPilot to optimize it. MetaPilot provides a more efficient SQL query version that can be implemented immediately.
+- **Benefit**: Enhances query performance, reduces costs associated with inefficient queries, and speeds up data processing.
+
+### 4. Fixing SQL Queries
+
+{% image
+src="/images/v1.5/metapilot/fixing-sql-queries-4.png"
+alt="Fixing SQL Queries"
+caption="Fixing SQL Queries"
+/%}
+
+- **Problem**: Complex queries can cause issues even for SQL experts.
+- **Solution**: MetaPilot can fix problematic SQL queries, ensuring they run correctly and efficiently.
+- **How It Works**: Users can ask MetaPilot, "Can you fix this query for me?" It analyzes the query, detects issues, and returns a corrected version ready for use.
+- **Benefit**: Simplifies writing and maintaining queries, allowing data teams to focus on analysis rather than troubleshooting.

openmetadata-docs/content/v1.5.x/collate-menu.md

@@ -1860,4 +1860,8 @@ site_menu:
     url: /sdk/java
   - category: SDK / Go
     url: /sdk/go
+  - category: Metapilot
+    url: /metapilot
+  - category: Metapilot / How to Use MetaPilot
+    url: /metapilot/how-to-use-metapilot
 ---

openmetadata-docs/content/v1.6.x-SNAPSHOT/Metapilot/how-to-use-metapilot.md

@@ -0,0 +1,110 @@
+---
+title: How to Use MetaPilot
+slug: /metapilot/how-to-use-metapilot
+collate: true
+---
+
+# How to Use MetaPilot
+
+## 1. Setting Up MetaPilot
+- Navigate to **Settings > Applications** in the Collate platform.
+
+{% image
+src="/images/v1.6/metapilot/setting-up-metapilot-1.png"
+alt="setting up metapilot"
+caption="Navigate to Settings > Applications"
+/%}
+
+- Install MetaPilot by following the on-screen instructions.
+
+{% image
+src="/images/v1.6/metapilot/setting-up-metapilot-2.png"
+alt="Install MetaPilot"
+caption="Install MetaPilot"
+/%}
+
+- Select the databases for which you want MetaPilot to automatically generate descriptions. You can configure multiple databases and set a default database for the MetaPilot chatbot to work with.
+
+{% image
+src="/images/v1.6/metapilot/setting-up-metapilot-3.png"
+alt="automatically generate descriptions"
+caption="automatically generate descriptions"
+/%}
+
+- **Scheduling**: Schedule MetaPilot to run regularly, automatically generating metadata at predefined intervals (e.g., weekly).
+
+{% image
+src="/images/v1.6/metapilot/setting-up-metapilot-4.png"
+alt="Schedule MetaPilot"
+caption="Schedule MetaPilot"
+/%}
+
+## 2. Using the MetaPilot Chatbot
+- The MetaPilot chatbot icon appears on every page after installation.
+
+{% image
+src="/images/v1.6/metapilot/using-metapilot-chatbot-1.png"
+alt="chatbot icon"
+caption="chatbot icon"
+/%}
+
+- Interact with the chatbot by typing natural language questions. For example:
+  - “Show me sales data for Q1.”
+  - “What is the average revenue per customer?”
+
+- MetaPilot will generate the corresponding SQL query and provide a detailed explanation of the query logic.
+
+{% image
+src="/images/v1.6/metapilot/using-metapilot-chatbot-2.png"
+alt="natural language questions"
+caption="natural language questions"
+/%}
+
+- Users can refine queries by providing further instructions, and the chatbot will adjust the SQL query accordingly.
+
+## 3. Optimizing and Fixing SQL Queries
+
+- The **Metadata Usage** workflows will ingest the queries being run against the tables. You can see how long each query has been executed in the **Queries** tab.
+
+{% image
+src="/images/v1.6/metapilot/fixing-sql-queries-1.png"
+alt="metadata usage workflows"
+caption="metadata usage workflows"
+/%}
+
+- If a query runs inefficiently, ask the chatbot to optimize it by typing: “Optimize this query.”
+
+{% image
+src="/images/v1.6/metapilot/fixing-sql-queries-3.png"
+alt="Optimize this query"
+caption="Optimize this query"
+/%}
+
+- MetaPilot will return a more efficient version of the SQL query, which you can then copy and execute in your database.
+
+- If the query contains errors or isn’t functioning correctly, ask MetaPilot: “Can you fix this query?” 
+
+- MetaPilot will correct the query and provide a working version.
+
+{% image
+src="/images/v1.6/metapilot/fixing-sql-queries-4.png"
+alt="Can you fix this query"
+caption="Can you fix this query"
+/%}
+
+## 4. Reviewing Generated Metadata
+- Once MetaPilot generates descriptions for tables and columns, navigate to the **database view** to review the metadata.
+- You can accept or reject each suggestion individually or choose to accept all suggestions in bulk.
+
+{% image
+src="/images/v1.6/metapilot/reviewing-generated-metadata.png"
+alt="reviewing Generated Metadata"
+caption="reviewing Generated Metadata"
+/%}
+
+- MetaPilot allows you to document entire datasets in a matter of minutes, significantly reducing the manual effort required to maintain metadata.
+
+## Best Practices
+- **Regular Scheduling**: Schedule MetaPilot to run at regular intervals to ensure your metadata is always up-to-date, especially when dealing with frequently changing datasets.
+- **Leverage the Chatbot for Query Writing**: Encourage both technical and non-technical users to use the MetaPilot chatbot for SQL query generation. It simplifies complex query writing and ensures accurate results.
+- **Optimize Queries Regularly**: Monitor the performance of your queries and use MetaPilot to optimize them, especially when working with large datasets or queries that require complex joins and filters.