Skip to content

Commit

Permalink
[skip ci] Updates
Browse files Browse the repository at this point in the history
  • Loading branch information
airflow-oss-bot committed Jul 16, 2024
1 parent 8fab9f0 commit d6c929b
Show file tree
Hide file tree
Showing 19 changed files with 77 additions and 77 deletions.
20 changes: 10 additions & 10 deletions README.html
Original file line number Diff line number Diff line change
Expand Up @@ -145,9 +145,9 @@
<li class="toctree-l1"><a class="reference internal" href="api/google_firestore.html">Setting Up Firestore for the Ask Astro Application on Google Cloud Run</a></li>
<li class="toctree-l1"><a class="reference internal" href="api/setup_slack_bot.html">Create an ask-astro Slack bot</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">💻 Develop Airflow DAGs</span></p>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">💻 Develop Apache Airflow® DAGs</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="airflow/README.html">ask-astro Airflow</a></li>
<li class="toctree-l1"><a class="reference internal" href="airflow/README.html">ask-astro Apache Airflow®</a></li>
</ul>

</div>
Expand Down Expand Up @@ -302,7 +302,7 @@ <h2> Contents </h2>
<li class="toc-h1 nav-item toc-entry"><a class="reference internal nav-link" href="#deployment">Deployment</a><ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#backend-api">Backend API</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#user-interface">User Interface</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#apache-airflow">Apache Airflow</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#apache-airflow">Apache Airflow®</a></li>
</ul>
</li>
<li class="toc-h1 nav-item toc-entry"><a class="reference internal nav-link" href="#slackbot">Slackbot</a></li>
Expand All @@ -322,7 +322,7 @@ <h2> Contents </h2>
<img src="_static/logo.svg" />
</p>
<br />
<p>Ask Astro is an open-source reference implementation of <a class="reference external" href="https://a16z.com/emerging-architectures-for-llm-applications/">Andreessen Horowitz’s LLM Application Architecture</a> built by <a class="reference external" href="https://astronomer.io">Astronomer</a>. It provides an end-to-end example of a Q&amp;A LLM application used to answer questions about Apache Airflow and Astronomer, including:</p>
<p>Ask Astro is an open-source reference implementation of <a class="reference external" href="https://a16z.com/emerging-architectures-for-llm-applications/">Andreessen Horowitz’s LLM Application Architecture</a> built by <a class="reference external" href="https://astronomer.io">Astronomer</a>. It provides an end-to-end example of a Q&amp;A LLM application used to answer questions about <a class="reference external" href="https://airflow.apache.org/">Apache Airflow®</a> and Astronomer, including:</p>
<ul class="simple">
<li><p>Airflow DAGs for data ingestion from various sources to a vector database</p></li>
<li><p>An API containing the business logic of taking a user’s prompt and ultimately generating a response</p></li>
Expand All @@ -340,11 +340,11 @@ <h1>Data Retrieval &amp; Embedding<a class="headerlink" href="#data-retrieval-em
<p>In order to make the responses as factual and accurate as possible, it’s generally best practice to use <a class="reference external" href="https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html">Retrieval Augmented Generation (RAG)</a>. However, in order for RAG to be effective, a vector database needs to be populated with the most up-to-date and relevant information.</p>
<p>Ask Astro uses a set of Airflow DAGs that: ingest data from a source via an API or Python library, preprocesses and splits the data into smaller chunks, embeds those chunks, and writes the embeddings to Weaviate. As of today, Ask Astro retrieves data from the following sources:</p>
<ul class="simple">
<li><p><a class="reference external" href="https://airflow.apache.org/docs/">Apache Airflow docs</a></p></li>
<li><p><a class="reference external" href="https://airflow.apache.org/docs/">Apache Airflow® docs</a></p></li>
<li><p><a class="reference external" href="https://docs.astronomer.io">Astronomer docs</a></p></li>
<li><p><a class="reference external" href="https://www.astronomer.io/blog/">Astronomer blog</a></p></li>
<li><p><a class="reference external" href="https://registry.astronomer.io">Astronomer Registry</a></p></li>
<li><p><a class="reference external" href="https://github.com/apache/airflow">Apache Airflow GitHub</a> pull requests</p></li>
<li><p><a class="reference external" href="https://github.com/apache/airflow">Apache Airflow® GitHub</a> pull requests</p></li>
<li><p><a class="reference external" href="https://github.com/OpenLineage/OpenLineage">OpenLineage GitHub</a></p></li>
<li><p><a class="reference external" href="https://github.com/OpenLineage/docs">OpenLineage GitHub docs</a></p></li>
<li><p><a class="reference external" href="https://archive.org/details/stackexchange">StackOverflow’s Stack Exchange Data Dump</a></p></li>
Expand Down Expand Up @@ -409,7 +409,7 @@ <h1>Local dev env<a class="headerlink" href="#local-dev-env" title="Permalink to
<ul class="simple">
<li><p><a class="reference internal" href="ui/README.html"><span class="std std-doc">Run ask-astro UI</span></a></p></li>
</ul>
<p><strong>Apache Airflow</strong></p>
<p><strong>Apache Airflow®</strong></p>
<ul class="simple">
<li><p><a class="reference internal" href="airflow/README.html"><span class="std std-doc">Run airflow</span></a></p></li>
</ul>
Expand All @@ -431,9 +431,9 @@ <h2>User Interface<a class="headerlink" href="#user-interface" title="Permalink
</ul>
</section>
<section id="apache-airflow">
<h2>Apache Airflow<a class="headerlink" href="#apache-airflow" title="Permalink to this heading">#</a></h2>
<h2>Apache Airflow®<a class="headerlink" href="#apache-airflow" title="Permalink to this heading">#</a></h2>
<ul class="simple">
<li><p><a class="reference internal" href="airflow/README.html"><span class="std std-doc">Airflow</span></a></p></li>
<li><p><a class="reference internal" href="airflow/README.html"><span class="std std-doc">Apache Airflow®</span></a></p></li>
</ul>
</section>
</section>
Expand Down Expand Up @@ -478,7 +478,7 @@ <h1>Slackbot<a class="headerlink" href="#slackbot" title="Permalink to this head
<li class="toc-h1 nav-item toc-entry"><a class="reference internal nav-link" href="#deployment">Deployment</a><ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#backend-api">Backend API</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#user-interface">User Interface</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#apache-airflow">Apache Airflow</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#apache-airflow">Apache Airflow®</a></li>
</ul>
</li>
<li class="toc-h1 nav-item toc-entry"><a class="reference internal nav-link" href="#slackbot">Slackbot</a></li>
Expand Down
12 changes: 6 additions & 6 deletions _sources/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

<br />

Ask Astro is an open-source reference implementation of [Andreessen Horowitz's LLM Application Architecture](https://a16z.com/emerging-architectures-for-llm-applications/) built by [Astronomer](https://astronomer.io). It provides an end-to-end example of a Q&A LLM application used to answer questions about Apache Airflow and Astronomer, including:
Ask Astro is an open-source reference implementation of [Andreessen Horowitz's LLM Application Architecture](https://a16z.com/emerging-architectures-for-llm-applications/) built by [Astronomer](https://astronomer.io). It provides an end-to-end example of a Q&A LLM application used to answer questions about [Apache Airflow®](https://airflow.apache.org/) and Astronomer, including:

- Airflow DAGs for data ingestion from various sources to a vector database
- An API containing the business logic of taking a user's prompt and ultimately generating a response
Expand All @@ -25,11 +25,11 @@ In order to make the responses as factual and accurate as possible, it's general

Ask Astro uses a set of Airflow DAGs that: ingest data from a source via an API or Python library, preprocesses and splits the data into smaller chunks, embeds those chunks, and writes the embeddings to Weaviate. As of today, Ask Astro retrieves data from the following sources:

- [Apache Airflow docs](https://airflow.apache.org/docs/)
- [Apache Airflow® docs](https://airflow.apache.org/docs/)
- [Astronomer docs](https://docs.astronomer.io)
- [Astronomer blog](https://www.astronomer.io/blog/)
- [Astronomer Registry](https://registry.astronomer.io)
- [Apache Airflow GitHub](https://github.com/apache/airflow) pull requests
- [Apache Airflow® GitHub](https://github.com/apache/airflow) pull requests
- [OpenLineage GitHub](https://github.com/OpenLineage/OpenLineage)
- [OpenLineage GitHub docs](https://github.com/OpenLineage/docs)
- [StackOverflow's Stack Exchange Data Dump](https://archive.org/details/stackexchange)
Expand Down Expand Up @@ -99,7 +99,7 @@ python3 scripts/local_dev.py run-airflow # To run Airflow

- [Run ask-astro UI](./ui/README.md)

**Apache Airflow**
**Apache Airflow®**

- [Run airflow](./airflow/README.md)

Expand All @@ -113,8 +113,8 @@ The following sections describe how to deploy the various components of Ask Astr
### User Interface
- [UI](./ui/README.md)

### Apache Airflow
- [Airflow](./airflow/README.md)
### Apache Airflow®
- [Apache Airflow®](./airflow/README.md)

## Slackbot
- [Slackbot](./api/setup_slack_bot.md)
10 changes: 5 additions & 5 deletions _sources/airflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,23 @@

<br />

# ask-astro Airflow
# ask-astro Apache Airflow®

Ask Astro is an open-source reference implementation of [Andreessen Horowitz's LLM Application Architecture](https://a16z.com/emerging-architectures-for-llm-applications/) built by [Astronomer](https://astronomer.io). It provides an end-to-end example of a Q&A LLM application used to answer questions about Apache Airflow and Astronomer.
Ask Astro is an open-source reference implementation of [Andreessen Horowitz's LLM Application Architecture](https://a16z.com/emerging-architectures-for-llm-applications/) built by [Astronomer](https://astronomer.io). It provides an end-to-end example of a Q&A LLM application used to answer questions about [Apache Airflow®](https://airflow.apache.org/) and Astronomer.

## Ingest and Embedding
In order to make the responses as factual and accurate as possible, Ask Astro uses [Retrieval Augmented Generation (RAG)](https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html). Use the following setup procedures to ingest and embed documents for RAG with Apache Airflow and Astro.

Ask Astro uses a set of Airflow DAGs that ingests data from a source via an API or Python library, preprocesses and splits the data into smaller chunks, embeds those chunks, and writes the embeddings to Weaviate. Ask Astro retrieves data from the following sources:

- [Apache Airflow docs](https://airflow.apache.org/docs/)
- [Apache Airflow® docs](https://airflow.apache.org/docs/)
- [Astronomer docs](https://docs.astronomer.io)
- [Astronomer blog](https://www.astronomer.io/blog/)
- [Astronomer Registry](https://registry.astronomer.io)
- [Apache Airflow GitHub](https://github.com/apache/airflow) issues and pull requests
- [Apache Airflow® GitHub](https://github.com/apache/airflow) issues and pull requests
- [OpenLineage GitHub](https://github.com/OpenLineage/OpenLineage)
- [OpenLineage GitHub docs](https://github.com/OpenLineage/docs)
- Apache Airflow Slack's [#troubleshooting channel](https://app.slack.com/client/TCQ18L22Z/CCQ7EGB1P)
- Apache Airflow® Slack's [#troubleshooting channel](https://app.slack.com/client/TCQ18L22Z/CCQ7EGB1P)
- [StackOverflow's Stack Exchange Data Dump](https://archive.org/details/stackexchange)
- [astro-cli](https://docs.astronomer.io/astro/cli/overview)

Expand Down
2 changes: 1 addition & 1 deletion _sources/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ api/setup_slack_bot.md

```{toctree}
:hidden:
:caption: 💻 Develop Airflow DAGs
:caption: 💻 Develop Apache Airflow® DAGs
airflow/README.md
```
2 changes: 1 addition & 1 deletion _sources/local_development.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ $ poetry install

![docs.generate-tasks-help-screenshot](_static/images/task_help_message/docs-generate-tasks-help-screenshot.svg)

### Airflow DAGs Tasks
### Apache Airflow® DAGs Tasks

- Run ask-astro airflow

Expand Down
6 changes: 3 additions & 3 deletions _sources/monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ This document outlines the monitoring and observability practices implemented fo

## Infrastructure Monitoring

### Airflow DAGs
### Apache Airflow® DAGs

- **Monitoring DAG**: This [DAG](../airflow/dags/monitor/monitor.py) runs every 5 minutes to check the health of all infrastructure components.
- **Ingestion DAG Monitoring**: This [DAG](../airflow/dags/monitor/monitor_ingestion_dags.py) monitors the health of ingestion processes.

- ![Airflow DAGs](./_static/images/monitoring/airflow_dags.png)
- ![Apache Airflow® DAGs](./_static/images/monitoring/airflow_dags.png)

### Alerting

Expand All @@ -31,7 +31,7 @@ This document outlines the monitoring and observability practices implemented fo
4. **UI Monitoring**: Regular checks of the UI for a 200 response status.
- UI Link: [https://ask.astronomer.io/](https://ask.astronomer.io/)

5. **Airflow Data Ingestion and Feedback DAGs**: Monitors for completeness and errors.
5. **Apache Airflow® Data Ingestion and Feedback DAGs**: Monitors for completeness and errors.

6. **Open AI Integration**: Ensures the availability and response quality of Open AI services.

Expand Down
20 changes: 10 additions & 10 deletions airflow/README.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

<title>ask-astro Airflow &#8212; ask-astro 0.0.1 documentation</title>
<title>ask-astro Apache Airflow® &#8212; ask-astro 0.0.1 documentation</title>



Expand Down Expand Up @@ -146,9 +146,9 @@
<li class="toctree-l1"><a class="reference internal" href="../api/google_firestore.html">Setting Up Firestore for the Ask Astro Application on Google Cloud Run</a></li>
<li class="toctree-l1"><a class="reference internal" href="../api/setup_slack_bot.html">Create an ask-astro Slack bot</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">💻 Develop Airflow DAGs</span></p>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">💻 Develop Apache Airflow® DAGs</span></p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active"><a class="current reference internal" href="#">ask-astro Airflow</a></li>
<li class="toctree-l1 current active"><a class="current reference internal" href="#">ask-astro Apache Airflow®</a></li>
</ul>

</div>
Expand Down Expand Up @@ -285,7 +285,7 @@


<div id="jb-print-docs-body" class="onlyprint">
<h1>ask-astro Airflow</h1>
<h1>ask-astro Apache Airflow®</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
Expand Down Expand Up @@ -315,22 +315,22 @@ <h2> Contents </h2>
<img src="../_static/ingestion.png" />
</p>
<br />
<section id="ask-astro-airflow">
<h1>ask-astro Airflow<a class="headerlink" href="#ask-astro-airflow" title="Permalink to this heading">#</a></h1>
<p>Ask Astro is an open-source reference implementation of <a class="reference external" href="https://a16z.com/emerging-architectures-for-llm-applications/">Andreessen Horowitz’s LLM Application Architecture</a> built by <a class="reference external" href="https://astronomer.io">Astronomer</a>. It provides an end-to-end example of a Q&amp;A LLM application used to answer questions about Apache Airflow and Astronomer.</p>
<section id="ask-astro-apache-airflow">
<h1>ask-astro Apache Airflow®<a class="headerlink" href="#ask-astro-apache-airflow" title="Permalink to this heading">#</a></h1>
<p>Ask Astro is an open-source reference implementation of <a class="reference external" href="https://a16z.com/emerging-architectures-for-llm-applications/">Andreessen Horowitz’s LLM Application Architecture</a> built by <a class="reference external" href="https://astronomer.io">Astronomer</a>. It provides an end-to-end example of a Q&amp;A LLM application used to answer questions about <a class="reference external" href="https://airflow.apache.org/">Apache Airflow®</a> and Astronomer.</p>
<section id="ingest-and-embedding">
<h2>Ingest and Embedding<a class="headerlink" href="#ingest-and-embedding" title="Permalink to this heading">#</a></h2>
<p>In order to make the responses as factual and accurate as possible, Ask Astro uses <a class="reference external" href="https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html">Retrieval Augmented Generation (RAG)</a>. Use the following setup procedures to ingest and embed documents for RAG with Apache Airflow and Astro.</p>
<p>Ask Astro uses a set of Airflow DAGs that ingests data from a source via an API or Python library, preprocesses and splits the data into smaller chunks, embeds those chunks, and writes the embeddings to Weaviate. Ask Astro retrieves data from the following sources:</p>
<ul class="simple">
<li><p><a class="reference external" href="https://airflow.apache.org/docs/">Apache Airflow docs</a></p></li>
<li><p><a class="reference external" href="https://airflow.apache.org/docs/">Apache Airflow® docs</a></p></li>
<li><p><a class="reference external" href="https://docs.astronomer.io">Astronomer docs</a></p></li>
<li><p><a class="reference external" href="https://www.astronomer.io/blog/">Astronomer blog</a></p></li>
<li><p><a class="reference external" href="https://registry.astronomer.io">Astronomer Registry</a></p></li>
<li><p><a class="reference external" href="https://github.com/apache/airflow">Apache Airflow GitHub</a> issues and pull requests</p></li>
<li><p><a class="reference external" href="https://github.com/apache/airflow">Apache Airflow® GitHub</a> issues and pull requests</p></li>
<li><p><a class="reference external" href="https://github.com/OpenLineage/OpenLineage">OpenLineage GitHub</a></p></li>
<li><p><a class="reference external" href="https://github.com/OpenLineage/docs">OpenLineage GitHub docs</a></p></li>
<li><p>Apache Airflow Slack’s <a class="reference external" href="https://app.slack.com/client/TCQ18L22Z/CCQ7EGB1P">#troubleshooting channel</a></p></li>
<li><p>Apache Airflow® Slack’s <a class="reference external" href="https://app.slack.com/client/TCQ18L22Z/CCQ7EGB1P">#troubleshooting channel</a></p></li>
<li><p><a class="reference external" href="https://archive.org/details/stackexchange">StackOverflow’s Stack Exchange Data Dump</a></p></li>
<li><p><a class="reference external" href="https://docs.astronomer.io/astro/cli/overview">astro-cli</a></p></li>
</ul>
Expand Down
4 changes: 2 additions & 2 deletions api/README.html
Original file line number Diff line number Diff line change
Expand Up @@ -147,9 +147,9 @@
<li class="toctree-l1"><a class="reference internal" href="google_firestore.html">Setting Up Firestore for the Ask Astro Application on Google Cloud Run</a></li>
<li class="toctree-l1"><a class="reference internal" href="setup_slack_bot.html">Create an ask-astro Slack bot</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">💻 Develop Airflow DAGs</span></p>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">💻 Develop Apache Airflow® DAGs</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../airflow/README.html">ask-astro Airflow</a></li>
<li class="toctree-l1"><a class="reference internal" href="../airflow/README.html">ask-astro Apache Airflow®</a></li>
</ul>

</div>
Expand Down
4 changes: 2 additions & 2 deletions api/cloudbuild_and_run.html
Original file line number Diff line number Diff line change
Expand Up @@ -147,9 +147,9 @@
<li class="toctree-l1"><a class="reference internal" href="google_firestore.html">Setting Up Firestore for the Ask Astro Application on Google Cloud Run</a></li>
<li class="toctree-l1"><a class="reference internal" href="setup_slack_bot.html">Create an ask-astro Slack bot</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">💻 Develop Airflow DAGs</span></p>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">💻 Develop Apache Airflow® DAGs</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../airflow/README.html">ask-astro Airflow</a></li>
<li class="toctree-l1"><a class="reference internal" href="../airflow/README.html">ask-astro Apache Airflow®</a></li>
</ul>

</div>
Expand Down
Loading

0 comments on commit d6c929b

Please sign in to comment.