Create TaskFlow API to use WeaviateHook and use taskflow API for ask-astro-load.py #132

sunank200 · 2023-11-16T13:54:12Z

I have taken airflow Docs ingestion for my implementation. Hence commits from Michael.
Create AskAstroWeaviateHook which inherits from OSS WeaviateHook.
Add get_schema, check_schema, create_schema, handle_upsert_rollback and ingest_data
Use Taskflow API for ask-astro-load.py.

closes: #134

cloudflare-workers-and-pages · 2023-11-17T21:26:04Z

Deploying with Cloudflare Pages

Latest commit:	`93ea3d8`
Status:	✅ Deploy successful!
Preview URL:	https://3db4b985.ask-astro.pages.dev
Branch Preview URL:	https://oss-weaviate-integration.ask-astro.pages.dev

View logs

airflow/Dockerfile

airflow/README.md

airflow/requirements.txt

airflow/include/airflow_provider_weaviate-0.0.1-py3-none-any.whl

airflow/dags/ingestion/ask-astro-load-airflow-docs.py

airflow/dags/ingestion/ask-astro-load-blogs.py

airflow/dags/ingestion/ask-astro-load-airflow-docs.py

airflow/dags/ingestion/ask-astro-load-github.py

airflow/dags/ingestion/ask-astro-load.py

airflow/include/tasks/extract/github.py

airflow/include/tasks/extract/stack_overflow.py

airflow/include/tasks/extract/utils/html_helpers.py

mpgreg

Some logic errors but mostly I'm concerned about readablity and generalizability. For the sake of the reference implementation we want to be very clear about DAG/task code and what should (eventually) go into the hook.

airflow/dags/ingestion/ask-astro-load.py

airflow/include/data/schema.json

airflow/include/tasks/extract/stack_overflow.py

airflow/include/tasks/ingest.py

airflow/dags/ingestion/ask-astro-load.py

airflow/include/tasks/extract/utils/weaviate/ask_astro_weaviate_hook.py

airflow/dags/ingestion/ask-astro-load.py

airflow/dags/ingestion/ask-astro-load-github.py

sunank200 · 2023-11-21T21:48:25Z

Some logic errors but mostly I'm concerned about readablity and generalizability. For the sake of the reference implementation we want to be very clear about DAG/task code and what should (eventually) go into the hook.

@mpgreg I have fixed most of your comments. Please check. Also just to be clear. This hook is not a drop-in replacement for LLM providers. It requires work contextual change to be added for LLM provider

Lee-W

There are only some nitpicks from my side. other parts look good to me

airflow/include/tasks/extract/utils/weaviate/ask_astro_weaviate_hook.py

- Use `AskAstroWeaviateHook` as the taskflow API for [ask-astro-load-slack.py](https://github.com/astronomer/ask-astro/blob/main/airflow/dags/ingestion/ask-astro-load-slack.py) closes: #140

- Use `AskAstroWeaviateHook` as the taskflow API for [ask-astro-load-stackoverflow.py](https://github.com/astronomer/ask-astro/blob/main/airflow/dags/ingestion/ask-astro-load-stackoverflow.py) closes: #139

- Use `AskAstroWeaviateHook` as the taskflow API for [ask-astro-load-registry.py](https://github.com/astronomer/ask-astro/blob/main/airflow/dags/ingestion/ask-astro-load-registry.py) closes: #138

- Use `AskAstroWeaviateHook` as the taskflow API for [ask-astro-load-github.py](https://github.com/astronomer/ask-astro/blob/main/airflow/dags/ingestion/ask-astro-load-github.py) closes: #137

- Use `AskAstroWeaviateHook` as the taskflow API for [ask-astro-load-blogs.py](https://github.com/astronomer/ask-astro/blob/main/airflow/dags/ingestion/ask-astro-load-blogs.py) closes: #135

- Implement AskAstroWeaviateHook which inherits from WeavaiteHook from OSS. - Use AskAstroWeaviateHook as the taskflow API for [ask-astro-load-airflow-docs.py](https://github.com/astronomer/ask-astro/blob/45ce6543d044a977d97e9314e443f629604e84a9/airflow/dags/ingestion/ask-astro-load-airflow-docs.py) closes: #141 Note: Merge this PR before merging [132](#132)

mpgreg

Looks great @sunank200. Just a couple of small suggestions.

airflow/include/tasks/extract/utils/weaviate/ask_astro_weaviate_hook.py

…e_hook.py

sunank200 force-pushed the oss_weaviate_integration branch from fa7a06d to 5fcfb18 Compare November 17, 2023 21:54

Lee-W reviewed Nov 20, 2023

View reviewed changes

sunank200 requested a review from mpgreg November 20, 2023 08:58

sunank200 marked this pull request as ready for review November 20, 2023 11:20

sunank200 requested a review from pankajastro as a code owner November 20, 2023 11:20

sunank200 force-pushed the oss_weaviate_integration branch from 281639d to 090cced Compare November 20, 2023 13:11

sunank200 changed the title ~~Create TaskFlow API to use WeaviateHook.~~ Create TaskFlow API to use WeaviateHook and use taskflow API for ask-astro-load.py Nov 20, 2023

This was referenced Nov 20, 2023

Use AskAstroWeaviateHook for ask_astro_load_airflow_docs DAG #155

Merged

Use OSS airflow hook for ask-astro-load-blogs.py #135

Closed

Lee-W reviewed Nov 21, 2023

View reviewed changes

mpgreg mentioned this pull request Nov 21, 2023

Add ingest for stack overflow #126

Closed

11 tasks

mpgreg suggested changes Nov 21, 2023

View reviewed changes

sunank200 force-pushed the oss_weaviate_integration branch 4 times, most recently from 57af49b to 64d97f0 Compare November 21, 2023 21:35

sunank200 requested review from mpgreg and Lee-W November 21, 2023 21:45

Lee-W approved these changes Nov 22, 2023

View reviewed changes

sunank200 force-pushed the oss_weaviate_integration branch 2 times, most recently from d15958d to c036d67 Compare November 23, 2023 08:08

mpgreg added 6 commits November 23, 2023 13:54

fixes #105, #106, #107, #108

b16e431

fixes #109

33766a4

fixes #110

5bc00bb

removed test code

41c28d8

fixes #113, fixes #114, fixes #115, fixes #116

76ae2ae

fixes #126

9f9704f

sunank200 added 15 commits November 23, 2023 13:54

Fix pre-commit error

c8bc5ee

Add ingest and upsert for weavaite

c0ccb2e

Update dockerfile

4a9d428

Fix comments from Wei

2ffceb8

Add context manager for batch ingestion

9945a3b

Remove the code which is not in scope for this PR

e2fc253

Fix comments from Michael

b238278

Use AskAstroWeaviateHook for ask-astro-load-slack.py DAG (#161)

d4c9c31

- Use `AskAstroWeaviateHook` as the taskflow API for [ask-astro-load-slack.py](https://github.com/astronomer/ask-astro/blob/main/airflow/dags/ingestion/ask-astro-load-slack.py) closes: #140

Use AskAstroWeaviateHook for ask-astro-load-registry.py DAG (#159)

efcf8f9

- Use `AskAstroWeaviateHook` as the taskflow API for [ask-astro-load-registry.py](https://github.com/astronomer/ask-astro/blob/main/airflow/dags/ingestion/ask-astro-load-registry.py) closes: #138

Use AskAstroWeaviateHook for ask-astro-load-github.py DAG (#158)

2849ec6

- Use `AskAstroWeaviateHook` as the taskflow API for [ask-astro-load-github.py](https://github.com/astronomer/ask-astro/blob/main/airflow/dags/ingestion/ask-astro-load-github.py) closes: #137

Use AskAstroWeaviateHook for ask-astro-load-blogs.py (#156)

d8f4fd6

- Use `AskAstroWeaviateHook` as the taskflow API for [ask-astro-load-blogs.py](https://github.com/astronomer/ask-astro/blob/main/airflow/dags/ingestion/ask-astro-load-blogs.py) closes: #135

Fix the comments from Wei

f5ddc1e

Fix comments from Michael

63bcb55

sunank200 force-pushed the oss_weaviate_integration branch from c036d67 to 63bcb55 Compare November 23, 2023 08:09

mpgreg suggested changes Nov 23, 2023

View reviewed changes

Resolve comments from Michael

0d10815

sunank200 requested a review from mpgreg November 23, 2023 09:28

mpgreg approved these changes Nov 23, 2023

View reviewed changes

Add a note to use directly in future releases

1bfd4ca

mpgreg approved these changes Nov 23, 2023

View reviewed changes

mpgreg suggested changes Nov 23, 2023

View reviewed changes

airflow/include/tasks/extract/utils/weaviate/ask_astro_weaviate_hook.py Outdated Show resolved Hide resolved

mpgreg approved these changes Nov 23, 2023

View reviewed changes

sunank200 force-pushed the oss_weaviate_integration branch from f9f1b2a to 96cc7d7 Compare November 23, 2023 11:00

Update airflow/include/tasks/extract/utils/weaviate/ask_astro_weaviat…

93ea3d8

…e_hook.py

sunank200 force-pushed the oss_weaviate_integration branch from 96cc7d7 to 93ea3d8 Compare November 23, 2023 11:16

sunank200 merged commit 1067794 into main Nov 23, 2023
4 checks passed

sunank200 deleted the oss_weaviate_integration branch November 23, 2023 11:24

sunank200 mentioned this pull request Nov 27, 2023

Switch to context manager for batch import #173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create TaskFlow API to use WeaviateHook and use taskflow API for ask-astro-load.py #132

Create TaskFlow API to use WeaviateHook and use taskflow API for ask-astro-load.py #132

sunank200 commented Nov 16, 2023 •

edited

Loading

cloudflare-workers-and-pages bot commented Nov 17, 2023 •

edited

Loading

mpgreg left a comment

sunank200 commented Nov 21, 2023

Lee-W left a comment

mpgreg left a comment

Create TaskFlow API to use WeaviateHook and use taskflow API for ask-astro-load.py #132

Create TaskFlow API to use WeaviateHook and use taskflow API for ask-astro-load.py #132

Conversation

sunank200 commented Nov 16, 2023 • edited Loading

cloudflare-workers-and-pages bot commented Nov 17, 2023 • edited Loading

Deploying with Cloudflare Pages

mpgreg left a comment

Choose a reason for hiding this comment

sunank200 commented Nov 21, 2023

Lee-W left a comment

Choose a reason for hiding this comment

mpgreg left a comment

Choose a reason for hiding this comment

sunank200 commented Nov 16, 2023 •

edited

Loading

cloudflare-workers-and-pages bot commented Nov 17, 2023 •

edited

Loading