Data Ingestion Improvement/Cleanup/Bug Fix - Part 2

[davidgxue]

Code ready for review, evaluations/testing is still in progress

Description

• This is part 2 of the data ingestion improvements. The goal is to analyze and investigate any potential issues with the data ingestion process, remove noisy data such as badly formatted content or gibberish text, find bugs, and enforce standards on the process to have predictable cost estimation/reduction.

Technical Changes

• Renamed the original split.py file to chunking_utils.py (multiple files changed due to import naming)
• During the above change, found 2 DAGs somehow were NOT changed. Provider docs and SDK docs's incremental ingest DAGs somehow never had a chunking/splitting task (even though the bulk ingest does). I added chunking to these 2 DAGs
• Airflow Docs extraction process has been improved(see code for details)
• Provider Docs extraction process has been improved (see code for details)
• Astro SDK Docs extraction process has been improved (see code for details) -> Astro SDK docs are not currently being ingested, but fixing code since they exist.
• Provider docs used to ingest all past versions leading to many duplicate docs in the DB. It is now changes to only ingest the stable version of the docs.

Evaluations

• Answer quality generally improved with some add on of more concise answering and linking hyperlinks when the model knows the answer is not exhaustive.
• Please attached for a quick list of results

data_ingest_comparison_part_2.csv
data_ingest_results_part_2.csv

• Bulk ingest also runs with no issues

Related Issues

closes #221
closes #258
closes #295 (Reranker has been addressed in GCP environment variables, embedding model change completed in a different PR)
closes #285 (This PR prevents empty docs from being ingested)

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Pages

Latest commit:	3142709
Status:	✅  Deploy successful!
Preview URL:	https://dc48aa3a.ask-astro.pages.dev
Branch Preview URL:	https://improve-data-ingestion-pt2.ask-astro.pages.dev

View logs

[Lee-W]

Left 1 nitpick. Rest looks good to me

[pankajastro]

LGTM. But this might require a fresh ingestion for testing

[sunank200]

@davidgxue requesting you to do the fresh ingestion to the new Database and test the changes before merging.

[davidgxue]

Updated the PR description with test results!!