Kaggle Competition Assistant

Small toy utility to help with Kaggle competition onboarding.
Intended to simplify diving into the new competition information and keeping up to date with discussions and comments.

Current functionality is limited to simple questions about the competition & discussion forum info.
Internally built as RAG tool (retrieval augmented generation).

Main current use cases:

1. high-level competition overview
2. learn about topics in the discussion forum
3. leaderboard info
4. text & voice-based interaction: quick & easy way to get information without sifting through many pages of competition

Demo

demo.mp4

Setup & dependencies

1. Python 3.11
2. Docker
3. RAM available for Docker: ~9GB
4. Create API keys and put them in the .env file (see .env_example)
   • Kaggle API key: how to create
   • Gemini API key (no credit card required, free quota for >1k gemini-1.5-flash requests per day): how to create

Run the assistant

1. Dockerized implementation

   Streamlit UI assistant with hybrid retrieval (via OpenSearch), monitoring and feedback collection (via PostgreSQL and Grafana).

   1. make up (or docker compose --env-file .env up)
   2. Wait until all services are ready (takes ~10 mins)
   3. Open localhost:8501 to view streamlit UI
   4. Open the kaggle-assistant dashboard from the Grafana UI localhost:3000/dashboards to view the monitoring dashboard
      • Grafana credentials: login - admin, password - admin (keep unchanged when prompted)
   5. To shut down the application run make down (or docker compose down)

2. Core standalone Python library

   It's also possible to run the core logic of the assistant just as a Python library.

   For the retrieval, instead of hybrid retrieval via OpenSearch, a simple lexical search in-memory implementation will be used.

   No monitoring and feedback collection.

   See example in full_example_simple.py

Data scraping

The repository contains scrapped data for 3 competitions (snapshots Sep 15/30, 2024) in data directory:

• Rohlik Orders Forecasting Challenge
• Arc Prize 2024
• LLM Zoomcamp 2024 competition

To scrape a new version / new competition, use the utility function from the library:

from kaggle_competition_assistant import scrape_competition_data

scrape_competition_data(competition_slug='arc-prize-2024', path='data')

After calling this function the scrapped data will be saved into data/arc-prize-2024 directory and will be ready for ingestion by the assistant.

Depending on the number of pages in the discussion forum scrapping can take some time (~1 hour for arc-prize-2024).
Scraping speed is limited by Kaggle rate limits.

Currently, the following information is scrapped:

1. Competition overview
2. Dataset information
3. Leaderboard
4. Discussion forum

Project structure

Key folders and files:

• data - scrapped data + evaluation data & results
• kaggle_competition_assistant - core python library that contains the assistant (scraping utilities & RAG creation)
  • scrape - scraping logic scripts
  • index/minsearch.py - simple in-memory lexical search engine
  • index/opensearch_index.py - implementation of various retrieval modes via OpenSearch
  • llm.py - LLM utilities
  • assistant.py - top-level API & logic (ingestion + RAG creation)
• app - streamlit demo app to chat with assistant and provide feedback
  • app.py - streamlit UI & voice support
  • assistants.py - assistants query & relevance evaluation
  • db.py - logic for logging requests and responses to postgres / monitoring
  • storage_init.py - script for initializing the database & assistants creation
  • install_tts.sh - script to install text to speech dependencies
• monitoring - monitoring dashboard
  • dashboard.json - configurations of dashboard to load by grafana
  • init_dashboard.py - script to initialize dashboard from dashboard.json

Technologies & techniques used

1. Knowledge base creation
   • web scrapping: selenium, tenacity
   • API access: kaggle api
2. Retrieval
   • data preprocessing & chunking: langchain
   • lexical & semantic search: OpenSearch
   • document embeddings for semantic search: sentence-transformers/multi-qa-mpnet-base-cos-v1
   • reranking: custom implementation of Reciprocal rank fusion (RRF) algorithm
   • query rewriting: generating rewritten variations of original user question (using LLM) to retrieve additional context from index and potentially improve RAG results
3. Text generation
   • LLM models for generation: gemini-1.5-flash-latest/gemini-1.5-pro-latest (free tier via Google AI Studio)
4. Voice input (speech-to-text)
   • faster-whisper
5. Voice output (text-to-speech)
   • MeloTTS
6. Interface
   • Demo UI: streamlit
7. Feedback collection & monitoring
   • PostgreSQL: PostgreSQL
   • Grafana: Grafana
8. Containerization
   • Docker: Docker

Documentation

1. Evaluation
2. Monitoring
3. Voice support

TODOs

See potential future improvements in TODO.md