GenAI Experiments

This repository contains a number of Gen AI Experiments, mainly focused on integrated LLMs text generation capablities with SPARK.

• SPARK dataframe automated summarization
• Generation of a SPARK job from natural language
• Distributed LLM inference with SPARK UDFs - Enrichment of dataframe with zero shot learning
• Distributed LLM inference with SPARK UDFs - Enrichment of dataframe with few shot learning
• Minimal example of Vertex AI LLM integration on langchain