spark_codegen

SPARK code generation Vertex Generative AI

This repository contains an example that demonstrate how to generate pyspark code using PaLM-2 Generative AI, powered by Vertex AI and run it using dataproc serverless SPARK

Sample outputs

Initial call:

python3  src/code_gen.py --bq_table_full_name="bigquery-public-data.new_york.citibike_trips" --question="What are the most popular Citi Bike stations?"

• Execution - see prompt design:
• Code generated:
• dataproc serverless execution log: