Initial goex commit (ShishirPatil#336)

We introduce GoEx, a runtime for LLMs with an intuitive `undo` and
`blast-radius confinement` abstractions, enabling the safer deployment
of LLM agents in practice.

Paper: https://arxiv.org/abs/2404.06921

---------

Co-authored-by: Noppapon (Neun) Chalermchockcharoenkit <76277934+Noppapon@users.noreply.github.com>
Co-authored-by: Roy Huang <hyfdudu@gmail.com>
Co-authored-by: Aaron Hao <aaronhao@berkeley.edu>
Co-authored-by: Tianjun Zhang <tianjunz@Tianjuns-MacBook-Pro.local>
Co-authored-by: Gil Vernik <gvernik@gmail.com>

.gitignore

@@ -6,9 +6,14 @@ dist
 **/*.lic
 .vscode
 .idea
+.env
 .editorconfig
 .DS_Store
 **/*.pyc
 ./berkeley-function-call-leaderboard/function_credential_config.json
 ./berkeley-function-call-leaderboard/eval_checker/tree-sitter-java
-./berkeley-function-call-leaderboard/eval_checker/tree-sitter-javascript
+./berkeley-function-call-leaderboard/eval_checker/tree-sitter-javascript
+goex/exec_engine/checkpoints/
+goex/exec_engine/credentials/secret_store.json
+goex/docker/*/requirements.txt
+goex/docker/misc/images.json

README.md

@@ -3,6 +3,8 @@
 
 <img src="https://github.com/ShishirPatil/gorilla/blob/gh-pages/assets/img/logo.png" width=50% height=50%>
 
+**🚒  GoEx: A Runtime for executing LLM generated actions like code & API calls** GoEx presents “undo” and “damage confinement” abstractions for mitigating the risk of unintended actions taken in LLM-powered systems. [Release blog](https://gorilla.cs.berkeley.edu/blogs/10_gorilla_exec_engine.html) [Paper](https://arxiv.org/abs/2404.06921).
+
 **🎉 Berkeley Function Calling Leaderboard** How do models stack up for function calling? :dart: Releasing the [Berkeley Function Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard). Read more in our [Release Blog](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html). 
 
 **:trophy: Gorilla OpenFunctions v2** Sets new SoTA for open-source LLMs :muscle: On-par with GPT-4 :raised_hands: Supports more languages :ok_hand: [Blog](https://gorilla.cs.berkeley.edu/blogs/7_open_functions_v2.html). 

goex/.env.example

@@ -0,0 +1,8 @@
+# mysql
+DATABASE_USER=root
+DATABASE_PASSWORD=abcdefg
+DATABASE_HOST=localhost
+DATABASE_NAME=testdb
+
+# sqlite
+DATABASE_PATH=goex_demo.db

goex/CONTRIBUTE.md

@@ -0,0 +1,16 @@
+# Contributing to the Repo
+
+## Adding a new database
+
+Adding a new database will require the following to be added:
+
+1. A new Manager subclass (like the DBManager classes in `exec_engine/db_manager.py`)
+   1. Needs to include logic for supporting transactional commits/rollbacks regardless of the database type (Ensure reversibility for all database types)
+   2. Implementation should be similar to ` MySQLManager` or `SQLiteManager`
+   3. Be able to format a user prompt with additional database specific info (like schema) to inform the LLM how to generate an accurate API call.
+2. \[Optional\] If dry-running is wanted:
+   1. Add a new folder inside `docker/` much like `docker/mysql_docker`
+   2. Make sure the dockerfile sets up the environment for the database deployment
+   3. Create a copy of the database and copy it to the docker container
+   4. Create a reversibility script for dry-running like `exec_engine/db_reversion_test.txt`
+      1. This script is written in txt since it will be passed into the docker environment and run with `python_executor.py`

goex/README.md

@@ -0,0 +1,241 @@
+# 🦍 GoEx
+
+| [**Website**](https://goex.gorilla-llm.com/index) |
+
+GoEx provides a new way to interact with your favorite APIs! Powered by LLMs, GoEx executes generated code, taking into consideration any credentials you may need to access APIs. Execution security is ensured by creating reverse calls that undo any unwanted effects. Ask GoEx for anything!
+
+## Contents <!-- omit from toc -->
+
+- [Install](#install)
+- [CLI Usage](#cli-usage)
+  - [RESTful API](#restful-api)
+  - [Database](#database)
+  - [Filesystem](#filesystem)
+- [Credentials \& Authorization Token](#credentials--authorization-token)
+
+## Install
+
+1.) Navigate inside the goex folder and set up a clean environment with **Conda** or **Python venv**
+
+```sh
+python3 -m venv goex-env
+source goex-env/bin/activate
+```
+
+**OR**
+
+```sh
+conda create --name goex python=3.10 -y
+conda activate goex
+```
+
+2.) Install the `goex` CLI tool
+
+```sh
+pip install -e .
+```
+
+3.) Install and initialize Mkcert to support OAuth2 token exchange [required for services that require https for redirect URL e.g Slack]
+
+Mac:
+
+```sh
+brew install mkcert
+mkcert -install
+mkcert localhost
+```
+
+Windows:
+
+```sh
+choco install mkcert
+mkcert -install
+mkcert localhost
+```
+
+Mkcert creates a local certificate authority, enabling Gorilla to establish secure communication for OAuth2 token exchange between localhost and interacting services. The command may prompt the user to pass in their password.
+
+## CLI Usage
+
+List all commands Gorilla currently supports and their usage:
+
+```sh
+goex -h
+```
+
+### RESTful API
+
+Give authorizations and perform OAuth2 token exchanges with services Gorilla currently support.
+
+```sh
+goex -authorize <service> # gmail, slack, spotify, github, dropbox
+```
+
+After a service is authorized, user will be able to interact with it by providing a prompt.
+
+```sh
+# example 1
+# please first run goex -authorize slack
+goex execute -prompt send a funny joke to the user with email gorilla@yahoo.com on slack -type rest
+```
+
+```sh
+# example 2
+# this action needs user to authorize gmail
+goex execute -prompt who are the senders of the 5 latest emails in my gmail inbox -type rest
+```
+
+**[Beta]** User can also specify `-generate_mode function_in_context` to generate API calls with function calling. Functions are stored in the **functions** folder, and it currently only supports a limited number of Slack features.
+
+```sh
+#example 3 (function calling)
+goex execute -prompt add a smile emoji to the latest message sent to channel <CHANNEL_ID> on slack -type rest -generate_mode function_in_context
+```
+
+List all commands Gorilla current supports and their usages
+
+### Database
+
+To test out database interactions locally, each database server requires its own setup
+
+#### SQLite
+
+- If you need to create a new SQLite DB, do the following:
+
+  ```
+  sqlite3 ./goex_demo.db
+  ```
+
+  Then after, you will need to use the sqlite CLI:
+
+  ```
+  sqlite> .database
+  ```
+
+  Press `Ctrl-D` to exit out and now your `.db` file will be created!
+
+- Run `demo/env_setup.py` to get your environment variables set up
+  Use default values if you are running just the demo.
+
+  ```
+  python3 demo/env_setup.py
+  ```
+
+- Set GoEx to use SQLite for DB operations
+  ```
+  goex -set_config dbtype sqlite
+  ```
+
+#### Try it out!
+
+After setting up your SQL database, you can try the following examples!
+
+```
+goex execute -prompt "Create a new table for storing user information called users, with each user having a current balance column denoting their bank balance" -type db
+```
+
+```
+goex execute -prompt "Add 3 example users to the users table" -type db
+```
+
+```
+goex execute -prompt "In the users table, add 500 to everyone's current balance" -type db
+```
+
+#### MySQL
+
+- Install MySQL Server
+
+  - For non-Mac, [install server here](https://dev.mysql.com/downloads/mysql/)
+    - **Make sure to add `mysql` to path!**
+  - Mac:
+
+  ```
+  brew install mysql
+  ```
+
+- If you don't have your own server, import the example database using `demo/mysql_setup.py` by running:
+  ```sh
+  cd demo
+  python3 mysql_setup.py testdb
+  ```
+- Put the user, password, host, and database name info into `.env` by running this script
+
+  ```
+  python3 demo/env_setup.py
+  ```
+
+- Set GoEx to use MySQL for DB operations
+  ```
+  goex -set_config dbtype mysql
+  ```
+
+### Filesystem
+
+The goex command will be executed at the path pointed to by `fs_path` in `user_config.json`. If `fs_path` is not specified or is equal to `""`, execution will occur in the user's current working directory. Below is an example of how to set this up.
+
+Make sure to have Git LFS installed on your machine!
+
+Mac: `brew install git-lfs`
+Windows/Linux: git-lfs.com and download
+
+#### Try it out!
+
+Let's first create a testing folder.
+
+```
+mkdir test
+goex -set_config fs_path test
+```
+
+##### Create a Simple File
+
+```
+goex execute -prompt "Write a witty LLM joke into joke.txt" -type fs
+```
+
+##### Code Writing
+
+You can tell Gorilla-Ex to write code for you, and directly have it be written onto your chosen directory!
+
+```
+goex execute -prompt "Write a Python program that is a calculator into a file called calculator.py" -type fs
+```
+
+##### Log Compilation
+
+```
+goex execute -prompt "Create 5 log files named according to some fake dates and insert some placeholder log content into them" -type fs
+```
+
+Here you will have 5 log files with some placeholder content. You can execute the command below to compile the log file contents into a single CSV for easy analysis.
+
+```
+goex execute -prompt "Compile the log file contents into a single CSV with the date as the identifying column here." -type fs
+```
+
+## Credentials & Authorization Token
+
+There are two types of credentials the user can add to the execution engine.
+
+1.) [**OAuth 2.0**](https://oauth.net/2/)
+
+Gorilla-ex follows the standard OAuth 2.0 token exchanges flow. Running the command below will redirect the user to the browser, where they will be prompted to grant permissions to Gorilla for a specific set of scopes.
+
+```sh
+goex -authorize <gmail,slack,spotify,github,dropbox>
+```
+
+After Gorilla-ex receives the token for a service, it will automatically be able to recognize the keyword in future prompts, enabling the user to perform actions on that particular platform. Additionally, the token will not be exposed to the LLM and will only be visible during execution.
+
+We continually seek to expand our support for additional services. The authorization logic for each service resides in the authorization/scripts folder. We warmly welcome interested contributors to submit a pull request introducing a new authorization flow. **Your contributions are highly appreciated 🚀**
+
+2.) **Raw API Key**
+
+If the user wants to add services not supported by OAuth2, they can do so by calling the function below with `service` and `key` as parameters:
+
+```sh
+goex -insert_creds alpha_vantage {API_KEY}
+```
+
+The key will be stored in Gorilla-ex's secret store and passed on to the LLM during prompting.

goex/__init__.py

@@ -0,0 +1 @@
+from .cli import main

goex/authorizations/scripts/authorization_utils.py

@@ -0,0 +1,25 @@
+
+import os
+from pathlib import Path
+import sys
+import inspect
+
+currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
+parentdir = os.path.dirname(currentdir)
+sys.path.insert(0, os.path.dirname(parentdir))
+
+from exec_engine.credentials.credentials_utils import CREDS_FOLDER_PATH, insert_creds
+
+AUTHORIZATION_FOLDER_PATH = os.path.dirname(Path(os.path.realpath(__file__)))
+
+def authorize_service(name: str):
+    name = name.lower()
+    authorization_file = "{name}_authorization.py".format(name = name)
+    authorization_path = os.path.join(AUTHORIZATION_FOLDER_PATH, authorization_file)
+
+    if not os.path.exists(authorization_path):
+        return False
+    else:
+        print(exec(open(authorization_path).read(), globals()))
+        insert_creds(name, os.path.join(CREDS_FOLDER_PATH, name), target = CREDS_FOLDER_PATH, cred_type="path")
+        return True

goex/authorizations/scripts/consts.py

@@ -0,0 +1,7 @@
+import os
+from pathlib import Path
+
+AUTH_URL = "https://goex-services.gorilla-llm.com/authorize"
+# AUTH_URL = "http://localhost:443/authorize"
+CERT_FILE_PATH = os.path.join(os.path.dirname(Path(os.path.realpath(__file__)).parent.parent), "localhost.pem")
+KEY_FILE_PATH = os.path.join(os.path.dirname(Path(os.path.realpath(__file__)).parent.parent), "localhost-key.pem")