feat: adds initial support for gpt-4-vison (#184)

* feat: adds support for gpt-4-vison * chore: update gitleaksignore * chore: mark gpt-4-vision as beta
tbckr · Dec 4, 2023 · 09b809a · 09b809a
1 parent 29f969c
commit 09b809a
Show file tree

Hide file tree

Showing 16 changed files with 632 additions and 83 deletions.
diff --git a/.bash_aliases b/.bash_aliases
@@ -21,3 +21,6 @@ gsum() {
         echo "Commit cancelled."
     fi
 }
+
+# Create a alias for access to the GPT-4 Vision API
+alias vision='sgpt -m "gpt-4-vision-preview"'
diff --git a/.gitleaksignore b/.gitleaksignore
@@ -0,0 +1 @@
+f1cf67714d70ef1087887abd008d80cfd2483a9b:pkg/fs/fs_test.go:aws-access-token:92
diff --git a/README.md b/README.md
@@ -171,6 +171,8 @@ To use the OpenAI API, you must first obtain an API key.
 After completing these steps, you'll have an OpenAI API key that can be used to interact with the OpenAI models through
 the SGPT tool.
 
+**Note:** Your API key is sensitive information. Do not share it with anyone.
+
 ### Querying OpenAI Models
 
 SGPT allows you to ask simple questions and receive informative answers. For example:
@@ -190,6 +192,36 @@ The mass of the sun is approximately 1.989 x 10^30 kilograms.
 If you want to stream the completion to the command line, you can add the `--stream` flag. This will stream the output
 to the command line as it is generated.
 
+## GPT-4 Vision API
+
+SGPT additionally facilitates the utilization of the GPT-4 Vision API. Include input images using the `-i` or `--input`
+flag, supporting both URLs and local images.
+
+```shell
+$ sgpt -m "gpt-4-vision-preview" -i "https://upload.wikimedia.org/wikipedia/en/c/cb/Marvin_%28HHGG%29.jpg" "what can you see on the picture?"
+The image shows a figure resembling a robot with a humanoid form. It has a
+$ sgpt -m "gpt-4-vision-preview" -i pkg/fs/testdata/marvin.jpg "what can you see on the picture?"
+The image shows a figure resembling a robot with a sleek, metallic surface. It
+```
+
+It is also possible to combine URLs and local images:
+
+```shell
+$ sgpt -m "gpt-4-vision-preview" -i "https://upload.wikimedia.org/wikipedia/en/c/cb/Marvin_%28HHGG%29.jpg" -i pkg/fs/testdata/marvin.jpg "what is the difference between those two pictures"
+The two images provided appear to be identical. Both show the same depiction of a
+```
+
+To avoid specifying the `-m "gpt-4-vision-preview"` for each request, you can streamline the process by creating a bash
+alias:
+
+```shell
+alias vision='sgpt -m "gpt-4-vision-preview"'
+```
+
+For more bash examples, see [.bash_aliases](https://github.com/tbckr/sgpt/blob/main/.bash_aliases).
+
+**Important:** The GPT-4-vision API integration is currently in beta and may change in the future.
+
 ### Chat Capabilities
 
 SGPT provides chat functionality that enables interactive conversations with OpenAI models. You can use the `--chat`

diff --git a/docs/getting-started.md b/docs/getting-started.md
@@ -20,6 +20,8 @@ export OPENAI_API_KEY="sk-..."
 After completing these steps, you'll have an OpenAI API key that can be used to interact with the OpenAI models through
 the SGPT tool.
 
+**Note:** Your API key is sensitive information. Do not share it with anyone.
+
 ## Querying OpenAI Models
 
 SGPT allows you to ask simple questions and receive informative answers. For example:
@@ -39,6 +41,36 @@ The mass of the sun is approximately 1.989 x 10^30 kilograms.
 If you want to stream the completion to the command line, you can add the `--stream` flag. This will stream the output
 to the command line as it is generated.
 
+## GPT-4 Vision API
+
+SGPT additionally facilitates the utilization of the GPT-4 Vision API. Include input images using the `-i` or `--input`
+flag, supporting both URLs and local images.
+
+```shell
+$ sgpt -m "gpt-4-vision-preview" -i "https://upload.wikimedia.org/wikipedia/en/c/cb/Marvin_%28HHGG%29.jpg" "what can you see on the picture?"
+The image shows a figure resembling a robot with a humanoid form. It has a
+$ sgpt -m "gpt-4-vision-preview" -i pkg/fs/testdata/marvin.jpg "what can you see on the picture?"
+The image shows a figure resembling a robot with a sleek, metallic surface. It
+```
+
+It is also possible to combine URLs and local images:
+
+```shell
+$ sgpt -m "gpt-4-vision-preview" -i "https://upload.wikimedia.org/wikipedia/en/c/cb/Marvin_%28HHGG%29.jpg" -i pkg/fs/testdata/marvin.jpg "what is the difference between those two pictures"
+The two images provided appear to be identical. Both show the same depiction of a
+```
+
+To avoid specifying the `-m "gpt-4-vision-preview"` for each request, you can streamline the process by creating a bash
+alias:
+
+```shell
+alias vision='sgpt -m "gpt-4-vision-preview"'
+```
+
+For more bash examples, see [.bash_aliases](https://github.com/tbckr/sgpt/blob/main/.bash_aliases).
+
+**Important:** The GPT-4-vision API integration is currently in beta and may change in the future.
+
 ## Code Generation Capabilities
 
 By adding the `code` command to your prompt, you can generate code based on given instructions by using the

diff --git a/docs/usage/gpt4-vision-api.md b/docs/usage/gpt4-vision-api.md
@@ -0,0 +1,29 @@
+# GPT-4 Vision API
+
+SGPT additionally facilitates the utilization of the GPT-4 Vision API. Include input images using the `-i` or `--input`
+flag, supporting both URLs and local images.
+
+```shell
+$ sgpt -m "gpt-4-vision-preview" -i "https://upload.wikimedia.org/wikipedia/en/c/cb/Marvin_%28HHGG%29.jpg" "what can you see on the picture?"
+The image shows a figure resembling a robot with a humanoid form. It has a
+$ sgpt -m "gpt-4-vision-preview" -i pkg/fs/testdata/marvin.jpg "what can you see on the picture?"
+The image shows a figure resembling a robot with a sleek, metallic surface. It
+```
+
+It is also possible to combine URLs and local images:
+
+```shell
+$ sgpt-m "gpt-4-vision-preview" -i "https://upload.wikimedia.org/wikipedia/en/c/cb/Marvin_%28HHGG%29.jpg" -i pkg/fs/testdata/marvin.jpg "what is the difference between those two pictures"
+The two images provided appear to be identical. Both show the same depiction of a
+```
+
+To avoid specifying the `-m "gpt-4-vision-preview"` for each request, you can streamline the process by creating a bash
+alias:
+
+```shell
+alias vision='sgpt -m "gpt-4-vision-preview"'
+```
+
+For more bash examples, see [.bash_aliases](https://github.com/tbckr/sgpt/blob/main/.bash_aliases).
+
+**Important:** The GPT-4-vision API integration is currently in beta and may change in the future.
diff --git a/go.mod b/go.mod
@@ -7,7 +7,7 @@ require (
 	github.com/jarcoal/httpmock v1.3.1
 	github.com/muesli/mango-cobra v1.2.0
 	github.com/muesli/roff v0.1.0
-	github.com/sashabaranov/go-openai v1.17.9
+	github.com/sashabaranov/go-openai v1.17.10-0.20231126084528-a09cb0c528c1
 	github.com/spf13/cobra v1.8.0
 	github.com/spf13/viper v1.17.0
 	github.com/stretchr/testify v1.8.4

diff --git a/go.sum b/go.sum
@@ -173,8 +173,8 @@ github.com/sagikazarmark/locafero v0.3.0 h1:zT7VEGWC2DTflmccN/5T1etyKvxSxpHsjb9c
 github.com/sagikazarmark/locafero v0.3.0/go.mod h1:w+v7UsPNFwzF1cHuOajOOzoq4U7v/ig1mpRjqV+Bu1U=
 github.com/sagikazarmark/slog-shim v0.1.0 h1:diDBnUNK9N/354PgrxMywXnAwEr1QZcOr6gto+ugjYE=
 github.com/sagikazarmark/slog-shim v0.1.0/go.mod h1:SrcSrq8aKtyuqEI1uvTDTK1arOWRIczQRv+GVI1AkeQ=
-github.com/sashabaranov/go-openai v1.17.9 h1:QEoBiGKWW68W79YIfXWEFZ7l5cEgZBV4/Ow3uy+5hNY=
-github.com/sashabaranov/go-openai v1.17.9/go.mod h1:lj5b/K+zjTSFxVLijLSTDZuP7adOgerWeFyZLUhAKRg=
+github.com/sashabaranov/go-openai v1.17.10-0.20231126084528-a09cb0c528c1 h1:F07vUreAjYQtO6Ny9czwt08RlQyOOoJdYpwxSkloDZI=
+github.com/sashabaranov/go-openai v1.17.10-0.20231126084528-a09cb0c528c1/go.mod h1:lj5b/K+zjTSFxVLijLSTDZuP7adOgerWeFyZLUhAKRg=
 github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo=
 github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0=
 github.com/spf13/afero v1.10.0 h1:EaGW2JJh15aKOejeuJ+wpFSHnbd7GE6Wvp3TsNhb6LY=

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -69,6 +69,7 @@ nav:
   - Getting Started: 'getting-started.md'
   - Usage Guide:
       - Query Models: 'usage/query-models.md'
+      - GPT-4 Vision API: 'usage/gpt4-vision-api.md'
       - Chat: 'usage/chat.md'
       - Docker: 'usage/docker.md'
       - Personas: 'usage/personas.md'