Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: upgrade cobalt v7->v10 #26

Merged
merged 3 commits into from
Nov 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ https://t.me/save_it_playground

- [Elixir](https://elixir-lang.org/)
- [ex_gram](https://github.com/rockneurotiko/ex_gram)
- [cobalt api](https://github.com/imputnet/cobalt/blob/current/docs/api.md)
- [cobalt api](https://github.com/imputnet/cobalt)
- [Typesense](https://typesense.org/)

## Development
Expand All @@ -61,6 +61,7 @@ docker compose up

```sh
# Run
export COBALT_API_URL=<your_cobalt_api_url>
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Please enhance the environment variable documentation.

The new COBALT_API_URL environment variable needs more context:

  1. Add a comment explaining what value to use
  2. Mention that it should point to the cobalt-api service (running on port 9001)
  3. Consider providing an example value like http://localhost:9001
 # Run
-export COBALT_API_URL=<your_cobalt_api_url>
+# Point this to your cobalt-api service (default: http://localhost:9001)
+export COBALT_API_URL=<cobalt_api_url>
 export TELEGRAM_BOT_TOKEN=<YOUR_TELEGRAM_BOT_TOKEN>
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
export COBALT_API_URL=<your_cobalt_api_url>
# Point this to your cobalt-api service (default: http://localhost:9001)
export COBALT_API_URL=<cobalt_api_url>
export TELEGRAM_BOT_TOKEN=<YOUR_TELEGRAM_BOT_TOKEN>

export TELEGRAM_BOT_TOKEN=<YOUR_TELEGRAM_BOT_TOKEN>

iex -S mix run --no-halt
Expand Down
4 changes: 3 additions & 1 deletion config/runtime.exs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@ import Config
config :save_it, :telegram_bot_token, System.get_env("TELEGRAM_BOT_TOKEN")
config :ex_gram, token: System.get_env("TELEGRAM_BOT_TOKEN")

config :save_it, :typesense_url, System.get_env("TYPESENSE_URL", "http://localhost:8100")
config :save_it, :cobalt_api_url, System.get_env("COBALT_API_URL", "http://localhost:9001")

config :save_it, :typesense_url, System.get_env("TYPESENSE_URL", "http://localhost:8101")
config :save_it, :typesense_api_key, System.get_env("TYPESENSE_API_KEY", "xyz")

# optional
Expand Down
19 changes: 16 additions & 3 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,15 +1,28 @@
services:
cobalt-api:
image: ghcr.io/imputnet/cobalt:10

init: true
read_only: true
restart: unless-stopped

ports:
- 9001:9000/tcp
environment:
API_URL: "http://localhost:9001/"

typesense:
image: typesense/typesense:27.1

restart: on-failure
hostname: typesense

ports:
- "8100:8108"
- "8101:8108"
volumes:
- ./data/typesense-data:/data
command: "--data-dir /data --api-key=xyz --enable-cors"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8100/health"]
test: ["CMD", "curl", "-f", "http://localhost:8101/health"]
interval: 30s
timeout: 10s
retries: 5
Binary file added docs/assets/save_it_demo.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 5 additions & 4 deletions lib/save_it/bot.ex
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
defmodule SaveIt.Bot do
require Logger
alias SaveIt.CobaltClient

alias SaveIt.FileHelper
alias SaveIt.GoogleDrive
alias SaveIt.GoogleOAuth2DeviceFlow

alias SaveIt.PhotoService

alias SmallSdk.Telegram
alias SmallSdk.Cobalt

@bot :save_it_bot

Expand Down Expand Up @@ -225,8 +226,8 @@ defmodule SaveIt.Bot do
{:ok, progress_message} = send_message(chat.id, Enum.at(@progress, 0))
url = List.first(urls)

case CobaltClient.get_download_url(url) do
{:ok, url, download_urls} ->
case Cobalt.get_download_url(url) do
{:ok, purge_url, download_urls} ->
case FileHelper.get_downloaded_files(download_urls) do
nil ->
update_message(chat.id, progress_message.message_id, Enum.slice(@progress, 0..1))
Expand All @@ -244,7 +245,7 @@ defmodule SaveIt.Bot do
bot_send_files(chat.id, files)

delete_messages(chat.id, [message_id, progress_message.message_id])
FileHelper.write_folder(url, files)
FileHelper.write_folder(purge_url, files)
# TODO: 给图片添加 emoji
GoogleDrive.upload_files(chat.id, files)

Expand Down
73 changes: 0 additions & 73 deletions lib/save_it/cobalt_client.ex

This file was deleted.

2 changes: 1 addition & 1 deletion lib/save_it/migration/typesense.ex
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
defmodule SaveIt.Migration.Typesense do
alias SmallSdk.Typesense

import Tj.UrlHelper, only: [validate_url!: 1]
import SaveIt.SmallHelper.UrlHelper, only: [validate_url!: 1]

def create_collection!(schema) do
req = build_request("/collections")
Expand Down
2 changes: 1 addition & 1 deletion lib/save_it/photo_service.ex
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ defmodule SaveIt.PhotoService do
require Logger
alias SmallSdk.Typesense

import Tj.UrlHelper, only: [validate_url!: 1]
import SaveIt.SmallHelper.UrlHelper, only: [validate_url!: 1]

def create_photo!(
%{
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
defmodule Tj.UrlHelper do
defmodule SaveIt.SmallHelper.UrlHelper do
def validate_url!(url) do
uri = URI.parse(url)

Expand Down
98 changes: 98 additions & 0 deletions lib/small_sdk/cobalt.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
defmodule SmallSdk.Cobalt do
require Logger

import SaveIt.SmallHelper.UrlHelper, only: [validate_url!: 1]

def get_download_url(text) do
url = String.split(text, "?") |> hd()

req = build_request("/")
res = Req.post(req, json: %{url: url})

body = handle_response(res)

case body do
%{"url" => download_url} ->
{:ok, download_url}

%{"status" => "picker", "picker" => picker_items} ->
{:ok, url, Enum.map(picker_items, &Map.get(&1, "url"))}

%{"status" => "error", "text" => msg} ->
Logger.warning("response.body is status error, text: #{msg}")
{:error, msg}

_ ->
Logger.warning("response.body: #{inspect(body)}")
{:error, "inner service error"}
end
end

defp get_env() do
api_url = Application.fetch_env!(:save_it, :cobalt_api_url) |> validate_url!()

{api_url}
end

defp build_request(path) do
{api_url} = get_env()

Req.new(
base_url: api_url,
url: path,
headers: [
{"Accept", "application/json"},
{"Content-Type", "application/json"}
]
)
end

@doc """
Handle response from Cobalt API return body if status is 200..209
"""
def handle_response({:ok, %{status: status, body: body}}) do
case status do
status when status in 200..209 ->
body

400 ->
Logger.warning("Bad Request: #{inspect(body)}")
raise "Bad Request"

401 ->
raise "Unauthorized"

404 ->
nil

409 ->
raise "Conflict"

422 ->
raise "Unprocessable Entity"

503 ->
raise "Service Unavailable"

_ ->
Logger.error("Unhandled status code #{status}: #{inspect(body)}")
raise "Unknown error: #{status}"
end
end

def handle_response({:error, reason}) do
Logger.error("Request failed: #{inspect(reason)}")
raise "Request failed"
end

def handle_response!(%{status: status, body: body}) do
case status do
status when status in 200..209 ->
body

status ->
Logger.warning("Request failed with status #{status}: #{inspect(body)}")
raise "Request failed with status #{status}"
end
end
end
2 changes: 1 addition & 1 deletion lib/small_sdk/typesense.ex
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
defmodule SmallSdk.Typesense do
require Logger

import Tj.UrlHelper, only: [validate_url!: 1]
import SaveIt.SmallHelper.UrlHelper, only: [validate_url!: 1]

def create_document!(collection_name, document) do
req = build_request("/collections/#{collection_name}/documents")
Expand Down
13 changes: 8 additions & 5 deletions zeabur/readme.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
# Zeabur
# SaveIt template on zeabur

## Typesense Template
- zeabur template: https://zeabur.com/templates/FTAONK
- github: https://github.com/ThaddeusJiang/save_it/

## Develop

```sh
npx zeabur template update -c FTAONK -f typesense-template.yaml
npx zeabur template update -c FTAONK -f template.yaml
```

docs:
## Docs

- https://zeabur.com/docs/template/template-in-code
- [zeabur docs | template ](https://zeabur.com/docs/template/template-in-code)
30 changes: 23 additions & 7 deletions zeabur/template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ metadata:
name: SaveIt
spec:
description: SaveIt is a telegram bot that helps you save photos.
icon: https://github.com/user-attachments/assets/fae196b8-716e-4be7-a8c2-3b141984c0e5
icon: https://raw.githubusercontent.com/ThaddeusJiang/save_it/refs/heads/main/docs/assets/savt_it_bot_logo.jpg
tags:
- Bot
- Telegram
- Photos Storage
- Search Engine
coverImage:
coverImage: https://raw.githubusercontent.com/ThaddeusJiang/save_it/refs/heads/main/docs/assets/savt_it_demo.gif

readme: |-
# SaveIt
Expand All @@ -22,14 +22,11 @@ spec:
- Search photos using semantic search
- Find similar photos by photo

## Learn more
https://github.com/ThaddeusJiang/save_it

<video src="https://github.com/user-attachments/assets/b0dedcc0-3305-42b2-8101-6b0b5d32f17a" controls="controls" width="100%" height="auto" ></video>
[more](https://github.com/ThaddeusJiang/save_it)

services:
- name: save_it
icon: https://github.com/user-attachments/assets/fae196b8-716e-4be7-a8c2-3b141984c0e5
icon: https://raw.githubusercontent.com/ThaddeusJiang/save_it/refs/heads/main/docs/assets/savt_it_bot_logo.jpg
template: GIT
spec:
source:
Expand All @@ -41,6 +38,9 @@ spec:
TELEGRAM_BOT_TOKEN:
default: ""
expose: false
COBALT_API_URL:
default: http://cobalt-api.zeabur.internal:9000
expose: false
TYPESENSE_URL:
default: http://typesense.zeabur.internal:8108
expose: false
Expand Down Expand Up @@ -81,3 +81,19 @@ spec:
expose: false

configs: []
- name: cobalt-api
icon: https://github.com/imputnet/cobalt/raw/main/web/static/favicon.png
template: PREBUILT_V2
spec:
source:
image: ghcr.io/imputnet/cobalt:10
ports:
- id: api
port: 9000
type: TCP
env:
API_URL:
default: undefined
expose: false

Comment on lines +84 to +98
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Several improvements needed for the cobalt-api service configuration.

  1. The API_URL environment variable has an invalid default value of "undefined".
  2. Missing important production configurations:
    • Resource limits (CPU/memory)
    • Health checks
    • Security configurations (read-only filesystem, non-root user)

Here's the suggested improvement:

    - name: cobalt-api
      icon: https://github.com/imputnet/cobalt/raw/main/web/static/favicon.png
      template: PREBUILT_V2
      spec:
        source:
          image: ghcr.io/imputnet/cobalt:10
        ports:
          - id: api
            port: 9000
            type: TCP
+        resources:
+          limits:
+            cpu: "1"
+            memory: "1Gi"
+          requests:
+            cpu: "200m"
+            memory: "256Mi"
+        healthCheck:
+          httpGet:
+            path: /health
+            port: 9000
+          initialDelaySeconds: 10
+          periodSeconds: 30
+        securityContext:
+          readOnlyRootFilesystem: true
+          runAsNonRoot: true
+          runAsUser: 1000
        env:
          API_URL:
-            default: undefined
+            default: http://0.0.0.0:9000
            expose: false
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: cobalt-api
icon: https://github.com/imputnet/cobalt/raw/main/web/static/favicon.png
template: PREBUILT_V2
spec:
source:
image: ghcr.io/imputnet/cobalt:10
ports:
- id: api
port: 9000
type: TCP
env:
API_URL:
default: undefined
expose: false
- name: cobalt-api
icon: https://github.com/imputnet/cobalt/raw/main/web/static/favicon.png
template: PREBUILT_V2
spec:
source:
image: ghcr.io/imputnet/cobalt:10
ports:
- id: api
port: 9000
type: TCP
resources:
limits:
cpu: "1"
memory: "1Gi"
requests:
cpu: "200m"
memory: "256Mi"
healthCheck:
httpGet:
path: /health
port: 9000
initialDelaySeconds: 10
periodSeconds: 30
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
env:
API_URL:
default: http://0.0.0.0:9000
expose: false

configs: []
Loading