Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing streaming by migrating to Typhoeus #12

Merged
merged 2 commits into from
Jan 24, 2024
Merged

Fixing streaming by migrating to Typhoeus #12

merged 2 commits into from
Jan 24, 2024

Conversation

gbaptista
Copy link
Owner

@gbaptista gbaptista commented Jan 24, 2024

New option to customize Faraday adapter:

require 'faraday/net_http'

client = Gemini.new(
  credentials: { service: 'vertex-ai-api', region: 'us-east4' },
  options: {
    model: 'gemini-pro',
    connection: { adapter: :net_http }
  }
)

New option to set service version:

client = Gemini.new(
  credentials: {
    service: 'generative-language-api',
    api_key: ENV['GOOGLE_API_KEY'],
    version: 'v1beta'
  },
  options: { model: 'gemini-pro', server_sent_events: true }
)

Why is Typhoeus the default rather than being used only for streaming with server-sent events?
I prefer using a single adapter for all use cases to ensure a consistent experience across different scenarios, e.g., to avoid having different timeout behaviors due to an unknown change of adapters under the hood. But, users can still choose another adapter if they want to.

Why Typhoeus?
Faraday's default adapter (Net::HTTP), specifically for the peculiar way that Google implements streaming in their APIs, is not capable of actually streaming data. We need an adapter with support for Response Streaming + Parallel Requests, which leads to the following options:

async-http adds async, which feels like adding an extra layer of dependency and potential complexity (Async do) unnecessary to the gem.

httpx seems promising, but for some reason, it does not work out of the box for the purpose of understanding Google's particular way of streaming.

Typhoeus just works out of the box, backed by libcurl that is something widely available and supported. Both timeouts and image/video modes work as expected.

@gbaptista gbaptista merged commit ac1b81b into main Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Streaming is not working
1 participant