Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make use of OpenAI's "predicted outputs" feature #15

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Make use of OpenAI's "predicted outputs" feature #15

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1c961fd
feat: extract markdown at server side and use predicted output feature
CNSeniorious000 Nov 9, 2024
956c0fd
feat: use `turndown` in the browser
CNSeniorious000 Nov 9, 2024
fca371d
fix: avoid accessing `delta` from usage chunk
CNSeniorious000 Nov 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/routes/(static)/read/Right.svelte
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

async function fetchLLMResult() {
llmResult = ""
const res = await fetch("/api/extract", { body: cleanedHtml, method: "POST" })
const res = await fetch("/api/extract", { body: JSON.stringify({ html, markdown: readabilityResult }), method: "POST" })
for await (const delta of responseToTextStream(res))
llmResult += delta
}
Expand Down
9 changes: 6 additions & 3 deletions src/routes/api/extract/+server.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { OpenAI } from "openai"

const client = new OpenAI()

async function * extract(html: string) {
async function * extract(html: string, markdown: string) {
for await (const chunk of await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
Expand All @@ -16,19 +16,22 @@ async function * extract(html: string) {
],
stream: true,
temperature: 0,
prediction: { type: "content", content: markdown },
stream_options: { include_usage: true },
})) {
chunk.usage && console.error(chunk.usage)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

建议： 考虑使用适当的日志系统而不是 console.error 进行使用跟踪

对于生产环境，结构化日志将使收集和分析使用数据更容易

    if (chunk.usage) {
      logger.info('API Usage', { usage: chunk.usage })
    }
Original comment in English

suggestion: Consider using a proper logging system instead of console.error for usage tracking

For production environments, structured logging would make it easier to collect and analyze usage data

    if (chunk.usage) {
      logger.info('API Usage', { usage: chunk.usage })
    }

const delta = chunk.choices[0].delta.content
if (delta)
yield delta
}
}

export const POST: RequestHandler = async ({ request }) => {
const html = await request.text()
const { html, markdown } = await request.json()
if (!html)
error(400, "Missing request body")

return new Response(iteratorToStream(extract(html)), { headers: { "content-type": "text/markdown" } })
return new Response(iteratorToStream(extract(html, markdown)), { headers: { "content-type": "text/markdown" } })
}

export const config = { runtime: "edge" }