[feature] web fetch agent

[sinedied]

Often times, I need to write a simple agent that extract links to web pages in a prompt to fetch the text data from it and augment the context with the fetched files.

There's a built-in web search agent, but it would be nice to also have a simple "web fetch" agents that recognize URLs, fetch the text from it and add it to FILES. This would also removes the need for an API key needed for the web search.

[pelikhan]

Yes there's a bunch of things we could do to make scrapping easier.

• You can try add urls to def and it will fetch them def("FILE", { filename: "https://..." }).
• You can also try parseReadableContent from 'genaiscript/runtime` which uses playwright and @mozilla/readability to extract the HTML of a page.

Other idea: fetchText should have an option to convert HTML to markdown as well.

[pelikhan]

• allowed list of domains
• fetch vs browse vs browse/screenshot/ocr

[pelikhan]

Added system.fetch
#1474

[github-actions]

The issue involves creating a simple “web fetch” agent within the genaiscript project to extract content from URLs, augmenting the context with fetched files. Existing web search functionality utilizes an API key.

Comments suggest using def("FILE", { filename: "https://..." }) to fetch content or leveraging parseReadableContent from genaiscript/runtime (via Playwright and @mozilla/readability). Additionally, there's a proposal to add an option to convert HTML to markdown during the fetching process.

A pull request (PR) titled “system.fetch” has been added to the repository.

AI-generated content by issue-bot may be incorrect