Websearch Improvements

[Josh-XT]

Websearch Improvements

There should be no breaking changes. There are feature additions and refinements of current functionality around websearch and memory in this PR.

Notes

• Only browse link if starts with http. Some scraped links started with javascript: and were causing issues. This resolves that problem.
• Created a verify_link function to check if a link has been browsed instead of using the same code in multiple places.
• Removed most of the requirements to initialize the websearch agent to make it more flexible.
• Directly learning URLs now instead of over the ApiClient to speed up websearches.
• Increased the minimum relevance score for context injection to 0.3 to get higher quality context injection based on user input.
• Added stored tracking of what URLs have been browsed per agent.
• Added GET /api/agent/{agent_name}/browsed_links endpoint to get a list of URLs that the agent has browsed along with timestamps of when the URL was browsed
• Added DELETE /api/agent/{agent_name}/browsed_links endpoint to delete a URL from the list of URLs that the agent has browsed.
• Added GET /api/agent/{agent_name}/memory/external_sources endpoint to list unique external sources (such as memories from a specific URL or file) of agent memories over API.
• Added DELETE /api/agent/{agent_name}/memory/external_source endpoint to delete all memories of a specific external source over API.

Todo

• Review tests to confirm there are no errors on this branch.
• Test websearch and browsing links in URLs on this branch.
• Create issue on AGiXT Python SDK repository to add new endpoints.
  • Create issue on AGiXT repository to create tests for new endpoints after Python SDK is updated.
• Create issue on AGiXT TypeScript SDK repository to add new endpoints.
• Create issue on AGiXT C# SDK repository to add new endpoints.
• Create issue on AGiXT Dart SDK repository to add new endpoints.
• Create issue on AGiXT Streamlit repository to add ability to list unique external sources, and delete selected ones per agent.
• Create issue on AGiXT Streamlit repository to add ability to list and delete browsed links per agent.