Add 2 firecrawl tools : Scrape and Search #6016
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implement the ScrapeTool class inheriting from BuiltinTool in scrape.py, which uses the FirecrawlApp to scrape data from a given URL. The tool supports custom scraping preferences and can return either scraped documents or URLs. Also, include the scrape.yaml configuration file to define the tool's identity, description, and parameters. BREAKING CHANGE: The addition of the scrape tool may affect existing workflows that do not account for this new tool. Ensure that your environment is prepared to handle the scrape tool beforedeploying this change. By tongyi
Introduce the SearchTool class within the firecrawl tools, implementing functionality for searching data using the Firecrawl API. This update also changes the author field for the scrape tool from 'Richards Tu' to 'ahasasjeb'. BREAKING CHANGE: The addition of the search tool and modification of the scrape tool's author field may affect existing configurations or dependencies. Ensure to review and update accordingly before deploying. By tongyi
dosubot
bot
added
the
size:M
5 tasks
ahasasjeb
changed the title
ahasasjeb
changed the title
crazywoola
reviewed
Jul 5, 2024
It shouldn't be involved here |
crazywoola
approved these changes
Jul 6, 2024
ZhouhaoJiang
added a commit
that referenced
this pull request
Jul 7, 2024
* refs/heads/fix/dataset_operator: (33 commits) feat: update dataset sort feat: add dataset_permissions tenant_id chore: optimize memory fetch performance (#6039) feat: support moonshot and glm base models for volcengine provider (#6029) Optimize db config (#6011) fix: token count includes base64 string of input images (#5868) chore: skip pip upgrade preparation in api dockerfile (#5999) feat(*): Swtich to dify_config. (#6025) fix: the input field of tool panel not worked as expected (#6003) Add 2 firecrawl tools : Scrape and Search (#6016) test(test_rerank): Remove duplicate test cases. (#6024) chore: optimize memory messages fetch count limit (#6021) Revert "feat: knowledge admin role" (#6018) feat: add Llama 3 and Mixtral model options to ddgo_ai.yaml (#5979) fix: add status_code 304 (#6000) 6014 i18n add support for spanish (#6017) [Feature] Support loading for mermaid. (#6004) fix: update workflow trace query (#6010) Removed firecrawl-py, fixed and improved firecrawl tool (#5896) fix API tool's schema not support array (#6006) ...
Merged
laipz8200
added a commit
that referenced
this pull request
Jul 10, 2024
Co-authored-by: -LAN- <laipz8200@outlook.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist:
Important
Please review the checklist below before submitting your pull request.
dev/reformat(backend) andcd web && npx lint-staged(frontend) to appease the lint godsDescription
Add 2 Firecrawler tools : Scrape and Search.
I have tested it using Docker.
Fixes #6015
Type of Change
Testing Instructions
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration