Firecrawl example docs #1438
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,103 @@ | ||||||||||||||
| --- | ||||||||||||||
| title: "Crawl a URL using Firecrawl" | ||||||||||||||
| sidebarTitle: "Firecrawl URL crawl" | ||||||||||||||
| description: "This example demonstrates how to crawl a URL using Firecrawl with Trigger.dev." | ||||||||||||||
| --- | ||||||||||||||
|
|
||||||||||||||
| ## Overview | ||||||||||||||
|
|
||||||||||||||
| Firecrawl is a tool for crawling websites and extracting clean markdown that's structured in an LLM-ready format. | ||||||||||||||
|
|
||||||||||||||
| Here are two examples of how to use Firecrawl with Trigger.dev: | ||||||||||||||
|
|
||||||||||||||
| ## Prerequisites | ||||||||||||||
|
|
||||||||||||||
| - A project with [Trigger.dev initialized](/quick-start) | ||||||||||||||
| - A [Firecrawl](https://firecrawl.dev/) account | ||||||||||||||
|
|
||||||||||||||
| ## Example 1: crawl an entire website with Firecrawl | ||||||||||||||
|
|
||||||||||||||
| This task crawls a website and returns the `crawlResult` object. You can set the `limit` parameter to control the number of URLs that are crawled. | ||||||||||||||
|
|
||||||||||||||
| ```ts trigger/firecrawl-url-crawl.ts | ||||||||||||||
| import FirecrawlApp from "@mendable/firecrawl-js"; | ||||||||||||||
| import { task } from "@trigger.dev/sdk/v3"; | ||||||||||||||
|
|
||||||||||||||
| // Initialize the Firecrawl client with your API key | ||||||||||||||
| const firecrawlClient = new FirecrawlApp({ | ||||||||||||||
| apiKey: process.env.FIRECRAWL_API_KEY, // Get this from your Firecrawl dashboard | ||||||||||||||
| }); | ||||||||||||||
|
|
||||||||||||||
| export const firecrawlCrawl = task({ | ||||||||||||||
| id: "firecrawl-crawl", | ||||||||||||||
| run: async (payload: { url: string }) => { | ||||||||||||||
| const { url } = payload; | ||||||||||||||
|
|
||||||||||||||
| // Crawl: scrapes all the URLs of a web page and return content in LLM-ready format | ||||||||||||||
| const crawlResult = await firecrawlClient.crawlUrl(url, { | ||||||||||||||
| limit: 100, // Limit the number of URLs to crawl | ||||||||||||||
| scrapeOptions: { | ||||||||||||||
| formats: ["markdown", "html"], | ||||||||||||||
| }, | ||||||||||||||
| }); | ||||||||||||||
|
|
||||||||||||||
| if (!crawlResult.success) { | ||||||||||||||
| throw new Error(`Failed to crawl: ${crawlResult.error}`); | ||||||||||||||
| } | ||||||||||||||
|
|
||||||||||||||
| return { | ||||||||||||||
| data: crawlResult, | ||||||||||||||
| }; | ||||||||||||||
| }, | ||||||||||||||
| }); | ||||||||||||||
|
Comment on lines
+31
to
+52
There was a problem hiding this comment. 🛠️ Refactor suggestion Add type safety and improve error handling. The implementation could benefit from:
+import { CrawlResponse } from "@mendable/firecrawl-js";
+
export const firecrawlCrawl = task({
id: "firecrawl-crawl",
- run: async (payload: { url: string }) => {
+ run: async (payload: { url: string; limit?: number }) => {
const { url } = payload;
// Crawl: scrapes all the URLs of a web page and return content in LLM-ready format
- const crawlResult = await firecrawlClient.crawlUrl(url, {
+ const crawlResult = await firecrawlClient.crawlUrl(url, {
limit: 100, // Limit the number of URLs to crawl
scrapeOptions: {
formats: ["markdown", "html"],
},
- });
+ }) as CrawlResponse;
|
||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ### Testing your task | ||||||||||||||
|
|
||||||||||||||
| You can test your task by triggering it from the Trigger.dev dashboard. | ||||||||||||||
|
|
||||||||||||||
| ```json | ||||||||||||||
| "url": "<url-to-crawl>" // Replace with the URL you want to crawl | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ## Example 2: scrape a single URL with Firecrawl | ||||||||||||||
|
|
||||||||||||||
| This task scrapes a single URL and returns the `scrapeResult` object. | ||||||||||||||
|
|
||||||||||||||
| ```ts trigger/firecrawl-url-scrape.ts | ||||||||||||||
| import FirecrawlApp, { ScrapeResponse } from "@mendable/firecrawl-js"; | ||||||||||||||
| import { task } from "@trigger.dev/sdk/v3"; | ||||||||||||||
|
|
||||||||||||||
| // Initialize the Firecrawl client with your API key | ||||||||||||||
| const firecrawlClient = new FirecrawlApp({ | ||||||||||||||
| apiKey: process.env.FIRECRAWL_API_KEY, // Get this from your Firecrawl dashboard | ||||||||||||||
| }); | ||||||||||||||
|
|
||||||||||||||
| export const firecrawlScrape = task({ | ||||||||||||||
| id: "firecrawl-scrape", | ||||||||||||||
| run: async (payload: { url: string }) => { | ||||||||||||||
| const { url } = payload; | ||||||||||||||
|
|
||||||||||||||
| // Scrape: scrapes a URL and get its content in LLM-ready format (markdown, structured data via LLM Extract, screenshot, html) | ||||||||||||||
| const scrapeResult = (await firecrawlClient.scrapeUrl(url, { | ||||||||||||||
| formats: ["markdown", "html"], | ||||||||||||||
| })) as ScrapeResponse; | ||||||||||||||
|
Comment on lines
+82
to
+84
There was a problem hiding this comment. Avoid type casting and document the response structure. The type casting to - const scrapeResult = (await firecrawlClient.scrapeUrl(url, {
+ const scrapeResult: ScrapeResponse = await firecrawlClient.scrapeUrl(url, {
formats: ["markdown", "html"],
- })) as ScrapeResponse;
+ });📝 Committable suggestion
Suggested change
|
||||||||||||||
|
|
||||||||||||||
| if (!scrapeResult.success) { | ||||||||||||||
| throw new Error(`Failed to scrape: ${scrapeResult.error}`); | ||||||||||||||
| } | ||||||||||||||
|
|
||||||||||||||
| return { | ||||||||||||||
| data: scrapeResult, | ||||||||||||||
| }; | ||||||||||||||
| }, | ||||||||||||||
| }); | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ### Testing your task | ||||||||||||||
|
|
||||||||||||||
| You can test your task by triggering it from the Trigger.dev dashboard. | ||||||||||||||
|
|
||||||||||||||
| ```json | ||||||||||||||
| "url": "<url-to-scrape>" // Replace with the URL you want to scrape | ||||||||||||||
| ``` | ||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🛠️ Refactor suggestion
Consider making the
limitparameter configurable.The hard-coded limit of 100 URLs might not suit all use cases. Consider:
const crawlResult = await firecrawlClient.crawlUrl(url, { - limit: 100, // Limit the number of URLs to crawl + limit: payload.limit ?? 100, // Use provided limit or default to 100 scrapeOptions: { formats: ["markdown", "html"], }, });