appmixer.firecrawl.core.CrawlWebsite
Crawl Website
1.0.0
Public
| Field | Type | Description |
|---|---|---|
| URL | text | The base URL to start crawling from, e.g. https://example.com/docs. |
| Max Pages | number | Maximum number of pages to crawl. Defaults to 100. Each crawled page consumes Firecrawl credits. |
| Max Discovery Depth | number | Maximum depth to follow links from the base URL. Leave empty for no depth limit. |
| Include Paths | text | Optional comma separated list of URL path regex patterns to include, e.g. blog/.*, docs/.*. Only matching URLs are crawled. |
| Exclude Paths | text | Optional comma separated list of URL path regex patterns to exclude from the crawl. |
| Crawl Entire Domain | toggle | Also follow links to sibling and parent pages of the base URL, not only its children. |
| Allow Subdomains | toggle | Also follow links to subdomains of the website. |
| Only Main Content | toggle | Scrape only the main content of each page, excluding headers, navigation and footers. |
| Correlation ID | text | Any value of your own - an order number, a record ID - echoed on both output ports. One component instance has one callback URL, so parallel crawls report back to the same place in completion order; this is how a downstream component tells them apart. |
| Field | Type | Description |
|---|---|---|
| Job ID | string | - |
| Status | string | - |
| URL | string | - |
| Correlation ID | string | - |
| Field | Type | Description |
|---|---|---|
| Job ID | string | - |
| Status | string | - |
| Total Pages | number | - |
| Completed Pages | number | - |
| Credits Used | number | - |
| Pages | array | - |
| Truncated | boolean | - |
| URL | string | - |
| Correlation ID | string | - |
| Error | string | - |