appmixer.tavily.core.FindCrawledPages
Crawl Website
1.0.0
Public
| Field | Type | Description |
|---|---|---|
| URL | text | The root URL to start from, e.g. https://docs.tavily.com. |
| Instructions | text | Optional natural language guidance for the crawler, e.g. only pages about pricing. Doubles the credit cost. |
| Max Depth | number | How many link hops away from the root URL to explore, between 1 and 5. |
| Max Breadth | number | How many links to follow per page level, between 1 and 500. |
| Max Pages | number | Total number of pages to process before stopping, between 1 and 500. |
| Select Paths | textarea | One regular expression per line. Only URL paths matching one of them are followed, e.g. /documentation/.*. A comma is a legal part of a pattern (a{1,3}), so patterns are separated by newlines, not commas. |
| Exclude Paths | textarea | One regular expression per line. URL paths matching any of them are skipped. |
| Select Domains | textarea | One regular expression per line. Only matching domains are followed, e.g. ^docs\.example\.com$. |
| Exclude Domains | textarea | One regular expression per line. Domains matching any of them are skipped. |
| Allow External Domains | toggle | Whether links pointing outside the root domain may be followed. |
| Extract Depth | select | Advanced retrieves more data per page at a higher credit cost. |
| Format | select | The format of the extracted page content. |
| Output Type | select | Choose whether you want to receive the result set as one complete list, or first item only or one item at a time or stream the items to a file. |
Dynamic variables loaded from /component/appmixer/tavily/core/FindCrawledPages?outPort=out&ignoreAuth=true
| Field | Type | Description |
|---|---|---|
| No fields defined for this port | ||