> Markdown versions of all pages of this connector library are available by appending `.md` to the page URL. For the full index of connectors, see [llms.txt](https://connectors.appmixer.com/llms.txt).

# Map Website

- Component: `appmixer.tavily.core.FindSiteLinks`
- Kind: Action
- Version: 1.0.0
- Visibility: Public
- Connector: [appmixer.tavily 1.0.2](https://connectors.appmixer.com/connector/appmixer.tavily/1.0.2.md)
- Component page: <https://connectors.appmixer.com/connector/appmixer.tavily/1.0.2/component/appmixer.tavily.core.FindSiteLinks>

Map the link structure of a website and return the URLs found, without extracting page content. Returns a maximum of 500 links per call.

## Input ports

### Port `in`

| Field | Type | Description |
| --- | --- | --- |
| URL (`url`) | text | The root URL to start from, e.g. `https://docs.tavily.com`. |
| Instructions (`instructions`) | text | Optional natural language guidance for the crawler, e.g. `only pages about pricing`. Doubles the credit cost. |
| Max Depth (`maxDepth`) | number | How many link hops away from the root URL to explore, between 1 and 5. |
| Max Breadth (`maxBreadth`) | number | How many links to follow per page level, between 1 and 500. |
| Max Pages (`maxPages`) | number | Total number of pages to process before stopping, between 1 and 500. |
| Select Paths (`selectPaths`) | textarea | One regular expression per line. Only URL paths matching one of them are followed, e.g. `/documentation/.*`. A comma is a legal part of a pattern (`a{1,3}`), so patterns are separated by newlines, not commas. |
| Exclude Paths (`excludePaths`) | textarea | One regular expression per line. URL paths matching any of them are skipped. |
| Select Domains (`selectDomains`) | textarea | One regular expression per line. Only matching domains are followed, e.g. `^docs\.example\.com$`. |
| Exclude Domains (`excludeDomains`) | textarea | One regular expression per line. Domains matching any of them are skipped. |
| Allow External Domains (`allowExternal`) | toggle | Whether links pointing outside the root domain may be followed. |
| Output Type (`outputType`) | select | Choose whether you want to receive the result set as one complete list, or first item only or one item at a time or stream the items to a file. |

## Output ports

### Port `out`

Variables are loaded dynamically from `/component/appmixer/tavily/core/FindSiteLinks?outPort=out&ignoreAuth=true`.

### Port `notFound`

_No fields defined for this port._

## Full metadata (component.json)

```json
{
  "name": "appmixer.tavily.core.FindSiteLinks",
  "label": "Map Website",
  "description": "Map the link structure of a website and return the URLs found, without extracting page content. Returns a maximum of 500 links per call.",
  "author": "Appmixer <info@appmixer.com>",
  "version": "1.0.0",
  "auth": {
    "service": "appmixer:tavily"
  },
  "quota": {
    "manager": "appmixer:tavily",
    "resources": "requests",
    "scope": {
      "userId": "{{userId}}"
    }
  },
  "inPorts": [
    {
      "name": "in",
      "schema": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string"
          },
          "instructions": {
            "type": "string"
          },
          "maxDepth": {
            "type": "integer",
            "minimum": 1,
            "maximum": 5
          },
          "maxBreadth": {
            "type": "integer",
            "minimum": 1,
            "maximum": 500
          },
          "maxPages": {
            "type": "integer",
            "minimum": 1,
            "maximum": 500
          },
          "selectPaths": {
            "type": "string"
          },
          "excludePaths": {
            "type": "string"
          },
          "selectDomains": {
            "type": "string"
          },
          "excludeDomains": {
            "type": "string"
          },
          "allowExternal": {
            "type": "boolean"
          },
          "outputType": {
            "type": "string"
          }
        },
        "required": [
          "url"
        ]
      },
      "inspector": {
        "inputs": {
          "url": {
            "type": "text",
            "index": 1,
            "label": "URL",
            "tooltip": "The root URL to start from, e.g. <code>https://docs.tavily.com</code>."
          },
          "instructions": {
            "type": "text",
            "index": 2,
            "label": "Instructions",
            "tooltip": "Optional natural language guidance for the crawler, e.g. <code>only pages about pricing</code>. Doubles the credit cost."
          },
          "maxDepth": {
            "type": "number",
            "index": 3,
            "label": "Max Depth",
            "defaultValue": 1,
            "tooltip": "How many link hops away from the root URL to explore, between 1 and 5."
          },
          "maxBreadth": {
            "type": "number",
            "index": 4,
            "label": "Max Breadth",
            "defaultValue": 20,
            "tooltip": "How many links to follow per page level, between 1 and 500."
          },
          "maxPages": {
            "type": "number",
            "index": 5,
            "label": "Max Pages",
            "defaultValue": 50,
            "tooltip": "Total number of pages to process before stopping, between 1 and 500."
          },
          "selectPaths": {
            "type": "textarea",
            "index": 6,
            "label": "Select Paths",
            "tooltip": "One regular expression per line. Only URL paths matching one of them are followed, e.g. <code>/documentation/.*</code>. A comma is a legal part of a pattern (<code>a{1,3}</code>), so patterns are separated by newlines, not commas."
          },
          "excludePaths": {
            "type": "textarea",
            "index": 7,
            "label": "Exclude Paths",
            "tooltip": "One regular expression per line. URL paths matching any of them are skipped."
          },
          "selectDomains": {
            "type": "textarea",
            "index": 8,
            "label": "Select Domains",
            "tooltip": "One regular expression per line. Only matching domains are followed, e.g. <code>^docs\\.example\\.com$</code>."
          },
          "excludeDomains": {
            "type": "textarea",
            "index": 9,
            "label": "Exclude Domains",
            "tooltip": "One regular expression per line. Domains matching any of them are skipped."
          },
          "allowExternal": {
            "type": "toggle",
            "index": 10,
            "label": "Allow External Domains",
            "defaultValue": true,
            "tooltip": "Whether links pointing outside the root domain may be followed."
          },
          "outputType": {
            "type": "select",
            "label": "Output Type",
            "index": 11,
            "defaultValue": "array",
            "tooltip": "Choose whether you want to receive the result set as one complete list, or first item only or one item at a time or stream the items to a file.",
            "options": [
              {
                "label": "First Item Only",
                "value": "first"
              },
              {
                "label": "All items at once",
                "value": "array"
              },
              {
                "label": "One item at a time",
                "value": "object"
              },
              {
                "label": "Store to CSV file",
                "value": "file"
              }
            ]
          }
        }
      }
    }
  ],
  "outPorts": [
    {
      "name": "out",
      "source": {
        "url": "/component/appmixer/tavily/core/FindSiteLinks?outPort=out&ignoreAuth=true",
        "data": {
          "properties": {
            "generateOutputPortOptions": true
          },
          "messages": {
            "in/outputType": "inputs/in/outputType",
            "in/url": "dummy"
          }
        }
      }
    },
    {
      "name": "notFound"
    }
  ],
  "icon": "(data URI omitted)"
}
```
