Skip to navigation
Documents

crawl_website

Ingest a public website by following links from a starting URL, up to the requested depth or page limit. The source_url must start with http:// or https://. Website crawling is not supported on-prem. Supported Document Types and Ingest Capacities

Authentication

X-API-Keystring
API Key authentication via header

Request

This endpoint expects an object.
websiteslist of objectsRequired
callbackUrlstringOptionalformat: "uri"
The URL that will receive processing event updates.
callbackDatastringOptional
Custom string included in processing updates sent to the callback URL.

Response

Website successfully queued
ingestobject

Errors

400
Bad Request Error
401
Unauthorized Error
403
Forbidden Error
503
Service Unavailable Error