website-crawler-api

Topics in the source

• Website Crawler API - Map, Crawl and Batch Extract to Markdown
• When to trigger
• Setup
• If you do not have a key yet
• Endpoints
• Cost
• Plan limits
• Parameters

Crawl a whole website and get every page back as clean Markdown, plain text or HTML; list a site's URLs from its sitemap; or extract a list of URLs in one job. Three operations on the Extract family: Map - a site's page URLs from its sitemaps and start page. Synchronous, 1 credit. Crawl - discover and extract one site's pages breadth-first as a job. Billed per successful page. Batch - extract up to 1,000 URLs you already have as a job. Billed per successful page. For a single page, use the url-to-markdown-api skill (POST /api/v1/extract) instead. Use this skill when the user asks to: Crawl a website, a docs site, a blog or a help center Turn a whole site (or a section of it) into Markdown for an LLM, a vector store or RAG List every URL on a site, read its sitemap, or find pages under a path Convert a sitemap into Markdown Scrape or extract a list of URLs in one go (batch scrape) Build a knowledge base or keep one fresh from a site If the user pastes one link and asks about it, that is single-page Extract, not this skill. Get a free API key at scavio.dev (50 free credits to get started, no card required): An agent running this skill without SCAVIOAPIKEY set will get 401 on every ca

This is an excerpt. Refer to the complete source and its current requirements.