SKILL

website-crawler-api

Website crawler API: crawl a whole site to clean Markdown, map every URL from its sitemap, or batch extract up to 1,000 URLs in one job, with a status URL or a signed webhook. Map 1 credit; crawl and batch 1 credit per successful page (2 on ultra), failed pages free; robots.txt obeyed.

Target: Software development & testing. Automatically collected · unverified.

Use with Codex

A guide for this combination is not available yet, and compatibility has not been verified.

Requirements

Only the author’s Skill metadata is included. Read the full instructions to check dependencies, licensing and conditions of use.

Choose an AI tool with documented instructions, or save the resource and request a guide after the page loads.

Read the author’s documentation ↗