Website to Markdown Crawler
Give your agent a whole website as clean markdown
Website to Markdown is a crawler API that returns every page under a start URL as markdown, for $0.002 per page. Send the URL of a docs site, help center or blog and get every page under it back as clean markdown, one item per page. For agents and RAG pipelines that need a whole site, not one page.
$0.002 per page delivered. Failed pages are not charged.
{
"url": "https://example.com/docs/setup",
"depth": 1,
"title": "Setup",
"markdown": "# Setup\n\n...",
"truncated": false
}Values shortened. depth is 0 for the start URL, 1 for the pages it links to, and so on. truncated is true when the page passed 3 MB or the markdown passed 200,000 characters and was cut.
Use cases
Give an agent a library's docs
One run turns a documentation site into pages the agent can read and quote, instead of guessing from what it was trained on.
Feed a RAG pipeline
Main content only: menus, footers and cookie banners are left out; headings, lists, tables and links stay.
Crawl one section, not the whole site
Only pages under the start URL's folder are crawled: https://example.com/docs/ crawls /docs/..., not /blog/.
Input and output
{
"startUrl": "https://example.com/docs/",
"maxPages": 50
}startUrl is where the crawl starts (http or https). maxPages is the most pages visited in a run, from 1 to 500 (default 50).
{
"url": "https://accounts.example.com/login",
"depth": 2,
"error": "This page redirected outside the crawled site. Nothing was charged for it."
}Too slow, unreachable, outside the crawled site or without readable text: the item says why.
Pricing
$0.002 per page
Pay per event on Apify, no subscription: one page event per page delivered as markdown. Pages that fail, redirect outside the crawled site or have no readable text are not charged, pages still waiting when the run stops are not fetched or charged, links that robots.txt disallows are skipped without a charge, and a run never charges more than the maximum you set.
Limits
- 1 to 500 pages per run (default 50), from one start URL, on its site and under its folder.
- Each page gets 10 seconds, at most 5 redirects and 3 MB; markdown longer than 200,000 characters is cut and marked
truncated. - No browser and no sitemaps: JavaScript is not run, so pages that only render in a browser may come back without text (and are not charged), and pages are found only through links.
- Fetches 5 pages at a time and respects robots.txt for
Skimmer(andFrameskimCrawl, its earlier name).
Use it by API or from Claude
curl -X POST "https://api.apify.com/v2/acts/emttn~website-markdown-crawler/run-sync-get-dataset-items" \
-H "Authorization: Bearer $APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"startUrl": "https://example.com/docs/", "maxPages": 50}'{ "mcpServers": { "apify": {
"url": "https://mcp.apify.com/?actors=emttn/website-markdown-crawler"
} } }Then ask: "Crawl https://fastapi.tiangolo.com/tutorial/ and explain how dependencies work."
FAQ
What is charged and what is not?
One page event per page delivered as markdown. A page that fails, redirects outside the crawled site or has no readable text is not charged, links that robots.txt disallows are skipped without a charge, and pages still waiting when the run stops are not fetched or charged. What was delivered stays charged if the run later stops or fails, and the run never charges more than the maximum you set: it stops before a page that would not fit.
Which pages does it crawl?
It starts at your URL and follows the links it finds on the same site, only under that URL's folder after its redirects: https://example.com/docs/ crawls /docs/..., not /blog/. Pages are found through links only, not sitemaps, up to the maxPages you set (at most 500). Each item carries its depth from the start URL.
How are the pages read?
Pages are fetched over HTTP through Apify's datacenter proxy with the user agent Skimmer, at most 5 at a time and at most one batch per second. JavaScript is not run, so pages that only render in a browser may come back without text, and are not charged. robots.txt is read once per host and run, and a page is fetched only if the rules for Skimmer, FrameskimCrawl (its earlier name), FrameskimFetch and * all allow it, for the page and again on every redirect.
What about slow, large or empty pages?
Each page gets 10 seconds, at most 5 redirects and 3 MB, and one more try after a timeout, a server error or a dropped connection when there is time left. A page past 3 MB, or with markdown past 200,000 characters, is cut, marked truncated and still charged. A page with no readable text comes back with an error and is not charged. The run's status message says how many pages were delivered, how many links robots.txt kept out and why the crawl ended.
How can a site keep its pages out?
Disallow Skimmer in your robots.txt; rules for FrameskimCrawl, its earlier name, are still honored, and disallowing FrameskimFetch works too. Each run reads a host's robots.txt once, so a run already in progress may not see the change; runs that start after it read the new file. Rights holders and people whose personal data appears in a page can also write to [email protected]. Neither a robots.txt rule nor a request removes pages from datasets that users already hold.