Web Search & Page Markdown

Give your agent the top web pages as clean markdown

Q: What is charged and what is not?

One search event per query that returned results and one page event per page delivered as markdown. A query that fails or finds nothing, and every page that fails, is blocked by its site or has no readable text, is not charged. A query with at least one result is charged even if none of its pages can be delivered, and what was delivered stays charged if the run later stops or fails. The run never charges more than the maximum you set.

Q: What is the search that Apify bills apart?

Each query runs Apify's Google Search Results Scraper (apify/google-search-scraper) once, on your own Apify account, for one page of results. Apify bills that run to you apart, at that actor's prices, also when the query finds nothing. The run asks for one page with every paid add-on off and caps its time and charge. Apify does not count it in the maximum you set for this run, so this run takes what each search may cost off that maximum before it starts the next query. The search run, its input and its results stay in your account like any other run.

Q: How are the pages read?

Pages are fetched over HTTP with the user agent FrameskimFetch. JavaScript is not run, so pages that only render in a browser may come back without text, and are not charged. robots.txt is respected for FrameskimFetch and for *. Each page gets 10 seconds, at most 5 redirects and 3 MB.

Q: How can a site keep its pages out?

Disallow FrameskimFetch in your robots.txt. Each run reads a host's robots.txt once, so a run already in progress may not see the change; runs that start after it read the new file. Rights holders and people whose personal data appears in a page can also write to support@frameskim.dev. Neither a robots.txt rule nor a request removes pages from datasets that users already hold.

Web Search & Page Markdown is a web search API for LLMs that returns the top result pages as markdown, for $0.005 per search and $0.004 per page. Send a question and get the top result pages already turned into markdown, in the order the search ranked them, in one call. Or send your own URLs and skip the search.

Run it on Apify View API example

$0.004 per page delivered and $0.005 per search with results, plus the Apify search run billed apart. Failed pages are not charged.

Each query: the top pages, in the order the search ranked them, as clean markdown.

Output, one item per page

{
  "source": "search",
  "query": "how does robots.txt work",
  "rank": 1,
  "url": "https://www.rfc-editor.org/rfc/rfc9309.html",
  "title": "RFC 9309: Robots Exclusion Protocol",
  "markdown": "# Robots Exclusion Protocol\n\n...",
  "truncated": false
}

Values shortened. rank is the position in the results. For a page you listed, source is url, and query and rank are null.

Use cases

Let an agent read the web

One tool call returns the pages, not just links, so the agent can answer from what they say.

Feed a RAG pipeline

Main content only: menus, footers and cookie banners are left out; headings, lists, tables and links stay.

Turn known URLs into markdown

Send up to 50 URLs in a run and get each page as markdown, with no search at all.

Input and output

Input

{
  "query": "markdown table syntax",
  "maxResults": 3
}

Or queries, up to 20 queries per run, and urls, up to 50 URLs per run. maxResults is 1 to 10 pages per query (default 5).

A page that failed

{
  "source": "url",
  "query": null,
  "rank": null,
  "url": "https://example.com/slow",
  "title": null,
  "error": "This page took longer than 10 s. Nothing was charged for it."
}

Blocked by robots.txt, too slow, no readable text: the item says why.

Pricing

$0.005 per search,
$0.004 per page

Pay per event on Apify, no subscription: one search event per query that returned results and one page event per page delivered as markdown. Pages that fail, are blocked or have no readable text are not charged, and a run never charges more than the maximum you set. AI agents can pay for each run on their own, per request, via x402.

Each search also runs Apify's apify/google-search-scraper once on your own Apify account, billed by Apify apart at that actor's prices, even when it finds nothing.

Limits

Up to 20 queries and up to 50 URLs per run, 1 to 10 pages per query.
Each page gets 10 seconds, at most 5 redirects and 3 MB; longer markdown is cut and marked truncated.
No crawling and no JavaScript: pages that only render in a browser may come back without text, and are not charged.
Respects robots.txt for FrameskimFetch.

Use it by API or from Claude

API

curl -X POST "https://api.apify.com/v2/acts/emttn~web-search-markdown/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "markdown table syntax", "maxResults": 3}'

MCP

{ "mcpServers": { "apify": {
  "url": "https://mcp.apify.com/?actors=emttn/web-search-markdown"
} } }

Then ask: "Search the web for how robots.txt works and summarize the top pages."

FAQ

What is charged and what is not?

One search event per query that returned results and one page event per page delivered as markdown. A query that fails or finds nothing, and every page that fails, is blocked by its site or has no readable text, is not charged. A query with at least one result is charged even if none of its pages can be delivered, and what was delivered stays charged if the run later stops or fails. The run never charges more than the maximum you set.

What is the search that Apify bills apart?

Each query runs Apify's Google Search Results Scraper (apify/google-search-scraper) once, on your own Apify account, for one page of results. Apify bills that run to you apart, at that actor's prices, also when the query finds nothing. The run asks for one page with every paid add-on off and caps its time and charge. Apify does not count it in the maximum you set for this run, so this run takes what each search may cost off that maximum before it starts the next query. The search run, its input and its results stay in your account like any other run.

How are the pages read?

Pages are fetched over HTTP with the user agent FrameskimFetch. JavaScript is not run, so pages that only render in a browser may come back without text, and are not charged. robots.txt is respected for FrameskimFetch and for *. Each page gets 10 seconds, at most 5 redirects and 3 MB.

What are the limits?

One results page per query, so at most 10 results per query; results come from the United States edition of Google. There is no crawling: only the result pages and the URLs you list. The search interface runs in English; which results come back is up to the search engine.

How can a site keep its pages out?

Disallow FrameskimFetch in your robots.txt. Each run reads a host's robots.txt once, so a run already in progress may not see the change; runs that start after it read the new file. Rights holders and people whose personal data appears in a page can also write to [email protected]. Neither a robots.txt rule nor a request removes pages from datasets that users already hold.