URL Metadata (Open Graph, JSON-LD, SEO tags)

Give your agent any page's metadata as clean JSON

Q: What is charged and what is not?

One page event per page whose metadata is delivered, charged right after its item is saved. A page that fails, is not HTML, has no title, description or Open Graph, or that robots.txt does not allow is not charged and comes back as an item that says why, and pages not started when the run stops are not fetched or charged. The run never charges more than the maximum you set: it stops before a page that would pass it.

Q: How can a site keep its pages out?

Disallow FrameskimFetch in your robots.txt; that also stops Web Search & Page Markdown, Website to Markdown Crawler and Document to Markdown. Each run reads a host's robots.txt once, so a run already in progress may not see the change; runs that start after it read the new file. Rights holders and people whose personal data appears in a page's metadata can also write to support@frameskim.dev. Neither a robots.txt rule nor a request removes metadata from datasets that users already hold.

URL Metadata is an API that returns the title, description, Open Graph and JSON-LD of a page as data, for $0.002 per page. Send page URLs and get back each page's title, description, canonical URL, Open Graph, Twitter card, JSON-LD and hreflang as data, one item per page. For agents that need to know what a URL is about without reading the whole page.

Run it on Apify View API example

$0.002 per page delivered. Failed pages are not charged.

A list of URLs: what each page says about itself, as data.

Output, one item per page

{
  "url": "https://example.com/blog/launch",
  "status": 200,
  "title": "We launched",
  "description": "What is new in version 2.",
  "canonical": "https://example.com/blog/launch",
  "lang": "en",
  "robots": "index, follow",
  "favicon": "https://example.com/favicon.ico",
  "openGraph": { "image": "https://example.com/og.png" },
  "twitter": { "card": "summary_large_image" },
  "jsonLd": [{ "@type": "BlogPosting" }],
  "hreflang": [{ "lang": "es", "url": "https://example.com/es/" }],
  "truncated": false
}

Values shortened. url is the final URL after redirects, and every URL in the item is absolute. A missing field is null, or empty for openGraph, twitter, jsonLd and hreflang. Without an icon link, favicon is the host's /favicon.ico. truncated is true when some JSON-LD was left out or the page passed 3 MB and was cut.

Use cases

Link previews for an agent or a chat

Title, description, image and icon of each link, so an agent can say what a URL is or show a preview card without reading the page.

Check the SEO tags of a list of URLs

Canonical URL, meta robots, language and hreflang of every page in one run, and the pages that are missing a title or a description.

Know what a page is without reading it

The structured data a page declares (a product, an article, an organization, an event) comes back as JSON objects, ready to use.

Input and output

Input

{
  "urls": [
    "https://example.com/",
    "https://example.com/blog/launch"
  ]
}

urls takes from 1 to 100 URLs of HTML pages (http or https). A repeated URL is read once.

A page that failed

{
  "url": "https://example.com/empty",
  "error": "No metadata was found on this page. Nothing was charged for it."
}

Too slow, unreachable, not HTML, blocked by robots.txt or without a title, description or Open Graph: the item says why.

Pricing

$0.002 per page

Pay per event on Apify, no subscription: one page event per page whose metadata is delivered. Pages that fail, are not HTML, have no title, description or Open Graph, or that robots.txt does not allow are not charged, pages not started when the run stops are not fetched or charged, and a run never charges more than the maximum you set.

Limits

1 to 100 URLs per run; a repeated URL is read once.
Each page gets 10 seconds, at most 5 redirects and the first 3 MB, with no retry; JSON-LD is kept up to 100,000 characters.
No browser: JavaScript is not run, so only the metadata in the page's HTML comes back.
Fetches 5 pages at a time and respects robots.txt for FrameskimFetch.

Use it by API or from Claude

API

curl -X POST "https://api.apify.com/v2/acts/emttn~url-metadata/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/"]}'

MCP

{ "mcpServers": { "apify": {
  "url": "https://mcp.apify.com/?actors=emttn/url-metadata"
} } }

Then ask: "Read the metadata of https://www.python.org/ and tell me its Open Graph image and canonical URL."

FAQ

What is charged and what is not?

One page event per page whose metadata is delivered, charged right after its item is saved. A page that fails, is not HTML, has no title, description or Open Graph, or that robots.txt does not allow is not charged and comes back as an item that says why, and pages not started when the run stops are not fetched or charged. The run never charges more than the maximum you set: it stops before a page that would pass it.

Which metadata does it return?

What the page declares in its HTML: title, description, canonical URL, language, meta robots, icon, Open Graph and Twitter card tags, every valid JSON-LD object (a list is flattened) and hreflang alternates, with the final URL and the HTTP status. When the page declares no icon, favicon is the host's /favicon.ico. The page itself is read only during the run and is not stored or returned.

How are the pages read?

Pages are fetched over HTTP through Apify's datacenter proxy with the user agent FrameskimFetch, at most 5 at a time, and JavaScript is not run. robots.txt is read once per host and run, and a page is fetched only if the rules for FrameskimFetch and * allow it, for the page and again on every redirect.

What about slow, large or non-HTML pages?

Each page gets 10 seconds and at most 5 redirects, with no retry; only its first 3 MB are read, and a page cut there is marked truncated and still charged if it has metadata. A URL that is not an HTML page comes back with an error and is not charged. The run's status message says how many pages were delivered and failed, and why the run stopped early, if it did.

How can a site keep its pages out?

Disallow FrameskimFetch in your robots.txt; that also stops Web Search & Page Markdown, Website to Markdown Crawler and Document to Markdown. Each run reads a host's robots.txt once, so a run already in progress may not see the change; runs that start after it read the new file. Rights holders and people whose personal data appears in a page's metadata can also write to [email protected]. Neither a robots.txt rule nor a request removes metadata from datasets that users already hold.