26 services carry this tag in their source record. Tags come from the provider, not from us — a tag is a claim about what the service does, not proof that it does it well.
Importance-ranked crypto headlines with per-article source links and publish times, plus a sentiment-scored market brief. Regenerated hourly; meta carries the snapshot's age and a fresh/stale verdict.
Crawl a website from a starting URL and return each page as clean markdown: breadth-first over internal links, bounded by page count and depth, honouring robots.txt, with per-page title, status, depth and outbound links....
Bounded async crawl with a job id, a page cap quoted up front, and per-page Markdown.
Metalift scrape: URL → LLM-ready markdown with WAF/retail escalation, JS render, and residential proxy when needed. Use instead of a plain HTTP fetch when Cloudflare, bot checks, or SPAs block cheap scrapers. Also works...
Use to list links on a page as structured rows: url, title, desc, rel, nofollow, internal/external, and kind. Not cleaned article body. An empty page returns item error no_links. After filter and dedupe, all unique links...
Read any public URL as clean Markdown for an LLM: articles, docs, product pages, news, PDFs (text by page), RSS feeds and JSON. Real tables, code with language, math as TeX, absolute links; author, publish date and other...
Website crawler: read a whole website or one section of it (for example every page under /docs) as clean Markdown: starts at a URL, follows same-site links breadth-first and returns up to 20 pages with title and descript...
Discover a website's URLs in one call: reads robots.txt, its declared sitemap(s) (sitemap indexes and gzipped sitemaps included, /sitemap.xml as the fallback) and the start page's internal links, then returns a same-host...
List the URLs of a website without reading them: robots.txt, sitemaps and sitemap indexes plus homepage links, up to 1,000 same-site URLs with lastmod and source. A cheap first step to decide which pages to read with web...
Internal linking audit: crawls up to max_pages same-site pages (robots.txt respected, 30 s cap): broken internal links, orphan pages vs XML sitemap, click depth, dead ends, generic/empty anchors, internal nofollow, dupli...
Firecrawl v2 API proxy — map. JSON body matches https://docs.firecrawl.dev/api-reference/v2-introduction. Poll GET routes are free; POST/PUT/DELETE are x402. $0.010 USDC per request. Agent search: site map, crawl map, fi...
Web page to clean text: title, description, headings, links and readable text from any public URL — Genesis402 / UnyKorn Operator Network
Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN an...
Extract every link from a public page for crawling agents: de-duplicated absolute URLs with anchor text, internal/external counts, declared feeds and a sitemap hint.
Sitemap URL extractor (robots.txt + sitemap index): Finds a site's sitemaps (robots.txt Sitemap lines, /sitemap.xml, /sitemap_index.xml, or a sitemap URL you give), follows sitemap indexes (up to 6 files) and returns up...
Request an https URL and report whether it is reachable, its status, every redirect hop with timings, key and security headers (HSTS, CSP and others), and the TLS certificate with its issuer, the names it covers and the...
Breached password check: is this password in known data breaches, and how often? Send only the SHA-1 hash (never the password); just the first 5 characters reach Have I Been Pwned (k-anonymity). Up to 50 hashes per call.
Encode text or a URL as a QR code: PNG (base64 and a data URL) or SVG, with the size and error correction level you choose.
DNS records for a domain over DNS-over-HTTPS: A, AAAA, CNAME, MX, TXT, NS, SOA, CAA, SRV, PTR or HTTPS, with TTLs and DNSSEC validation flag.
Is a specific bot allowed to fetch a URL? Fetches the site's robots.txt and evaluates the path for the named user-agent (GPTBot, ClaudeBot, Googlebot, Bingbot, CCBot, PerplexityBot or any token), returning allowed/disall...
Sitemap URL list: fetches sitemap.xml (or the URL you give), expands sitemap indexes up to two levels, and returns the page URLs (up to 500 shown), total count, sitemaps visited and lastmod range. Crawl planning and site...
robots.txt fetched and parsed: every user-agent group with its allow/disallow rules and crawl-delay, declared sitemaps, and whether the file exists at all. Understand a site's crawl policy before fetching it. $0.01 per s...
Discover and parse a site's sitemap: pass a site root (found via robots.txt or /sitemap.xml) or a direct .xml. Returns URLs with lastmod/priority; handles <sitemapindex> (one level).
Is a URL crawlable? Fetches the site's robots.txt and evaluates the given path for a specific user-agent (Googlebot, GPTBot, ClaudeBot, Bingbot or any token; default *), returning allowed/disallowed, the matching rule, t...