27 services carry this tag in their source record. Tags come from the provider, not from us — a tag is a claim about what the service does, not proof that it does it well.
Is a URL crawlable? Fetches the site's robots.txt and evaluates the given path for a specific user-agent (Googlebot, GPTBot, ClaudeBot, Bingbot or any token; default *), returning allowed/disallowed, the matching rule, t...
Extract all outbound links from a web page's main content as absolute URLs, plus its headings outline and title. Crawling, link discovery, sitemap building, backlink and citation research. $0.01 per page.
Extract every link from a public page for crawling agents: de-duplicated absolute URLs with anchor text, internal/external counts, declared feeds and a sitemap hint.