Robots TXT Deep x402 REST
Paid deep robots.txt audit — per-UA directive counts, sitemap, 34 AI-bot directive detection (GPTBot/ClaudeBot/Google-Extended/CCBot/PerplexityBot/Meta-ExternalAgent/Bytespider/etc), crawl-delay, anti-pattern flags, robo...
862 services carry this tag in their source record. Tags come from the provider, not from us — a tag is a claim about what the service does, not proof that it does it well.
Paid deep robots.txt audit — per-UA directive counts, sitemap, 34 AI-bot directive detection (GPTBot/ClaudeBot/Google-Extended/CCBot/PerplexityBot/Meta-ExternalAgent/Bytespider/etc), crawl-delay, anti-pattern flags, robo...
Paid HTTP cache-control audit — probes favicon/robots/sitemap headers (Cache-Control+max-age+s-maxage+immutable+stale-while-revalidate / ETag / Last-Modified / Vary / Surrogate-Control / CDN-specific) returns cache_valid...
Paid network-timing breakdown for a URL — DNS resolve + TCP connect + TLS handshake + TTFB (time to first byte) + total download phase-by-phase milliseconds + CDN/edge-region detection from CF-Ray/X-Amz-Cf-Pop/X-Vercel-I...
Paid exhaustive meta-tag coverage audit — title + meta_description + meta_robots + meta_keywords + meta_author + meta_generator + meta_theme_color + meta_viewport + 17 OG fields (og:title/description/image/url/type/site_...
Paid deep WCAG 2.1/2.2 AA accessibility audit — forms+inputs (label/aria-label/aria-labelledby/placeholder), autocomplete token validity, button+link accessible-name, heading hierarchy (skip-level + multiple-h1 detection...
Paid OpenAPI 3.x / Swagger spec diff — accepts two URLs (url_a + url_b) or two inline payloads (spec_a + spec_b) as JSON or YAML, reports breaking changes (removed paths, removed operations, type/format changes, enum nar...
Paid security.txt RFC 9116 deep audit — path discovery tries /.well-known/security.txt then /security.txt, parses all fields with line-continuation support, strict Contact format validation (mailto: + https://), strict E...
Paid robots.txt vs meta robots vs X-Robots-Tag conflict detector — 4 conflict types (robots-disallow-vs-meta-index, robots-allow-vs-meta-noindex, X-Robots-Tag-noindex-vs-meta-index, X-Robots-Tag-index-vs-meta-noindex) +...
Paid structural-diff between two URL snapshots — fetches both with 10s timeout, extracts 17 comparable dimensions (title/description/canonical/OG title/desc/image/h1-count/heading-count/image-count/link-count/word-count/...
Paid JavaScript sourcemap exposure audit — fetches every external <script src> + probes inline-script //# sourceMappingURL= references + tries each .map URL via HEAD to surface public source-leak risk, sourcemaps_referen...
Paid URL safety audit (sandbox-friendly) — 12-signal heuristic preflight for AI agents deciding whether to render/embed a URL: scheme allowlist (http = medium risk, mailto/file/javascript/data = high), Suspicious-TLD cat...
Paid form security + accessibility audit — per-form action analysis (empty/#/javascript:/data:/mixed-content http-on-https), method (GET with passwords = credential-in-URL), enctype (text/plain CRLF injection), target=_b...
Paid cache-freshness strategy audit — probes HTML page + /favicon.ico + /robots.txt + /sitemap.xml + first stylesheet URL + first script URL; for each captures Cache-Control directives + ETag + Last-Modified + Age + Vary...
Paid Wayback Machine CDX-API archive availability audit — snapshot_count, first_snapshot, last_snapshot, age_at_first_snapshot_days, age_at_last_snapshot_days, years_active, snapshots_per_year, is_archived, is_recently_a...
Paid social-share preview image audit — parses og:image / og:image:url / og:image:secure_url / twitter:image / twitter:image:src / twitter:image:alt / link rel=image_src / link rel=icon / largest inline <img>, HEAD-valid...
Paid AI/LLM crawl-policy synthesis (x402 $0.001) - GET ?domain=<DOMAIN> reads robots.txt, llms.txt, agents.txt and ai.txt, resolves policy for 40+ AI-bot User-Agents, and returns allow/block counts plus an A-F llm_crawl_...
Paid composite AI-agent friendliness score — rolls up 9 AI-relevant discovery surfaces (llms.txt + llms-full.txt + sitemap.xml + RSS/Atom + robots.txt AI-bot allow-rate + openapi.json + webmanifest + .well-known/agents.t...
Paid redirect-chain map (cycle 95) — follows 3xx chain up to 15 hops without auto-follow (custom HTTPRedirectHandler); per-hop status+location+scheme_change+latency_ms+content_length; detects redirect loops, cross-domain...
Paid llms.txt grader (cycle 95) — probes /llms.txt + /llms-full.txt + /agents.txt per Answer.AI proposed LLM-discovery spec; parses H1 + blockquote summary + markdown list of [Name](URL): description; validates per-line...
Paid hreflang <-> sitemap.xml cross-validator (cycle 94) — extracts page hreflang tags, validates BCP47 + self-reference + x-default + canonical agreement, locates sitemap (sitemap.xml + sitemap_index.xml + robots.txt Si...
Paid DNT/GPC policy probe (cycle 96) — fetches target URL 3x (baseline, with DNT:1, with Sec-GPC:1); compares Set-Cookie count + body size to detect server-side honoring of privacy signals; probes 5 common privacy-policy...
Paid mixed-content body scanner (cycle 97) — parses HTML body for http:// subresources on https:// pages across 12 element types (img/script/link/iframe/source/object/embed/video/audio/form + CSS url() + inline event han...
Paid prompt-safe URL preview (cycle 99) — fetches a URL and returns a plain-text preview with ALL scripts/styles/iframes/objects/embeds/event-handlers/javascript:-URLs/data:-URLs STRIPPED before text extraction; reports...
Paid CSP violation-report ROUND-TRIP replay (cycle 170, x402 $0.001) - fetches target page; extracts CSP header + meta CSP + Reporting-Endpoints header + report-uri + report-to group; for each declared endpoint REPLAYS 4...