api.crawlwitness.com · 2 listed services · provider site
Use when an agent is about to crawl, train on or ingest a website into RAG: checks the domain's AI consent and licensing signals (robots.txt per AI bot, RSL license, IETF AIPREF, TDMRep, noai meta tags, llms.txt, C2PA),...
AI consent and licensing check for a web domain before crawling, training or RAG ingestion: robots.txt rules per AI bot, RSL license, IETF AIPREF, TDMRep, noai meta tags, llms.txt and C2PA, combined into a 0-100 score an...
Provider identity is derived from the domain or repository in the source record. If two vendors share a host, they appear together here — that is a limitation of the data, not a statement about ownership.