henu-wang/ai-crawlers-reference
ai-crawlers-reference
0 catalogued skills
Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples
1HTMLLicense unavailable
GITHUB PROJECTS / RECORDED SOURCE METADATA
Projects worth exploring
Discover projects by topic and publisher. Inspect the source, license and activity before using their code or skills.
cited-io/ai-search-crawlers-robots-txt
ai-search-crawlers-robots-txt
0 catalogued skills
robots.txt for AI crawlers: copy-paste templates that allow AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and let you block AI training (GPTBot, ClaudeBot, Google-Extended), a verified AI crawler list, an llms.txt template and access tests. For generative engine optimization (GEO) and AI SEO.
0TypeScriptMIT