# Repository-owned crawler policy. Cloudflare may prepend its Managed Content # block in production. Its Content-Signal field is a non-standard policy signal; # the standard User-agent + Allow/Disallow groups below remain the testable crawl # directives and are kept semantically compatible with that managed block. User-agent: * Allow: / # Search indexing and user-initiated answer/retrieval crawlers. User-agent: Googlebot User-agent: Bingbot User-agent: Baiduspider User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / # Google crawlers used to review pages and serve AdSense inventory. User-agent: Mediapartners-Google Allow: / User-agent: Google-Display-Ads-Bot Allow: / # Training and other automated data-use crawlers currently excluded by policy. User-agent: Amazonbot User-agent: Applebot-Extended User-agent: Bytespider User-agent: CCBot User-agent: ClaudeBot User-agent: cohere-ai User-agent: GPTBot User-agent: meta-externalagent User-agent: anthropic-ai Disallow: / # Google-Extended covers both some Gemini grounding and generative-AI training # or related data use. The current business choice is to keep both opted out; # this group does not control ordinary Google Search crawling by Googlebot. User-agent: Google-Extended Disallow: / Sitemap: https://getgiffgaff.com/sitemap.xml