User-agent: * Allow: / Disallow: /api/ Disallow: /login/ Sitemap: https://weareocta.com/sitemap.xml # AI retrieval crawlers — allowed for citation and indexing User-agent: GPTBot Allow: / Disallow: /api/ Disallow: /login/ User-agent: ChatGPT-User Allow: / Disallow: /api/ Disallow: /login/ User-agent: Google-Extended Allow: / Disallow: /api/ Disallow: /login/ User-agent: ClaudeBot Allow: / Disallow: /api/ Disallow: /login/ User-agent: Claude-Web Allow: / Disallow: /api/ Disallow: /login/ User-agent: Bingbot Allow: / Disallow: /api/ Disallow: /login/ User-agent: PerplexityBot Allow: / Disallow: /api/ Disallow: /login/ User-agent: Amazonbot Allow: / Disallow: /api/ Disallow: /login/ User-agent: anthropic-ai Allow: / Disallow: /api/ Disallow: /login/ User-agent: Applebot-Extended Allow: / Disallow: /api/ Disallow: /login/ User-agent: cohere-ai Allow: / Disallow: /api/ Disallow: /login/ User-agent: Meta-ExternalFetcher Allow: / Disallow: /api/ Disallow: /login/ # Social media link-preview scrapers User-agent: LinkedInBot Allow: / Disallow: /api/ Disallow: /login/ User-agent: Twitterbot Allow: / Disallow: /api/ Disallow: /login/ User-agent: facebookexternalhit Allow: / Disallow: /api/ Disallow: /login/ User-agent: Facebot Allow: / Disallow: /api/ Disallow: /login/ User-agent: Slackbot Allow: / Disallow: /api/ Disallow: /login/ User-agent: Slackbot-LinkExpanding Allow: / Disallow: /api/ Disallow: /login/ User-agent: Discordbot Allow: / Disallow: /api/ Disallow: /login/ User-agent: WhatsApp Allow: / Disallow: /api/ Disallow: /login/ User-agent: TelegramBot Allow: / Disallow: /api/ Disallow: /login/ User-agent: Applebot Allow: / Disallow: /api/ Disallow: /login/ User-agent: Pinterestbot Allow: / Disallow: /api/ Disallow: /login/ User-agent: redditbot Allow: / Disallow: /api/ Disallow: /login/ # Google Ads crawlers — every /go/ landing page stays crawlable for ad approval User-agent: AdsBot-Google Allow: / Disallow: /api/ Disallow: /login/ User-agent: AdsBot-Google-Mobile Allow: / Disallow: /api/ Disallow: /login/