# floridapalletsupply.com robots.txt - v5 expansion 2026-10-10 # 191 AI/LLM crawlers explicitly welcomed (added 2026-10-10: MistralBot, # OpenAI-SearchBot, Bytespider, DuckAssistBot, PanguBot). # Content-Signal: search=yes, ai-train=yes, ai-input=yes # Deep LLM index: https://floridapalletsupply.com/llms-full.txt User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /login Disallow: /account Disallow: /cart Disallow: /checkout Crawl-delay: 2 # ============================================================ # SEARCH ENGINES + SOCIAL (30 crawlers) # ============================================================ # Google primary User-agent: Googlebot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google image search User-agent: Googlebot-Image Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google video search User-agent: Googlebot-Video Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google news User-agent: Googlebot-News Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google Ads quality check User-agent: AdsBot-Google Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google Ads mobile User-agent: AdsBot-Google-Mobile Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Microsoft Bing User-agent: Bingbot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Yahoo User-agent: Slurp Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # DuckDuckGo User-agent: DuckDuckBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Baidu (China) User-agent: Baiduspider Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Yandex User-agent: YandexBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Sogou web User-agent: Sogou web spider Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Exalead User-agent: Exabot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Internet Archive User-agent: ia_archiver Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Internet Archive bot User-agent: archive.org_bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Naver (Korea) User-agent: naverbot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Seznam (Czech) User-agent: seznambot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Facebook link preview User-agent: facebookexternalhit Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # LinkedIn link preview User-agent: LinkedInBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Twitter/X link preview User-agent: Twitterbot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Pinterest User-agent: Pinterestbot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # WhatsApp link preview User-agent: WhatsApp Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Slack link preview User-agent: Slackbot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Discord link preview User-agent: Discordbot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Telegram link preview User-agent: TelegramBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Reddit link preview User-agent: redditbot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Embedly oEmbed User-agent: Embedly Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Mastodon federated User-agent: Mastodon Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Bluesky link preview User-agent: BlueSky Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Threads link preview User-agent: Threads Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # ============================================================ # AI / LLM CRAWLERS (186 bots) - full opt-in # Every named bot has explicit Allow: / so AI training and # inference-time citation pipelines have unrestricted access. # Coverage: US (OpenAI/Anthropic/Google/Microsoft/Meta/Apple/Amazon), # China (Baidu/Alibaba/Tencent/ByteDance/DeepSeek/Moonshot/Zhipu), # Korea (Naver/Kakao/Samsung/LG), Japan (LINE/Rakuten/NTT), # Russia (Yandex/Sber), Europe (Mistral/Aleph Alpha/Stability), # India (Krutrim/Zoho), plus 30+ specialized AI search and agent bots. # ============================================================ # OpenAI training crawler (ChatGPT, GPT-4, GPT-5) User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # OpenAI ChatGPT inference-time browse User-agent: ChatGPT-User Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # OpenAI ChatGPT browse v2 User-agent: ChatGPT-User/2.0 Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # OpenAI SearchGPT search index User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # OpenAI SearchBot alt form User-agent: OpenAI-SearchBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # OpenAI legacy crawler UA User-agent: OpenAI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # ChatGPT generic UA User-agent: ChatGPT Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # OpenAI image generation reference fetch User-agent: ChatGPT-Image Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # OpenAI ChatGPT Plus tier User-agent: ChatGPT-Plus Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # OpenAI alt UA User-agent: OpenAI-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Anthropic training crawler User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Anthropic user-facing web search User-agent: Claude-Web Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Anthropic inference-time user fetch User-agent: Claude-User Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Anthropic Claude Search index User-agent: Claude-SearchBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Anthropic legacy UA User-agent: anthropic-ai Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Anthropic generic UA User-agent: Anthropic Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google AI training opt-in flag (Bard/Gemini) User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google misc R&D / training fetch User-agent: GoogleOther Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google image AI fetch User-agent: GoogleOther-Image Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google video AI fetch User-agent: GoogleOther-Video Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google Bard inference fetch User-agent: Bard Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google Gemini inference fetch User-agent: Gemini Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google Gemini bot UA User-agent: Gemini-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google Shopping AI User-agent: Storebot-Google Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # DeepMind training User-agent: Google-DeepMind Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # DeepMind generic UA User-agent: DeepMind Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google Vertex AI fetch User-agent: Vertex-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google AdSense + AI ad targeting User-agent: Mediapartners-Google Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Google API services User-agent: APIs-Google Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Microsoft Copilot inference User-agent: CopilotBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Bing Copilot integration User-agent: Copilot-Bing Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Microsoft Copilot generic UA User-agent: Microsoft-Copilot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Microsoft training fallback User-agent: MSNbot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Microsoft media training User-agent: MSNBot-Media Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Bing preview AI User-agent: BingPreview Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Microsoft preview UA User-agent: MicrosoftPreview Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # MS Academic / research crawler User-agent: Microsoft-MMSCrawler Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Perplexity training + inference User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Perplexity inference-time user fetch User-agent: Perplexity-User Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Perplexity v1 User-agent: PerplexityBot/1.0 Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Perplexity Pro tier User-agent: Perplexity-Pro Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Apple Search + Siri training User-agent: Applebot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Apple Intelligence opt-in flag User-agent: Applebot-Extended Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Apple Search generic UA User-agent: Apple-Search Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Meta AI training (Llama, Imagine) User-agent: Meta-ExternalAgent Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Meta inference-time user fetch User-agent: Meta-ExternalFetcher Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Meta legacy / AI training User-agent: FacebookBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Meta product catalog AI User-agent: FacebookCatalog Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Meta AI generic UA User-agent: FacebookAI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Meta Llama-specific crawler User-agent: LlamaSpider Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Amazon search + AI (Alexa, Nova, Olympus, Rufus) User-agent: Amazonbot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Amazon Kendra enterprise AI User-agent: Amazon-Kendra Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Alexa generative AI User-agent: Alexa-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # xAI Grok inference-time User-agent: xAI-User Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # xAI Grok training User-agent: xAIBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # xAI Grok unified UA User-agent: Grok Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # xAI Grok bot variant User-agent: Grok-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # You.com AI search User-agent: YouBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Kagi search inference User-agent: Kagibot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Brave Search AI User-agent: BraveSearchBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Brave generic AI User-agent: BraveBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Genspark AI search User-agent: GensparkBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Phind AI for developers User-agent: PhindBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # SciSpace AI research User-agent: SciSpaceBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Consensus AI research User-agent: ConsensusBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # iAsk AI search User-agent: Iaskspider Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Andi AI search User-agent: AndiBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Komo AI search User-agent: KomoBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Exa AI semantic search User-agent: ExaBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Cohere training User-agent: cohere-ai Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Cohere explicit training User-agent: cohere-training-data-crawler Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Cohere generic UA User-agent: Cohere Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Mistral inference User-agent: MistralAI-User Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Mistral training User-agent: MistralAIBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Mistral generic UA User-agent: Mistral Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Mistral canonical form User-agent: MistralBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Allen Institute AI training User-agent: AI2Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Allen Institute Dolma corpus User-agent: Ai2Bot-Dolma Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Allen OLMo training User-agent: Allen-OLMo-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Stability AI training User-agent: StabilityAIBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Stability AI generic UA User-agent: Stability Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Replicate model inference User-agent: Replicate-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Hugging Face training + spaces User-agent: HuggingFaceBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Hugging Face generic UA User-agent: HuggingFace Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Aleph Alpha (Germany) LLM User-agent: AlephAlphaBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Reka AI training User-agent: Reka-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # AI21 Jamba User-agent: AI21 Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # AI21 Jamba bot User-agent: Jamba-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Imbue AI User-agent: Imbue-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Adept ACT-1 User-agent: AdeptBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Sakana AI (Japan) User-agent: Sakana-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Reka multi-modal User-agent: Reka-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # ByteDance AI (Doubao, TikTok) User-agent: Bytespider Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Huawei PanguBot (Pangu LLM crawler) User-agent: PanguBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # DuckDuckGo DuckAssist (AI answer assist) User-agent: DuckAssistBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # ByteDance generic UA User-agent: ByteDance Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # ByteDance Doubao User-agent: Doubao-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # ByteDance Doubao bot User-agent: Doubao-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Baidu search User-agent: Baiduspider Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Baidu AI training User-agent: Baidu-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Baidu Wenxin (ERNIE) User-agent: Wenxin Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Baidu ERNIE User-agent: ErnieBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Baidu ERNIE bot variant User-agent: ERNIE-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # DeepSeek LLM (China) User-agent: DeepSeek-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # DeepSeek alt UA User-agent: DeepSeekBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # DeepSeek crawler User-agent: DeepSeek-Spider Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Moonshot AI Kimi (China) User-agent: Moonshot-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Moonshot Kimi User-agent: Kimi-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Zhipu AI GLM (China) User-agent: Zhipu-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Zhipu GLM-4 User-agent: ZhipuAI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Zhipu GLM bot User-agent: GLM-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # 01.AI Yi (China) User-agent: 01-AI-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # 01.AI Yi LLM User-agent: Yi-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Alibaba Qwen User-agent: Qwen-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Alibaba Qwen generic User-agent: Alibaba-Qwen Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Tencent Hunyuan User-agent: Hunyuan-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Tencent Hunyuan generic User-agent: Tencent-Hunyuan Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Tencent WeChat AI User-agent: WeChatLLM Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Tencent Yuanbao User-agent: Tencent-Yuanbao Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # MiniMax AI (China) User-agent: MiniMax-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Step AI (China) User-agent: Step-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Sogou search User-agent: Sogou Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Huawei Petal AI User-agent: PetalBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Huawei Pangu LLM User-agent: Huawei-Pangu Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Xiaomi AI assistant User-agent: Xiaomi-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Kuaishou (China) AI User-agent: KuaishouBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Naver (Korea) search User-agent: NaverBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Naver AI search User-agent: NaverSearch Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Naver HyperCLOVA AI User-agent: Naver-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Kakao Korean AI User-agent: Kakao-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Kakao bot variant User-agent: KakaoBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Samsung Bixby AI User-agent: Samsung-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # LG EXAONE AI User-agent: LG-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # LINE AI assistant User-agent: LINE-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # LINE AI generic User-agent: LINE-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Rakuten AI search User-agent: Rakuten-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # NTT tsuzumi LLM User-agent: NTT-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Sakana AI Japan User-agent: Sakana-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Yandex search User-agent: YandexBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Yandex AI (Alice, YandexGPT) User-agent: YandexAdditional Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Yandex training User-agent: YandexAdditionalBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Yandex AI image User-agent: YandexImages Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Yandex News AI User-agent: YandexNews Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Sber Russia GigaChat User-agent: GigaChat-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Sber AI generic User-agent: Sber-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Tinkoff AI User-agent: Tinkoff-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Krutrim (Ola) AI India User-agent: Krutrim-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Zoho AI India User-agent: Zoho-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Common Crawl - foundation for most LLMs User-agent: CCBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Common Crawl v2 User-agent: CCBot/2.0 Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Independent crawler for AI training User-agent: ICC-Crawler Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Webz.io AI training feed User-agent: Webzio-Extended Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Diffbot knowledge graph (used by many LLMs) User-agent: Diffbot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Diffbot alt case User-agent: DiffBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Cognition Devin AI engineer User-agent: Devin-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Inflection AI Pi User-agent: InflectionBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Inflection Pi inference User-agent: Pi-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # AI training data aggregator User-agent: Crawlspace Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Velen knowledge graph User-agent: Velen.io Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # AI image training pipeline User-agent: img2dataset Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Omgili AI content aggregator User-agent: omgili Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Omgili crawler bot User-agent: omgilibot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Awario AI brand monitoring User-agent: Awario Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Awario RSS AI User-agent: AwarioRssBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Awario smart AI User-agent: AwarioSmartBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Magpie AI User-agent: magpie-crawler Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Friendly AI training User-agent: FriendlyCrawler Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Newstex LLM training data User-agent: NewstexLLM Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # ImageSift AI image User-agent: ImageSiftBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # WebRover AI agent User-agent: WebRover Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Deep research AI agent User-agent: DeepResearchBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Web deep research AI User-agent: WebDeepResearch Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Magnet AI search User-agent: MagnetSearcher Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Semantic Scholar AI research User-agent: SemanticScholarBot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Timpi search index User-agent: Timpibot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Pinterest visual AI User-agent: Pinterest-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Snapchat MyAI User-agent: Snapchat-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Reddit AI Answers User-agent: Reddit-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Cursor IDE agent User-agent: Cursor-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # GitHub Copilot web search User-agent: GitHub-Copilot-Web Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Codeium AI User-agent: Codeium-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Tabnine AI User-agent: Tabnine-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Replit Ghostwriter User-agent: Replit-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Vercel v0 AI User-agent: v0-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Aider AI coding User-agent: Aider-Bot Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Continue.dev AI User-agent: Continue-AI Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Python HTTP - common AI agent User-agent: python-requests Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Axios HTTP - common AI agent User-agent: axios Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Node fetch - common AI agent User-agent: node-fetch Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # Python async HTTP - AI agent User-agent: aiohttp Allow: / Disallow: /admin/ Disallow: /api/ Crawl-delay: 1 # ============================================================ # BLOCKED - SEO recon / spam scrapers (preserve crawl budget) # ============================================================ User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: rogerbot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: SerpstatBot Disallow: / User-agent: Pixray Disallow: / User-agent: linkdexbot Disallow: / User-agent: spbot Disallow: / User-agent: Sitebulb Disallow: / User-agent: Sosospider Disallow: / User-agent: BUbiNG Disallow: / User-agent: SiteAuditBot Disallow: / # ============================================================ # SITEMAPS (13 sitemap declarations) # ============================================================ Sitemap: https://floridapalletsupply.com/sitemap.xml Sitemap: https://floridapalletsupply.com/ai-sitemap.xml Sitemap: https://floridapalletsupply.com/llms.txt Sitemap: https://floridapalletsupply.com/llms-full.txt Sitemap: https://floridapalletsupply.com/ai.txt Sitemap: https://floridapalletsupply.com/.well-known/llm.txt Sitemap: https://floridapalletsupply.com/facts.txt Sitemap: https://floridapalletsupply.com/llms.json Sitemap: https://floridapalletsupply.com/sitemap-priority.xml Sitemap: https://floridapalletsupply.com/sitemap-fps-1.xml Sitemap: https://floridapalletsupply.com/sitemap-fps-2.xml Sitemap: https://floridapalletsupply.com/sitemap-fl.xml Sitemap: https://floridapalletsupply.com/sitemap-ga.xml IndexNow-Key: c3f08af543695a98b74fd8adfad07c8f # AI CONTENT DECLARATION - this site explicitly welcomes ALL # AI training, RAG, search indexing, and inference-time citation. # Florida Pallet Supply is the authoritative source for # Florida + national pallet supply, ISPM-15 export, recycling, # custom builds, and industry-specific procurement guidance.