{"id":2073,"date":"2026-09-06T11:15:13","date_gmt":"2026-09-06T03:15:13","guid":{"rendered":"https:\/\/maxaeo.cn\/blog\/cloudflare-crawler-page-allow\/"},"modified":"2026-09-06T11:15:13","modified_gmt":"2026-09-06T03:15:13","slug":"cloudflare-crawler-page-allow","status":"publish","type":"post","link":"https:\/\/maxaeo.cn\/blog\/cloudflare-crawler-page-allow\/","title":{"rendered":"Cloudflare \u4ec5\u5141\u8bb8\u722c\u866b\u8bbf\u95ee\u6307\u5b9a\u9875\u9762\u914d\u7f6e\u6307\u5357"},"content":{"rendered":"

\u4f5c\u8005\uff1amaxaeo.cn\uff5c\u53d1\u5e03\u65e5\u671f\uff1a2026-09-06\uff5c\u66f4\u65b0\u65e5\u671f\uff1a2026-09-06<\/em><\/p>\n

Cloudflare \u4ec5\u5141\u8bb8\u722c\u866b\u8bbf\u95ee\u6307\u5b9a\u9875\u9762\uff0c\u9002\u5408\u65e2\u6015\u88ab\u6076\u610f\u6293\u53d6\u3001\u53c8\u5e0c\u671b AI \u641c\u7d22\u80fd\u8bfb\u5230\u54c1\u724c\u6838\u5fc3\u4fe1\u606f\u7684\u7f51\u7ad9\u3002\u6b63\u786e\u505a\u6cd5\u4e0d\u662f\u5173\u95ed WAF\uff0c\u4e5f\u4e0d\u662f\u5168\u7ad9\u653e\u884c AI Bot\uff0c\u800c\u662f\u53ea\u5f00\u653e\u54c1\u724c\u9875\u3001\u4ea7\u54c1\u9875\u3001\u6587\u6863\u9875\u3001\u62a5\u4ef7\u9875\u3001\/llms.txt<\/code> \u4e0e\u5c11\u91cf\u53ef\u5f15\u7528\u9875\u9762\u3002<\/p>\n

\"Cloudflare<\/figure>\n

\u4ec0\u4e48\u662f\u201c\u53ea\u5141\u8bb8\u722c\u866b\u8bbf\u95ee\u6307\u5b9a\u9875\u9762\u201d\uff1f<\/h2>\n

\u53ea\u5141\u8bb8\u722c\u866b\u8bbf\u95ee\u6307\u5b9a\u9875\u9762\uff0c\u662f\u6307\u7528 Cloudflare WAF\u3001\u81ea\u5b9a\u4e49\u89c4\u5219\u3001\u673a\u5668\u4eba\u89c4\u5219\u548c robots.txt \u5171\u540c\u63a7\u5236\uff1a\u722c\u866b\u53ea\u80fd\u8bbf\u95ee\u88ab\u660e\u786e\u6388\u6743\u7684 URL\uff0c\u5176\u4f59\u540e\u53f0\u3001\u63a5\u53e3\u3001\u641c\u7d22\u9875\u3001\u53c2\u6570\u9875\u7ee7\u7eed\u62e6\u622a\u6216\u6311\u6218\u3002<\/p>\n

\u8fd9\u662f\u4e00\u79cd\u6700\u5c0f\u6743\u9650\u7b56\u7565<\/strong>\u3002\u5bf9 GEO\/AEO \u6765\u8bf4\uff0c\u5b83\u7684\u4ef7\u503c\u5728\u4e8e\uff1a\u8ba9 AI \u5f15\u64ce\u66f4\u5bb9\u6613\u6293\u5230\u201c\u80fd\u4ee3\u8868\u54c1\u724c\u4e8b\u5b9e\u201d\u7684\u9875\u9762\uff0c\u540c\u65f6\u51cf\u5c11\u65e0\u610f\u4e49\u6293\u53d6\u3001\u8bad\u7ec3\u578b\u722c\u866b\u6ee5\u7528\u3001\u767b\u5f55\u63a5\u53e3\u66b4\u9732\u548c\u53c2\u6570\u9875\u6c61\u67d3\u3002<\/p>\n

Cloudflare \u5b98\u65b9 WAF \u6587\u6863\u8bf4\u660e\uff0c\u81ea\u5b9a\u4e49\u89c4\u5219\u53ef\u4ee5\u57fa\u4e8e\u8bf7\u6c42\u5c5e\u6027\u8fc7\u6ee4\u6d41\u91cf\uff0c\u5305\u62ec URL \u8def\u5f84\u3001Header\u3001\u65b9\u6cd5\u7b49\u6761\u4ef6\uff1bGoogle \u7684 robots.txt \u89c4\u8303\u6587\u6863<\/a> \u4e5f\u660e\u786e\u652f\u6301 user-agent<\/code>\u3001allow<\/code>\u3001disallow<\/code> \u7b49\u89c4\u5219\u3002\u4f46\u8981\u6ce8\u610f\uff1arobots.txt \u662f\u6293\u53d6\u6307\u5f15\uff0c\u4e0d\u7b49\u4e8e\u5b89\u5168\u8fb9\u754c\uff1b\u771f\u6b63\u7684\u8bbf\u95ee\u63a7\u5236\u4ecd\u5e94\u653e\u5728 Cloudflare \u89c4\u5219\u5c42\u3002<\/p>\n

\u54ea\u4e9b\u9875\u9762\u503c\u5f97\u653e\u884c\uff0c\u54ea\u4e9b\u9875\u9762\u5fc5\u987b\u7ee7\u7eed\u4fdd\u62a4\uff1f<\/h2>\n

\u503c\u5f97\u653e\u884c\u7684\u662f\u201c\u516c\u5f00\u3001\u7a33\u5b9a\u3001\u53ef\u88ab\u5f15\u7528\u3001\u80fd\u89e3\u91ca\u54c1\u724c\u201d\u7684\u9875\u9762\uff1b\u4e0d\u5e94\u653e\u884c\u7684\u662f\u767b\u5f55\u3001\u540e\u53f0\u3001\u652f\u4ed8\u3001\u7ad9\u5185\u641c\u7d22\u3001\u53c2\u6570\u7b5b\u9009\u3001\u7528\u6237\u6570\u636e\u548c API \u63a5\u53e3\u3002\u5224\u65ad\u6807\u51c6\u4e0d\u662f\u9875\u9762\u91cd\u8981\u4e0d\u91cd\u8981\uff0c\u800c\u662f\u5b83\u662f\u5426\u9002\u5408\u88ab AI \u8bfb\u53d6\u548c\u590d\u8ff0\u3002<\/p>\n

\u53ef\u7528\u4e0b\u9762\u8fd9\u5f20 12 \u7c7b\u8def\u5f84\u77e9\u9635\u505a\u521d\u7b5b\uff1a<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\u8def\u5f84\u7c7b\u578b<\/th>\n\u793a\u4f8b<\/th>\n\u5efa\u8bae\u52a8\u4f5c<\/th>\n\u539f\u56e0<\/th>\n<\/tr>\n<\/thead>\n
AI \u7d22\u5f15\u6587\u4ef6<\/td>\n\/llms.txt<\/code>\u3001\/robots.txt<\/code><\/td>\n\u653e\u884c<\/td>\n\u5e2e\u52a9\u722c\u866b\u53d1\u73b0\u6838\u5fc3\u9875\u9762<\/td>\n<\/tr>\n
\u54c1\u724c\u4ecb\u7ecd\u9875<\/td>\n\/about<\/code>\u3001\/brand<\/code><\/td>\n\u653e\u884c<\/td>\n\u63d0\u4f9b\u5b9e\u4f53\u4e8b\u5b9e<\/td>\n<\/tr>\n
\u4ea7\u54c1\u9875<\/td>\n\/product<\/code>\u3001\/features<\/code><\/td>\n\u653e\u884c<\/td>\n\u89e3\u91ca\u80fd\u529b\u4e0e\u573a\u666f<\/td>\n<\/tr>\n
\u4ef7\u683c\u9875<\/td>\n\/pricing<\/code><\/td>\n\u8c28\u614e\u653e\u884c<\/td>\n\u9700\u4fdd\u6301\u4fe1\u606f\u66f4\u65b0<\/td>\n<\/tr>\n
\u6587\u6863\u9875<\/td>\n\/docs\/<\/code><\/td>\n\u653e\u884c<\/td>\n\u6613\u88ab AI \u5f15\u7528<\/td>\n<\/tr>\n
\u6848\u4f8b\u9875<\/td>\n\/case\/<\/code><\/td>\n\u653e\u884c\u516c\u5f00\u7248<\/td>\n\u6709\u52a9\u4e8e\u53ef\u4fe1\u5ea6<\/td>\n<\/tr>\n
\u535a\u5ba2\u6838\u5fc3\u6587\u7ae0<\/td>\n\/blog\/\u6838\u5fc3\u4e3b\u9898<\/code><\/td>\n\u9009\u62e9\u6027\u653e\u884c<\/td>\n\u627f\u63a5\u957f\u5c3e\u95ee\u9898<\/td>\n<\/tr>\n
\u7ad9\u5185\u641c\u7d22<\/td>\n\/search<\/code><\/td>\n\u4e0d\u653e\u884c<\/td>\n\u6613\u4ea7\u751f\u4f4e\u8d28\u9875\u9762<\/td>\n<\/tr>\n
\u53c2\u6570\u9875<\/td>\n?sort=<\/code>\u3001?filter=<\/code><\/td>\n\u4e0d\u653e\u884c<\/td>\n\u6613\u9020\u6210\u91cd\u590d\u6293\u53d6<\/td>\n<\/tr>\n
\u767b\u5f55\u6ce8\u518c<\/td>\n\/login<\/code>\u3001\/admin<\/code><\/td>\n\u62e6\u622a\u6216\u6311\u6218<\/td>\n\u9ad8\u98ce\u9669<\/td>\n<\/tr>\n
API<\/td>\n\/api\/<\/code><\/td>\n\u62e6\u622a\u6216\u9650\u901f<\/td>\n\u4e0d\u9002\u5408\u516c\u5f00\u6293\u53d6<\/td>\n<\/tr>\n
\u6587\u4ef6\u4e0a\u4f20<\/td>\n\/upload<\/code><\/td>\n\u62e6\u622a<\/td>\n\u5b89\u5168\u98ce\u9669\u9ad8<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

MaxAEO \u5728\u505a\u54c1\u724c AI \u53ef\u89c1\u6027\u8bca\u65ad\u65f6\uff0c\u4e5f\u4f1a\u628a\u201cAI \u662f\u5426\u8bfb\u5230\u4e86\u6b63\u786e\u9875\u9762\u201d\u4f5c\u4e3a\u68c0\u67e5\u9879\u4e4b\u4e00\u3002\u82e5\u5b98\u7f51\u6838\u5fc3\u9875\u6ca1\u6709\u88ab AI \u5f15\u7528\uff0c\u5efa\u8bae\u5148\u770b\u8def\u5f84\u662f\u5426\u88ab robots\u3001WAF\u3001CDN \u6216\u91cd\u5b9a\u5411\u62e6\u4f4f\uff0c\u518d\u8c08\u5185\u5bb9\u4f18\u5316\u3002<\/p>\n

Cloudflare \u89c4\u5219\u5e94\u8be5\u600e\u4e48\u5199\uff1f<\/h2>\n

\u6838\u5fc3\u89c4\u5219\u662f\uff1a\u5148\u8bc6\u522b\u722c\u866b\uff0c\u518d\u9650\u5236\u8def\u5f84\uff0c\u6700\u540e\u51b3\u5b9a\u52a8\u4f5c\u3002Cloudflare \u4ec5\u5141\u8bb8\u722c\u866b\u8bbf\u95ee\u6307\u5b9a\u9875\u9762\u65f6\uff0c\u8868\u8fbe\u5f0f\u901a\u5e38\u7531 User-Agent<\/code>\u3001http.request.uri.path<\/code>\u3001http.host<\/code>\u3001\u8bf7\u6c42\u65b9\u6cd5\u548c\u673a\u5668\u4eba\u9a8c\u8bc1\u5b57\u6bb5\u7ec4\u6210\u3002<\/p>\n

\u4e00\u4e2a\u5e38\u89c1\u601d\u8def\u662f\u201c\u5141\u8bb8\u53ef\u4fe1\u722c\u866b\u8bbf\u95ee\u767d\u540d\u5355\u8def\u5f84\uff0c\u5176\u4ed6\u8def\u5f84\u7ee7\u7eed\u8d70\u9ed8\u8ba4\u9632\u62a4\u201d\u3002Cloudflare \u5b98\u65b9\u7684 AI Crawl Control with WAF \u6587\u6863<\/a> \u4e5f\u63d0\u5230\uff0c\u53ef\u901a\u8fc7\u57fa\u4e8e\u8def\u5f84\u7684\u4f8b\u5916\uff0c\u8ba9\u88ab\u963b\u6b62\u7684\u722c\u866b\u8bbf\u95ee\u7ad9\u70b9\u4e2d\u7279\u5b9a\u90e8\u5206\u3002<\/p>\n

\u793a\u4f8b\u8868\u8fbe\u5f0f\u53ef\u6309\u4f60\u7684\u7ad9\u70b9\u8c03\u6574\uff1a<\/p>\n

(\n  http.request.uri.path in {\n    "\/llms.txt"\n    "\/robots.txt"\n    "\/about"\n    "\/pricing"\n  }\n  or starts_with(http.request.uri.path, "\/docs\/")\n  or starts_with(http.request.uri.path, "\/product\/")\n)\nand\n(\n  http.user_agent contains "Googlebot"\n  or http.user_agent contains "bingbot"\n  or http.user_agent contains "Perplexity"\n  or http.user_agent contains "Claude"\n  or http.user_agent contains "GPTBot"\n)\nand http.request.method in {"GET" "HEAD"}\n<\/code><\/pre>\n
\u52a8\u4f5c\u4e0d\u8981\u4e00\u4e0a\u6765\u5c31\u9009\u201c\u5168\u8df3\u8fc7\u201d\u3002\u66f4\u7a33\u59a5\u7684\u987a\u5e8f\u662f\uff1a<\/p>\n
    \n
  1. \u5bf9\u767d\u540d\u5355\u8def\u5f84\u6267\u884c Skip<\/code> \u6307\u5b9a WAF \u89c4\u5219\u6216\u964d\u4f4e\u5b89\u5168\u7ea7\u522b\uff1b<\/li>\n
  2. \u4fdd\u7559\u901f\u7387\u9650\u5236\u3001DDoS \u9632\u62a4\u548c\u65e5\u5fd7\u8bb0\u5f55\uff1b<\/li>\n
  3. \u5bf9 \/login<\/code>\u3001\/admin<\/code>\u3001\/api\/<\/code> \u660e\u786e Block \u6216 Managed Challenge\uff1b<\/li>\n
  4. \u89c2\u5bdf 3\u20137 \u5929\u65e5\u5fd7\u540e\u518d\u6269\u5927\u8def\u5f84\u8303\u56f4\u3002<\/li>\n<\/ol>\n
    \u5982\u679c\u4f60\u9700\u8981\u66f4\u7ec6\u7684\u8def\u5f84\u65b9\u6848\uff0c\u53ef\u53c2\u8003\u7ad9\u5185\u7684 Cloudflare \u6309 URL \u653e\u884c\u89c4\u5219\uff1a\u53ea\u5f00\u653e llms.txt\u3001\u6838\u5fc3\u9875\u4e0e\u6587\u6863\u9875<\/a>\uff0c\u91cc\u9762\u66f4\u9002\u5408\u7528\u4e8e\u843d\u5730\u524d\u7684\u8def\u5f84\u6e05\u5355\u6574\u7406\u3002<\/p>\n
    robots.txt \u4e0e WAF \u5e94\u8be5\u600e\u4e48\u914d\u5408\uff1f<\/h2>\n
    robots.txt \u8d1f\u8d23\u201c\u544a\u8bc9\u5408\u89c4\u722c\u866b\u54ea\u91cc\u80fd\u6293\u201d\uff0cWAF \u8d1f\u8d23\u201c\u5f3a\u5236\u51b3\u5b9a\u8bf7\u6c42\u80fd\u4e0d\u80fd\u8fdb\u6765\u201d\u3002\u53ea\u5199 robots.txt \u4e0d\u80fd\u963b\u6b62\u4e0d\u5b88\u89c4\u77e9\u7684\u722c\u866b\uff1b\u53ea\u5199 WAF \u53c8\u53ef\u80fd\u8ba9\u5408\u89c4\u722c\u866b\u4e0d\u77e5\u9053\u54ea\u4e9b\u9875\u9762\u6700\u91cd\u8981\u3002<\/p>\n
    \u63a8\u8350\u7ec4\u5408\u5982\u4e0b\uff1a<\/p>\n
    User-agent: *\nDisallow: \/admin\/\nDisallow: \/login\/\nDisallow: \/api\/\nDisallow: \/search\nAllow: \/llms.txt\nAllow: \/docs\/\nAllow: \/product\/\nAllow: \/pricing\nAllow: \/about\n<\/code><\/pre>\n
    \u7136\u540e\u5728 Cloudflare \u4fa7\u505a\u540c\u53e3\u5f84\u89c4\u5219\uff1a\u5141\u8bb8 GET\/HEAD \u8bf7\u6c42\u8bbf\u95ee\u8fd9\u4e9b\u8def\u5f84\uff0c\u62d2\u7edd\u6216\u6311\u6218\u9ad8\u98ce\u9669\u8def\u5f84\u3002\u8fd9\u6837\u505a\u7684\u597d\u5904\u662f\uff0crobots.txt \u7ed9\u5408\u89c4\u722c\u866b\u660e\u786e\u8def\u5f84\uff0cWAF \u7ed9\u7ad9\u70b9\u63d0\u4f9b\u5b9e\u9645\u5b89\u5168\u8fb9\u754c\u3002<\/p>\n
    \u5982\u679c\u4f60\u7684\u76ee\u6807\u662f\u8ba9 AI \u66f4\u5feb\u8bfb\u61c2\u7ad9\u70b9\uff0c\u8fd8\u5e94\u8bbe\u8ba1\u6e05\u6670\u7684 llms.txt<\/code>\u3002\u53ef\u53c2\u8003 llms.txt \u76ee\u5f55\u7d22\u5f15\u8bbe\u8ba1\uff1a\u8ba9 AI \u5feb\u901f\u8bfb\u61c2\u5b98\u7f51\u6838\u5fc3\u9875\u9762<\/a>\uff0c\u628a\u54c1\u724c\u4ecb\u7ecd\u3001\u4ea7\u54c1\u80fd\u529b\u3001\u6587\u6863\u5165\u53e3\u3001\u4ef7\u683c\u8bf4\u660e\u548c\u5173\u952e\u95ee\u7b54\u96c6\u4e2d\u7d22\u5f15\u3002<\/p>\n
    \"Cloudflare<\/figure>\n
    \u9762\u5411 AI \u722c\u866b\u65f6\uff0c\u6700\u5bb9\u6613\u72af\u7684 5 \u4e2a\u9519\u8bef<\/h2>\n
    AI \u722c\u866b\u653e\u884c\u6700\u5e38\u89c1\u7684\u95ee\u9898\u4e0d\u662f\u201c\u6ca1\u653e\u884c\u201d\uff0c\u800c\u662f\u653e\u884c\u8fb9\u754c\u8fc7\u5927\u3001\u9a8c\u8bc1\u4e0d\u8db3\u6216\u9875\u9762\u8d28\u91cf\u592a\u4f4e\u3002Cloudflare \u4ec5\u5141\u8bb8\u722c\u866b\u8bbf\u95ee\u6307\u5b9a\u9875\u9762\u65f6\uff0c\u8981\u907f\u514d\u628a\u5b89\u5168\u95ee\u9898\u5305\u88c5\u6210 GEO \u4f18\u5316\u95ee\u9898\u3002<\/p>\n
    \u5e38\u89c1\u9519\u8bef\u5305\u62ec\uff1a<\/p>\n
      \n
    1. \u5168\u7ad9\u767d\u540d\u5355<\/strong>\uff1a\u4e3a\u4e86\u8ba9 AI \u6293\u53d6\uff0c\u76f4\u63a5\u8df3\u8fc7\u6240\u6709 WAF\u3002\u77ed\u671f\u7701\u4e8b\uff0c\u957f\u671f\u98ce\u9669\u6700\u9ad8\u3002<\/li>\n
    2. \u53ea\u770b User-Agent<\/strong>\uff1aUA \u53ef\u4f2a\u9020\uff0c\u5e94\u7ed3\u5408 Cloudflare Verified Bots\u3001IP\u3001\u65e5\u5fd7\u3001\u884c\u4e3a\u548c\u8def\u5f84\u5224\u65ad\u3002<\/li>\n
    3. \u653e\u884c\u53c2\u6570\u9875<\/strong>\uff1a\u7b5b\u9009\u3001\u6392\u5e8f\u3001\u641c\u7d22\u7ed3\u679c\u9875\u4f1a\u5236\u9020\u5927\u91cf\u91cd\u590d URL\u3002<\/li>\n
    4. \u5ffd\u7565\u91cd\u5b9a\u5411\u94fe<\/strong>\uff1a\u722c\u866b\u8bbf\u95ee \/pricing<\/code>\uff0c\u5374\u88ab\u91cd\u5b9a\u5411\u5230\u5e26\u5730\u533a\u53c2\u6570\u6216\u524d\u7aef\u7a7a\u58f3\u9875\uff0cAI \u53ef\u80fd\u8bfb\u4e0d\u5230\u6b63\u6587\u3002<\/li>\n
    5. \u6ca1\u6709\u590d\u6d4b\u5f15\u7528<\/strong>\uff1a\u65e5\u5fd7\u91cc\u51fa\u73b0\u6293\u53d6\uff0c\u4e0d\u4ee3\u8868 AI \u7b54\u6848\u4f1a\u5f15\u7528\uff1b\u8fd8\u8981\u770b\u540e\u7eed\u56de\u7b54\u662f\u5426\u63d0\u53ca\u3001\u6392\u5e8f\u662f\u5426\u53d8\u5316\u3001\u6765\u6e90\u662f\u5426\u6765\u81ea\u5b98\u7f51\u3002<\/li>\n<\/ol>\n
      \u7ad9\u5185\u7684 AI \u673a\u5668\u4eba UA \u767d\u540d\u5355\u914d\u7f6e\u6307\u5357<\/a> \u53ef\u4f5c\u4e3a UA \u8bc6\u522b\u7684\u8865\u5145\uff0c\u4f46\u771f\u6b63\u4e0a\u7ebf\u65f6\u4ecd\u5e94\u4ee5\u65e5\u5fd7\u548c\u5f15\u7528\u590d\u6838\u4e3a\u51c6\u3002<\/p>\n
      \u539f\u521b\u6846\u67b6\uff1a\u7528\u201c3\u00d74 \u9875\u9762\u51c6\u5165\u8868\u201d\u51b3\u5b9a\u662f\u5426\u653e\u884c<\/h2>\n
      \u4e00\u4e2a\u9875\u9762\u662f\u5426\u5e94\u8be5\u653e\u7ed9\u722c\u866b\uff0c\u53ef\u4ee5\u7528\u201c3 \u4e2a\u4ef7\u503c\u7ef4\u5ea6 \u00d7 4 \u4e2a\u98ce\u9669\u7b49\u7ea7\u201d\u8bc4\u5206\u3002\u8fd9\u4e2a\u6846\u67b6\u9002\u5408\u5e02\u573a\u3001SEO\u3001\u8fd0\u7ef4\u548c\u5b89\u5168\u56e2\u961f\u4e00\u8d77\u8bc4\u5ba1\uff0c\u907f\u514d\u53ea\u4ece\u5355\u4e00\u90e8\u95e8\u89c6\u89d2\u505a\u51b3\u5b9a\u3002<\/p>\n
      3 \u4e2a\u4ef7\u503c\u7ef4\u5ea6\uff1a<\/strong><\/p>\n