HomeCloudflare Targets Mixed AI Crawlers with New Bulk Blocking Rules
Cloudflare has rolled out a significant update to its default service rules, giving AI companies across the industry until September 15 to comply. All AI vendors are now required to differentiate between search crawlers, model training crawlers, and AI agent-specific crawlers. Any mixed crawlers that fail to distinguish themselves will be automatically blocked by the system when attempting to access pages that contain advertisements.
The new rules have a broad scope, applying equally to new platform customers, existing users who create new sites, and all free user websites. If a website administrator wishes to permit mixed crawlers, the only option is to manually adjust the backend configuration—a change that directly impacts how AI companies collect web content for training materials.
Many site administrators are happy to allow traditional search engines to index their content, but they do not want their intellectual property harvested and used for training without compensation. Cloudflare has pointed out that Google's crawler serves both search and AI data collection purposes, making it challenging for websites to allow search indexing while blocking AI training. In response, Google introduced a dedicated robot tool that lets sites block AI training access without affecting their search indexing.
However, Google's core crawler will continue to collect data for the search engine's built-in AI features, making it hard to fully separate search and AI data requirements. The platform's CEO noted that robot traffic has long surpassed human visits, and the industry ecosystem urgently needs to regulate various crawling behaviors.
Cloudflare is continuing to improve its content protection tools, evolving from the 2024 anti-AI crawler tool to a new value-based billing model. Previously, the platform charged based on crawl volume, but now it has adopted a pay-per-use approach, calculating fees according to the actual revenue that the content generates in AI applications. Data indicates that over half of AI crawlers repeatedly scrape pages that haven't been updated, and the payment mechanism can reduce invalid traffic while boosting creators' income.
The current payment plan has already been tested with two AI companies, enabling site administrators to receive direct revenue when their content is used by AI products. In an environment where copyright regulations are tightening, the new rules compel AI companies to increase the transparency of their crawling activities, giving web creators greater control over their content.
Related article
ByteDance Boosts Core AI Incentives as Doubao Surges 14.6%
ByteDance recently convened a DouBao equity briefing to unveil fresh incentive policies for staff involved in the DouBao division. The strike price for DouBao shares has been lifted from $14.85 in June 2026 to $17.02, marking an approximate 14.6% inc
MiniMax Unveils 10x Team Program to Incentivize Global AI Experts
MiniMax (Xiyu Technology), the General Artificial Intelligence Lab, has officially launched "10x Team," a global talent collaboration initiative. This program aims to recruit top experts across industries to explore the deep application of large mode
South Korea Breaks Ground on National AI Computing Center, Investing 2.5 Trillion Won with 2028 Target
South Korean outlet EtNews reports that groundbreaking for the Korea AI Computing Center (KOACC) took place on August 3 at the Solar City data center park in Sunan, Jeollanam-do. Backed by a total investment of 2.5 trillion KRW (roughly 11.838 billio
Related Special Topic Recommendations
Education and Learning
AI Study Tools for Homework and Exam Prep
2026 Latest Best AI Study Tools for Homework and Exam Prep! XIX.AI curates a top-rated list of powerful, game-changing tools that help students boost productivity, streamline homework completion, and ace exams through real-world tests. Get a free vs paid comparison, detailed rankings, and must-try options to unlock your AI edge. Explore now!
10 tools
xix.ai
Music composition
AI Vocal Demo Tools for Songwriters, Hooks, Toplines, and Multilingual Draft Sessions
2026 Latest Best AI Vocal Demo Tools for Songwriters, Hook Creators, and Multi-Language Content Teams! XIX.AI has curated a top-rated list of powerful game-changing tools that go through rigorous real-world tests. You’ll find detailed free vs paid comparison data, comprehensive rankings, and must-try options to help you boost writing efficiency and unlock your creative potential. Explore now to discover your perfect tool for all your content needs!
9 tools
xix.ai
Business
Best AI Competitive Research Tools for Small Businesses
2026 Latest Best Top-rated AI Competitive Research Tools for Small Businesses! XIX.AI has curated a highly powerful game-changing collection, updated weekly with rigorous real-world tests and detailed rankings. You can find a comprehensive free vs paid comparison to help you identify the must-try tools that boost your productivity and give you a competitive edge. Explore now to discover your perfect tool!
9 tools
xix.ai
Image editing
Photoshop AI Retouch Tools for Ecommerce Apparel, Skin Cleanup, and Color Consistency
2026 Latest Best Photoshop AI retouch tools for ecommerce apparel, skin cleanup, and color consistency! This top-rated curated list features powerful game-changing solutions that help you boost writing efficiency, streamline content creation, and achieve perfect visual results effortlessly. Each tool has undergone real-world tests through weekly updated rankings, complete with free vs paid comparison details. Backed by XIX.AI, it’s the must-try guide for anyone aiming to unlock your AI edge. Explore now!
10 tools
xix.ai
Prompt
Best AI Prompt Libraries for ChatGPT Workflows
2026 Latest Best Top-Rated AI Prompt Libraries for optimizing all types of ChatGPT workflows. XIX.AI has curated a powerful, game-changing collection that goes through rigorous real-world tests to ensure top performance. You can find detailed free vs paid comparisons and expert rankings to help you choose the must-try tools that boost your productivity and unlock your AI edge. Explore now!
11 tools
xix.ai
Education and Learning
AI Quiz Builder Platforms for Teachers, Tutors, and Cohort-Based Learning Programs
2026 Latest Best AI Quiz Builder Platforms for Teachers, Tutors, and Cohort-Based Learning Programs! XIX.AI has curated a top-rated list of powerful game-changing tools that go through real-world tests to deliver accurate rankings. These must-try platforms help boost writing efficiency, streamline content creation, and simplify quiz design across all learning scenarios. Explore now to discover your perfect tool for unlocking your AI edge in teaching!
13 tools
xix.ai
Comments (0)
0/500
Cloudflare has rolled out a significant update to its default service rules, giving AI companies across the industry until September 15 to comply. All AI vendors are now required to differentiate between search crawlers, model training crawlers, and AI agent-specific crawlers. Any mixed crawlers that fail to distinguish themselves will be automatically blocked by the system when attempting to access pages that contain advertisements.
The new rules have a broad scope, applying equally to new platform customers, existing users who create new sites, and all free user websites. If a website administrator wishes to permit mixed crawlers, the only option is to manually adjust the backend configuration—a change that directly impacts how AI companies collect web content for training materials.
Many site administrators are happy to allow traditional search engines to index their content, but they do not want their intellectual property harvested and used for training without compensation. Cloudflare has pointed out that Google's crawler serves both search and AI data collection purposes, making it challenging for websites to allow search indexing while blocking AI training. In response, Google introduced a dedicated robot tool that lets sites block AI training access without affecting their search indexing.
However, Google's core crawler will continue to collect data for the search engine's built-in AI features, making it hard to fully separate search and AI data requirements. The platform's CEO noted that robot traffic has long surpassed human visits, and the industry ecosystem urgently needs to regulate various crawling behaviors.
Cloudflare is continuing to improve its content protection tools, evolving from the 2024 anti-AI crawler tool to a new value-based billing model. Previously, the platform charged based on crawl volume, but now it has adopted a pay-per-use approach, calculating fees according to the actual revenue that the content generates in AI applications. Data indicates that over half of AI crawlers repeatedly scrape pages that haven't been updated, and the payment mechanism can reduce invalid traffic while boosting creators' income.
The current payment plan has already been tested with two AI companies, enabling site administrators to receive direct revenue when their content is used by AI products. In an environment where copyright regulations are tightening, the new rules compel AI companies to increase the transparency of their crawling activities, giving web creators greater control over their content.
ByteDance Boosts Core AI Incentives as Doubao Surges 14.6%
ByteDance recently convened a DouBao equity briefing to unveil fresh incentive policies for staff involved in the DouBao division. The strike price for DouBao shares has been lifted from $14.85 in June 2026 to $17.02, marking an approximate 14.6% inc
MiniMax Unveils 10x Team Program to Incentivize Global AI Experts
MiniMax (Xiyu Technology), the General Artificial Intelligence Lab, has officially launched "10x Team," a global talent collaboration initiative. This program aims to recruit top experts across industries to explore the deep application of large mode
South Korea Breaks Ground on National AI Computing Center, Investing 2.5 Trillion Won with 2028 Target
South Korean outlet EtNews reports that groundbreaking for the Korea AI Computing Center (KOACC) took place on August 3 at the Solar City data center park in Sunan, Jeollanam-do. Backed by a total investment of 2.5 trillion KRW (roughly 11.838 billio
2026 Latest Best AI Study Tools for Homework and Exam Prep! XIX.AI curates a top-rated list of powerful, game-changing tools that help students boost productivity, streamline homework completion, and ace exams through real-world tests. Get a free vs paid comparison, detailed rankings, and must-try options to unlock your AI edge. Explore now!
10 tools
xix.ai
2026 Latest Best AI Vocal Demo Tools for Songwriters, Hook Creators, and Multi-Language Content Teams! XIX.AI has curated a top-rated list of powerful game-changing tools that go through rigorous real-world tests. You’ll find detailed free vs paid comparison data, comprehensive rankings, and must-try options to help you boost writing efficiency and unlock your creative potential. Explore now to discover your perfect tool for all your content needs!
9 tools
xix.ai
2026 Latest Best Top-rated AI Competitive Research Tools for Small Businesses! XIX.AI has curated a highly powerful game-changing collection, updated weekly with rigorous real-world tests and detailed rankings. You can find a comprehensive free vs paid comparison to help you identify the must-try tools that boost your productivity and give you a competitive edge. Explore now to discover your perfect tool!
9 tools
xix.ai
2026 Latest Best Photoshop AI retouch tools for ecommerce apparel, skin cleanup, and color consistency! This top-rated curated list features powerful game-changing solutions that help you boost writing efficiency, streamline content creation, and achieve perfect visual results effortlessly. Each tool has undergone real-world tests through weekly updated rankings, complete with free vs paid comparison details. Backed by XIX.AI, it’s the must-try guide for anyone aiming to unlock your AI edge. Explore now!
10 tools
xix.ai
2026 Latest Best Top-Rated AI Prompt Libraries for optimizing all types of ChatGPT workflows. XIX.AI has curated a powerful, game-changing collection that goes through rigorous real-world tests to ensure top performance. You can find detailed free vs paid comparisons and expert rankings to help you choose the must-try tools that boost your productivity and unlock your AI edge. Explore now!
11 tools
xix.ai
2026 Latest Best AI Quiz Builder Platforms for Teachers, Tutors, and Cohort-Based Learning Programs! XIX.AI has curated a top-rated list of powerful game-changing tools that go through real-world tests to deliver accurate rankings. These must-try platforms help boost writing efficiency, streamline content creation, and simplify quiz design across all learning scenarios. Explore now to discover your perfect tool for unlocking your AI edge in teaching!
13 tools
xix.ai
