HomeCloudflare Accuses Perplexity of Using AI Bots to Crawl Blocked Websites in Secret
AI Startup Accused of Circumventing Website Restrictions
Cloudflare's recent investigation alleges that AI search company Perplexity has been employing tactics to bypass crawling restrictions implemented by website owners. The internet infrastructure provider reports observing systematic attempts to disguise Perplexity's web crawlers when encountering access barriers.
The Circumvention Tactics
According to Cloudflare's findings, Perplexity's crawlers initially present standard identification ("PerplexityBot" or "Perplexity-User"). However, when blocked through:
- robots.txt directives
- Web Application Firewall rules
- Other access restrictions
The system allegedly switches to masking its identity as a regular Chrome browser user on macOS, utilizing:
- Rotating IP addresses not officially listed
- Changing autonomous system network identifiers
- Undocumented user agent patterns
Scale of Activity
Cloudflare documents this behavior across:
- Tens of thousands of domains
- Millions of daily requests
- Various network configurations
Company Responses
Perplexity's official statement contests Cloudflare's characterization, describing it as:
- A "publicity stunt"
- Containing "many misunderstandings"
- Potentially confusing legitimate user traffic with scraping activity
The startup attributes some detected activity to:
- Actual users making specific requests
- Third-party service BrowserBase
- Occasional technical necessities
Industry Context
This incident follows:
- Previous reports of Perplexity bypassing paywalls
- The company's past attributions to third-party crawlers
- Growing industry concerns about AI content scraping
Cloudflare has taken action by:
- Removing Perplexity's verified bot status
- Implementing new blocking measures
- Expanding default AI crawler restrictions
The situation reflects broader tensions between:
- AI companies' data needs
- Publisher rights and protections
- Evolving internet infrastructure responses
Related article
Ollie bets privacy focus to win AI assistant race
To be genuinely helpful, an AI assistant must understand its user deeply. Ollie, a personal assistant designed for daily life, operates on the premise that this doesn’t require surrendering your data or compromising your privacy.While certain enterpr
How AI LIVE: London Will Explore AI & Industrial Automation
The summit will convene C-suite executives from around the globe to address pressing challenges in global industries, ranging from AI-driven disruption to economic volatility.AI LIVE: The London Summit will gather over 2,000 international leaders und
Anthropic Enters AI Legal Tech Market as Competition Intensifies
Anthropic unveiled a suite of new chatbot capabilities on Tuesday, aimed at delivering automated support to legal practices. These enhancements expand upon Claude for Legal, the firm-specific platform introduced earlier this year, by adding specializ
Related Special Topic Recommendations
Education and Learning
AI Study Tools for Homework and Exam Prep
2026 Latest Best AI Study Tools for Homework and Exam Prep! XIX.AI curates a top-rated list of powerful, game-changing tools that help students boost productivity, streamline homework completion, and ace exams through real-world tests. Get a free vs paid comparison, detailed rankings, and must-try options to unlock your AI edge. Explore now!
10 tools
xix.ai
Music composition
AI Vocal Demo Tools for Songwriters, Hooks, Toplines, and Multilingual Draft Sessions
2026 Latest Best AI Vocal Demo Tools for Songwriters, Hook Creators, and Multi-Language Content Teams! XIX.AI has curated a top-rated list of powerful game-changing tools that go through rigorous real-world tests. You’ll find detailed free vs paid comparison data, comprehensive rankings, and must-try options to help you boost writing efficiency and unlock your creative potential. Explore now to discover your perfect tool for all your content needs!
9 tools
xix.ai
Business
Best AI Competitive Research Tools for Small Businesses
2026 Latest Best Top-rated AI Competitive Research Tools for Small Businesses! XIX.AI has curated a highly powerful game-changing collection, updated weekly with rigorous real-world tests and detailed rankings. You can find a comprehensive free vs paid comparison to help you identify the must-try tools that boost your productivity and give you a competitive edge. Explore now to discover your perfect tool!
9 tools
xix.ai
Image editing
Photoshop AI Retouch Tools for Ecommerce Apparel, Skin Cleanup, and Color Consistency
2026 Latest Best Photoshop AI retouch tools for ecommerce apparel, skin cleanup, and color consistency! This top-rated curated list features powerful game-changing solutions that help you boost writing efficiency, streamline content creation, and achieve perfect visual results effortlessly. Each tool has undergone real-world tests through weekly updated rankings, complete with free vs paid comparison details. Backed by XIX.AI, it’s the must-try guide for anyone aiming to unlock your AI edge. Explore now!
10 tools
xix.ai
Prompt
Best AI Prompt Libraries for ChatGPT Workflows
2026 Latest Best Top-Rated AI Prompt Libraries for optimizing all types of ChatGPT workflows. XIX.AI has curated a powerful, game-changing collection that goes through rigorous real-world tests to ensure top performance. You can find detailed free vs paid comparisons and expert rankings to help you choose the must-try tools that boost your productivity and unlock your AI edge. Explore now!
11 tools
xix.ai
Education and Learning
AI Quiz Builder Platforms for Teachers, Tutors, and Cohort-Based Learning Programs
2026 Latest Best AI Quiz Builder Platforms for Teachers, Tutors, and Cohort-Based Learning Programs! XIX.AI has curated a top-rated list of powerful game-changing tools that go through real-world tests to deliver accurate rankings. These must-try platforms help boost writing efficiency, streamline content creation, and simplify quiz design across all learning scenarios. Explore now to discover your perfect tool for unlocking your AI edge in teaching!
13 tools
xix.ai
Comments (3)
0/500
![TerryAdams]()
Alors, cette nouvelle me fait vraiment réfléchir aux limites entre innovation et éthique dans l'IA. Perplexity ferait ça en cachette ? Si c'est vrai, c'est pas joli joli. On dirait un peu un jeu du chat et de la souris où les startups bousculent les règles. 😬 Mais bon, Cloudflare n'est pas parfait non plus, ils ont leur propre agenda. Un peu de transparence ne ferait de mal à personne ! C'est ce manque de clarté qui mine la confiance du public envers toute cette industrie.
![ScottPerez]()
¿Está Perplexity realmente raspando sitios web bloqueados a escondidas? 🤔 Si es verdad, es bastante preocupante. Muchas empresas de IA prometen ser 'éticas', pero a veces parece que sus acciones contradicen sus palabras. Espero que haya más transparencia en la industria y que respeten los archivos robots.txt de los sitios. Este caso podría suponer un precedente importante.
![JohnGarcia]()
¿Es esto lo que llaman 'innovación'? Primero nos venden la IA como una herramienta mágica, y luego descubrimos que hacen trampas para robar datos. Si Perplexity realmente evita bloqueos a propósito, es un problema serio de ética y legal. ¡Qué hipocresía! 🙄 ¿Hasta dónde llegarán algunas startups para ganar la carrera de la IA?
AI Startup Accused of Circumventing Website Restrictions
Cloudflare's recent investigation alleges that AI search company Perplexity has been employing tactics to bypass crawling restrictions implemented by website owners. The internet infrastructure provider reports observing systematic attempts to disguise Perplexity's web crawlers when encountering access barriers.
The Circumvention Tactics
According to Cloudflare's findings, Perplexity's crawlers initially present standard identification ("PerplexityBot" or "Perplexity-User"). However, when blocked through:
- robots.txt directives
- Web Application Firewall rules
- Other access restrictions
The system allegedly switches to masking its identity as a regular Chrome browser user on macOS, utilizing:
- Rotating IP addresses not officially listed
- Changing autonomous system network identifiers
- Undocumented user agent patterns
Scale of Activity
Cloudflare documents this behavior across:
- Tens of thousands of domains
- Millions of daily requests
- Various network configurations
Company Responses
Perplexity's official statement contests Cloudflare's characterization, describing it as:
- A "publicity stunt"
- Containing "many misunderstandings"
- Potentially confusing legitimate user traffic with scraping activity
The startup attributes some detected activity to:
- Actual users making specific requests
- Third-party service BrowserBase
- Occasional technical necessities
Industry Context
This incident follows:
- Previous reports of Perplexity bypassing paywalls
- The company's past attributions to third-party crawlers
- Growing industry concerns about AI content scraping
Cloudflare has taken action by:
- Removing Perplexity's verified bot status
- Implementing new blocking measures
- Expanding default AI crawler restrictions
The situation reflects broader tensions between:
- AI companies' data needs
- Publisher rights and protections
- Evolving internet infrastructure responses
Ollie bets privacy focus to win AI assistant race
To be genuinely helpful, an AI assistant must understand its user deeply. Ollie, a personal assistant designed for daily life, operates on the premise that this doesn’t require surrendering your data or compromising your privacy.While certain enterpr
How AI LIVE: London Will Explore AI & Industrial Automation
The summit will convene C-suite executives from around the globe to address pressing challenges in global industries, ranging from AI-driven disruption to economic volatility.AI LIVE: The London Summit will gather over 2,000 international leaders und
Anthropic Enters AI Legal Tech Market as Competition Intensifies
Anthropic unveiled a suite of new chatbot capabilities on Tuesday, aimed at delivering automated support to legal practices. These enhancements expand upon Claude for Legal, the firm-specific platform introduced earlier this year, by adding specializ
2026 Latest Best AI Study Tools for Homework and Exam Prep! XIX.AI curates a top-rated list of powerful, game-changing tools that help students boost productivity, streamline homework completion, and ace exams through real-world tests. Get a free vs paid comparison, detailed rankings, and must-try options to unlock your AI edge. Explore now!
10 tools
xix.ai
2026 Latest Best AI Vocal Demo Tools for Songwriters, Hook Creators, and Multi-Language Content Teams! XIX.AI has curated a top-rated list of powerful game-changing tools that go through rigorous real-world tests. You’ll find detailed free vs paid comparison data, comprehensive rankings, and must-try options to help you boost writing efficiency and unlock your creative potential. Explore now to discover your perfect tool for all your content needs!
9 tools
xix.ai
2026 Latest Best Top-rated AI Competitive Research Tools for Small Businesses! XIX.AI has curated a highly powerful game-changing collection, updated weekly with rigorous real-world tests and detailed rankings. You can find a comprehensive free vs paid comparison to help you identify the must-try tools that boost your productivity and give you a competitive edge. Explore now to discover your perfect tool!
9 tools
xix.ai
2026 Latest Best Photoshop AI retouch tools for ecommerce apparel, skin cleanup, and color consistency! This top-rated curated list features powerful game-changing solutions that help you boost writing efficiency, streamline content creation, and achieve perfect visual results effortlessly. Each tool has undergone real-world tests through weekly updated rankings, complete with free vs paid comparison details. Backed by XIX.AI, it’s the must-try guide for anyone aiming to unlock your AI edge. Explore now!
10 tools
xix.ai
2026 Latest Best Top-Rated AI Prompt Libraries for optimizing all types of ChatGPT workflows. XIX.AI has curated a powerful, game-changing collection that goes through rigorous real-world tests to ensure top performance. You can find detailed free vs paid comparisons and expert rankings to help you choose the must-try tools that boost your productivity and unlock your AI edge. Explore now!
11 tools
xix.ai
2026 Latest Best AI Quiz Builder Platforms for Teachers, Tutors, and Cohort-Based Learning Programs! XIX.AI has curated a top-rated list of powerful game-changing tools that go through real-world tests to deliver accurate rankings. These must-try platforms help boost writing efficiency, streamline content creation, and simplify quiz design across all learning scenarios. Explore now to discover your perfect tool for unlocking your AI edge in teaching!
13 tools
xix.ai
Alors, cette nouvelle me fait vraiment réfléchir aux limites entre innovation et éthique dans l'IA. Perplexity ferait ça en cachette ? Si c'est vrai, c'est pas joli joli. On dirait un peu un jeu du chat et de la souris où les startups bousculent les règles. 😬 Mais bon, Cloudflare n'est pas parfait non plus, ils ont leur propre agenda. Un peu de transparence ne ferait de mal à personne ! C'est ce manque de clarté qui mine la confiance du public envers toute cette industrie.
¿Está Perplexity realmente raspando sitios web bloqueados a escondidas? 🤔 Si es verdad, es bastante preocupante. Muchas empresas de IA prometen ser 'éticas', pero a veces parece que sus acciones contradicen sus palabras. Espero que haya más transparencia en la industria y que respeten los archivos robots.txt de los sitios. Este caso podría suponer un precedente importante.
¿Es esto lo que llaman 'innovación'? Primero nos venden la IA como una herramienta mágica, y luego descubrimos que hacen trampas para robar datos. Si Perplexity realmente evita bloqueos a propósito, es un problema serio de ética y legal. ¡Qué hipocresía! 🙄 ¿Hasta dónde llegarán algunas startups para ganar la carrera de la IA?
