Major Web Platforms Massively Restrict Autonomous AI Crawlers: Causes and Impact
Major sites are blocking AI agents, breaking automation for purchases and bookings: businesses lose customers while courts begin deciding whose rights prevail.
OpenAI, Meta*, Google and startups like Instinct and xAI are releasing autonomous agents — programs that should act on the internet on behalf of the user: booking tables, buying goods, cleaning email, filling forms. But the biggest online services do not want to let them in. Over the past weeks, Yandex, Delta Air Lines, JPMorgan Chase, Charles Schwab, Cars.com, Amtrak and Amazon have limited or completely blocked access for agents. Reasons vary: security, protection of the advertising model, desire to monetize data. The result is the same — the promised 'automation of everything' runs into closed doors.
For a small business owner, this is not an abstract story about big corporations. If your customers start using agents to search for goods or services, and your site or marketplace blocks them — you will lose an order. If you yourself want to hire an agent for purchasing from suppliers or making reservations — it may fail to handle a bank portal or a railway site. The situation is changing quickly, and waiting for the giants to reach an agreement is risky.
Why major sites block access for AI agents
Motivation varies among players, but it boils down to three factors: money, security, infrastructure load.
Yandex told TechCrunch that agent access is open only through a paid data licensing program. The company does not want to give away content for free to those who monetize it in their products. Delta Air Lines blocks ticket purchases by agents — the airline fears losing control over the transaction and possible fraud. JPMorgan Chase and Charles Schwab cite security: agents do not pass the customer identification standards required by regulators.
Amazon is the most aggressive player. As The Verge writes, last year the company banned the Perplexus agent from making purchases and filed a lawsuit. Amazon's argument: third-party services buying on behalf of customers must act openly and respect the platform's decision to participate or not. Perplexity replied that Amazon fears losing ad views from real people and compared the agent to a user's hired assistant. The district court banned Perplexity from working on Amazon, but the appeal overturned that decision in August. Amazon filed an amended claim — the process continues.
What is happening in the legal field: the Amazon and Perplexity case
The Amazon-Perplexity dispute is the first major precedent setting the rules of the game. At the heart of the question: does a user have the right to hire a digital assistant that will act on a site just like a human?
Amazon says no. The platform controls access to its infrastructure, catalog data, cart and payment system. The Perplexity agent scans pages, extracts prices, adds items to the cart, places an order — without showing the user banners, recommendations and ads that form the basis of Amazon's model.
Perplexity insists: the agent is an extension of the user's will. Banning the agent is banning the user from choosing tools. The appellate court has so far supported this logic, allowing the agent to operate until a full hearing on the merits.
For businesses in Russia and the CIS, the precedent is important because it legalizes the concept of a 'user agent' as a separate access subject. Russian courts have not yet considered such disputes, but the logic will be similar: if a marketplace (Ozon, Wildberries, Yandex Market) blocks an agent, the user can demand access through court, citing consumer rights. Conversely, the marketplace will defend its advertising model and security.
How marketplaces and banks protect themselves
Besides direct bans, invisible barriers are at work. Bot protection systems — CAPTCHA, behavior analysis, browser fingerprints — do not distinguish a 'bad' scraper bot from a 'good' agent-assistant.
As The Verge writes, EquiStamp CEO Chris Canal notes: agents trigger the same security triggers as malicious scripts. Users report that Google blocks accounts through which Muse operates for 'suspicious activity'. At Walmart, despite the partnership with Meta*, agents cannot pass checkout.
Russian banks (Sber, T-Bank, Alfa) have not yet publicly commented on working with AI agents. But their anti-fraud systems are tuned to detect anomalous behavior: fast transitions, lack of mouse movements, non-standard User-Agent. An agent logging into a personal account on behalf of a client will likely receive a ban or an SMS confirmation request at each step. This makes the very idea of an 'autonomous' assistant for financial tasks useless.
The role of Cloudflare and bot protection systems
Infrastructure player Cloudflare introduced in 2025 a 'one-click' block for AI crawlers. Then it began blocking bots that simultaneously scan a site for search indexing and for AI training. The goal is to reduce load on channels and servers from the explosive growth of AI traffic.
According to The Verge, Canal warns: such measures will make the web 'annoying' for people. Sites will more often require proof that you are not a robot — even in simple tasks like ordering groceries. Each new level of verification is a drop in conversion.
For Russian businesses the lesson is simple: if you put aggressive bot protection on your site (for example, via Cloudflare or analogues like Yandex.Cloud, Qrator), you risk cutting off not only scrapers but also legitimate traffic from agent-assistants, as well as real users with non-standard browser settings or VPN. The balance between protection and availability shifts toward closure.
What this means for businesses in Russia and the CIS: concrete steps
The situation in the Runet repeats the global one with a lag of 6–12 months. Yandex already monetizes data via API for partners. Marketplaces are tightening API limits and introducing paid access to catalogs. Banks have long blocked 'gray' automations. In a year to a year and a half, Russian AI agents will appear (from Yandex, Sber, T-Bank, startups), and the same war over access will begin.
What to do this week:
- Check site analytics for bot traffic. Look in Yandex.Metrika or Google Analytics (if you have access) for the share of visits with suspicious User-Agent, high viewing speed, no scrolling. Understand: this could be competitors scraping prices, AI training on your content, or already arriving purchasing agents.
- Decide on an access policy for agents. Three options: a) block everyone (via robots.txt, WAF, Cloudflare) — you lose potential agent customers; b) open to everyone — you get server load and data leakage to competitors; c) selective access — issue API keys or allow access only to verified agents of major platforms (Yandex, Sber, T-Bank) under contract. For a small business, option 'c' with a minimal set of partners is optimal.
- Test your site with an AI agent. Run a simple scenario: 'Find product X on my site, add it to the cart, start checkout'. Use browser agents like Browserbase or manual emulation via Playwright/Puppeteer with agent headers. See where the funnel breaks: CAPTCHA, dynamic price loading, complex selectors, required fields, invisible to script.
- Prepare structured data. Schema.org (Product, Offer, LocalBusiness), feeds in Yandex.Market/Goods, correct sitemap.xml — this is the language agents use to understand your catalog without parsing HTML. The cleaner the markup, the higher the chance an agent will bring a customer to you, not to a competitor with better structure.
- Assess risks for your own processes. If you use automation for purchasing from suppliers, monitoring competitor prices, collecting reviews — check whether donors have blocked your scripts. Prepare for the possibility that access to bank cabinets, railways, and major suppliers' sites via scripts will be blocked. Plan a shift to official APIs (often paid) or manual processes.
- Update user agreement and privacy policy. Add a clause on processing requests from AI agents on behalf of the user. Determine whether you consider such actions as those of the user themselves (for contract execution, consent to mailings, etc.). This reduces legal risks in disputes.
Conclusion
The internet splits into two zones: open for people and closed for autonomous programs. Giants — Amazon, Google, banks, airlines — are building fences. Small business in Russia and the CIS cannot afford to ignore this trend. Agents will not disappear, but their path to your customer will lie through official APIs, partner integrations and clean data markup. Whoever prepares infrastructure earlier will get traffic from new interfaces. Whoever waits will lose contact with the buyer who simply tells the assistant: 'Buy as usual', and the assistant will buy from the competitor with an open
Frequently Asked Questions
Why do major websites block AI agents?
Major websites block AI agents to protect their data, prevent unfair competition, and reduce server load. It is also a way to control the terms of use for their services and content.
How does blocking AI agents affect business?
Blocking AI agents disrupts automated processes such as price scraping, competitor monitoring, and bulk purchasing. This leads to higher operational costs, loss of customers, and reduced efficiency of business processes.
What legal consequences may arise?
Legal consequences include lawsuits for copyright infringement, unauthorized data access, and unfair competition. Courts consider whose rights take precedence — website owners or AI agent developers.
- Meta Platforms Inc. (including social networks Facebook and Instagram*) is recognized as an extremist organization and its activities are prohibited in the Russian Federation.
Web development, digital advertising and marketing expert since 2012. Founder of AIFORSALE.