Meta* updates terms: it can now scan business websites to train AI for free
Meta* has updated its terms to allow scanning of business websites for AI training without paying, ignoring robots.txt and without owners' consent.
Meta introduces new commercial terms effective October 30. According to The Verge, the changes give the company the right to automatically scan and collect content from any web pages “linked” to business accounts on Facebook, Instagram, WhatsApp and Messenger. Collected material Meta can use for development, improvement and training of its AI products. For website owners and media this means: their content may end up in the training set without separate consent and without payment.
Previously Meta* had licensing agreements with major outlets such as CNN, Fox News, USA Today and People Inc. Experts say the new terms allow the company to access a much larger data set — including small and medium business sites — bypassing license negotiations. As The Verge reports citing Nieman Lab, lawyers believe the wording in the terms can override technical bans such as robots.txt.
What exactly changed in the terms
The text of the new commercial terms explicitly states that the business authorizes Meta* (or parties acting on its behalf) to use automated software — scrapers, bots, spiders, crawlers — to access domains, pages and URLs “related to your products or services”. The company gains the right to index, cache, analyze, collect and periodically update data from these resources.
Then comes the key phrase: Meta* may use any content from such domains to “provide, develop, improve, research, train and maintain the integrity” of its products and services. The publication does not specify how exactly the algorithm determines which site is “linked” to the account — whether a link in the profile, in an ad or in an organic post is enough.
Meta spokesperson Daniel Roberts said the move is merely a consolidation of existing advertising and commercial terms, not a new way to use crawlers. He added that content owners can still use robots.txt flags to indicate preferences. James Grimmelmann, professor of digital and information law at Cornell University, told Nieman Lab: for news publishers this clearly looks like Meta’s permission to train AI on any content from their sites and possibly deploy it even in competition with the publishers themselves.
Why robots.txt may not work
The robots.txt file is a long-standing standard: it tells search engine bots which parts of a site not to index. Until recently major AI companies, including OpenAI and Meta, said they respect this protocol. Meta’s new terms contain wording that, according to lawyers, gives the company a legal argument: commercial terms override technical blocking instructions.
If a court or arbitrator accepts this logic, a robots.txt ban ceases to be a guarantee. Meta* would gain legal access to full article texts, product descriptions, reviews, prices and any other data businesses publish on their domains. For small businesses lacking a legal department to track case law, this creates an asymmetry: you set a ban, the company cites your agreed terms and continues collecting.
Connection to the paid Meta* One subscription
Last month Meta launched the Meta One subscription. It requires businesses and creators to pay for the ability to place links in posts and Reels. As Nieman Lab notes, this creates a paradox: Meta* starts charging for the right to link to content that it itself gains the right to scan for free to train its models.
This model changes the economics of traffic. Previously businesses paid for ads to bring users to their sites. Now the platform can use site content to train AI that then generates answers or recommendations inside the Meta* ecosystem — without sending traffic back to the source. For an e-commerce store this means: you pay to attract a customer, while your product catalog freely feeds the knowledge base of a competitor’s AI assistant or the platform itself.
Meta*’s position: goal is products, not news
Communications vice president Andy Stone wrote on X that the update is not aimed at news sites. He said the goal is to ensure accurate retrieval of product information from third-party sites for advertising recommendations and content collections on the platform.
However the terms text is not limited to product pages only. The wording “domains, web pages or URLs related to your products or services” covers blogs, articles, case studies, documentation — everything a business publishes for marketing. Lawyers interviewed by Nieman Lab point out: legally the text lets Meta* argue for access to any content on a domain if the business account promotes that domain. A clarification from October 17, reported by The Verge, narrowed the wording to “ads and business products”, but the basic authorization mechanism for crawling remains in force.
What this means for small business in Russia and the CIS
There is currently no direct legal action by Meta against Russian companies — the platform has left the market, the ad account is blocked, payments do not go through. Nevertheless Meta’s crawlers (facebookexternalhit, Meta-ExternalAgent and others) continue to traverse the open web. The new terms cement their right to use what they find for training global models, including Llama and future multimodal systems.
For businesses operating domestically or reaching abroad via mirrors, parallel domains or partners, the risks are concrete:
- Content goes into training without payment. Product descriptions, blog articles, FAQs, technical specifications — all become training data for AI that may later serve answers to users on WhatsApp, Instagram or Meta search without citing you.
- Loss of uniqueness in results. If AI learns to generate answers based on your content, users will not need to visit your site. Traffic will shift to a chatbot or summarizing widget.
- No leverage for negotiations. Large holdings can strike license deals. Small business accepts the terms by creating a business account or running ads — and loses control.
Steps worth taking this week
- Check robots.txt. Add directives
User-agent: Meta-ExternalAgentandUser-agent: facebookexternalhitwithDisallow: /. This is not a guarantee after the new terms, but basic hygiene. Also addUser-agent: CCBot,User-agent: GPTBot,User-agent: ClaudeBot— other major crawlers also collect data. - Close admin and technical sections. Ensure robots.txt disallows
/admin/,/bitrix/,/wp-admin/,/api/,/search/, cart, checkout, personal account pages. This is not about AI but about security and index cleanliness. - Mark authorship and rights. Use schema.org
WebPage,Article,Productwith fieldsauthor,publisher,copyrightHolder. This does not block scraping but fixes authorship in structured form — useful for future disputes or attribution systems. - Set server-side blocking rules. In nginx or Apache return 403 for User-Agent containing
Meta,facebookexternalhit,Meta-ExternalAgentbased on IP ranges AS32934 (Meta Platforms). The subnet list is public in Meta’s developer documentation. This is a hard block: the bot gets an error, not content. - Assess the value of open content. If you publish unique methods, calculations, expertise — consider which fragment to leave open for SEO and customers and which to put behind authentication, in PDF or a closed section. AI will not index what it cannot download.
- Save screenshots of the terms. Record the version of Meta*’s commercial terms dated October 30. If a future copyright claim arises, you will have proof of what you “agreed to” by acceptance.
- Follow court practice in the EU and US. Early lawsuits over using content for AI training are underway. Court rulings on robots.txt versus Terms of Service will set precedent. For Russian business this is a signal: how they decide there will shape how it works here — models are trained on a global corpus.
Conclusion
Meta has legalized the collection of data from business websites for AI training through acceptance of its commercial terms. Technical bans such as robots.txt may prove legally ineffective. There is no direct impact on the Russian market yet, but the crawlers operate globally. Protecting content now means combining server-level blocks by User-Agent and IP, authorship markup, and conscious control over what you place in open access. Paying for links in Meta One so the platform can freely ingest your catalog into its model is economically unsound. Better to block bots at the server level and keep unique content close to you.
Frequently Asked Questions
What do the new Meta* terms for business mean?
The new terms allow Meta* to scan public web resources of businesses to train its AI models. This is done for free and may ignore robots.txt directives, meaning owners have less control over how their content is used.
How can a business protect its data from scanning?
Completely preventing scanning is difficult because Meta can ignore robots.txt. It is recommended to review the privacy policy, use noindex/nofollow meta tags on pages that should not be used, and consider restricting public access to sensitive information.
Does this affect the SEO rankings of a website?
There is no direct negative impact on SEO rankings in search engines such as Google or Yandex. However, using the site’s content to train AI may result in similar or paraphrased content appearing in AI service responses, which could potentially reduce traffic to the original source.
- Meta Platforms Inc. (including social networks Facebook and Instagram*) is recognized as an extremist organization and its activities are prohibited in the Russian Federation.
Web development, digital advertising and marketing expert since 2012. Founder of AIFORSALE.