Skip to main content
    Insights · SEO

    SEO News Week 45: Block AI Training With robots.txt

    9 min read
    Quick answer:

    Google, Microsoft and OpenAI plan to honor a shared signal that lets site owners block AI training with robots.txt without losing visibility in Google Search. Google plans to support it in early 2027. Bing is also separating AI training from AI grounding and will stop treating NoArchive and NoCache as AI settings on March 1, 2027. Advertisers and site owners should review their robots.txt now.

    At a glance

    NewsWhat changesWho it affectsAction neededSource + date
    Shared AI training opt-out in robots.txtA new User-Agent: __usage_ai-training group blocks AI training at Google, Microsoft and OpenAIAll website ownersMedium: decide before early 2027Search Engine Roundtable, October 30, 2026; Search Engine Journal, November 1, 2026
    Bing separates AI training and AI groundingNew options in Bing Webmaster Tools; NoArchive/NoCache lose their AI effect on March 1, 2027Sites using NoArchive or NoCacheHigh for those sites, deadline March 1, 2027Search Engine Roundtable, October 30, 2026; Search Engine Journal, November 1, 2026
    Google expands its user-generated spam guidanceNow explicitly covers tools and platforms where users publish pagesShops with reviews, portals, forums, SaaSMedium: check indexing of user pagesSearch Engine Roundtable, October 26, 2026; Search Engine Journal, October 28, 2026
    Google on 404 errorsA 404 is not a ranking problem; redirecting to the homepage is discouragedEveryone, especially during a relaunchLow: improve your 404 pageSearch Engine Journal, October 29, 2026
    Test: opting out of AI featuresOrganic clicks also fell after the opt-out, single caseAnyone considering an AI opt-outLow: don't switch off hastilySearch Engine Roundtable, October 30, 2026

    Block AI training with robots.txt: What are Google, Microsoft and OpenAI planning?

    Google announced on the Google Search Central Blog that it will support a shared signal against AI training. According to Search Engine Roundtable on October 30, 2026, the signal is being developed at the IETF, the Internet Engineering Task Force, the body that develops technical internet standards. The working group in charge is called AIPREF. According to Search Engine Journal on November 1, 2026, Google, Microsoft and OpenAI plan to honor the new line. At the time of reporting, none of the three supported it yet.

    Technically, the signal is an ordinary rule group in robots.txt. The entry reads User-Agent: __usage_ai-training, followed by the usual Disallow and Allow lines. According to Search Engine Journal, the wildcard rule User-Agent: * does not apply to this signal. The signal also does not control crawling. It only governs whether content is used for AI training.

    Google plans to support the signal in early 2027, according to its crawler documentation as reported by Search Engine Roundtable. If a site opts out with the new entry and does not specify Google-Extended, Google's blog post says it will treat the site as if it had disallowed Google-Extended. Google-Extended is Google's existing control for training future Gemini models. According to Search Engine Journal, Google-Extended does not affect inclusion in Google Search. OpenAI aims to honor the signal in the next few months, according to Search Engine Journal.

    What does this mean for your business?

    You get a single line that blocks AI training across several providers. According to Google, your visibility in Google Search is not affected. The change takes effect in 2027, so you have time to make a deliberate decision. Stroe Media recommends settling the question as a team: Do you want your content to feed AI models or not?

    Bing separates AI training and AI grounding: Which deadlines apply?

    Microsoft is introducing three new settings for Bing: AI Training, AI Grounding and Limited Grounding. Search Engine Roundtable reported this on October 30, 2026. Grounding means an AI system pulls your content live while answering a question and uses it as a source, for example in Microsoft Copilot. Training, by contrast, means your content is used to develop an AI model.

    Search Engine Journal listed these dates on November 1, 2026:

    1. December 15, 2026: A setting opens in Bing Webmaster Tools, available only to sites still using NoArchive or NoCache on that date.
    2. February 1, 2027: Microsoft starts honoring the new training and grounding options as it crawls.
    3. March 1, 2027: Microsoft stops recognizing NoArchive and NoCache as training or grounding settings. According to Search Engine Journal, pages without a new preference then default to allowing grounding until they are recrawled.

    According to Search Engine Roundtable, blocking grounding makes your content less visible in AI answers. Classic Bing rankings are not affected.

    What does this mean for your business?

    If your website uses NoArchive or NoCache today, that setting loses its effect on Bing's AI features on March 1, 2027. Without a change, your content will then be open to grounding. If you want to appear in Copilot answers, allow grounding deliberately. For most local businesses, visibility in AI answers is an opportunity, not a risk.

    Google expands its spam guidance for user-generated content

    Google has expanded its documentation on preventing abuse by users. According to Search Engine Roundtable on October 26, 2026, it now explicitly covers tools that let users publish shared pages, generated content or web apps. Search Engine Journal summarized three changes on October 28, 2026: content sharing platforms are now explicitly covered, a new section lists steps to avoid manual actions, and new guidance explains how to detect abuse. A manual action is a penalty applied by hand by a Google employee.

    The new section is titled "Only index content meant for Search." According to Search Engine Journal, pages built only for sharing, collaboration or utility should default to noindex or be blocked in robots.txt. Noindex is an instruction in a page's code that keeps it out of Google's index. Google also recommends verifying the site in Google Search Console and watching the Page Indexing report for unusual spikes. According to Search Engine Journal, the change does not mean a wave of penalties is imminent.

    What does this mean for your business?

    This affects online shops with customer reviews, industry portals, forums and software providers with user pages. Check Google Search Console for a sudden rise in indexed pages you did not create yourself. Pages such as shared shopping carts or profile pages usually don't belong in the index.

    Google on 404 errors: Use a custom 404 page instead of redirecting to the homepage

    Gary Illyes and Cherry Sireetorn Prommawin of Google discussed 404 errors on a podcast. A 404 error is the response a server sends when a page does not exist. According to Search Engine Journal on October 29, 2026, Illyes said Google does not penalize 404 errors and that fixing them is optional.

    Search Engine Journal summarized the advice this way: A redirect only makes sense when there is a true successor, such as a newer model of the same product. If there is no replacement, a 404 or 410 status code is the expected response. Prommawin advised against redirecting deleted pages to the homepage across the board, because it confuses visitors and muddies data. A custom 404 page should include a friendly message, a search bar and links to key sections.

    What does this mean for your business?

    After a website relaunch, you don't need to redirect every old URL to the homepage. Redirect only pages with a true successor and let all others return a 404 or 410. Invest instead in a helpful 404 page with links to your services and contact details.

    Opting out of Google's AI features: Clicks fell in one test

    Cyrus Shepard opted a single website out of Google's AI features. Search Engine Roundtable reported this on October 30, 2026. According to the report, the click-through rate on AI impressions was about 0.9 percent, and AI impressions made up about 8 percent of all impressions. After the opt-out, regular organic clicks also declined, according to Search Engine Roundtable.

    Search Engine Roundtable describes the result as a single case with a small sample. There is no confirmation from Google or a larger study so far. So far, only Search Engine Roundtable has reported this, with no official confirmation.

    What does this mean for your business?

    Opting out of AI Overviews or AI Mode can cost more than the AI clicks themselves. Don't test this without clear before-and-after measurement. Stroe Media currently advises small and mid-sized businesses (SMEs) to stay visible in AI answers and optimize their content for them.

    What you should do this week

    1. 1Open your robots.txt and note which AI crawlers and entries such as Google-Extended are currently blocked or allowed.
    2. 2Check your site's source code or HTTP headers for NoArchive or NoCache, and plan the switch before March 1, 2027.
    3. 3Add your website to Bing Webmaster Tools if you haven't yet, so you can use the new settings from December 15, 2026.
    4. 4Review the Page Indexing report in Google Search Console for an unusual number of new pages, especially user profiles or shared pages.
    5. 5Open a URL on your site that doesn't exist and check whether a helpful 404 page with search and links appears.

    Frequently asked questions

    The new signal is a separate rule group with User-Agent: __usage_ai-training followed by Disallow: / for the entire site. According to Search Engine Journal on November 1, 2026, Google, Microsoft and OpenAI plan to honor this line. Google plans to support it in early 2027. The wildcard User-Agent: * does not apply to this signal, so you have to add the entry explicitly.

    No, according to Google. At Google, the new signal works like blocking Google-Extended. According to Search Engine Journal, Google-Extended controls the training of future Gemini models and does not affect inclusion in Google Search. AI Overviews and AI Mode have a separate control in Google Search Console.

    AI training means content is used to develop an AI model. AI grounding means an AI system pulls your content live while answering a specific question and uses it as a source, such as Microsoft Copilot drawing on Bing. Starting February 1, 2027, Microsoft lets site owners control both uses separately.

    According to Search Engine Journal on November 1, 2026, Microsoft will stop recognizing NoArchive and NoCache as settings for AI training or grounding on March 1, 2027. Pages without a new preference will then allow grounding by default until Bing recrawls them. If you want to keep restricting AI use, you need to adopt the new options in time.

    No. According to Search Engine Journal on October 29, 2026, Google advises against redirecting deleted pages to the homepage across the board. A redirect only makes sense when there is a true successor. Without a replacement, a 404 or 410 status code is correct. A custom 404 page with search and links helps visitors move on.

    Conclusion

    The key development this week is a shared standard that lets you block AI training with robots.txt without putting your Google rankings at risk. Bing sets a firm deadline of March 1, 2027, for anyone using NoArchive or NoCache today. Google is also raising expectations for platforms with user content. Stroe Media recommends setting your website's AI preferences deliberately now instead of leaving them to chance.

    Want to know how your website performs in Google and in AI answers? Our SEO and GEO team can review your robots.txt and your visibility. For paid reach, we help with Google Ads. Get in touch with us directly. Last week's news is in SEO News Week 44.

    Sources

    1. Google adopting shared AI training control, Search Engine Roundtable, October 30, 2026
    2. Google, Microsoft and OpenAI back a shared AI training opt-out, Search Engine Journal, November 1, 2026
    3. Supporting a shared AI training control for websites, Google Search Central Blog, October 2026
    4. AI preference signal draft (draft-kelley-aipref-training-preference-signal-00), IETF, October 30, 2026
    5. Microsoft Bing AI controls for training and grounding, Search Engine Roundtable, October 30, 2026
    6. Google user-generated spam guidance expands to tools, Search Engine Roundtable, October 26, 2026
    7. Google signals a wider crackdown on UGC SEO spam, Search Engine Journal, October 28, 2026
    8. Google recommends using custom 404 error pages, Search Engine Journal, October 29, 2026
    9. Google AI Overviews and AI Mode opt-out CTR test, Search Engine Roundtable, October 30, 2026

    Author: Stroe Media Editorial Team, Last updated: November 2, 2026