Blocking AI Training While Wanting AI Recommendations: Websites Don’t Have to Be All Open or All Closed
Anxiety about talent often frames the question as “What do people need to learn?” I would rather look at the other end: a capable professional may have a website that leaves AI no way to understand that expertise.
My position may please neither side: Wanting AI to recommend you does not mean you must first agree to let every AI freely use your content; refusing model training does not mean you must also shut off access for search.
Bundling these two things together is convenient for platforms, but not necessarily for businesses.
A Notice at the Entrance, Not a Security System
The robots.txt file in a website’s root directory can tell different crawlers which paths they are allowed or prohibited from crawling. But these are rules for crawlers to follow, not a password lock or a comprehensive licensing agreement. When visitors ignore the rules, the notice will not suddenly grow arms to stop them. Sensitive data still requires logins, access controls, and server protection.
Another distinction that is easy to blur: Building an index for search and collecting data for model training serve different purposes. For example, OpenAI’s official documentation describes its search crawler, OAI-SearchBot, separately from its training crawler, GPTBot; website operators can set rules according to purpose. Implementation still requires checking each provider’s documentation—you cannot apply one provider’s names to every platform. Official crawler documentation
That takes some of the intimidation out of the claim that “blocking AI means disappearing.”
Being Read May Still Bring No Visitors
Imagine a shoe repair shop in Tainan. The craftsperson clearly describes leather sole replacement, handling specialty shoe materials, and how to book an appointment. Someone asks AI, “Where nearby can I get old leather shoes repaired?” If that content can be accessed and understood, it could provide a basis for a recommendation.
But “allowing crawling” is only an entry point, not a guarantee of recommendations. Whether a page can be read, whether its information is explicit, and whether its service area is clear all affect how it is subsequently interpreted; a single Allow line will not create credibility for the craftsperson.
The real awkwardness for marketers comes afterward: AI may spell out the services, address, and things to keep in mind, leaving users with no reason to click through to the website.
The craftsperson may consider that a good deal: “As long as the phone rings.” A professional writer who earns advertising revenue from instructional articles, however, may feel that their content has become someone else’s unpaid service clerk. Both need exposure, but the terms of exchange they can afford to accept are completely different.
Who Has the Power to Negotiate the Exchange?
I tend to see robots.txt as a notice about permitted uses, because what small businesses most often lack is not generosity, but the ability to distinguish between terms of exchange.
They may want their business hours and service areas to be searchable; paid course materials and detailed client case studies may require different access arrangements. Technical settings express crawling preferences; copyright and personal data responsibilities still cannot be left to a few lines of text.
The regulatory tension here is very practical: if platforms bind exposure too tightly to data use, a business’s “voluntary openness” may simply reflect fear of disappearing from recommendation lists. Conversely, blocking everything also has a cost—a small shop may lose even the chance to be compared alongside chain brands.
The shoe repair craftsperson opens the admin dashboard and sees that website traffic has not increased, but two more phone calls have come in. What he wants to know is not “Did AI read my website?” but which particular skill that customer had seen when they said, “AI recommended you.”
