How to stop AI bots from crawling your website

What is AI training?

AI training is a process of teaching an artificial intelligence system to correctly analyze data and learn from it so that the AI program can carry out various tasks that include making decisions based on the information provided. To successfully conduct AI training, three things are required:

• artificial intelligence model that needs training,
• suitable data,
• powerful computing platform.

Okay, but the question remains – what do AI models train on? For instance, in the case of ChatGPT (GPT-3), the answer is mainly Common Crawl – a web archive that has been collecting data since 2008. Among other sources from which it crawled data, “Wikipedia, The Free Encyclopedia” is worth mentioning.

Publishers’ AI-related concerns

Digital content creators can benefit from AI tools in various ways, but not all of them are willing to share their unique content for the purpose of training AI models. A worry arises that the artificially-generated suggestions could be too similar to original content, although AI companies state that copying and pasting of their creations is not possible due to the fact that they don’t store information AI models train on. Regardless of their assurances, in some cases, a text that is solely based on AI output might be, in fact, very similar to the original one. In worst-case scenarios, if the new text becomes more popular, search engine ranking systems may consider the original content as plagiarism! Consequently, your content will rank lower and may become less attractive for both users and, if you monetize it, for advertisers!

It’s worth noting that the White House stated that several US-based AI companies, including OpenAI, committed to creating a watermarking system indicating that the content was generated by AI. Still, they have not promised to discontinue the use of internet data for training purposes. In fact, Meta and OpenAI faced legal action from many authors in a California Court for utilizing their books in AI training, including Sarah Silverman, an American comedian and writer. When it comes to Europe, recently, the European Parliament voted over a draft which, among others, states that artificial intelligence programs generating content would have to clearly indicate its artificial origin.

Nonetheless, OpenAI has earned praise from many creators for sharing a code that can prevent ChatGPT from learning from their websites. All you have to do is follow the steps we guided on below!

How to stop GPTBot?

OpenAI released a way for publishers to prevent GPTBot – their web crawler – from reading your valuable content and later using it to generate responses in ChatGPT. This bot’s main task is to perfect GPT-4 and GPT-5 language models. It is possible to restrict GPTBot from your entire website or specific parts of it by adding it to your site’s robots.txt – a kind of guide for web crawlers that informs which areas of your website are accessible. In order to do that in the first place, you have to copy the text from the chosen frame below and save it as a .txt file. Afterward, you have to add it to your site. The process may differ, depending on the server architecture and solutions applied by your hosting provider. This is why, in order to add a robots.txt file, contact your hosting company or find the company’s documentation regarding the process.

In order to ensure that you have restricted bot access to your website successfully, you can use robots.txt testing tools like Logeix.

Protect your flock!

Just as a shepherd watches over their flock, you should safeguard your creation. The internet is a rapidly changing environment, and it is worth staying up-to-date with all the newly emerging threats so that you don’t fall prey to traps you can easily omit. If you’re interested in this kind of content, don’t hesitate to visit our blog, where we continuously discuss various issues and news from the world of monetization, digital content creation, and its optimization!