Creating a robots.txt
What is a robots.txt?
You use a robots.txt to shield parts of your website from search engine spiders. That way they don't index unimportant, similar or forbidden pages. In short, a robots.txt text file keeps the crawler traffic on your website under control.
Before your website shows up in the search results, it first has to be indexed. Search engines do that based on data collected by their crawlers, spiders or robots. With a robots.txt, you tell those crawlers: “You can visit these URLs, but not these”. That's why it's also called the Robots Exclusion Protocol (REP) or Robots Exclusion Standard (RES). Without such a file, all spiders are free to crawl, index and rank all your pages.
Creating a robots.txt is fairly easy, but not without risk. You don't want to accidentally block Googlebot everywhere. In this technical SEO guide, you'll find out whether you need a robots.txt and how to write and test one yourself.
Where do you store the robots.txt?
A robots.txt belongs in the main directory or root of your website. That's normally where a crawler lands first. If it finds a robots.txt, it reads it right away. That way it knows which actions it definitely may or may not take.
By the way, the main directory is also where you'll find the XML sitemap. It's sometimes mentioned in the robots.txt as well.
What does a robots.txt look like?
A robots.txt usually contains only a few lines. You'll find a clear example at www.digital-climax.be/robots.txt. Want to check whether you have a robots.txt yourself? Enter your website's home page followed by /robots.txt.
What's in a robots.txt?
A robots.txt usually contains a few different terms.
- User-agent: indicates the crawler or crawlers the following rule applies to
- Allow: the robot in the line above may crawl this page or these pages
- Disallow: the robot in the line above isn't welcome on this page or these pages
- Crawl-budget: gives spiders a waiting time after each action, for example 5 seconds
- Sitemap: add the URL of your sitemap here once more
You combine the first three commands in different ways.
Robots.txt examples
User-agent: *
Allow: /
All spiders can crawl everything. Nice of you to point that out, but this is the default setting. Save yourself the trouble and leave the file empty. Or simply don't add one at all.
User-agent: spambot
Disallow: /
The ‘spambot’ spider can't crawl a single page. Say ‘spambot’ crawls around so ‘spamsearch’ can index and rank websites. Anyone who searches for your company on ‘spamsearch’ then won't find your website in the search results.
User-agent: Googlebot
Disallow: /secret.html
Google's robot can crawl everything, except the page ‘secret’.
User-agent: Baiduspider
Disallow: /secret/
Baidu's spider can crawl everything, except the subfolder ‘secret’.
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /
Google's crawler gets a warm welcome on your website. All other spiders are kindly asked to leave your pages alone.
Creating your own robots.txt
Once you understand the rules, creating a robots.txt is not that hard. Just follow these four steps.
1) Open your robots.txt
Notepad is the ideal tool for creating a robots.txt. Make sure you use the right name:
- Always use the plural ‘robots’: crawlers won't find a file called robot.txt.
- Use lowercase letters, no capitals.
- Use the right format: it's important that it's a .txt file.
2) Write your robots.txt
A robots.txt consists of several rules, and several lines: each command goes on a separate line. You always start by specifying the ‘user-agent’, followed by ‘allow’ or ‘disallow’. That way the spider knows which pages it may and may not index.
Tip
A classic: don't forget to save your file. It has to be saved as a UTF-8 encoded text file.
3) Upload your robots.txt
How you add the file to your website depends on your site, server and host. A quick online search will probably tell you more. If not, pass your question on to your host. Or get in touch with us, we're happy to help.
Your robots.txt only applies to the domain you add it to. These three domains each need their own robots.txt file:
- https://www.voorbeeld.be
- http://www.voorbeeld.be
- https://voorbeeld.be
You'll find the differences between these three in our domain guide.
4) Test your robots.txt
A first test is of course simply browsing to www.yourdomainname.yourextension/robots.txt. If your file shows up there, you're ready for round two.
Google's robots.txt Tester flags any errors. You can also test whether specific URLs are blocked correctly or not.
Another handy tool for this is the robots.txt Validator and Testing Tool. Enter your URL and the agent, and edit your text file if needed.
All good? Then you don't need to do anything else. The crawlers will find and use your robots.txt on their own.
What can't a robots.txt do?
A robots.txt is definitely not a lock on the door that keeps unwanted visitors out. It's a set of guidelines, not hard laws. Most malware bots are more than happy to ignore these files.
On top of that, your robots.txt is accessible to everyone. Simply by entering the address, your visitor can see which parts of your website you'd rather keep out of the search results. So you're better off protecting sensitive information with a password, or simply not putting it online.
Having a robots.txt is no guarantee that the parts you ‘disallow’ won't appear in the search results. That can still happen, for example when Google finds a backlink to those pages. The URLs then show up in the SERP without a description. Still want to remove some pages from the search results? Then meta tags like noindex are a better option.
Do you need a robots.txt?
The answer to that question depends on several factors. What kind of website do you have, what pages are on it, which visitors do you want to bring to which page and where would you rather not see them?
In general, the answer is: yes, you need a robots.txt file. It helps you steer crawlers, so it has an impact on your overall search engine optimization.
Wondering whether you need a robots.txt? Contact us and we'll look at it together.