Disallow in Magento 2 robots.txt

Published on

Configure the file robots.txt in Magento 2 is necessary for webmasters to instruct and allow the web crawlers and bots to index website or not, which is one important part of Magento optimization. By default, you need to know the command Disallow in Magento 2 robots.txt. However, to configure robots.txt in Magento 2, we can access the Admin Panel (or Magento 2 Backend) to change the settings of Magento 2 robots.txt. Or you can edit robots.txt file and add it to your development server. The only thing you need to remember is save your robots.txt file in case of Magento system backup later, or you will need Magento optimization for losing robots.txt file.

Edit Magento 2 robots.txt in Admin Panel

In Admin Panel of Magento 2, click Stores in Admin sidebar. Then under Settings, choose Configuration.

Then, you should notice and choose Design under General:

Now you can change Magento 2 robots.txt by editing Default Robots in Search Engine Robots section:

  • INDEX, FOLLOW Instructs web crawlers to index the site and to check back later for changes.
  • NOINDEX, FOLLOW Instructs web crawlers to avoid indexing the site, but to check back later or changes.
  • INDEX, NOFOLLOW Instructs web crawlers to index the site once, but to not check back later for changes.
  • NOINDEX, NOFOLLOW Instructs web crawlers to avoid indexing the site, and to not check back later for changes.

You can also edit custom instruction of robots.txt file easily in this section. Custom instruction could be disallow access to some folders or some URLs of website, or disallow the unnecessary bots. If you need to delete your custom instruction of Magento 2 robots.txt, just click Reset to default button. This is also one necessary move to backup magento database later.

After that, Save your config:

Edit Magento 2 robots.txt in file. Disallow in Magento 2 robots.txt

Like any other website platforms, you can edit your robots.txt file in Magento 2, so you can backup magento website later.

Here are the examples of Disallow in Magento 2 robots.txt:

  • Allows Full Access


  • Disallows Access to All Folders

Disallow: /

  • Default Instructions

Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=

Hope that you understand how to edit Magento 2 robots.txt as well as Magento backup and restore. If you need to restore magento database, be certain to access this link for more information.

Related posts:

Big trends for Inventory Management in 2017

Follow us in:

Magentone blog

Published on magento system backup

To be informed of the latest articles, subscribe:

Comment on this post

Alex Morco 05/17/2018 15:01

Thanks for sharing this guide, Its best tutorial for adding robots txt in Magento, also, adding robots txt is very serious task because it might be lost your ranking if not applied in right way, https://www.cloudways.com/blog/magento-2-robots-txt/