Configure the file robots.txt in Magento 2 is necessary for webmasters to instruct and allow the web crawlers and bots to index website or not, which is one important part of Magento optimization. By default, you need to know the command Disallow in Magento 2 robots.txt. However, to configure robots.txt in Magento 2, we can access the Admin Panel (or Magento 2 Backend) to change the settings of Magento 2 robots.txt. Or you can edit robots.txt file and add it to your development server. The only thing you need to remember is save your robots.txt file in case of Magento system backup later, or you will need Magento optimization for losing robots.txt file.
Edit Magento 2 robots.txt in Admin Panel
In Admin Panel of Magento 2, click Stores in Admin sidebar. Then under Settings, choose Configuration.
Then, you should notice and choose Design under General:
Now you can change Magento 2 robots.txt by editing Default Robots in Search Engine Robots section:
You can also edit custom instruction of robots.txt file easily in this section. Custom instruction could be disallow access to some folders or some URLs of website, or disallow the unnecessary bots. If you need to delete your custom instruction of Magento 2 robots.txt, just click Reset to default button. This is also one necessary move to backup magento database later.
After that, Save your config:
Edit Magento 2 robots.txt in file. Disallow in Magento 2 robots.txt
Like any other website platforms, you can edit your robots.txt file in Magento 2, so you can backup magento website later.
Here are the examples of Disallow in Magento 2 robots.txt:
Follow us in: