Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough

The RobotsTxt Module

Parent Feed: 

https://www.drupal.org/project/robotstxt

Credits & Thanks

Thank you to:

About the RobotsTxt module

The RobotsTxt module is great when you are running multiple Drupal sites from a single code base (multisite) and you need a different robots.txt file for each one. RobotsTxt can generate the robots.txt file for each and gives you the ability to edit on a site by site basis from within the Drupal admin interface.

Volacci uses this module to make changes to the default robots.txt because it is not completely optimized for SEO.

WARNING:
Before installing the RobotsTXT module, you’ll need to delete or rename the existing robots.txt file in the root of your Drupal installation. The module will not work properly until this is done.

  1. Install and Enable the RobotsTxt module on your server. (See this section for more instructions on installing modules.)
     
  2. Go to the Extend page: Click Manage > Extend (Coffee: “extend”) or visit https://yourDrupalSite.dev/admin/modules.

    installing and enabling the robotstxt module in Drupal
     

  3. Select the checkbox next to RobotsTxt and click the Install button at the bottom of the page.
     

Permissions

If necessary, give yourself permissions to use the XML Sitemap module.

  1. Click Manage > People > Permissions (Coffee: perm”) or visit https://yourDrupalSite.dev/admin/people/permissions.
     
  2. Select the appropriate checkboxes for "Administer robots.txt".

    setting the robotstxt module permissions
     

  3. Click the Save permissions button at the bottom of the page.
     

Adding the XML Sitemap to your robots.txt file using the RobotsTxt Module

Note: If you do not use the RobotsTxt module, you’ll need to do things the old-fashioned way. Skip to the next section for information on how to make changes to your robots.txt file.

  1. Click Manage > Configuration > Search and metadata > Robotstxt (coffee:robots) or go to https://yourDrupalSite.dev/admin/config/search/robotstxt in your browser.  
     
  2. Put your cursor within the Contents of robots.txt window and scroll to the bottom of it.

    editing your robotstxt file text in the Drupal interface
     

  3. On a new line, add this to the bottom of the field:

    sitemap: https:///sitemap.xml
     

  4. Click the Save configuration button at the bottom of the page.
     

Adding the XML Sitemap to your robots.txt file without a module

  1. Download your robots.txt file. One way to do that is to visit https://yourDrupalSite.dev/robots.txt in your browser and select File > Save Page As... from the browser’s menu.

    downloading your robots.txt file within a browser
     

  2. Using a text editor like Notepad or TextEdit, open your robots.txt file.

    DANGER!
    Avoid complex word processing programs to edit this file because they will add invisible markup that makes the file unusable by crawlers.

  3. Add this line to the bottom of your robots.txt file and save the file:

    sitemap: https://yourDrupalSite.dev/sitemap.xml
     

  4. Here’s what Volacci’s robots.txt file looks like.

    Note: You will always want to use the https version of your site because not doing so will impact your SEO rankings. Contact your web developer or hosting company to make sure your site has an SSL certificate and that your site defaults to the https protocol.

  5. Upload the new file to the root level of your Drupal site, replacing your existing robots.txt file. If you don’t have FTP access, ask your developer or hosting company to help you.
     
  6. Verify that you did it properly by visiting https://yourDrupalSite.dev/robots.txt, refresh the page, and look for your changes.

That’s it! Now, any other search engines can find the location of your XML sitemap by visiting your robots.txt file.

Author: 
Original Post: 

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web