Contact Us:

670 Lafayette Ave, Brooklyn,
NY 11216

+1 800 966 4564
+1 800 9667 4558

White Label DM services

The robots.txt file is a crucial element for website owners who want to control how search engines crawl their site. It acts like a set of instructions, telling search engine bots which pages they can and cannot access.

Yoast SEO, a popular WordPress plugin, offers a convenient way to edit the robots.txt file directly from your WordPress dashboard. However, sometimes you might encounter an issue where Yoast SEO can’t find the file. This can happen for a couple of reasons.

This guide will walk you through the steps of creating or editing the robots.txt file using Yoast SEO and offer alternative methods if the Yoast functionality is unavailable.

Finding and Editing the robots.txt with Yoast SEO (if available)

  • Log in to your WordPress Dashboard.

wordpress login

  • Navigate to Yoast SEO: In the left-hand admin menu, click on “Yoast SEO”.
  • Access the File Editor: Under “Yoast SEO” tools, look for “File editor”. If you don’t see this option, it means file editing is disabled on your WordPress installation, or your server doesn’t allow it (we’ll address this later).
  • Check for Existing File (Optional): If a robots.txt file already exists, you’ll see its content displayed here. You can edit the existing directives or add new ones.
  • Create a New File (if needed): If there’s no robots.txt file, click the “Create robots.txt file” button. Yoast will generate a basic file with some default instructions.

Important Note: Editing the robots.txt file can impact how search engines crawl your site. Make sure you understand the directives before making any changes.

Adding or Editing Directives in the robots.txt File

Once you have access to the robots.txt file (created or existing), you can add directives to control crawling. Here’s a basic structure:

User-agent: *  # This applies to all search engine bots
Disallow: /path/to/folder/   # Disallows bots from accessing this folder
Allow: /specific/page.html  # Allows access to a specific page even if disallowed in a folder

Yoast SEO provides a user-friendly interface to add these directives without needing to write complex code.

Saving Your Changes

Once you’ve made the necessary edits to your robots.txt file, click the “Save” button at the bottom of the File Editor. Yoast SEO will attempt to save the changes to your server.

Alternative Methods if Yoast SEO File Editor is Unavailable

If you can’t use the Yoast SEO File Editor, there are two alternative methods to create or edit the robots.txt file:

1. Manual File Creation via FTP:

  • Access your server via FTP client: This requires FTP access details provided by your web hosting provider.
  • Locate the root directory: This is usually the public_html folder

directory.

  • Create a new text file: Use a text editor like Notepad and name it exactly “robots.txt” (lowercase and with the .txt extension).
  • Add your desired directives: Following the structure mentioned earlier.
  • Upload the file: Upload the created “robots.txt” file to your server’s root directory.

2. Contact your Web Host:

If you’re not comfortable using FTP, you can contact your web hosting provider’s customer support. They might be able to create or edit the robots.txt file for you.

By following these steps, you should be able to create or edit the robots.txt file for your website, either using Yoast SEO or alternative methods. Remember to make changes cautiously and understand the impact they have on search engine crawling.

Leave a comment

Your email address will not be published. Required fields are marked *