SEO

Yoast SEO: Can't Find the Robots.txt File? Here's How to Create and Edit It

By the White Label DM team ··3 min read

The robots.txt file is a crucial element for website owners who want to control how search engines crawl their site. It acts like a set of instructions, telling search engine bots which pages they can and cannot access.

Yoast SEO, a popular WordPress plugin, offers a convenient way to edit the robots.txt file directly from your WordPress dashboard. However, sometimes you might encounter an issue where Yoast SEO can't find the file. This can happen for a couple of reasons.

This guide will walk you through the steps of creating or editing the robots.txt file using Yoast SEO and offer alternative methods if the Yoast functionality is unavailable.

Finding and Editing the robots.txt with Yoast SEO (if available)

  • Log in to your WordPress Dashboard.
  • Navigate to Yoast SEO: In the left-hand admin menu, click on "Yoast SEO".
  • Access the File Editor: Under "Yoast SEO" tools, look for "File editor". If you don't see this option, it means file editing is disabled on your WordPress installation, or your server doesn't allow it (we'll address this later).
  • Check for Existing File (Optional): If a robots.txt file already exists, you'll see its content displayed here. You can edit the existing directives or add new ones.
  • Create a New File (if needed): If there's no robots.txt file, click the "Create robots.txt file" button. Yoast will generate a basic file with some default instructions.

Important Note: Editing the robots.txt file can impact how search engines crawl your site. Make sure you understand the directives before making any changes.

Adding or Editing Directives in the robots.txt File

Once you have access to the robots.txt file (created or existing), you can add directives to control crawling. Here's a basic structure:

User-agent: * # This applies to all search engine bots Disallow: /path/to/folder/ # Disallows bots from accessing this folder Allow: /specific/page.html # Allows access to a specific page even if disallowed in a folder

Yoast SEO provides a user-friendly interface to add these directives without needing to write complex code.

Saving Your Changes

Once you've made the necessary edits to your robots.txt file, click the "Save" button at the bottom of the File Editor. Yoast SEO will attempt to save the changes to your server.

Alternative Methods if Yoast SEO File Editor is Unavailable

If you can't use the Yoast SEO File Editor, there are two alternative methods to create or edit the robots.txt file:

1. Manual File Creation via FTP

  • Access your server via FTP client: This requires FTP access details provided by your web hosting provider.
  • Locate the root directory: This is usually the public_html folder
.
  • Create a new text file: Use a text editor like Notepad and name it exactly "robots.txt" (lowercase and with the .txt extension).
  • Add your desired directives: Following the structure mentioned earlier.
  • Upload the file: Upload the created "robots.txt" file to your server's root directory.

2. Contact your Web Host

If you're not comfortable using FTP, you can contact your web hosting provider's customer support. They might be able to create or edit the robots.txt file for you.

By following these steps, you should be able to create or edit the robots.txt file for your website, either using Yoast SEO or alternative methods. Remember to make changes cautiously and understand the impact they have on search engine crawling.

Written by the White Label DM team

We deliver SEO, AI SEO, paid ads and websites for agencies under their own brand.

Ready to add a delivery team without hiring one?

Tell us about your clients. We will reply within one business day with a plan and pricing. No pressure, no lock-in.