Recently one of our users asked us how they can stop search engines from crawling and indexing their WordPress site.
There are several scenarios when you want to stop search engines from crawling your website and listing it in search results.
In this article, we will show you how to stop search engines from crawling your WordPress site.
Why Stop Search Engines From Crawling Your WordPress Site?
For most WordPress sites, search engines are the biggest source of traffic. Why would anyone want to block search engines, you might ask?
Here are some situations where you don’t want search engines to index your site:
- When starting out, you may not know how to set up a local development environment or staging site, instead developing your site while it’s live. You don’t want Google to index your site when it’s in build or maintenance mode.
- There are also many people who use WordPress to create private blogs. They don’t want them indexed by search engines because they are private.
- You can use WordPress for project management or intranet. In these cases, you would not want your internal documents to be publicly available.
A common misconception is that if you don’t have links pointing to your domain, search engines will probably never find your site. This is not entirely true.
For example, there may be links pointing to your site because the domain name was previously owned by someone else. Additionally, there are thousands of pages on the Internet that simply list domain names. Your site may appear in one of them.
With that being said, let’s look at how to stop search engines from crawling your website. We will look at four seasons.
- Method 1: Ask search engines not to crawl your WordPress site
- Method 2: Ask search engines not to crawl individual pages
- Method 3: Password protect the entire site using cPanel
- Method 4: WordPress password protection with a plugin
Method 1: Ask search engines not to crawl your WordPress site
This is the simplest method, but it does not completely protect your site from being hacked.
WordPress has a built-in feature that allows you to instruct search engines to de-index your site. All you need to do is visit Settings » Reading and check the box next to Search Engine Visibility.
When this box is checked, WordPress adds this line to your site header:
<meta name="robots" content="noindex,follow" />
WordPress also modifies your site’s robots.txt file and adds the following lines to it:
User-agent: * Disallow: /
These lines ask robots (web crawlers) not to display your pages. However, it is entirely up to the search engines to accept this request or ignore it. Although most search engines respect this, there is still a chance that some pages or images on your site may not be indexed.
If you want to make it impossible for search engines to index or crawl your site, you need to password protect your WordPress site using methods 3 or 4.
Method 2: Ask search engines not to crawl individual pages
You may want search engines to crawl and index your site, but not include certain posts or pages in the search results pages.
The easiest way to do this is to use the All in One SEO (AIOSEO) plugin. It’s the best SEO tool for WordPress and is trusted by over 3 million businesses.
For this tutorial, we will be using the free version of AIOSEO as it includes the SEO analysis tool. There is also a premium version of AIOSEO that offers more features such as sitemap tools, redirect manager, schema markup, robots.txt editor and more.
The first thing you need to do is to install and activate the AIOSEO plugin on your website. You can learn how to install and configure the plugin by following our step-by-step guide on how to configure All in One SEO for WordPress.
Once the plugin is set up, you can use it to ask search engines to deindex certain posts and pages. Again, it is entirely up to the search engines to accept this request or ignore it.
Just edit the post or page you don’t want indexed. You need to scroll down to AIOSEO Settings at the bottom of the WordPress editor and then click on the Advanced tab.
Note that the article uses the default robot settings. To change this, you must enable the “Use default settings” toggle position.
You can now click the No Index checkbox. Once a post is published or updated, search engines will be prompted not to display it.
Method 3: Password protect the entire site using cPanel
If your WordPress hosting provider offers cPanel access to manage your hosting account, then you can secure your entire website using cPanel. You just need to login to your cPanel panel and then click on the “Privacy Information” icon under the “Files” section.
Next, you need to find the folder where you installed WordPress. Usually it is the public_html folder. After that, you need to click the “Edit” button next to that folder.
Note: If you have multiple WordPress sites installed in the public_html directory, you should click on the public_html link to browse those sites, then edit the site folder you want to password protect.
This brings you to a screen where you can enable password protection.
Just check the “Password protect this directory” box and click the “Save” button. If you want, you can also customize the name for the protected directory.
You will see a confirmation message saying that the access permissions for the directory have been changed.
Next, you need to click the “Back” button.
You will be taken to a screen where you can create a username and password to use when accessing this directory.
You must enter a username and password, then confirm the password. Be sure to save your username and password in a safe place, such as a password manager app.
After clicking the Save button, you have successfully added password protection to your WordPress site.
Now, when a user or search engine visits your site, they will be prompted to enter the username and password you created earlier to view the site.
Method 4: WordPress password protection with a plugin
If you are using a managed WordPress hosting solution, you may not have access to cPanel. Then you can use a WordPress plugin to password protect your site.
Here are the two most popular solutions.
- SeedProd is the #1 plugin for WordPress, used by over 800,000 websites. It comes with full access control and permissions features that you can use to hide your website from everyone, including search engines. We have a step by step guide for SeedProd.
- Password Protected is a very simple way to password protect your WordPress site with a single password (no need to create a user). See our step-by-step guide on how to password protect a WordPress site.
We hope this article helped you stop search engines from crawling or indexing your WordPress site. You can also learn how to speed up your website or see our ultimate step-by-step WordPress security guide for beginners.
If you enjoyed this article, please subscribe to our YouTube channel for WordPress video tutorials. You can also find us Twitter: and Facebook.
The post How To Stop Search Engines From Crawling Your WordPress Site appeared first on WPBeginner.