ezgif.com gif maker 20 17

Optimal Crawlers and Indexing Settings for Your Blogger Blog

Would you like your BlogSpot blog content to be swiftly crawled, indexed, and ranked? If your answer is in the affirmative, you’ve come to the right destination.

To begin, let’s explore the concepts of crawling and indexing.

Optimal Crawlers and Indexing Settings for Your Blogger Blog

What is crawling?

In broad terms, crawling refers to the activity of identifying new or recently updated pages for inclusion in Google or Bing’s search indexes. This operation is carried out by search engine bots or spiders.

What is indexing?

Indexing essentially involves the task of processing information that search engines uncover during the crawling process and subsequently storing this information within an index. Think of an index as a vast digital library of web pages that can be retrieved when someone conducts a search.

At times, the terms crawling and indexing are used interchangeably, even though they pertain to distinct processes.

Shifting our focus away from definitions, it’s a common experience that when you establish a new website, there’s a time lag before search engines such as Google and Bing can crawl and index your web pages. However, you can expedite this process by configuring crawler and indexing settings within your Blogger dashboard.

Custom Robots.txt

Start by logging into your Blogger dashboard. Next, navigate to the blog settings, and scroll down to find the Crawlers and Indexing settings. To kick things off, activate the custom robots.txt option.
Click on “Custom Robot.txt,” then copy and paste the robot.txt file provided below.

Custom Robots.txt

Replace the Sitemap URL with your domain name.

Custom robot header tags

To start, enable the custom robot header tags. This will make the homepage tags, archive and search page tags, as well as post and page tags visible.

Custom robot header tags

Homepage Tags

Click on the homepage tags and enable all, along with “noodp,” then save your settings.

Homepage Tags

Archive and search page tags

For the archive and search page tags, enable “noindex” and “noodp,” then save your changes.

Archive and search page tags

Post and page tags

In this section, enable “all” and “noodp,” then save your settings.

Post and page tags