Crawl budget definition: What is crawl budget?

Crawl budget it is the number of resources (time/kb) that Google allocates to your web page to crawl on a daily basis. In other words, is the number of pages Google crawls and indexes on a website within a given timeframe.

Crawl budget for SEO: Why is crawl budget important for SEO?

In short: if Google doesn’t index a page, it’s not going to rank for anything. So if your number of pages exceed your site’s crawl budget, you’re going to have pages on your site that aren’t indexed.

As you can imagine this is a very important aspect, especially in large pages (> 1,000 pages). For small pages, it is not advisable to become obsessed with it because you will not notice much difference.

Crawl budget calculation: How does Google calculate/assign crawl budget?

This crawl budget that Google assigns to your website depends mainly on:

1. Page Authority
2. Frequency of publication or content update (you can have the same number of URLs, but keep them up to date, which increases the crawl frequency)
3. Web load time

In other words, the more links your website receives, the more content you publish and the less time it takes to load your website, the greater the crawl budget will be assigned by Google to its robots.

Crawl budget optimization: How can we optimize the crawl budget?

Let’s see the options (beyond the obvious ones) that we have to optimize our website’s crawl budget:

• Remove links to pages with errors or redirects
• Optimize web load time
• Increase the frequency of publication/update
• Increase the number of backlinks to your website

[irp posts=”24222″ name=”Linkbuilding for SEO: Free Backlinking and Guest Post Secrets by SEO Specialists”]

This is the beginning of the ‘beautiful’ part… It is VERY important to prevent Google from crawling pages that we are not interested in positioning (duplicate content, low value for SEO, etc.). For this I recommend:

• Add a no-index value to that page
• Exclude the page from the sitemap (if generated automatically)
• Block that page from your robots.txt:
  User-agent: *
  Disallow: /page/
• Obfuscate links

How to Obfuscate links to optimize our crawl budget?

To obfuscate a link, we need to add some Javascript code. The code to be used, illustrated with an example would be similar to this:

<a class=”pointer” onclick=”location.href=’/legal-notice/'” title=”Legal Notice”> Legal Notice</a>

However, there is a WordPress plugin created by the SEO expert Fede Gómez that I use to obfuscate links on any WordPress site without having to add a single line of Javascript or CSS code. You can download it here. Simply click on ‘DESCARGAR’.

This plugin allows you to obfuscate links in 3 parts of our WordPress:

• Elements of a menu.
• Links within the content of post and pages.
• Links within the content of HTML widgets.

To obfuscate a link, all you have to do is apply the ‘ljoptimizer’ class, then the plugin will do its job using PHP and javascript functions. You can read more information about how to use this plugin here. (sorry, the post is only available in Spanish, but I hope you can use Google to translate it!)