How to optimise a website for its crawl budget and get your most important pages indexed »
Answer by Jason Darrell:
Hi,. Thanks for the invitation to answer.
One of the lesser known elements of SEO is a site’s ‘crawl budget‘. Every website has a threshold that bots will crawl. If you’ve reached that limit, your site may seem not to have been indexed for a while.
Assuming that your robots.txt files are allowing indexers to crawl your site and haven’t been accidentally switched to noindex, it could be that your website has reached that saturation point. That’s the best way I can think of describing what I mean.
Imagine you’re targeting specific search terms across your site. If the only new content you’re producing is similar to that which already exists, why would Google or other search engines reward the new content with QDF/displaying it in SERPs?
The way Google is headed is that it’s feeding spiders on awesome, quality content. The more you feed it, the more its capacity for lunch. If you’re not telling the web anything new, your site will starve.
So, if you’re adding no new value to your website, the indexers will look at the overall power of your website, deem they’ve crawled all they want to and show no signs of having crawled the deeper pages.
How to optimise your website to make best use of its crawl budget
There is a way you can point the indexers in the direction of the pages that you want crawling. That’s whether you have fresh content or not.
But if you have got new content that you want indexing, it’s imperative that you know how to send the bots there. A share on Google+ is often a quick fix for a specific page, but doesn’t serve your long term purpose for your entire website.
Here’s whathas to say about freshness as a signal, in the context of what I’m trying to get across:
Now, hands up, the technical aspect of implementing 304s is beyond my remit – strictly an on-page semantic copy guy.
But my CTO at
The above video is one of three in the article that help underline why crawl budget is so important to understand and to optimise for. That goes for any webmaster who wants organic traffic.
As Matt himself would say, #HopeThatHelps ☺
image credit: algohunters
Visit the question on Quora, here: