Why isn’t my site crawling since 1.5 months ago?

How to optimise a website for its crawl budget and get your most important pages indexed »

Answer by Jason Darrell:

Hi, Maulik Solanki. Thanks for the invitation to answer.

One of the lesser known elements of SEO is a site’s ‘crawl budget‘. Every website has a threshold that bots will crawl. If you’ve reached that limit, your site may seem not to have been indexed for a while.

Assuming that your robots.txt files are allowing indexers to crawl your site and haven’t been accidentally switched to noindex, it could be that your website has reached that saturation point. That’s the best way I can think of describing what I mean.

Imagine you’re targeting specific search terms across your site. If the only new content you’re producing is similar to that which already exists, why would Google or other search engines reward the new content with QDF/displaying it in SERPs?

The way Google is headed is that it’s feeding spiders on awesome, quality content. The more you feed it, the more its capacity for lunch. If you’re not telling the web anything new, your site will starve.

So, if you’re adding no new value to your website, the indexers will look at the overall power of your website, deem they’ve crawled all they want to and show no signs of having crawled the deeper pages.

How to optimise your website to make best use of its crawl budget

There is a way you can point the indexers in the direction of the pages that you want crawling. That’s whether you have fresh content or not.

But if you have got new content that you want indexing, it’s imperative that you know how to send the bots there. A share on Google+ is often a quick fix for a specific page, but doesn’t serve your long term purpose for your entire website.

Here’s what Matt Cutts has to say about freshness as a signal, in the context of what I’m trying to get across:

Now, hands up, the technical aspect of implementing 304s is beyond my remit – strictly an on-page semantic copy guy.

But my CTO at SEO Workers, John S. Britsios, has  written an in-depth article about the topic on algohunters, a subsidiary site:
Building a Solid Index Presence by Optimizing your Crawl Budget

The above video is one of three in the article that help underline why crawl budget is so important to understand and to optimise for. That goes for any webmaster who wants organic traffic.

As Matt himself would say, #HopeThatHelps ☺
image credit: algohunters

Visit the question on Quora, here: Why isn’t my site crawling since 1.5 months ago?


What do YOU think? Pray, tell...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s