Tag Archives: Google Webmaster Tools

spider-man

SEO with Screaming Frog

Leave a reply

If you’ve been following the series you’ll remember that we created and submitted XML sitemaps in the last post. Before I move into the spider simulation, let’s check on the status of the sitemap. Navigate to your Google Webmaster tools page and make note of any messages or warnings you see there. In my case, there is nothing new to report. No news is good news!

The Google Webmaster Tools dashboard is a great tool to quickly potential issues such as crawl errors or problems with your sitemap. The middle chart depicts the number of times my site is showing up in Google search. It’s low, but that’s to be expected with a new campaign. My focus is on the right graph, which shows the status of my sitemap.

It looks like things are running smoothly in the sitemap department, but my next step is to simulate a crawl of my website using a tool called Screaming Frog.

Screaming Frog SEO Spider

The free version of Screaming Frog will up to 500 URLs (pages), but since my website is new, that’s plenty. You can download Screaming Frog here: http://www.screamingfrog.co.uk/seo-spider/. To get an idea of what Screaming Frog can do, watch this video:

Starting from my home page, as directed, the spider will hop through hyperlinks much like Googlebot or another search engine spider would. Sitemaps and crawling a website are 2 ways that spiders can discover and report on content. For this exercise I’m going to filter my results to show only HTML. In other words, my pages. Screaming Frog found about 20 pages versus the 7 pages Google Webmaster Tools is showing in my sitemap.

What happened?

Robot Instructions

Earlier in the series I used Yoast’s WordPress SEO plugin to give the search engine spiders directions. I asked them (not ordered them) not to index certain types of pages: tags and categories. I also instructed my WordPress installation to exclude these post types from the sitemap. The image below shows that even though these pages were found, the directive “noindex” is present.

Looking good! Let’s do one final check to make sure this piece of the puzzle really is falling into place. I mentioned earlier that directives like “noindex” are requests, not really commands. It’s up to the search engine to follow these requests, so I’m going to do a site search of this domain to see what Google has in its index. (Google is partly a big, really big, database.)

You can see what Google has indexed for a website by typing site:http://yourwebsite as the search string. In my case, it’s site:http://marketingchris.com.

I see that there are 14 pages indexed versus 7 pages in my sitemap. Now what happened? Google must have crawled my site prior to me updating the settings asking it to not include categories in its index. Not a big deal! I know that these instructions are in place now, so I just need to be patient and wait for the next crawl.

Or not!

Fetch as Google

There is a tool in Google Webmaster Tools for the impatient SEO, like me, called Fetch as Googlebot.

 

I want Googlebot to re-spider my content to see that some of my requests of what to index have changed. Please let me know if you have any questions and I’ll see you next time!

 

 

 

site-map

Website Index and Crawl Health

3 Replies

Creating and Submitting XML Sitemaps

Since I’m building this site on WordPress with Yoast’s WordPress SEO plugin, setting up my XML sitemap has essentially been done for me. Check the box in the SEO settings of the XML sitemaps page and boom. A sitemap! I also checked off categories, tags and format as page types I didn’t want to include in my sitemap. I’ll get into that later on in this post, but my objective is clear: allow Google to index every possible piece of content on my website, but index it only once.

Now that I’ve got my sitemap file location from Yoast, http://marketingchris.com/sitemap_index.xml, it’s time to submit it to Google via Webmaster Tools, which I set up earlier in the campaign. From your GWT dashboard, click Optimization > Sitemaps > Add/Test Sitemap

Excluding Categories and Tags

It might seem strange that I didn’t want to include my category and tag pages in my sitemap, but I actually don’t want those indexed by Google. I mentioned earlier that I only one want my content indexed once and unless I make modifications to my category and tag pages, they will have the same exact content as the posts themselves. I want my pages to work together, not compete for link juice or traffic.

Look at how similar the category and tag pages are below.

 

The content I really want indexed are the articles themselves, not a list of articles. Thinking of it from a user experience perspective as well as respecting the resources needed to discover and store this information may be that which separates your website from others.

Create a Sitemap for a Non-WordPress Site

If you’re using another CMS (content management system), chances are it has a way for you to create XML sitemaps easily. However, if you (or your niece) built your website in a completely custom fashion, you may need to rely on another program to do the heavy lifting.

I recommend Xenu Link Sleuth as your starting point and keep it handy – you’ll probably use it later to detect potential issues with your website. Fire it up and launch your first crawl (yep, it’s a spider!) by clicking File > Check URL.

Xenu Link Sleuth

Once it’s finished crawling your website, click File > Create Google Sitemap File and then follow the directions above to submit your new XML sitemap to Google.

During my crawl I noticed some 404, Page Not Found errors. I’ll address those in my next post. If you want to skip ahead, please feel free to contact me! Thanks for reading.

google-webmaster-tools

Google Webmaster Tools

Leave a reply

In my previous post I showed you a cool WordPress plugin that captures traffic data. The next step in prepping my SEO campaign is to configure Google Webmaster Tools (GWT).

Set Up Google Webmaster Tools

Visit https://www.google.com/webmasters/tools/home?hl=en and click the Add a Site button. I add my URL and click next to get to the verification stage of the process. Now I have some options: I can download an HTML file (it’s a blank web page that proves you have some kind of ownership over the site);

or I can verify with an HTML tag, Google Analytics or via my domain name provider. Since I just set up Google Analytics using the asynchronous JavaScript, this method is going to be the best choice for me.

Since I just set up the account, the Google Webmaster Tools dashboard is likely to look a little boring. It needs data before it really becomes a resource, but I still have some configuring to do before I move on.

First I configure my preferred domain under settings. More than anything this is a reminder for myself to ensure I’m going to use a consistent version of my website’s URL from here on out. I’ll dive more into this later, but here is a handy SEO audit checklist with a section that speaks specifically to architecture and consistent URLs.

If you’re using WordPress your website is probably already using one version of the top level domain. In SEO we reference this as the www versus the non-www versions of a website. If we let it, Google will try to index each version of each page even though we know the pages are really the same.

How can you tell, though? The easiest test is to type your website into the browser with and without the www; www.marketingchris.com, for example. In my case, the www version of the website redirects to just http://marketingchris.com. I can watch as it happens using the built-in web developer tools in the Chrome browser. (CTRL + Shift + i will open this tool.)

Click the Network tab and type the version you want to test into the browser. Below you can see the original call for www.marketingchris.com followed by a 301 Permanent Redirect response to http://marketingchris.com. (Click image to enlarge)

If I’m using WordPress I can change which version I want to use by navigating to the Settings > General tab and adding or removing the www from the home page URL. You may need to verify the other version of your website as well.

If you have questions about this process and need help, please feel free to contact me. See you next time!