First time here? You may want to check out the blog archive, subscribe to the RSS feed, sign up for free email updates, or follow me on Twitter. Thanks for visiting!

Point Google to Your Sitemap Via Robots.txt 

Posted on Tue Apr 24, 2007 by Tom Kraak
In: DotNetNuke, SEO | 18 Comments

XML SitemapThe big four search engines (Google, Yahoo!, Microsoft Live Search and Ask.com) recently united behind a standard called sitemaps “autodiscovery.” So instead of submitting your xml sitemap to every search engine individually, you can now add the following line to robots.txt and your sitemap will be picked up automatically:

Sitemap: http://mysite.com/sitemap.xml.

This is not to say to you should forgo tools such as Google’s Webmaster Central or Yahoo’s Site Explorer, but it’s a great first step to expose your site to the major search engines quickly.

On a side note, back when DNN 4.5 was first announced I promised a review of DNN’s ability to create an xml sitemap dynamically. Well, it turns out there is not much to review after all. All DNN 4.5.x does at this point is create a sitemap according to your site’s navigation scheme as stored in the Tabs database table without Admin and Host menu items. For my site that means that all blog entries are not included, which is exactly the kind of content I want to tell Google about. Here is my “hand-massaged” sitemap created with xml-sitemaps.com.




Comments

Dan Tuesday, April 24, 2007 at 11:53 PM Dan says:

xml-sitemaps.com is a gem!

Thanks for the tip.

Tom Kraak Wednesday, April 25, 2007 at 12:35 AM Tom Kraak says:

I'm glad you found it helpful.

Dan Wednesday, April 25, 2007 at 11:09 AM Dan says:

Tom: the link pointed to in " All DNN 4.5.x does" is your sitemap... but this was NOT generated by DNN 4.5.1, but rather from the sitemaps-xml utility- correct?

Tom Kraak Wednesday, April 25, 2007 at 12:28 PM Tom Kraak says:

Sorry Dan, I knew this was going to be confusing.

Here is my sitemap generated by DNN:
http://seablick.com/sitemap.aspx

And here is my "custom" xml-sitemaps.com sitemap:
http://seablick.com/sitemap.xml

Feel free to ask more questions.

Dan Wednesday, April 25, 2007 at 12:52 PM Dan says:

Ah- my bad for not checking them... doh!

How about this: does it matter that the default sitemap is generated- if I am to use a different one (namely, one generated by xml-sitemap.com)?
As long as I submit/tell the 'bots which one to use, they'll ignore the other(s)?

Tom Kraak Wednesday, April 25, 2007 at 1:36 PM Tom Kraak says:

That's correct. No bot will "automatically scan" the root of your site for any sitemap file.

You'll have to tell the bot about the existence of your sitemap file by either submitting to the search engines directly or by pointing them to it via robots.txt.

Dan Wednesday, April 25, 2007 at 2:52 PM Dan says:

Great stuff!

Thanks for the enormously useful blog.

Tom Kraak Wednesday, April 25, 2007 at 2:58 PM Tom Kraak says:

My pleasure. Just keep telling your friends about it :)-

Also, what other topics would you like to see covered?

Dan Thursday, April 26, 2007 at 8:49 AM Dan says:

I found your short blog about "keywords" to be provocative... It seems that on the DNN "page settings" , there are lots of places for keywords (page title,description,keywords), and I wonder about their utility and relative effectiveness. Any chance you could elaborate upon this?

Tom Kraak Thursday, April 26, 2007 at 8:56 AM Tom Kraak says:

Sure Dan, I'll shed some light on DNN's Page Settings in terms of SEO. Look for it early next week. Thanks.

Michael P Thursday, April 26, 2007 at 11:51 AM Michael P says:

Tom, any idea how to handle pointing to the correct sitemap when you have multiple sites running on the same DNN installation?

Tom Kraak Friday, April 27, 2007 at 6:03 PM Tom Kraak says:

Mike, you'll have to submit the sitemaps by hand. I've never bought into the "feature" of running multiple sites from the same code base. Where are you going to put multiple robots.txt files?

Dan Sunday, April 29, 2007 at 6:27 PM Dan says:

eep!- one minor correction:
You wrote:
"Sitemaps: http://mysite.com/sitemap.xml"

Google no likey.

It appears that the line should read "Sitemap" not SitemapS (singular, not plural)

Cheers!

Tom Kraak Sunday, April 29, 2007 at 7:16 PM Tom Kraak says:

Good catch Dan. Thanks.

Website Marketing Monday, October 29, 2007 at 4:37 PM Website Marketing says:

Great stuff, very useful.

Bruce Chapman Tuesday, March 18, 2008 at 5:54 AM Bruce Chapman says:

Tom I keep running into your blog on Google in my daily trawl so you must be doing a good job of SEO. If you weren't aware, I have a free download for generating Blog sitemaps. I developed an extensible provider-model Sitemap generator before there was a sitemap.xml in DotNetNuke. The design I have implemented allows module developers to extend the base sitemap, so that modules like the blog module can have an entry generated for each separate Url in the blog. You can find it at http://www.ifinity.com.au/Products/articleType/ArticleView/articleId/3/

Tom Kraak Tuesday, March 18, 2008 at 12:19 PM Tom Kraak says:

Thanks Bruce, looks very promising. I'll take a closer look at your sitemap provider shortly.

Lugaluda Thursday, January 29, 2009 at 10:35 AM Lugaluda says:

Nice... thanks for sharing this tip dude... I'm currently having problems on google crawling my web pages. Will give this a shot...
Thanks....

Name (required)

Email (will not be published) (required)

Website

Enter the code shown above in the box below

Subscribe to our Feeds Follow on Twitter