Call Us 203.687.6053

Blog with a focus on DotNetNuke news, tips and tricks, DNN SEO, and insights and opinions about the DNN community at large.

Point Google to Your Sitemap Via Robots.txt

Posted by Tom on Tuesday, April 24, 2007 to DotNetNuke, SEO

XML SitemapThe big four search engines (Google, Yahoo!, Microsoft Live Search and Ask.com) recently united behind a standard called sitemaps “autodiscovery.” So instead of submitting your xml sitemap to every search engine individually, you can now add the following line to robots.txt and your sitemap will be picked up automatically:

Sitemap: http://mysite.com/sitemap.xml.

This is not to say to you should forgo tools such as Google’s Webmaster Central or Yahoo’s Site Explorer, but it’s a great first step to expose your site to the major search engines quickly.

On a side note, back when DNN 4.5 was first announced I promised a review of DNN’s ability to create an xml sitemap dynamically. Well, it turns out there is not much to review after all. All DNN 4.5.x does at this point is create a sitemap according to your site’s navigation scheme as stored in the Tabs database table without Admin and Host menu items. For my site that means that all blog entries are not included, which is exactly the kind of content I want to tell Google about. Here is my “hand-massaged” sitemap created with xml-sitemaps.com.


Permalink Permalink      Comments 17 Comments      RSS feeds RSS feeds      Email updates Email updates

Technorati tags
DotNetNuke, SEO
 

Comments

Comment By Dan on Tuesday, April 24, 2007 at 11:53 PM

xml-sitemaps.com is a gem!

Thanks for the tip.

Comment By Tom Kraak on Wednesday, April 25, 2007 at 12:35 AM

I'm glad you found it helpful.

Comment By Dan on Wednesday, April 25, 2007 at 11:09 AM

Tom: the link pointed to in " All DNN 4.5.x does" is your sitemap... but this was NOT generated by DNN 4.5.1, but rather from the sitemaps-xml utility- correct?

Comment By Tom Kraak on Wednesday, April 25, 2007 at 12:28 PM

Sorry Dan, I knew this was going to be confusing.

Here is my sitemap generated by DNN:
http://seablick.com/sitemap.aspx

And here is my "custom" xml-sitemaps.com sitemap:
http://seablick.com/sitemap.xml

Feel free to ask more questions.

Comment By Dan on Wednesday, April 25, 2007 at 12:52 PM

Ah- my bad for not checking them... doh!

How about this: does it matter that the default sitemap is generated- if I am to use a different one (namely, one generated by xml-sitemap.com)?
As long as I submit/tell the 'bots which one to use, they'll ignore the other(s)?

Comment By Tom Kraak on Wednesday, April 25, 2007 at 1:36 PM

That's correct. No bot will "automatically scan" the root of your site for any sitemap file.

You'll have to tell the bot about the existence of your sitemap file by either submitting to the search engines directly or by pointing them to it via robots.txt.

Comment By Dan on Wednesday, April 25, 2007 at 2:52 PM

Great stuff!

Thanks for the enormously useful blog.

Comment By Tom Kraak on Wednesday, April 25, 2007 at 2:58 PM

My pleasure. Just keep telling your friends about it :)-

Also, what other topics would you like to see covered?

Comment By Dan on Thursday, April 26, 2007 at 8:49 AM

I found your short blog about "keywords" to be provocative... It seems that on the DNN "page settings" , there are lots of places for keywords (page title,description,keywords), and I wonder about their utility and relative effectiveness. Any chance you could elaborate upon this?

Comment By Tom Kraak on Thursday, April 26, 2007 at 8:56 AM

Sure Dan, I'll shed some light on DNN's Page Settings in terms of SEO. Look for it early next week. Thanks.

Comment By Michael P on Thursday, April 26, 2007 at 11:51 AM

Tom, any idea how to handle pointing to the correct sitemap when you have multiple sites running on the same DNN installation?

Comment By Tom Kraak on Friday, April 27, 2007 at 6:03 PM

Mike, you'll have to submit the sitemaps by hand. I've never bought into the "feature" of running multiple sites from the same code base. Where are you going to put multiple robots.txt files?

Comment By Dan on Sunday, April 29, 2007 at 6:27 PM

eep!- one minor correction:
You wrote:
"Sitemaps: http://mysite.com/sitemap.xml"

Google no likey.

It appears that the line should read "Sitemap" not SitemapS (singular, not plural)

Cheers!

Comment By Tom Kraak on Sunday, April 29, 2007 at 7:16 PM

Good catch Dan. Thanks.

Comment By Website Marketing on Monday, October 29, 2007 at 4:37 PM

Great stuff, very useful.

Comment By Bruce Chapman on Tuesday, March 18, 2008 at 5:54 AM

Tom I keep running into your blog on Google in my daily trawl so you must be doing a good job of SEO. If you weren't aware, I have a free download for generating Blog sitemaps. I developed an extensible provider-model Sitemap generator before there was a sitemap.xml in DotNetNuke. The design I have implemented allows module developers to extend the base sitemap, so that modules like the blog module can have an entry generated for each separate Url in the blog. You can find it at http://www.ifinity.com.au/Products/articleType/ArticleView/articleId/3/

Comment By Tom Kraak on Tuesday, March 18, 2008 at 12:19 PM

Thanks Bruce, looks very promising. I'll take a closer look at your sitemap provider shortly.

Add a comment
Add to Technorati Favorites

Email Updates

Enter your email address below and find our blog updates in your inbox.