Include XML sitemap in robots.txt

Posted by & filed under Code Tips.

If you’ve got a robots.txt file and an XML sitemap, add the following line to robots.txt so that search engine spiders will pick up your sitemap without you having to submit. Of course, change the url to match your actual site.

Should I block Baidu?

Posted by & filed under Internet.

Well, I am for now. I noticed Baidu pretty much on a forum of mine 24/7, but I didn’t recall ever seeing them show up in the referrer logs.  Over the past year, across the sites on my server, they’ve made almost 35,000 requests for pages; chewed up 337 megs of bandwidth, admittedly not a… Read more »