Robots Meta Tag
Yesterday we looked at how to use the Robot Exclusion Standard (that is your “robots.txt” file to you and me) that prevents spiders from visiting certain pages on your web site. Today we’re going to look at how to do the same with the Robots Meta Tag. This is similar to your robots.txt file but [...]
Like this blog post?
Feel free to use the link below on your website or blog to link back to us:
http://www.returnondigital.com/blog/robots-meta-tag
Spiders – How They Index The Web
Spiders are the elements of Search Engines that find or index search engines. Though that’s pretty near the mark, if we’re being picky it’s not strictly true, in that they have nothing to do with the indexing process other than initial data capture of individual web pages. In fact they’re not even really called spiders [...]
Like this blog post?
Feel free to use the link below on your website or blog to link back to us:
http://www.returnondigital.com/blog/spiders-how-they-index-the-web
