Robots.txt - Is it Even Important Anymore?

http://www.site-reference.com/articles/Website-Development