well, there are some spiders that obey robots.txt.