Beware. Google does not seem to read and follow "robots.txt" files these days. Yes. I recently found that one of my websites is indexed by Google in spite of the fact that it has a robots.txt that prohibits all search engines from indexing it. I uploaded the robots.txt in July, and it is still in the Google index. I checked out Yahoo! to see how other search engines are doing, and I found that Yahoo! has already excluded that website.
I logged into my Google account to find out what Google is doing, and I learned that Google does seem to recognize the existence of the robots.txt and understand what it says. I wonder why Google has failed to exclude that website from their index. It might be a bug. At least we need to be cautious about Google in this matter.