I like this resource on the ins and outs of Google’s Crawlers or Spiders. The page referenced below explains in an easily understood way, how crawlers and spiders interact with both the robots.txt file and the robots meta tag.
“Crawler” is a generic term for any program (such as a robot or spider) used to automatically discover and scan websites by following links from one webpage to another.
This table lists information about the common Google crawlers you may see in your referrer logs, and how they should be specified in robots.txt, the robots meta tags, and the X-Robots-Tag HTTP directives.
If you want all of Google to be able to crawl your pages, you don’t need a robots.txt file at all.
If you want to block or allow all of Google’s crawlers from accessing some of your content, you can do this by specifying Googlebot as the user-agent.
For example, if you want all your pages to appear in Google search, and if you want AdSense ads to appear on your pages, you don’t need a robots.txt file.