. ..and more from giles walker >
http://www.gileswalker.org/gileswalker.org/ROBOTS/ROBOTS.html