robots.txt is not a security mechanism.

It is basically a little note taped to the door for polite software.

---