A file that is stored on a web server and contains instructions for search engine crawlers, or robots that index web pages for search engines. Web crawlers use robots.txt files to determine what should be included in their index and what should be ignored. Search engines can’t see pages like form results pages (“thank you” pages that appear when users submit a form, for example).