Robot.txt file technical seo using the robot exclusion standard to talk to Google Bing search engine crawlers

What is the robot.txt file? It is a file that your developer can generate and leave at the root directory of your site. The robot.txt can give instructions to Google website crawlers on how to properly access and index your site using a protocol and some basic commands called the Robot Exclusion Standard. Why would want Google to exclude your content from its search? Maybe the crawlers are generating too many requests and over burdening your API servers. Maybe you have images or webpages that you'd rather not appear at the top positions in a Google search result. However, these instructions don't prevent other websites from linking to your web pages and do not keep your files secure. It's just some basic instructions to google. It can be helpful when telling Google to avoid certain duplicated pages on your site which Google may otherwise punish you for "trying to game organic keywords and search results". There are many tools that can help web masters or developers generate this file. Use a tool can ensure correct syntax, ease of making robot.txt for different search engines such as Google Bing and Yahoo, conserving time and effort of generating this file correctly.

https://support.google.com/webmasters/answer/6062608?hl=en