Wednesday, January 27, 2016
Robot.txt file technical seo using the robot exclusion standard to talk to Google Bing search engine crawlers
What is the robot.txt file? It is a file that your developer can generate and leave at the root directory of your site. The robot.txt can give instructions to Google website crawlers on how to properly access and index your site using a protocol and some basic commands called the Robot Exclusion Standard. Why would want Google to exclude your content from its search? Maybe the crawlers are generating too many requests and over burdening your API servers. Maybe you have images or webpages that you'd rather not appear at the top positions in a Google search result. However, these instructions don't prevent other websites from linking to your web pages and do not keep your files secure. It's just some basic instructions to google. It can be helpful when telling Google to avoid certain duplicated pages on your site which Google may otherwise punish you for "trying to game organic keywords and search results". There are many tools that can help web masters or developers generate this file. Use a tool can ensure correct syntax, ease of making robot.txt for different search engines such as Google Bing and Yahoo, conserving time and effort of generating this file correctly.
What is a domain name system (DNS)? How stuff works explains it in a very good graph I was very confused by the Wikipedia explanatio...
The bogus request from P2PU to hunt for HTML tags in real life has yielded a lot of good thoughts. My first impression was that this is stup...
I recently submitted Project 1 of the Udacity Full Stack Nanodegree curriculum. This blog post is my reflection and review on the experience...