Webmaster Forum

Webmaster Forum (http://www.v7n.com/forums/index.php)
-   SEO Forum (http://www.v7n.com/forums/forumdisplay.php?f=11)
-   -   Robots Exclusion Protocol Specification (http://www.v7n.com/forums/showthread.php?t=429976)

HTMLBasicTutor 07-03-2019 08:21 AM

Robots Exclusion Protocol Specification
3 articles regarding changes to the Robots Exclusion Protocol from Google:


For 25 years, the Robots Exclusion Protocol (REP) has been one of the most basic and critical components of the web. It allows website owners to exclude automated clients, for example web crawlers, from accessing their sites - either partially or completely....

...the REP was never turned into an official Internet standard, which means that developers have interpreted the protocol somewhat differently over the years. And since its inception, the REP hasn't been updated to cover today's corner cases....

...Together with the original author of the protocol, webmasters, and other search engines, we've documented how the REP is used on the modern web, and submitted it to the IETF...
Full article with specifics on the changes: Formalizing the Robots Exclusion Protocol Specification July 01, 2019


... we open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files. This library has been around for 20 years and it contains pieces of code that were written in the 90's. Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense.
We also included a testing tool in the open source package to help you test a few rules....
Read more about Google's robots.txt parser is now open source July 01, 2019


...In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we're retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019. For those of you who relied on the noindex indexing directive in the robots.txt file, which controls crawling, there are a number of alternative options:
Read the full article related to unsupported rules in robots.txt July 02, 2019

HTMLBasicTutor 07-03-2019 09:44 AM

Bing: We Never Supported Noindex In Robots.txt
Barry Schwartz of Search Engine Roundtable's conversation with Frédéric Dubut from Bing regarding noindex in the Robots.txt file:

Frédéric Dubut from Bing said that its search engine never supported the noindex in a robots.txt file before. So nothing is going to be changing with Bing on that front...
Bing: We Never Supported Noindex In Robots.txt
Jul 3, 2019

All times are GMT -7. The time now is 09:23 AM.