Webmaster Forum

Go Back   Webmaster Forum > Marketing Forums > SEO Forum

SEO Forum Search engine optimization discussions.


Reply
 
Thread Tools Display Modes
Share |
  #1  
Old 07-03-2019, 08:21 AM
HTMLBasicTutor's Avatar
HTMLBasicTutor HTMLBasicTutor is offline
Administrator
 
Join Date: 10-29-07
Location: Canada
Posts: 26,637
iTrader: 5 / 100%
Robots Exclusion Protocol Specification

3 articles regarding changes to the Robots Exclusion Protocol from Google:

Quote:
For 25 years, the Robots Exclusion Protocol (REP) has been one of the most basic and critical components of the web. It allows website owners to exclude automated clients, for example web crawlers, from accessing their sites - either partially or completely....

...the REP was never turned into an official Internet standard, which means that developers have interpreted the protocol somewhat differently over the years. And since its inception, the REP hasn't been updated to cover today's corner cases....

...Together with the original author of the protocol, webmasters, and other search engines, we've documented how the REP is used on the modern web, and submitted it to the IETF...
Full article with specifics on the changes: Formalizing the Robots Exclusion Protocol Specification July 01, 2019

Quote:
... we open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files. This library has been around for 20 years and it contains pieces of code that were written in the 90's. Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense.
We also included a testing tool in the open source package to help you test a few rules....
Read more about Google's robots.txt parser is now open source July 01, 2019

Quote:
...In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we're retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019. For those of you who relied on the noindex indexing directive in the robots.txt file, which controls crawling, there are a number of alternative options:
Read the full article related to unsupported rules in robots.txt July 02, 2019
 
Reply With Quote

Advertisement

Advertisement

  #2  
Old 07-03-2019, 09:44 AM
HTMLBasicTutor's Avatar
HTMLBasicTutor HTMLBasicTutor is offline
Administrator
 
Join Date: 10-29-07
Location: Canada
Posts: 26,637
iTrader: 5 / 100%
Bing: We Never Supported Noindex In Robots.txt

Barry Schwartz of Search Engine Roundtable's conversation with Frédéric Dubut from Bing regarding noindex in the Robots.txt file:
Quote:
Frédéric Dubut from Bing said that its search engine never supported the noindex in a robots.txt file before. So nothing is going to be changing with Bing on that front...
Bing: We Never Supported Noindex In Robots.txt
Jul 3, 2019
 
Reply With Quote
Go Back   Webmaster Forum > Marketing Forums > SEO Forum

Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
How to Solve HTTP Protocol Errors..? swatijain2233 SEO Forum 2 12-12-2017 06:03 AM
Is Robots allow or not to all Search Engines ? Fadia Sheetal SEO Forum 6 09-28-2017 05:49 PM
What are Spiders, Robots and Crawlers and what are their functions? nazarseo SEO Forum 2 07-28-2017 05:06 AM


V7N Network
Get exposure! V7N I Love Photography V7N SEO Blog V7N Directory


All times are GMT -7. The time now is 12:30 AM.
Powered by vBulletin
Copyright © 2000-2014 Jelsoft Enterprises Limited.
Copyright © 2003 - 2018 VIX-WomensForum LLC