Webmaster Forum

Go Back   Webmaster Forum > Marketing Forums > Google Forum

Google Forum Discuss Google related issues.


Reply
 
Thread Tools Display Modes
Share |
  #1  
Old 11-24-2010, 05:13 PM
HTMLBasicTutor's Avatar
HTMLBasicTutor HTMLBasicTutor is offline
Administrator
 
Join Date: 10-29-07
Location: Canada
Posts: 26,710
iTrader: 5 / 100%
Controlling crawling and indexing now documented on code.google.com

Quote:
Do you know how Google's crawler, Googlebot, handles conflicting directives in your robots.txt file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot’s favorite song? The answers to these questions (except for the last one ), along with lots of other information about controlling the crawling and indexing of your site, are now available on code.google.com:
Read more: Official Google Webmaster Central Blog: Controlling crawling and indexing now documented on code.google.com
 
Reply With Quote

Advertisement

Advertisement

  #2  
Old 11-24-2010, 05:27 PM
HTMLBasicTutor's Avatar
HTMLBasicTutor HTMLBasicTutor is offline
Administrator
 
Join Date: 10-29-07
Location: Canada
Posts: 26,710
iTrader: 5 / 100%
Interesting Tidbits

Here's some interesting tidbits of info I found while reading through the documentation:
Quote:
Note: Pages may be indexed despite never having been crawled: the two processes are independent of each other. If enough information is available about a page, and the page is deemed relevant to users, search engine algorithms may decide to include it in the search results despite never having had access to the content directly. That said, there are simple mechanisms such as robots meta tags to make sure that pages are not indexed.
Quote:
URLs disallowed by the robots.txt file might still be indexed without being crawled, and the robots.txt file can be viewed by anyone, potentially disclosing the location of your private content.
Quote:
Note: Keep in mind that in order for a crawler to find a meta tag or HTTP header element, the crawler must be able to crawl the page—it cannot be disallowed from crawling with the robots.txt file.
http://code.google.com/web/controlcr...g_started.html

Appendix: Google's website crawlers
 
Reply With Quote
  #3  
Old 11-24-2010, 11:38 PM
ponmayil ponmayil is offline
Contributing Member
 
Join Date: 08-10-09
Location: India
Posts: 150
iTrader: 0 / 0%
Google giving useful information to all SEO experts. Thanks very much to HTMLBasicTutor for sharing useful information.
 
Reply With Quote
  #4  
Old 11-25-2010, 12:56 AM
TechWorm's Avatar
TechWorm TechWorm is offline
Junior Member
 
Join Date: 11-01-10
Posts: 34
iTrader: 0 / 0%
I knew that! Thanks for information, I suspected that crawling and indexing are different processes, now I know it exactly
 
Reply With Quote
Go Back   Webmaster Forum > Marketing Forums > Google Forum

Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
google crawling and indexing rajwebwiz SEO Forum 6 11-12-2010 06:15 AM
Google crawling and indexing question wahoo94 Google Forum 6 10-28-2009 12:48 AM
Google Title and Descirption crawling and indexing problem miccharlys SEO Forum 3 10-17-2009 03:17 AM
InfoAppenders on Indexing and crawling issues InfoAppenders SEO Forum 6 03-03-2009 03:52 AM
Yahoo and MSN are crawling my site but not indexing casperl SEO Forum 0 06-10-2007 11:10 AM


V7N Network
Get exposure! V7N I Love Photography V7N SEO Blog V7N Directory


All times are GMT -7. The time now is 05:15 AM.
Powered by vBulletin
Copyright © 2000-2014 Jelsoft Enterprises Limited.
Copyright © 2003 - 2018 VIX-WomensForum LLC