Webmaster Forum

Go Back   Webmaster Forum > Marketing Forums > SEO Forum

SEO Forum Search engine optimization discussions.

Closed Thread
Thread Tools Display Modes
Share |
Old 08-27-2017, 11:58 PM
Fadia Sheetal Fadia Sheetal is offline
Contributing Member
Join Date: 05-29-15
Posts: 136
iTrader: 0 / 0%
Is Robots allow or not to all Search Engines ?

Hay Users,

I start working on a Website and bit confuse about the robots. What is the right extension for search engines allow or not to crawl the website ?



Old 08-29-2017, 02:04 AM
Naksh Naksh is offline
Contributing Member
Join Date: 05-31-17
Posts: 114
iTrader: 0 / 0%
You can specify the webpages you want the crawlers to access by stating it in robots.txt file. You can also create XML Sitemap.
Old 08-29-2017, 03:49 AM
coreisolutions coreisolutions is offline
Junior Member
Join Date: 01-21-17
Posts: 14
iTrader: 0 / 0%
By robots.txt you can specify instructions related to crawling for all search engines. it is an Important on page fector.

User-Agent: *
Allow: /

In this we allow all search engines to crawl all pages of website.
Old 08-30-2017, 01:35 AM
gs_ng gs_ng is offline
Contributing Member
Join Date: 07-12-17
Posts: 54
iTrader: 0 / 0%
Hi, by default your site is crawlable by search engines unless you put "noindex". To block search engines to crawl certain directories, use:

User-agent: *
Disallow: /

For example:
User-agent: *
Disallow: /cgi-bin/

will block search engines to access your /cgi-bin/ directory.

Please note that the robots.txt is a public file (viewable by users), so don't use it to hide any sensitive URL path.
Old 09-04-2017, 12:45 AM
Sumit_Kumar Sumit_Kumar is offline
Contributing Member
Join Date: 09-03-17
Location: DELHI, INDIA
Posts: 63
iTrader: 0 / 0%
If you do not restrict any web page for crawling so just write the following code in your robots.txt file
User-agent: *
if need to restrict only one page for all the SE Spiders
Disallow: /contact-us.html/
Old 09-04-2017, 02:02 AM
NehaMalik's Avatar
NehaMalik NehaMalik is offline
Contributing Member
Join Date: 08-22-17
Location: Hyderabad
Posts: 91
iTrader: 0 / 0%
there are many meta tags like nofollow no index ....etc but you can easily create a robots.txt file and upload into your hosting that will be enough for Google bot.
Old 09-28-2017, 05:49 PM
HTMLBasicTutor's Avatar
HTMLBasicTutor HTMLBasicTutor is offline
Join Date: 10-29-07
Location: Canada
Posts: 26,710
iTrader: 5 / 100%
The bots will follow all your links within the site including links to your images.

If you want to learn how to control what is indexed and what is not you should read The Web Robots Pages.

Note: only polite bots obey your instructions in the robots.txt file so if there is anything super sensitive them you should put it behind a login.
Go Back   Webmaster Forum > Marketing Forums > SEO Forum

Closed Thread

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Malicious Image Defacement Hidden from Search Engines snakeair SEO Forum 0 04-16-2017 05:25 AM

V7N Network
Get exposure! V7N I Love Photography V7N SEO Blog V7N Directory

All times are GMT -7. The time now is 08:27 PM.
Powered by vBulletin
Copyright 2000-2014 Jelsoft Enterprises Limited.
Copyright © 2003 - 2018 VIX-WomensForum LLC