Webmaster Forum

Go Back   Webmaster Forum > Web Development > Blogging Forum

Blogging Forum Discuss general blogging issues here - design, integration, posting, trackbacks, ETC. Also discuss blogs you like.

Thread Tools Display Modes
Share |
Old 11-29-2014, 03:24 AM
rimidas's Avatar
rimidas rimidas is offline
Junior Member
Join Date: 11-11-14
Posts: 46
iTrader: 0 / 0%
Should I use custom robots.txt file or not ?

I have about 400+ post in my blog. Recently i have added a new domain to it.
Now I am using custom robots.txt :
User-agent: Mediapartners-Google

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://myblog.com/feeds/posts/default?orderby=UPDATED
Sitemap: http://myblog.com/atom.xml?redirect=false&start-index=1&max-results=500

And I have just submit a sitemap :


Is it good configuration ? Or I should use other configiration ? Pls siggest me. Thank you.

Last edited by J. H. Rasmussen; 11-29-2014 at 03:39 AM. Reason: Removed live part of 3 links.
Reply With Quote



Old 11-29-2014, 10:09 AM
iwe Magazine's Avatar
iwe Magazine iwe Magazine is offline
Contributing Member
Join Date: 03-12-14
Location: Clearwater, Florida
Posts: 198
iTrader: 0 / 0%
This all depends what you're trying to accomplish. Just be aware that Google has their own robots that watch your site, so make sure you aren't doing anything illegal or your site will be taking down. Some custom robots are good, some aren't.
Reply With Quote
Old 11-29-2014, 08:49 PM
kimseasok's Avatar
kimseasok kimseasok is offline
Contributing Member
Join Date: 11-23-14
Location: Cambodia
Posts: 215
iTrader: 0 / 0%
I think that the default robot.txt of Blogger is great but to make sure that you webmaster and nothing error such duplicate title or so on. Thus, check webmaster tool fist and consideration change
Reply With Quote
Old 11-30-2014, 05:15 AM
ScriptMan's Avatar
ScriptMan ScriptMan is online now
Super Moderator
Join Date: 02-10-07
Location: Central Kentucky
Posts: 13,954
iTrader: 4 / 100%
I think this is what Rimidas asked. Compliments of Hostgator:

The subdomain name and the primarydomain/subfolder name will not be seen by the search engines, unless you link to it.

EXAMPLE: Let's say you own mynewworld.com and decide to make it an addon domain. If you publish any links, purposefully or inadvertently, that point to http://primarydomain.com/mynewworld.com/
source and more: http://support.hostgator.com/article...-addon-domains
Reply With Quote
Old 12-03-2014, 08:41 AM
saifullahkhan saifullahkhan is offline
Junior Member
Join Date: 12-03-14
Location: Pakistan
Posts: 24
iTrader: 0 / 0%
You should go with default robots.txt file ......
Reply With Quote
Old 12-03-2014, 08:58 AM
J. H. Rasmussen J. H. Rasmussen is online now
Super Moderator
Join Date: 11-11-11
Location: Copenhagen, Denmark
Posts: 5,992
iTrader: 1 / 100%
Originally Posted by saifullahkhan View Post
You should go with default robots.txt file ......
Why? Please add some details to your replies.
Reply With Quote
Old 01-12-2015, 01:29 AM
RajatGaur's Avatar
RajatGaur RajatGaur is offline
Junior Member
Join Date: 12-17-14
Location: India
Posts: 11
iTrader: 0 / 0%
Thumbs down

I'm guessing you are on Blogger. Blogger's default robots.txt is fine, but bear in mind that it blocks label pages by disallowing /search. If you want Google to index your label pages you may allow /search pages (I would not recommend that, as this might sometimes cause duplicate content problem.). If there is any other page that you don't want to be indexed in search results, you may disallow that, otherwise you need not mess with it.
Reply With Quote
Go Back   Webmaster Forum > Web Development > Blogging Forum


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Is it necessary to use a robots.txt file? dominicyordz SEO Forum 21 08-30-2011 09:39 AM
What is robots file? juliacates1 Coding Forum 6 02-08-2011 04:04 PM
example of robots.txt file 3DProf4online Web Design Lobby 6 05-07-2008 10:10 PM
What is in your robots.txt file? m42 Web Design Lobby 4 05-05-2008 01:49 PM

V7N Network
Get exposure! V7N I Love Photography V7N SEO Blog V7N Directory

All times are GMT -7. The time now is 09:37 PM.
Powered by vBulletin
Copyright 2000-2014 Jelsoft Enterprises Limited.
Copyright © 2003 - 2018 VIX-WomensForum LLC