Webmaster Forum


Go Back   Webmaster Forum > Marketing Forums > SEO Forum
quick robots.txt question... please help :) quick robots.txt question... please help :)
Register FAQ Members List Calendar Search Today's Posts Mark Forums Read

SEO Forum Search engine optimization discussions.

Directory Submission Service   ClickBooth Network   V7N Directory

Reply
 
LinkBack Thread Tools Display Modes
Old 11-11-2006, 02:39 PM   #1 (permalink)
bwb
Contributing Member
 
bwb's Avatar
 
Join Date: 07-21-05
Location: USA
Posts: 57
iTrader: 0 / 0%
Latest Blog:
None

bwb is liked by somebodybwb is liked by somebodybwb is liked by somebodybwb is liked by somebodybwb is liked by somebody
quick robots.txt question... please help :)
Quick question on robots.txt files...

I have a URL I don't want bots to crawl:
www.testing.com/click.php?link=example.com

Is this what I should use for the robots.txt file?

User-agent: *
Disallow: /click.php


What about if the URL is:
http://www.test.com/contact?comment=yes&subject=blah

Is the below right?

User-agent: *
Disallow: /contact

even though I'm using a rewrite on contact.php?
bwb is offline  
Add Post to del.icio.us
Reply With Quote
Sponsored Links
SEO Hosting by HostGator  Advertise Here  Buy Blog Links
Old 11-11-2006, 04:00 PM   #2 (permalink)
Contributing Member
 
Blazed's Avatar
 
Join Date: 10-07-06
Posts: 188
iTrader: 0 / 0%
Latest Blog:
None

Blazed is liked by somebodyBlazed is liked by somebodyBlazed is liked by somebodyBlazed is liked by somebody
Send a message via Yahoo to Blazed
Quote:
Originally Posted by bwb View Post
Quick question on robots.txt files...

I have a URL I don't want bots to crawl:
www.testing.com/click.php?link=example.com

Is this what I should use for the robots.txt file?

User-agent: *
Disallow: /click.php


What about if the URL is:
http://www.test.com/contact?comment=yes&subject=blah

Is the below right?

User-agent: *
Disallow: /contact

even though I'm using a rewrite on contact.php?
For your first case..
Click.php is correct.
and for 2nd case
User-agent: *
Disallow: /contact.php
__________________
█ SEO Company ||
Blazed is offline  
Add Post to del.icio.us
Reply With Quote
Old 11-11-2006, 04:03 PM   #3 (permalink)
bwb
Contributing Member
 
bwb's Avatar
 
Join Date: 07-21-05
Location: USA
Posts: 57
iTrader: 0 / 0%
Latest Blog:
None

bwb is liked by somebodybwb is liked by somebodybwb is liked by somebodybwb is liked by somebodybwb is liked by somebody
Quote:
Originally Posted by Blazed View Post
For your first case..
Click.php is correct.
and for 2nd case
User-agent: *
Disallow: /contact.php
Even though it is written /contact and the .php revmoed?

for a directory should it be

Disallow: /directory/

or no slash on the end?

thanks!
bwb is offline  
Add Post to del.icio.us
Reply With Quote
Old 11-11-2006, 04:20 PM   #4 (permalink)
Contributing Member
 
Blazed's Avatar
 
Join Date: 10-07-06
Posts: 188
iTrader: 0 / 0%
Latest Blog:
None

Blazed is liked by somebodyBlazed is liked by somebodyBlazed is liked by somebodyBlazed is liked by somebody
Send a message via Yahoo to Blazed
Quote:
Originally Posted by bwb View Post
Even though it is written /contact and the .php revmoed?

for a directory should it be

Disallow: /directory/

or no slash on the end?

thanks!
thanx for the correction brother.
__________________
█ SEO Company ||
Blazed is offline  
Add Post to del.icio.us
Reply With Quote
Old 11-11-2006, 05:10 PM   #5 (permalink)
bwb
Contributing Member
 
bwb's Avatar
 
Join Date: 07-21-05
Location: USA
Posts: 57
iTrader: 0 / 0%
Latest Blog:
None

bwb is liked by somebodybwb is liked by somebodybwb is liked by somebodybwb is liked by somebodybwb is liked by somebody
so should have a / on the end?
bwb is offline  
Add Post to del.icio.us
Reply With Quote
Old 11-12-2006, 01:48 AM   #6 (permalink)
Contributing Member
 
Blazed's Avatar
 
Join Date: 10-07-06
Posts: 188
iTrader: 0 / 0%
Latest Blog:
None

Blazed is liked by somebodyBlazed is liked by somebodyBlazed is liked by somebodyBlazed is liked by somebody
Send a message via Yahoo to Blazed
haha really i'm not sure about it thought u did a correction to my statement.
__________________
█ SEO Company ||
Blazed is offline  
Add Post to del.icio.us
Reply With Quote
Old 11-12-2006, 01:50 AM   #7 (permalink)
Contributing Member
 
Blazed's Avatar
 
Join Date: 10-07-06
Posts: 188
iTrader: 0 / 0%
Latest Blog:
None

Blazed is liked by somebodyBlazed is liked by somebodyBlazed is liked by somebodyBlazed is liked by somebody
Send a message via Yahoo to Blazed
anyways one can get the answer from the following links...

http://www.google.com/support/webmas...364&topic=8846

http://www.google.com/support/webmas...367&topic=8846
__________________
█ SEO Company ||
Blazed is offline  
Add Post to del.icio.us
Reply With Quote
Old 11-12-2006, 04:42 AM   #8 (permalink)
Inactive
 
Join Date: 11-12-06
Posts: 276
iTrader: 0 / 0%
Latest Blog:
None

TheBrain is just really niceTheBrain is just really niceTheBrain is just really niceTheBrain is just really niceTheBrain is just really niceTheBrain is just really niceTheBrain is just really niceTheBrain is just really niceTheBrain is just really nice
I remember that somewhere out there on the net said that the slash at end prevents robots for getting in

If you omit the slash, robots will be able to crawl the directory just ignoring whatever file belonging to this directory is listed in your root robots.txt.
TheBrain is offline  
Add Post to del.icio.us
Reply With Quote
Old 11-12-2006, 08:26 PM   #9 (permalink)
Contributing Member
 
livePR.ezer.com's Avatar
 
Join Date: 10-14-06
Location: LivePR.ezer.com
Posts: 461
iTrader: 0 / 0%
Latest Blog:
None

livePR.ezer.com is on the right pathlivePR.ezer.com is on the right path
For your case

Maybe you can list all three together ...

User-agent: *
Disallow: /contact
Disallow: /contact.php
Disallow: /contact/
__________________
Live PageRank PR checker history of PR, backlinks | SeoSerp.com 58 countries Top 1000 SERPs checker
Google SERPs checker 37+ countries google search ranking results tool
livePR.ezer.com is offline  
Add Post to del.icio.us
Reply With Quote
Go Back   Webmaster Forum > Marketing Forums > SEO Forum

Reply

« Previous Thread | Next Thread »


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Quick first question... CommandD SEO Forum 3 08-18-2007 11:43 AM
Robots.txt Question Calisonder SEO Forum 3 05-04-2007 07:55 PM
quick css question gnznroses Coding Forum 10 02-14-2007 09:06 AM
Robots.txt question Thanol SEO Forum 21 06-01-2006 05:08 AM


Sponsor Links
Get exposure! Get exposure! Find Scripts Web Hosting Directory Get exposure! SEO Blog


Site Navigation: v7n Home .::. Graphics .::. Scripts .::. Web Designers .::. SEO Consultants .::. Web Hosting

Partners:V7N Web Directory .::. V7N Technology Blog .::. Host Ratings@V7N .::. V7N Affiliate Program .::. Advertise

All times are GMT -7. The time now is 12:28 PM.
© Copyright 2008 V7 Inc

Click Here
Contact Us - Internet Marketing - Top