Webmaster Forum


Go Back   Webmaster Forum > Marketing Forums > SEO Forum
Coincidence? Coincidence?
Register FAQ Members List Calendar Search Today's Posts Mark Forums Read

SEO Forum Search engine optimization discussions.

Directory Submission Service   ClickBooth Network   V7N Directory

Reply
 
LinkBack Thread Tools Display Modes
Old 04-16-2007, 06:26 PM   #1 (permalink)
Contributing Member
 
Comenius's Avatar
 
Join Date: 04-02-07
Location: San Francisco
Posts: 255
iTrader: 0 / 0%
Comenius is liked by somebodyComenius is liked by somebodyComenius is liked by somebodyComenius is liked by somebodyComenius is liked by somebody
Coincidence?
For one of my new sites, I was getting a bit frustrated as my good friend Yahoo Slurp didn't seem to want to crawl all of my pages. It just stuck to the same handful and would hit them very slowly every day or so.

Reading the other post in this forum about the ability to use the Sitemap: line in your robots.txt, and knowing that Yahoo was among the supports of this format, I decided to add my sitemap URL to it last night. Bear in mind I has submitted the exact same site map URL to Yahoo a couple weeks ago.

Checked my site logs again this afternoon and Mr. Slurp is now trucking through a lot more of my pages, and has been back multiple times today already.

Coincidence?
Comenius is offline  
Add Post to del.icio.us
Reply With Quote
Sponsored Links
SEO Hosting by HostGator  Advertise Here  Buy Blog Links
Old 04-16-2007, 06:43 PM   #2 (permalink)
gio
Contributing Member
 
gio's Avatar
 
Join Date: 06-06-06
Location: LinkChannels.com
Posts: 152
iTrader: 1 / 100%
gio is a jewel in the roughgio is a jewel in the roughgio is a jewel in the roughgio is a jewel in the roughgio is a jewel in the roughgio is a jewel in the rough
it could be. but then again it was really yahoo slurp's time to crawl your site.
gio is offline  
Add Post to del.icio.us
Reply With Quote
Old 04-16-2007, 08:16 PM   #3 (permalink)
Inactive
 
Join Date: 01-21-04
Posts: 779
iTrader: 0 / 0%
peter_d is a highly respected web propeter_d is a highly respected web propeter_d is a highly respected web propeter_d is a highly respected web propeter_d is a highly respected web propeter_d is a highly respected web propeter_d is a highly respected web propeter_d is a highly respected web propeter_d is a highly respected web propeter_d is a highly respected web propeter_d is a highly respected web pro
Quote:
Checked my site logs again this afternoon and Mr. Slurp is now trucking through a lot more of my pages, and has been back multiple times today already
Could be, but the result is desirable regardless, eh.
peter_d is offline  
Add Post to del.icio.us
Reply With Quote
Old 04-17-2007, 02:12 AM   #4 (permalink)
Senior Member
 
Join Date: 02-03-07
Posts: 392
iTrader: 0 / 0%
Latest Blog:
Matlock

Janna122003 is liked by many
Quote:
Originally Posted by Comenius View Post
For one of my new sites, I was getting a bit frustrated as my good friend Yahoo Slurp didn't seem to want to crawl all of my pages. It just stuck to the same handful and would hit them very slowly every day or so.

Reading the other post in this forum about the ability to use the Sitemap: line in your robots.txt, and knowing that Yahoo was among the supports of this format, I decided to add my sitemap URL to it last night. Bear in mind I has submitted the exact same site map URL to Yahoo a couple weeks ago.

Checked my site logs again this afternoon and Mr. Slurp is now trucking through a lot more of my pages, and has been back multiple times today already.

Coincidence?
Maybe your right it's just a coincidence.
Janna122003 is offline  
Add Post to del.icio.us
Reply With Quote
Old 04-17-2007, 04:10 AM   #5 (permalink)
Contributing Member
 
Join Date: 04-02-07
Posts: 186
iTrader: 0 / 0%
Mark07 is liked by somebodyMark07 is liked by somebodyMark07 is liked by somebody
Yeah it is a mere coincidence and I guess your sitemap must have been stuck somewhere earlier and just got a push from the new one you submitted.
Mark07 is offline  
Add Post to del.icio.us
Reply With Quote
Old 04-17-2007, 08:12 AM   #6 (permalink)
Inactive
 
Join Date: 02-13-07
Posts: 7
iTrader: 0 / 0%
Latest Blog:
None

AgentD is liked by many
Maybe not.

Maybe using the robots.txt way of forwarding their bots to your sitemap is now their "preferred" way...

Why not remove the sitemaps line from your robots.txt file as an experiment and report back to us?
AgentD is offline  
Add Post to del.icio.us
Reply With Quote
Old 04-18-2007, 04:00 PM   #7 (permalink)
Contributing Member
 
Comenius's Avatar
 
Join Date: 04-02-07
Location: San Francisco
Posts: 255
iTrader: 0 / 0%
Comenius is liked by somebodyComenius is liked by somebodyComenius is liked by somebodyComenius is liked by somebodyComenius is liked by somebody
Well I'm happy to report that my friend Mr. Slurp is still merrily breezing through my new site.

Seems like he's taking forever to get through the whole thing, but at least he's no longer a stranger!
Comenius is offline  
Add Post to del.icio.us
Reply With Quote
Go Back   Webmaster Forum > Marketing Forums > SEO Forum

Reply

« Previous Thread | Next Thread »


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Sponsor Links
Get exposure! Get exposure! Find Scripts Web Hosting Directory Get exposure! SEO Blog


Site Navigation: v7n Home .::. Graphics .::. Scripts .::. Web Designers .::. SEO Consultants .::. Web Hosting

Partners:V7N Web Directory .::. V7N Technology Blog .::. Host Ratings@V7N .::. V7N Affiliate Program .::. Advertise

All times are GMT -7. The time now is 02:53 PM.
© Copyright 2008 V7 Inc

Contact Us - Internet Marketing - Top