My pages arent getting crawled

ljastangs21 · 10-06-2008, 04:08 PM

Ok my rap site which allows users to post their own stuff isnt getting its pages crawled by any search engines. The only page that is being crawled is the homepage. I think it may have something to do with how the users submit the pages and not me but idk. So if anyone knows how to fix this please tell me.

zeruel · 10-06-2008, 04:39 PM

Go get your site an xml sitemap... Submit it to Google Webmaster Tools...

LailaScott · 10-06-2008, 09:24 PM

Quote:

Originally Posted by zeruel

Go get your site an xml sitemap... Submit it to Google Webmaster Tools...

no, that wouldn't be the solution. from what you said, i think the sub pages are user accounts and it looks like it do need a username and password. it's not getting crawled because sometimes, bots don't crawl private pages. your sub pages (which are the users accounts) are considered private because user name and passwords are required and that is why your home page is the only one getting crawled. but try to bookmark every user's page, it might get crawled. use stumbleupon, wait for at least a week. i hope it will work.

ljastangs21 · 10-06-2008, 09:32 PM

The pages are not private. If you are not logged in you can still view others pages and guests can post their own pages.

Quote:

Originally Posted by LailaScott

no, that wouldn't be the solution. from what you said, i think the sub pages are user accounts and it looks like it do need a username and password. it's not getting crawled because sometimes, bots don't crawl private pages. your sub pages (which are the users accounts) are considered private because user name and passwords are required and that is why your home page is the only one getting crawled. but try to bookmark every user's page, it might get crawled. use stumbleupon, wait for at least a week. i hope it will work.

global-seo · 10-06-2008, 10:57 PM

submitting sitemap to webmaster tool is one of the best way to crawl the sub pages of the site.. wait for 1-2 weeks

Quote:

Originally Posted by LailaScott

no, that wouldn't be the solution. from what you said, i think the sub pages are user accounts and it looks like it do need a username and password. it's not getting crawled because sometimes, bots don't crawl private pages. your sub pages (which are the users accounts) are considered private because user name and passwords are required and that is why your home page is the only one getting crawled. but try to bookmark every user's page, it might get crawled. use stumbleupon, wait for at least a week. i hope it will work.

lol.

.

alex27 · 10-06-2008, 11:15 PM

submit it to social bookmarking sites.

LailaScott · 10-13-2008, 12:24 AM

Quote:

Originally Posted by ljastangs21

The pages are not private. If you are not logged in you can still view others pages and guests can post their own pages.

did you check it in other search engine? if that so, social bookmarking is the only answer i believe that would help you.

cashgators · 10-06-2008, 09:24 PM

how long has your website been up? Sometimes it can take a very long time to get indexed and sometimes it almost seems instant.

C.Whyte · 10-06-2008, 11:22 PM

Just keep it up and give it time... the worst thing you can do is give up on a site. If a site is updated regularly and has strong content, it eventually will get indexed... you just can't rush these types of things.

MajorTom · 10-06-2008, 11:45 PM

If you posted the link to the site maybe some of us could help you better with the problem. It could be a million different things but I'd have to see the site to be able to give you any realistic advice.

ljastangs21 · 10-07-2008, 04:51 AM

Quote:

Originally Posted by MajorTom

If you posted the link to the site maybe some of us could help you better with the problem. It could be a million different things but I'd have to see the site to be able to give you any realistic advice.

I posted it to begin with but the mods took it out. Here it is my16bars.com

And to everyone who is telling me to social bookmark and all these other basic things please stop. Thanks for the help but im already aware of these very basic methods. I think my problem has something to do on my site.

grishamist · 10-07-2008, 12:16 AM

If you are on the confused stage right now on Google indexing, try to submit your pages on Digg. It's an alternative way to get indexed by Google easily.

mit · 10-07-2008, 12:28 AM

Try social bookmarking i.e. submitting your site to different social bookmarking sites like digg.com, stumbleupon.com and more...

jabo · 10-07-2008, 06:17 AM

Quote:

Originally Posted by mit

Try social bookmarking i.e. submitting your site to different social bookmarking sites like digg.com, stumbleupon.com and more...

very true, specially with digg. Everytime I would have a new page, I would digg it and it would be visited by Google quickly.

shakir · 10-07-2008, 09:56 AM

add site map and include all links . if do google will crawl with in a week

ppc-han · 10-07-2008, 02:05 PM

just keep on building backlink for that certain page and submit it to social bookmarking sites.

pinkfluffybunny · 10-10-2008, 06:12 AM

Google may not be spidering the site for many reasons

- The code is bad and the spiders can't get through
- the CMS is bad and has duplicate headers (no seo mod)
- the headers are called using java or xhtml
- your site is penalized for (add your own) violations
- your DNS is set up incorrectly
- links are bad (java or buried in crazy code)
- and on and on

I've never used a sitemap and never will - those peswky spiders will just have to crawl their way through.

post a link (hide it with h**p:// and I'll take a look)

ljastangs21 · 10-10-2008, 07:16 AM

my16bars . com

pinkfluffybunny · 10-10-2008, 07:46 AM

Hi,

duplicate headers - no unique meta tags - no description tag - no keyword tags

All say "My 16 Bars - Rap Battles - Underground Rap Lyrics"

as far as google is concerned you only have one page and it has no content

Build some static pages and link into your unspiderable site.

good luck.

kristeejo · 10-10-2008, 08:27 AM

good to see the pink bunny around again