Would someone please help, my SERPs are a mess.

[CheebaHawk]

Hey everyone, I feel like I am extremely lost right now...I have been having trouble with google and one of the SEO mods I installed on my Phpbb. I installed a static URL mod, using a Mod_rewrite. From my understanding the mod was supposed to change the dynamic urls ex.../forum/viewforum.php?F=11 to something like /forum/forum-11.html. Also it was supposed to change /forum/viewtopic.php?f=342 to /forum/about-342.html.

That part that I am confused about is that google keeps finding my pages like this /forum/post-342...I tried to dis-alow this style URL in my robots.txt because I wanted my pages to be indexed as /forum/about-324.html...Now, google has removed all of my SERPs that are formated with the "about" url and removed all of the SERPS with the "post-232.html" URLs?? Would someone please look at my .httaccess file and my robots.txt file below and tell me if there are any discrepancies. I would really appreciate it, the httaccess code is really foreign to me, and I think that there is something wrong with my setup...

htaccess...

RewriteEngine On 
RewriteRule ^sitemap.xml$ sitemap.php [L]
RewriteRule ^forum-([0-9]+).xml$ sitemap.php?fid=$1 [L]
RewriteRule ^forums.* index.php [L,NC]
RewriteRule ^post-([0-9]*).html&highlight=([a-zA-Z0-9]*) viewtopic.php?p=$1&highlight=$2 [L,NC]
RewriteRule ^post-([0-9]*).* viewtopic.php?p=$1 [L,NC]
RewriteRule ^view-poll([0-9]*)-([0-9]*)-([a-zA-Z]*).* viewtopic.php?t=$1&postdays=$2&postorder=$3&vote=viewresult [L,NC]
RewriteRule ^about([0-9]*).html&highlight=([a-zA-Z0-9]*) viewtopic.php?t=$1&highlight=$2 [L,NC]
RewriteRule ^about([0-9]*).html&view=newest viewtopic.php?t=$1&view=newest [L,NC]
RewriteRule ^about([0-9]*)-([0-9]*)-([a-zA-Z]*)-([0-9]*).* viewtopic.php?t=$1&postdays=$2&postorder=$3&start=$4 [L,NC]
RewriteRule ^about([0-9]*)-([0-9]*).* viewtopic.php?t=$1&start=$2 [L,NC]
RewriteRule ^about([0-9]*).* viewtopic.php?t=$1 [L,NC]
RewriteRule ^about([0-9]*).html viewtopic.php?t=$1&start=$2&postdays=$3&postorder=$4&highlight=$5 [L,NC]
RewriteRule ^mark-forum([0-9]*).html* viewforum.php?f=$1&mark=topics [L,NC]
RewriteRule ^updates-topic([0-9]*).html* viewtopic.php?t=$1&watch=topic [L,NC]
RewriteRule ^stop-updates-topic([0-9]*).html* viewtopic.php?t=$1&unwatch=topic [L,NC]
RewriteRule ^forum-([0-9]*).html viewforum.php?f=$1 [L,NC]
RewriteRule ^forum-([0-9]*).* viewforum.php?f=$1 [L,NC]
RewriteRule ^topic-([0-9]*)-([0-9]*)-([0-9]*).* viewforum.php?f=$1&topicdays=$2&start=$3 [L,NC]
RewriteRule ^ptopic([0-9]*).* viewtopic.php?t=$1&view=previous [L,NC]
RewriteRule ^ntopic([0-9]*).* viewtopic.php?t=$1&view=next [L,NC]

Robotx.txt...

user-agent: *
Disallow: /forum/post-*.html$ 
Disallow: /forum/updates-topic.html*$ 
Disallow: /forum/stop-updates-topic.html*$ 
Disallow: /forum/ptopic*.html$ 
Disallow: /forum/ntopic*.html$
Disallow: /forum/viewforum.php?t=*
Disallow: /forum/viewtopic.php?t=*
Disallow: /forum/admin/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/language/
Disallow: /forum/templates/
Disallow: /forum/common.php
Disallow: /forum/groupcp.php
Disallow: /forum/memberlist.php
Disallow: /forum/modcp.php
Disallow: /forum/posting.php
Disallow: /forum/profile.php
Disallow: /forum/privmsg.php
Disallow: /forum/viewonline.php
Disallow: /forum/faq.php
Disallow: /forum/updates-topic
Disallow: /forum/stop-updates-topic
Disallow: /forum/ptopic
Disallow: /forum/ntopic
Disallow: /post-

Thanks for the help in advance, I really appreciate it!

[BinkyMelnik]

CheeblaHawk:

It seems to me you're never getting an opportunity to rewrite the URLs because you're disallowing the spidering of /forum/post-*.html$. If the spider can't get there, how can it be rewritten?

I'm not sure you wanna disallow access to that directory.

[CheebaHawk]

Ok, cool, I knew I was doing something wrong. I was worried about duplicate content as I didn't want my site to be indexed twice with each URL style. So if I remove that line from the robots.txt it should be ok yeah?

P.S. Thanks for the reply...no one else seems to take the "stoner site" too seriously...I really appreciate the help.

P.P.S - I just went and looked over the SEO mod that I used for the static url rewrite...it said to add the following lines...which include the one we are worried about...

Disallow: forums/post-*.html$ 
Disallow: forums/updates-topic.html*$ 
Disallow: forums/stop-updates-topic.html*$ 
Disallow: forums/ptopic*.html$ 
Disallow: forums/ntopic*.html$

The rest of the robots.txt was from a generic suggestion from someone at the phpbb board, maybe the error resides there? I can see how the problem could be that line if the MOD instructed me to do it, yes no?

[BinkyMelnik]

CheeblaHawk:

I was worried about duplicate content as I didn't want my site to be indexed twice with each URL style

I went through this two years ago, and I have amnesia, so my details are fuzzy, and it's hurting my brain trying to remember all the details, but lemme see what I can pull out of my headpan for you:

The purpose behind the mod_rewrite is so that when the spider requests yoursite.com/forums/phpbb.php?topic.asp?TOPIC_ID=34521, mod_rewrite will turn it into yoursite.com/forums/about/weed-is-good.html (I'm making these up, obviously).

If the spider can't get in to do any reading, then no rewriting will take place. This is why your pages have disappeared from Google, I believe. There's no need to worry about your pages showing up twice (once with the funky URL, and once with the good one) because the URL will have been rewritten into the nice-looking one.

So no, I don't believe you wanna stop the spider via your robots.txt file. Where there's no access, there's no rewriting.

(I think. Like I said, I hurt my head, but common sense would seem to dictate that if something's off-limits, then it's not gonna be rewritten.)

Thanks for the reply...no one else seems to take the "stoner site" too seriously

We're all in this together, right? So long as folks aren't asking for help with child porn, terrorism, or sites devoted to Democrats*, they oughtta get it!

*I typed that with a smile.