how to stop crawling of https:// urls from google

[mitra]

While checking through the indexed pages of my site, I found lots of
urls with https:// are indexed by Google. It is creating content
duplicity as I found two cached version of the same page, one with http://
and another with https://. I checked for the reason and discovered
that there are some links present in my site with https:// . I can't
stop posting of such urls as I don't have any control over my
visitors.

So my questions are:

1) How do I stop google from crawling the https:// urls?
2) How do I remove the urls that are already indexed with https://?

Please help me with your valuable suggestions asap. I'm in need of it.

[WorldwideTrading]

This is quite a simple one. What you need to do is offer up 2 different versions of robots.txt depending if http or https has been used.

I can show you a live example from my website:-

Notice the following pages differ:
http://worldwidetrading.co.uk/robots.txt
https://worldwidetrading.co.uk/robots.txt

Why do they do this you ask?
Because I wanted to block googlebot from crawling the same info via http and https.


How did I do this?
Its actually quite simple. You create a second robots.txt file, I called mine robots_ssl.txt and I added entries to it to block all content.

Then add the following lines to your .htaccess file (in the root of your webhosting).
RewriteCond %{SERVER_PORT} ^443$
RewriteRule ^robots.txt$ robots_ssl.txt

If you dont have an .htaccess file, create a new one - be sure to put these 2 lines at the top of it:
Options +FollowSymLinks
RewriteEngine on

I hope this info is useful to you.

Cheers,


Gareth

[Eire-Web Design]

and add noindex,nofollow meta tag into the header of your ssl pages as well.

[mitra]

Thank you WorldwideTrading. You made it so simple for me. http://www.seoworkers.com/seo-articl...and-https.html this link was great for me. Implemented the redirection of robots_ssl.txt through .htaccess. It worked fine for me.

[mitra]

How do I remove the urls that are already indexed with https://? I heard google can only remove the 404 pages. I can't make those pages 404 as those pages are coming from the same section as normal pages.

[Rankenstein]

Hi Mitra,

Originally Posted by Eire-Web Design

and add noindex,nofollow meta tag into the header of your ssl pages as well.

Eire said it. Put this in the head tags of pages you don't want indexed:

<meta name="robots" content="noindex, nofollow">

That'll do it.

[mitra]

Rankenstein, I've already blocked crawling of https://version. The only point is how to remove the existing https:// urls from google index. I have with google webmaster tool. But we can remove only http:// versions there. what to do with the https:// version

[Rankenstein]

It is just a question of time - they will be removed the next time Googlebot comes round and finds itself blocked from the pages.

[Rankenstein]

I'm surprised that Google Webmaster tools can't remove https though!

[coolguy27]

You can use disallow on robot.txt OR METAS

put your site that you want to block from crawling..

robots.txt Code:

User-agent: *
Disallow: /


Meta Code:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

[mitra]

coolguy27 i've done that already. thanks for your help it was my mistake. Google webmaster tool can remove https:// urls as well. I've already added with http:// and now I'm going to add http://. After adding http://, if I remove entire site from removal tool, only the https:// url will be deleted. It won't affect the normal urls with http:// . I'm bit nervous.

[poksa]

Originally Posted by coolguy27

You can use disallow on robot.txt OR METAS

put your site that you want to block from crawling..

robots.txt Code:

User-agent: *
Disallow: /


Meta Code:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

Nice Idea Coolguy... but how about the indexed pages, is there a way to removed that?

[WorldwideTrading]

Originally Posted by Rankenstein

Hi Mitra,



Eire said it. Put this in the head tags of pages you don't want indexed:

<meta name="robots" content="noindex, nofollow">

That'll do it.

Actually NO IT WONT! The problem occurs when google indexes the same page twice, once under http and once via https. Your sugestion would prevent google indexing the page altogether which is uhmm wrong.

[WorldwideTrading]

Originally Posted by coolguy27

You can use disallow on robot.txt OR METAS

put your site that you want to block from crawling..

robots.txt Code:

User-agent: *
Disallow: /


Meta Code:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

This also will not work. As by design only a single robots.txt file is used per domain whether it is accessed via http and https. Your suggestion will not acomplish what was asked! What you have to do is offer up different robots.txt files dependant on if http or https has been used. This simplest way of doing this is to use an .htaccess rewrite as I suggested above.

[mitra]

I've submitted 63 https://urls in google webmaster url removal tool . It has been 24 hours since I posted those urls. How long it will take to remove those urls? The status is showing as pending. Analyze robots.txt section shows that the robots.txt last downloaded 17 hours ago.

[Rankenstein]

Originally Posted by WorldwideTrading

Actually NO IT WONT! The problem occurs when google indexes the same page twice, once under http and once via https. Your sugestion would prevent google indexing the page altogether which is uhmm wrong.

Doh! Yeah, you're right.

Put the robots text on the secure port instead. That's Google's official advice, IIRC.

[sleepyhead]

content="index, nofollow"

[Rankenstein]

The first answer by Worldwide Trading was spot on. Perfect example of me not reading the thread properly. Ignore anything after the second post. Thread solved and we're all too dumb to know it. 10/10 for Gareth.

[VirtualHoney]

If i were you i would put a secondary robots.txt file that has a nofollow on https:// and for the http:// file i would put a follow robots.txt file

[EnomSoft.com]

use Rebot.txt