Webmaster Forum

Go Back   Webmaster Forum > Web Development > Coding Forum

Coding Forum Problems with your code? Discuss coding issues, including JavaScript, PHP & MySQL, HTML & CSS, Flash & ActionScript, and more.


Reply
 
Thread Tools Display Modes
Share |
  #1  
Old 08-07-2011, 02:42 AM
markusl markusl is offline
Junior Member
 
Join Date: 11-07-10
Posts: 7
iTrader: 0 / 0%
Block Bad Bots in httpd.conf

Hi,

I use this lines in my httpd.conf (apache2):
PHP Code:

<directory /var/www/mysite.com>
SetEnvIf User-Agent "^Ezooms" badUA 
SetEnvIf User
-Agent "^proximic" badUA
SetEnvIf User
-Agent "^discobot" badUA
SetEnvIf User
-Agent "^Netseer" badUA
SetEnvIf User
-Agent "^Nutraspace" badUA
SetEnvIf User
-Agent "^TalkTalk" badUA
SetEnvIf User
-Agent "^Nutch" badUA
SetEnvIf User
-Agent "^LexxeBot" badUA
SetEnvIf User
-Agent "^BlogPulseLive" badUA

Order Allow
,Deny
Allow from all
Deny from env
=badUA
</directory

But it seems it doesn't work. These bots aren't blocked.

Can anyone tell me how block these bots to crawl my site?

Last edited by snakeair; 08-27-2011 at 08:19 AM.
 
Reply With Quote

Advertisement

Advertisement

  #2  
Old 08-07-2011, 08:40 AM
nafirici nafirici is offline
Contributing Member
 
Join Date: 03-06-11
Posts: 67
iTrader: 0 / 0%
It looks like according to this site: http://evolt.org/node/15126/
that you to use SetEnvIfNoCase and move all of those badUA vars out of the <directory> tags and move them to the top of your config. I don't know much about this but you could try that.
 
Reply With Quote
  #3  
Old 08-08-2011, 11:37 AM
Rukbat's Avatar
Rukbat Rukbat is offline
Contributing Member
 
Join Date: 08-08-11
Location: Long Island, NY, USA
Posts: 294
iTrader: 0 / 0%
Instead of SetEnv, have you tried using Mod-Rewrite?
PHP Code:
RewriteEngine On
RewriteCond 
%{HTTP_USER_AGENT} ^Ezooms [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^proximic [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^discobot.*Webster[NC]

RewriteRule abuse.txt [L
#or whatever rule you want for them

Last edited by snakeair; 08-27-2011 at 08:19 AM.
 
Reply With Quote
  #4  
Old 08-22-2011, 01:14 PM
markusl markusl is offline
Junior Member
 
Join Date: 11-07-10
Posts: 7
iTrader: 0 / 0%
Well...

1. proximic can be blocked from robots.txt
2. discobot can be blocked from robots.txt

For the others i have no solution at this time.

@rukbat and @nafirici
I've tried your ideas but it's not working. Anyway, thank you for your help.

__________________
Wallpapers Resource
 
Reply With Quote
  #5  
Old 08-22-2011, 03:17 PM
Defrag Defrag is offline
Contributing Member
 
Join Date: 08-15-11
Posts: 50
iTrader: 0 / 0%
Well I haven't looked at all your bots, but just using this as an example:
Quote:
SetEnvIf User-Agent "^Ezooms" badUA
That would fail because the user agent string is:

Quote:
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
And your regex uses ^ which says it has to match "Ezooms" at the beginning, which doesn't seem like it would ever work. In other words, you'd want to do something like:

Code:
SetEnvIf User-Agent "Ezooms/" badUA
Granted, I don't even know if the SetEnvIf directive works, or how, but as far as your regex I thought I'd point that out.
 
Reply With Quote
Go Back   Webmaster Forum > Web Development > Coding Forum

Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Googlebot blocking in httpd.conf file with some exception... xitdude Google Forum 1 11-13-2012 09:14 AM
WTS: 2 Articles "How a Browser Connects" Object-Oriented Programming Block by Block shafiqamiami Content 0 12-19-2009 05:51 AM
Megan Fox Block Media Block Out Day gggorosin22 Forum Lobby 0 07-29-2009 10:25 PM
Submit: infrasec-conf.org kiss Directory Announcements & Promotions 5 08-21-2008 02:36 AM


V7N Network
Get exposure! V7N I Love Photography V7N SEO Blog V7N Directory


All times are GMT -7. The time now is 11:27 AM.
Powered by vBulletin
Copyright 2000-2014 Jelsoft Enterprises Limited.
Copyright © 2003 - 2018 VIX-WomensForum LLC