Need mod_security rules to prevent Googlebot from crawling one file

[vectro]

I need mod_security rules to prevent Googlebot from indexing any file named browse.php anywhere on the server, while still allowing Googlebot to access anything else. I figured mod_security will do the trick because it can recognize user-agents and set rules accordingly.

Any ideas?

[vectro]

I did some research on creating mod_security rules and figured this out myself. Here is a server-wide mod_security rule for the main Apache configuration which will keep Googlebot off of 1 particular file. This only applies if the file shows up in the root directory of a domain, but it applies to all domains on the server.

<LocationMatch "/file.php">
SecRule REQUEST_HEADERS:User-Agent "@pm Googlebot" "deny,status:403"
</LocationMatch>

Change file.php to the name of the file you want to protect. The part that says "Googlebot" can also be changed to any user-agent. It's a pattern match and not an explicit match. This means the full user-agent simply needs to include the word for the rule to apply.

[vectro]

Originally Posted by mountainman

SecRule REQUEST_HEADERS:User-Agent "compatible; Googlebot" "nolog,allow"
If you add this rule it allows to crawl or if you remove this that blocks Google

I'm not exactly referring to allowing/denying all Googlebot traffic. I was only thinking about blocking it from one file.

[seriesn]

Originally Posted by vectro

I'm not exactly referring to allowing/denying all Googlebot traffic. I was only thinking about blocking it from one file.

Assuming you are talking about a Website, where the noted "File" is available, Why not setup a robots.txt file?
Something like this?

User-agent: *
Disallow:
Disallow: /file.extension.

[vectro]

robots.txt only applies to one site. I use the mod_security rule so that it applies to the entire server. There are a lot of sites with this one particular file that tends to confuse Googlebot.

[emilyrose9]

Make all the crawler IPs collectively in /etc/csf/csf.allow to whitelist them. You can alos try check your audit log and find out which rules blocks Google and then adjust it.

[vectro]

Originally Posted by emilyrose9

Make all the crawler IPs collectively in /etc/csf/csf.allow to whitelist them.

CSF is firewall, not mod_security.

Originally Posted by emilyrose9

You can alos try check your audit log and find out which rules blocks Google and then adjust it.

That could work, but audit log can be enormous, so make sure to use a filter like grep in Linux.