Quote:
|
Originally Posted by unrealfragmaster
? My robots.txt is a text file. Its a file made in notepad and saved as robots.txt.
I dunno why it is showing that in the source
I reuploaded anyway and it is fine now
|
It's actually only half a robots.txt:
Quote:
#
# psp-forum: robots.txt
#
#
#
User-agent: *
|
basically says "all spiders, here's your instructions", except there are no instructions.
At a minimum, it should say:
which says, "all you spiders, listen up! Disallow nothing (i.e., spider everything".
Of course, if that's how you want to instruct the bots, you don't really need a robots.txt file at all.