[mythtv-users] visit from googlebot

[chris at cpr.homelinux.net]

On Tue, Oct 03, 2006 at 09:05:29AM -0400, Peter Watkins wrote:
> On Mon, Oct 02, 2006 at 01:36:43PM -0400, Michael T. Dean wrote:
> > So, the best thing to do is keep the Google bot off your website.
> No doubt.

Don't the spiders just follow links?  Someone posted urls Google 
had found for MythWeb installations running on non-standard ports, 
but how do you suppose Google found them?  It's probably because 
people put MythWeb on a non-standard port and then put a convenient 
link to the non-standard port from their server's normal pages.

The best solution is to use .htaccess to require authentication 
and/or use deny/allow rules to limit MythWeb usage to known IP 
addresses, but if you can't do that then the least you could do is 
NOT put a link to your MythWeb installation out there where the 
bots can find it....

FWIW, if you're going to use .htaccess and require passwords, a 
good follow-up would be to install fail2ban, which is a daemon 
which watches the logs for authorization failures and firewalls 
offending IP addresses.  That way you keep out the bots and also 
block malicious users.