[mythtv-users] visit from googlebot

[Stef Coene]

On Saturday 30 September 2006 22:25, Matt Emmott wrote:

> What happened? Did everything get deleted? Maybe mythweb should
> include a robots.txt to keep this from happeneing?
Some lines from the access.log:
66.249.66.99 - - [30/Sep/2006:21:51:07 
+0200] "GET /mythweb/tv/recorded?delete=yes&chanid=1007&starttime=1158868800 
HTTP/1.1" 302 27 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)"
66.249.66.99 - - [30/Sep/2006:21:51:29 
+0200] "GET /mythweb/tv/recorded?delete=yes&chanid=1050&starttime=1159277880 
HTTP/1.1" 302 27 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)"
66.249.66.99 - - [30/Sep/2006:21:51:51 
+0200] "GET /mythweb/tv/recorded?delete=yes&chanid=1015&starttime=1159595580&forget_old 
HTTP/1.1" 302 27 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)"
66.249.66.99 - - [30/Sep/2006:21:52:13 
+0200] "GET /mythweb/tv/recorded?delete=yes&chanid=1023&starttime=1158605880&forget_old 
HTTP/1.1" 302 27 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; 
+http://www.google.com/bot.html)"

Once your site is known, the googlebot just visits every link.  The only 
solution is to password protect this.  You can make a robots.txt and it's not 
a bad idea to include one with mythweb, but I know some bots will ignore 
this.
So, for everybody with a public mythweb, put a password on it !!!!!!!
Searching google with the right keywords and you will find some more 
unprotected mythweb installations.


Stef