Bots are at the moment crawling everything, including all opearations available via twiki other then reading pages. Attempting those actions (see RewriteRule list below) at the moment returns HTTP 401 Unathorized. By using mod_rewrite, we're now instead returning HTTP 403 Forbidden, which we think think should be a less resource expensive operation.

• suggestion 403 is returned immidiately while with 401 HTTP header www-Authenticate is sent, and only on return of incorrect header details is 401 returned. In other words, one step more then with 403.
• 403 as absolute restriction, IP based

The issue here is not just the cost of 403/401, but the cost of it under mod_rewrite.

  RewriteCond %{HTTP_USER_AGENT} Googlebot
  RewriteRule ^/rename|viewauth|edit|search|attach|rdiff|oops - [F,L]

-- ToniPrug - 18 Nov 2005
to top