Revision 220034

Fixes for NUTCH-66, and some cleanup:

* apply the more lenient CookiePolicy. This fixes the problem with poorly
  formatted cookies being rejected.

* remove the while() loop, which was required only to handle retries related
  to authentication. commons-httpclient can handle this internally; and the
  present code was broken anyway. This is not a solution yet - the
  process of loading and presenting credentials needs to be developed further.

* cleaned up deprecation warnings by setting parameters at the correct level
  and in the new preferred way.

* correctly set the maximum number of threads.

* add MIME magic handling.