-----------------
                          Content Detection
                          -----------------

~~ Licensed to the Apache Software Foundation (ASF) under one or more
~~ contributor license agreements.  See the NOTICE file distributed with
~~ this work for additional information regarding copyright ownership.
~~ The ASF licenses this file to You under the Apache License, Version 2.0
~~ (the "License"); you may not use this file except in compliance with
~~ the License.  You may obtain a copy of the License at
~~
~~     http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License.

Content Detection

   This page gives you information on how content and language detection
   works with Apache Tika, and how to tune the behaviour of Tika.

%{toc|section=1|fromDepth=1}

* {The Detector Interface}

  The
  {{{./api/org/apache/tika/detect/Detector.html}org.apache.tika.detect.Detector}}
  interface is the basis for most of the content type detection in Apache
  Tika. All the different ways of detecting content all implement the
  same common method:

---
MediaType detect(java.io.InputStream input,
                 Metadata metadata) throws java.io.IOException
---

   The <<<detect>>> method takes the stream to inspect, and a 
   <<<Metadata>>> object that holds any additional information on
   the content. The detector will return a 
   {{{./api/org/apache/tika/mime/MediaType.html}MediaType}} object describing
   its best guess as to the type of the file.

   In general, only two keys on the Metadata object are used by Detectors.
   These are <<<Metadata.RESOURCE_NAME_KEY>>> which should hold the name
   of the file (where known), and <<<Metadata.CONTENT_TYPE>>> which should
   hold the advertised content type of the file (eg from a webserver or
   a content repository).


* {Mime Magic Detection}

  By looking for special ("magic") patterns of bytes near the start of
  the file, it is often possible to detect the type of the file. For
  some file types, this is a simple process. For others, typically
  container based formats, the magic detection may not be enough. (More
  detail on detecting container formats below)

  Tika is able to make use of a a mime magic info file, in the 
  {{{http://www.freedesktop.org/standards/shared-mime-info}Freedesktop MIME-info}} 
  format to peform mime magic detection. (Note that Tika supports a few
  more match types than Freedesktop does)

  This is provided within Tika by
  {{{./api/org/apache/tika/detect/MagicDetector.html}org.apache.tika.detect.MagicDetector}}. It is most commonly access via
  {{{./api/org/apache/tika/mime/MimeTypes.html}org.apache.tika.mime.MimeTypes}},
  normally sourced from the <<<tika-mimetypes.xml>>> and <<<custom-mimetypes.xml>>>
  files. For more information on defining your own custom mimetypes, see
  {{{./parser_guide.html#Add_your_MIME-Type}the new parser guide}}.
   

* {Resource Name Based Detection}

  Where the name of the file is known, it is sometimes possible to guess 
  the file type from the name or extension. Within the 
  <<<tika-mimetypes.xml>>> file is a list of patterns which are used to
  identify the type from the filename.

  However, because files may be renamed, this method of detection is quick
  but not always as accurate.

  This is provided within Tika by
  {{{./api/org/apache/tika/detect/NameDetector.html}org.apache.tika.detect.NameDetector}}.


* {Known Content Type "Detection}

  Sometimes, the mime type for a file is already known, such as when
  downloading from a webserver, or when retrieving from a content store.
  This information can be used by detectors, such as
  {{{./api/org/apache/tika/mime/MimeTypes.html}org.apache.tika.mime.MimeTypes}},


* {The default Mime Types Detector}

  By default, the mime type detection in Tika is provided by
  {{{./api/org/apache/tika/mime/MimeTypes.html}org.apache.tika.mime.MimeTypes}}.
  This detector makes use of <<<tika-mimetypes.xml>>> to power
  magic based and filename based detection.

  Firstly, magic based detection is used on the start of the file.
  If the file is an XML file, then the start of the XML is processed
  to look for root elements. Next, if available, the filename 
  (from <<<Metadata.RESOURCE_NAME_KEY>>>) is
  then used to improve the detail of the detection, such as when magic
  detects a text file, and the filename hints it's really a CSV. Finally,
  if available, the supplied content type (from <<<Metadata.CONTENT_TYPE>>>)
  is used to further refine the type.


* {Container Aware Detection}

  Several common file formats are actually held within a common container
  format. One example is the PowerPoint .ppt and Word .doc formats, which
  are both held within an OLE2 container. Another is Apple iWork formats,
  which are actually a series of XML files within a Zip file.

  Using magic detection, it is easy to spot that a given file is an OLE2
  document, or a Zip file. Using magic detection alone, it is very difficult
  (and often impossible) to tell what kind of file lives inside the container.

  For some use cases, speed is important, so having a quick way to know the
  container type is sufficient. For other cases however, you don't mind 
  spending a bit of time (and memory!) processing the container to get a 
  more accurate answer on its contents. For these cases, the additional
  container aware detectors contained in the <<<Tika Parsers>>> jar should
  be used.

  Tika provides a wrapping detector in the form of 
  {{{./api/org/apache/tika/detect/DefaultDetector.html}org.apache.tika.detect.DefaultDetector}}.
  This uses the service loader to discover all available detectors, including
  any available container aware ones, and tries them in turn. For container
  aware detection, include the <<<Tika Parsers>>> jar and its dependencies
  in your project, then use DefaultDetector along with a <<<TikaInputStream>>>.

  Because these container detectors needs to read the whole file to open and
  inspect the container, they must be used with a 
  {{{./api/org/apache/tika/io/TikaInputStream.html}org.apache.tika.io.TikaInputStream}}.
  If called with a regular <<<InputStream>>>, then all work will be done
  by the default Mime Magic detection only.

  For more information on container formats and Tika, see
  {{{http://wiki.apache.org/tika/MetadataDiscussion}}}


* {The default Tika Detector}

  Just as with Parsers, Tika provides a special detector
  {{{./api/org/apache/tika/detect/DefaultDetector.html}org.apache.tika.detect.DefaultDetector}}
  which auto-detects (based on service files) the available detectors at 
  runtime, and tries these in turn to identify the file type.

  If only <<<Tika Core>>> is available, the Default Detector will work only
  with Mime Magic and Resource Name detection. However, if <<<Tika Parsers>>>
  (and its dependencies!) are available, additional detectors which known about
  containers (such as zip and ole2) will be used as appropriate, provided that
  detection is being performed with a
  {{{./api/org/apache/tika/io/TikaInputStream.html}org.apache.tika.io.TikaInputStream}}.
  Custom detectors can also be used as desired, they simply need to be listed
  in a service file much as is done for
  {{{./parser_guide.html#List_the_new_parser}custom parsers}}.


* {Ways of triggering Detection}

  The simplest way to detect is through the 
  {{{./api/org/apache/tika/Tika.html}Tika Facade class}}, which provides methods to
  detect based on
  {{{./api/org/apache/tika/Tika.html##detect(java.io.File)}File}},
  {{{./api/org/apache/tika/Tika.html##detect(java.io.InputStream)}InputStream}},
  {{{./api/org/apache/tika/Tika.html##detect(java.io.InputStream, java.lang.String)}InputStream and Filename}},
  {{{./api/org/apache/tika/Tika.html##detect(java.lang.String)}Filename}} or a few others.
  It works best with a File or 
  {{{./api/org/apache/tika/io/TikaInputStream.html}TikaInputStream}}.

  Alternately, detection can be performed on a specific Detector, or using
  <<<DefaultDetector>>> to have all available Detectors used. A typical pattern
  would be something like:

---
TikaConfig tika = new TikaConfig();

for (File f : myListOfFiles) {
   Metadata metadata = new Metadata();
   metadata.set(Metadata.RESOURCE_NAME_KEY, f.toString());
   String mimetype = tika.getDetector().detect(
        TikaInputStream.get(f), metadata);
   System.out.println("File " + f + " is " + mimetype);
}
for (InputStream is : myListOfStreams) {
   String mimetype = tika.getDetector().detect(
        TikaInputStream.get(is), new Metadata());
   System.out.println("Stream " + is + " is " + mimetype);
}
---

* {Language Detection}

  Tika is able to help identify the language of a piece of text, which
  is useful when extracting text from document formats which do not include
  language information in their metadata.

  The language detection is provided by
  {{{./api/org/apache/tika/language/LanguageIdentifier.html}org.apache.tika.language.LanguageIdentifier}}

* {More Examples}

  For more examples of Detection using Apache Tika, please take a look at
  the {{{./examples.html}Tika Examples page}}.