ApacheCon NA 2010 Session

Scientific data curation and processing with Apache Tika

• Chris Mattman
• Fri, 05 November 2010 09:00
• No Materials Available
• Lucene

In the first part of the talk, I will provide detail on Apache Tika, a content analysis and detection toolkit from the ASF. Tika provides flexible interfaces in Java, from the command line, and interactive, GUI-based for exploring the information landscape. I'll cover Tika's major components including its MIME type detection system, its Parsing framework (that integrates several major third-party parsing libraries), its language identifer and its Metadata system. The second part of the talk will show how NASA processes 10s-100s of terabytes of scientific data in myriad formats (e.g., HDF4/HDF5, NetCDF 4, Grib) and how it harmonizes the data's associated metadata models (e.g., HDF-EOS, CF, FGDC, etc.) using Tika. The discussion will center on the architecture of NASA's processing systems for the Earth science Decadal Study missions including OCO and SMAP, and the important open source technologies like Tika that help to implement the architecture. At NASA, Tika is being used in concert with other Apache technologies including OODT (a grid technology used for science data processing currently incubating at Apache), Apache Lucene/Solr, and Apache SIS (another current Incubator project whose goal is to provide a computational library for geospatial data) to automate, virtualize and increase the efficiency of NASA's science data processing pipeline.