org.apache.nutch.indexer.geoip

Class GeoIPIndexingFilter

java.lang.Object

org.apache.nutch.indexer.geoip.GeoIPIndexingFilter

All Implemented Interfaces:

Configurable, IndexingFilter, Pluggable

public class GeoIPIndexingFilter
extends Object
implements IndexingFilter

This plugin implements an indexing filter which takes advantage of the GeoIP2-java API.

The third party library distribution provides an API for the GeoIP2 Precision web services and databases. The API also works with the free GeoLite2 databases.

Depending on the service level agreement, you have with the GeoIP service provider, the plugin can add a number of the following fields to the index data model:

1. Continent
2. Country
3. Regional Subdivision
4. City
5. Postal Code
6. Latitude/Longitude
7. ISP/Organization
8. AS Number
9. Confidence Factors
10. Radius
11. User Type

Some of the services are documented at the GeoIP2 Precision Services webpage where more information can be obtained.

You should also consult the following three properties in nutch-site.xml

  <!-- index-geoip plugin properties -->
 <property>
   <name>index.geoip.usage</name>
   <value>insightsService</value>
   <description>
   A string representing the information source to be used for GeoIP information
   association. Either enter 'cityDatabase', 'connectionTypeDatabase', 
   'domainDatabase', 'ispDatabase' or 'insightsService'. If you wish to use any one of the 
   Database options, you should make one of GeoIP2-City.mmdb, GeoIP2-Connection-Type.mmdb, 
   GeoIP2-Domain.mmdb or GeoIP2-ISP.mmdb files respectively available on the Hadoop classpath 
   and available at runtime. This can be achieved by adding it to $NUTCH_HOME/conf
   </description>
 </property>
 
 <property>
   <name>index.geoip.userid</name>
   <value></value>
   <description>
   The userId associated with the GeoIP2 Precision Services account.
   </description>
 </property>
 
 <property>
   <name>index.geoip.licensekey</name>
   <value></value>
   <description>
   The license key associated with the GeoIP2 Precision Services account.
   </description>
 </property>
 
 

Field Summary

Fields inherited from interface org.apache.nutch.indexer.IndexingFilter

X_POINT_ID

Constructor Summary

Constructors

Constructor and Description
GeoIPIndexingFilter() Default constructor for this plugin

Method Summary

Methods

Modifier and Type	Method and Description
NutchDocument	filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Adds fields or otherwise modifies the document that will be indexed for a parse.
Configuration	getConf()
void	setConf(Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GeoIPIndexingFilter

public GeoIPIndexingFilter()

Default constructor for this plugin

Method Detail

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

See Also:

Configurable.getConf()

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

See Also:

Configurable.setConf(org.apache.hadoop.conf.Configuration)

filter

public NutchDocument filter(NutchDocument doc,
                   Parse parse,
                   Text url,
                   CrawlDatum datum,
                   Inlinks inlinks)
                     throws IndexingException

Description copied from interface: IndexingFilter

Adds fields or otherwise modifies the document that will be indexed for a parse. Unwanted documents can be removed from indexing by returning a null value.

Specified by:

filter in interface IndexingFilter

Parameters:

doc - document instance for collecting fields

parse - parse data instance

url - page url

datum - crawl datum for the page (fetch datum from segment containing fetch status and fetch time)

inlinks - page inlinks

Returns:

modified (or a new) document instance, or null (meaning the document should be discarded)

Throws:

IndexingException

See Also:

IndexingFilter.filter(org.apache.nutch.indexer.NutchDocument, org.apache.nutch.parse.Parse, org.apache.hadoop.io.Text, org.apache.nutch.crawl.CrawlDatum, org.apache.nutch.crawl.Inlinks)