Pinpoint - a log context indexing, searching, and analysis tool =============================================================== Author: Paul Smith (psmith at apache.org) To build, you will need: * Java 5+ * Maven 2.0.x Step 1 - Install JMX into your Maven repository =============================================== Pinpoint relies on the Sun JMX jars which are not available in any 'official' Maven repository (at least none that I know of), so you will be forced to perform this manual step first. Painful I know, but it's not my fault, Sun requires you to physically accept a license to get access to the JARs. Go here: http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/download.jsp and download: JMX 1.2.1 Reference Implementation and JMX Remote API 1.0.1_04 Reference Implementation (select the 'accept' radio box and then click on the EXTREMELY obscure "down arrow" icon) Unpack the zips into a directory. Now 'mvn install' the jmxri.jar, jmxtools.jar into your local mvn repository. Assuming you unzip'd the 2 downloads into a 'jmx' sub-folder, this would be: mvn install:install-file -DgroupId=com.sun.jmx -DartifactId=jmxri \ -Dversion=1.2.1 -Dpackaging=jar -Dfile=jmx/jmx-1_2_1-bin/lib/jmxri.jar mvn install:install-file -DgroupId=com.sun.jdmk -DartifactId=jmxtools \ -Dversion=1.2.1 -Dpackaging=jar -Dfile=jmx/jmx-1_2_1-bin/lib/jmxtools.jar Step 2 - Note use of log4j SNAPSHOTS ========================= This project makes use of 2 log4j sub-projects that are not officially released. Maven Snapshot artifacts for them have been made available via: http://people.apache.org/www/people.apache.org/builds/logging/repo/ using log4j-component and log4j-receivers sub-projects. Nothing for you to do here except take note. Step 3 - Build Pinpoint (finally, thanks for your patience! :) ) ================================================================ From the top level directory you have extracted/checked-out Pinpoint: mvn package Building with Eclipse ===================== One issue I ran into with the Eclipse Maven integration plugin, is that it does not treat the json-lib dependency the same way Maven does, because of the use of the 'classifier' option in the dependency. If you have trouble getting it to build and it complains that it cannot find the json-lib dependency, even though from the command line it's ok, then try using symlinks, or manually copying/renaming the files in your maven repository. For example, if it complains it can't find: /Users/paulsmith/.m2/repository/net/sf/json-lib/json-lib-jdk15/2.2.1/json-lib-jdk15-2.2.1.jar I had to symlink the directory json-lib-jdk15 -> json-lib And then symlink the jar: json-lib-jdk15-2.2.1.jar -> json-lib-2.2.1-jdk15.jar Annoying! Now What? Using Pinpoint ======================== Basic Concepts ============== Pinpoint Home - A root directory on the local computer where all Pinpoint-related 'schtuff' goes. If not explicitly specified through command line options, then the a sub-directory under the users home directory is chosen (~/.pinpoint). Pinpoint Context - a Context is a logical grouping of indexed events. In most cases your context is probably going to represent a days' worth of logging events, each LoggingEvent observed by Pinpoint is routed to a particular Context store based on a selector. The default selector makes sure all LoggingEvents with the same day timestamp live together. This usually makes the most sense. Running Pinpoint Service to soak events from a production environment ===================================================================== The Pinpoint Service layer is designed to run as a service in a production environment and act as a LoggingEvent Clearing House. Many applications inside a production network could broadcast their events to a central Pinpoint Service instance aggregating events together for later searching and analysis. You run the Service layer as follows: sh pinpoint-service/target/appassembler/bin/pinpoint-service This starts up the Service layer with some defaults: * Starts a SocketReceiver listening on port 4560 Your applications in your environment can have their log4j configuration amended to use a matching SocketAppender pointing to the host running Pinpoint Service to stream LoggingEvents to it. Multiple applications can be run simultaneously to connect to a single Pinpoint Service instance. Each LoggingEvent will be tagged with the remote host details the events come from, so you can easily determine where the events are sourced from while analysing. You can also configure Pinpoint Service to automatically connect to known remote applications that are configured with a SocketHubAppender as follows: sh pinpoint-service/target/appassembler/bin/pinpoint-service --socketHub Once the Service layer is initialized and running, you can monitor it via JMX or you can simply browse to a simple web-server that is embedded within Pinpoint Service to show performance graphs. For example, if you have Pinpoint Service layer running on your local machine, simply browse to: http://localhost:8182/ A simple(very) webpage is displayed exposing graphs updated periodically showing the rate of LoggingEvents received. Because applications can do a LOT of logging, and because Pinpoint indexes these events & indexing can be computationally expensive, you might think that Pinpoint could slow down the sender application. This would be _bad_ in a production environment. However Pinpoint Service has been designed with this in mind and buffers the events locally using an internal JMS instance (using ActiveMQ), with an asynchronous process used to index the events. LoggingEvents should therefore be received as fast as possible, but the indexing process may be behind. The Performance graphs show the back log; allowing you to see how far behind the sending application event stream. This may be important when performing searches on a local Pinpoint Context that might not be up2date as yet. Importing Existing log4j log files into Pinpoint ================================================ If your logging environment is alreday using log files, you can import these into Pinpoint by using the Pinpoint Importer sub-project. This configures an internal instance of LogFilePatternReceiver class and you can specify the expression syntax to match the format. Unfortunately the syntax for LogFilePatternReceiver is not the same as log4j's PatternLayout class and it can be a bit fiddly to work it out. See the javadocs for that LogFilePatternReciever for more info or feel free to post to log4j-user@logging.apache.org to ask for help. Note: Importing LoggingEvent's this way may not bring across the complete richness of information available in a raw LoggingEvent object. The best approach is to use the Pinpoint Service module to soak complete LoggingEvent objects from a socket-based receiver. See section above. Once you have worked out your import pattern, you can run the Pinpoint Importer as follows (assumes you've run 'mvn package' and are currently in the top-level Pinpoint source directory)) sh pinpoint-importer/target/appassembler/bin/pinpoint-importer -h /tmp/pinpoint-home -p "[TIMESTAMP LEVEL ][LOGGER][THREAD]* MESSAGE" -r -c testlog /tmp/somelog.log This creates a new Pinpoint Home in /tmp/pinpoint-home and creates a context directory called 'ctx.testlog' using the contents of the log file 'somelog.log' based on the pattern conversion. OK, so what can I do with this Context? ======================================= Once a Pinpoint Context has been created, you can search and analyze the data stored within it. There is currently only a basic CLI working, but a Web-based, and Swing-based GUI are planned to make this vastly easier. To perform queries on this context: sh pinpoint-cli/target/appassembler/bin/pinpoint -h /temp/pinpoint-home/ -c testlog -s This uses the search option of Pinpoint, and provides you with command prompt: pinpoint> You can type 'search ' eg. pinpoint>search level:info message:foo .... The pinpoint-cli module has command completion like a bash shell, so try it out. Running/Debugging Pinpoint Web tier =================================== You can run the Pinpoint Web tier web application from within an IDEA such as Eclipse by using the Maven Jetty plugin. This requires a bit of pre-configuration in your Eclipse environment. There are screenshots provided in the docs/ConfiguringEclipse sub-directory that outline the steps to setup a run configuration. Steps: 1) Run->Run configurations 2) Create a new Java Application 3) Configure it following the screenshots (docs/ConfiguringEclipse/Eclipse-Run-1-Main.png, Eclipse-Run-2-Arguments.png,Eclipse-Run-3-Classpath.png) NOTE: Examples are based on Maven installed via Darwin ports. You should locate the relevant files from your local Maven installation. One limitation via this method is that the jetty plugin appears (to me at least) to require the Pinpoint sub-projects to be installed into the local repository _first_, even though you can make _some_ code changes on the fly, not all will be seamless. If you get runtime 'compilation' errors, try 'mvn install' from the top Pinpoint directory and re-launch. Once started, you can access the main page via: http://localhost:8080/pinpoint-web/searchContext.form FAQ === * OutOfMemoryError - You'll need to provide a different JVM Memory setting. The mavenised launcher scripts look for the standard JAVA_OPTS environment variable. So something of the order of: export JAVA_OPTS="-Xmx512m -Xms256m" before launching any Pinpoint applications will use those options So what's ThePlan(tm)? ====================== The core components are working well, but the GUI/CLI is extremely raw at this stage and not ready for general use. My current plans are to: * Focus on Web/Spring GUI implementation - the CLI does not appear to be as practical for general analysis as these mechanisms. It would help if I knew Spring MVC a lot better than I do. * Improve the Reporting module by allowing Regular expression patterns to spit out Scatter Plot Graphs so that you can visualize LoggingEvent sequences and distributions better. * The LoggingEvent serialization file used ends up consuming a lot of space. Usually 3x bigger than a standard log file text file format would use up. Be nice to be more efficient in this area.