Olio/Java
Install and Setup Guide From Tarball

Overview

Java Olio is a macro-level benchmark consisting of the following components:

1. The web application

2. The main database

3. Distributed storage servers (NFS or MogileFS)

4. An optional caching server (memcached)
   

5. Storage metadata database (for MogileFS)

6. Geocoder emulator

7. Workload driver

If your primary interest is in setting up the application alone, you need items 1-3 above and they can all be setup on a single system. If on the other hand, you would like to drive load against the application, you will need at least 2 systems. At higher loads, you may need multiple systems. At a minimum, we need to separate the SUT (System Under Test) components and the non-SUT components to get valid results. The non-SUT components are the Geocoder emulator and the workload driver. It is best to connect the driver machine to the SUT machine on a local private network. This ensures that latencies measured do not include arbitrary delays.

For a horizontally scaled workload, or to measure the performance of the individual components, you can deploy the SUT components on separate physical or virtual machines.  Keep in mind though that the bulk of cpu is spent in the application server component.

In the following sections, we'll go over the steps needed to configure each component :

Setting up the Driver

Setting up the Database

Loading the Database

Setting up Caching

Setting up the Geocoder Emulator

Setting up the Application Server

Building the Web Application

Testing the Web Application

Starting a benchmark run

Prerequisite Software

• Java SE 6
• Apache Ant 
• Subversion:  (if you are going to be installing from the source repository)
• Faban a facility for developing and running benchmarks.
• Java EE 5 compliant application server - eg. GlassFish
• Java Persistence API (JPA) provider - eg. Eclipselink
• Database server - e.g. MySQL(minimum 5.0)
  

Extracting the Contents

•  webapp.war contains the web application. This application will be deployed on the webserver.
  
  
•  OlioDriver.jar contains the load generator/driver (which we typically refer to simply as driver). The driver is implemented using Faban, an open source benchmarking toolkit.
  
  
• geocoder.war contains the simple servlet used to emulate the geocoder.
  
  
• java_setup_kit.html is this document.
  
  
• release contains the release name. LICENSE and NOTICE contain the licenses.

• RELEASE-NOTES-java-0.2.txt contains the release notes for this release.


Setting up the Driver

1. See http://faban.sunsource.net/docs/guide/harness/install.html for Faban installation instructions. Note that faban needs to be installed on all the machines used for the benchmark.  Please also read the Getting Started Guide to get a high-level understanding of Faban terminology and how it works.  From now on, we will refer to the faban install directory as $FABAN_HOME.

2. Install the required Faban services :
   
• cd $FABAN_HOME
• cp samples/services/GlassfishService/build/GlassfishService.jar services  (if using Glassfish)

• cp samples/services/MysqlService/build/MysqlService.jar services
• cp samles/services/MemcachedService/build/MemcachedService.jar services  (if using Memcached)


3. Copy $OLIO_HOME/OlioDriver.jar to the $FABAN_HOME/benchmarks directory. Also copy the OlioDriver.jar file to the $FABAN_HOME/benchmarks directory on the web server system.


4. For the driver to work, you will need JDK 1.6. Set JAVA_HOME to the path of the JDK in the faban user's environment.
   


5. Start the faban master on the master driver machine :

• $FABAN_HOME/master/bin/startup.sh

6. Test that you can connect to the master by pointing your browser at http://<driver_machine>:9980.  You should see the Faban welcome page or if you have already deployed the driver, a page with the text "Welcome to Java Olio Workload"

7. Download MySQL Connector/J (JDBC Driver for MySQL) and install the jar in $FABAN_HOME/benchmarks/OlioDriver/lib.

8. To deploy the Faban driver, you can either copy $WORKLOAD/build/OlioDriver.jar to the $FABAN_HOME/benchmarks directory, or execute ant deploy.  Executing the deploy target will deploy the build jar file on the Faban master that you specified as the 'faban.url' in the build.properties file.  To use ant deploy you must have completed steps 4, 6 and 7 and have Faban up and running.

Setting Up the Database

1. If you plan to run MySQL on a separate machine, install MySQL on that system. We will refer to the MySQL installation directory as MYSQL_HOME.

2. Setup the mysql user/group and permissions for its directories:
   # groupadd mysql
   # useradd -d $MYSQL_HOME -g mysql -s /usr/bin/bash mysql
   # chown -R mysql:mysql $MYSQL_HOME
   

3. Create the database :
   # su - mysql
   $ cd bin
   $ ./mysql_install_db

4. A sample MySQL configuration file (my.cnf) is included in the $WEBAPP/etc. Please read the embedded comments and edit the file appropriately after copying it to /etc.   Note that the data directory that the innodb_data_home_dir variable refers to must be writable by the mysql user.
   

5. Start the mysql server. Substitute your own password for pwd (we typically use adminadmin)
   $ ./mysqld_safe --defaults-file=/etc/my.cnf &
   $ ./mysqladmin -u root password <your-password>
   $ ./mysqladmin -u root -h <machine-name> password <your-password>

6. Create the olio user and grant privileges :
   $ ./mysql -uroot -ppwd
   mysql> create user 'olio'@'%' identified by 'olio';
   mysql> grant all privileges on *.* to 'olio'@'%' identified by 'olio' with grant option;
   
   
   In some cases the wildcard '%' does not work reliably as a substitution for all hosts. You need to grant the privileges to 'olio'@'<hostname>' individually, where hostname are the names of the driver and apache systems.
   Eg. mysql> grant all privileges on *.* to 'olio'@'localhost' identified by 'olio' with grant option;
   

7. Create database
   mysql> create database bpwebapp;
   mysql> use bpwebapp;
   
   
8. Although it is possible to create the database schema automatically by JPA upon deployment or when the application server starts, it recommended to create the schema via the schema.sql file provided found in the $WEBAPP/etc/, $WORKLOAD/bin or $FABAN_HOME/benchmarks/OlioDriver/bin if you have already deployed the driver to the Faban master.
   
   mysql> \. <location>/schema.sql
   mysql> exit
    Now, if you login as the user olio, you should be able to see the database created by the root user.

Loading the Database

It is best to load the database manually the first time so that we can test the web application. However, while doing benchmark runs, the benchmark driver can be configured to automatically re-load the database before the run.

1. Login to the machine running the Faban master driver. Only this machine has the loader at this time.

2. Go to the directory containing the loader script:
   # cd $FABAN_HOME/benchmarks/OlioDriver/bin
3. Ensure the script has execute permissions. Faban takes care of this for the runs, but since we have not yet started the first run, we will need to change that ourselves:
   # chmod +x dbloader.sh
4. Run the loader script:
   # ./dbloader.sh <dbserver> <load_scale>
   For a fully loaded T2000 (8x1.2Ghz) we are currently running a scale of around 1000. You can start small with a SCALE of 50 or so for initial testing.

Setting up the Geocoder Emulator

The Geocoder Emulator is a simple Java EE application that you can deploy on Glassfish or Tomcat. It is typically run on a driver machine. The following steps describe how to install it.

For Glassfish:

1. Download and install Glassfish V2.1.1 or GlassFish v3 from https://glassfish.dev.java.net/public/downloadsindex.html.
   
2. Copy $OLIO_HOME/geocoder.war from  to $GLASSFISH_HOME/domains/domain1/autodeploy (domain1 is the default domain created with Glassfish)
3. Start Glassfish by using $GLASSFISH_HOME/bin/asadmin start-domain domain1.

1. Download and install Tomcat (http://tomcat.apache.org) on the driver machine. The install directory doesn't matter – we will refer to it as $TOMCAT_HOME.

2. Copy $OLIO_HOME/geocoder.war to $TOMCAT_HOME/webapps.

3. Start Tomcat using $TOMCAT_HOME/bin/startup.sh.

Setting up the Application Server

1. Java EE compliant application server (eg. GlassFish v2.1.1 or GlassFish v3)
   
2. JPA provider  ( eg. Eclipselink)

 Glassfish Installation and Configuration

1. Install Glassfish application server V2.1.1 or GlassFish v3 from https://glassfish.dev.java.net/public/downloadsindex.html.
   The directory where Glassfish has been installed will now be referred to as $GLASSFISH_HOME.  

2. Download the JDBC driver for mysql from http://dev.mysql.com/downloads/connector/j/5.1.html.   Extract the mysql-connector-java-5.1.X-bin.jar and drop this is $GLASSFISH_HOME/domains/domain1/lib/ext directory (or whatever you have named your domain).

3. The benchmark utilizes JPA (Java Persistence API) and has been benchmarked with the EclipseLink implementation.  GlassFish v2.1.1 uses TopLink as the default JPA implementation so you will have to download the eclipselink.jar, and  put this in $GLASSFISH_HOME/lib directory. If you are using GlassFish v3, you need not download Eclipselink since the default JPA implementation is Eclipselink v2.0.
   
   
4. For ease of use, a sample script, config_glassfish.sh, is provided in the $WEBAPP/etc directory.  You can use this CLI script to create the JDBC pool and resources as well as configure basic JVM environment variables necessary to run the application in the event you do not want to edit the web.xml.  Please look at the script carefully before running it since you have to modify the values to fit your setup.
   


5. Instructions on how to use the Glassfish admin interface is https://glassfish.dev.java.net/javaee5/admin-gui/admin-gui.html#Usage.  To edit the classpath prefix, please consult  https://glassfish.dev.java.net/javaee5/docs/AG/ablwj.html#ablwm.  
   Note:  To use the admin interface, you will have to start Glassfish.  To do so,  use $GLASSFISH_HOME/bin/asadmin start-domain domain1 (or whatever your domain name is if you are using the default domain1).
              To stop Glassfish, use $GLASSFISH_HOME/bin/asadmin stop-domain domain1.

6. Create JDBC connection pool and resource.  Please consult https://glassfish.dev.java.net/javaee5/docs/AG/ablih.html for details.  MM MySQL Type 4 Driver (Non-XA)
   The JAR file for the MySQL driver is mysql-connector-java-version-bin.jar, for example, mysql-connector-java-5.1.6-bin.jar. Configure the connection pool using the following settings:
   

   • Name: Use this name when you configure the JDBC resource.  eg. BPwebappPool
   • Resource Type: Specify the appropriate value, eg. javax.sql.DataSource
   • Database Vendor: mysql
   • DataSource Classname: Specify one of the following:
   • com.mysql.jdbc.jdbc2.optional.MysqlDataSource
   • Properties:
   • serverName - Specify the host name or IP address of the database server.
   • port - Specify the port number of the database server eg. 3306
   • user - Set as appropriate,  eg. olio
   •  password - Set as appropriate, eg. olio
   • databaseName - Set as appropriate, eg. bpwebapp
   • URL - If you are using global transactions, you can set this property instead of serverName, port, and databaseName, eg. jdbc:mysql://<dbhost>:3306/bpwebapp
   • Keep the rest of properties created.
   6.  Create the JDBC resource with the JNDI name jdbc/BPWebappDB pointing to new resource created above.

Configuring other Application Servers

1. Download the JDBC driver for mysql from http://dev.mysql.com/downloads/connector/j/5.1.html.  Extract the mysql-connector-java-5.1.X-bin.jar and put this your application server classpath (or external library directory)
2. Download a Java Persistence API (JPA) provider, eg. EclipseLink.  You will have to download the eclipselink.jar and put this in your application server classpath.
3. Create JDBC connection pool and resource for connecting to mySQL database.  Below are examples for the properties used for the connection pool:
       * Name: Use this name when you configure the JDBC resource.  eg. BPwebappPool
       * Resource Type: Specify the appropriate value, eg. javax.sql.DataSource
       * Database Vendor: mysql
       * DataSource Classname: Specify one of the following:
             o com.mysql.jdbc.jdbc2.optional.MysqlDataSource
       * Properties:
             o serverName - Specify the host name or IP address of the database server.
             o port - Specify the port number of the database server eg. 3306
             o user - Set as appropriate.
             o  password - Set as appropriate.
             o databaseName - Set as appropriate, eg. bpwebapp
             o URL - If you are using global transactions, you can set this property instead of serverName, port, and databaseName, eg. jdbc:mysql://<dbhost>:3306/bpwebapp
             o Keep the rest of properties created.
   
   6.  Create the JDBC resource with the JNDI name jdbc/BPWebappDB pointing to new resource created above.

Deploying the Web Application

Tuning Glassfish

1. Go to the administration interface, eg. http://localhost:4848.

2. Ensure that you have the JVM option "-server" and the heap size has been adequately sized.  For benchmarking purposes, we have sized Xmx and Xms to 3G.

3. Start Glassfish. Check that you can connect to it from your browser (http://host:8080), but don't try to access any of the application pages yet !  To do so :
   

   • use $GLASSFISH_HOME/bin/asadmin start-domain domain1
   • To stop Glassfish, use $GLASSFISH_HOME/bin/asadmin stop-domain domain1.

Setting up Caching

• No cache. This option is useful if you want to stress MySQL. In this case, set -DcacheOlio=false.
• MapCache. This is the default and doesn't require any additional setting. It does a good job of off-loading the database. It is activated by the cacheFactorClass=org.apache.olio.webapp.cache.SimpleMapCacheFactory property.
  
• Memcache. In this case, the home page contents are cached in memcached. Set -DcacheFactoryClass=org.apache.olio.webapp.cache.MemCachedFactory and -DmemcachedInstances=<host>:<port> where <host>:<port> point to the memcached instance. <port> is 11211 if you use the default port when starting memcached.
  

1. Download and install memcached on the system you plan to use.

2. Start the memcached server thus :
   # nohup /opt/coolstack/bin/memcached -m 256 -u mysql &

Setting up the Filestore

Olio for Java EE can be configured to use either a local filesystem (typically NFS-mounted off a file server) or MogileFS for the object data. Our initial testing with MogileFS found some severe performance issues, so for now we advice using a normal filesystem. You will need about 50GB of space for the data, as the data does grow over runs.  Using a single spindle does work but may create performance bottlenecks.  We recommend striping the filesystem across at least 3 spindles to avoid such bottlenecks.  A local file system needs to be setup on the same machine as the web application.  A network file system can reside on a separate server but needs to be exported and mounted on the system running the web application.

1. Create a directory (or mount a filesystem) designated for storing the image and binary files. This directory is referred to as <filestore>. Any valid name for the OS should be fine. Ensure that everyone has read and write access to it:
   # mkdir -p <filestore>
   # chmod a+rwx <filestore>

2. Now load the filestore.  You should have the $FABAN_HOME/benchmarks/OlioDriver.jar file on this system.
   # cd $FABAN_HOME/benchmarks; mkdir olio
   
   # cd olio; jar xvf ../OlioDriver.jar; chmod a+x bin/*
   # $FABAN_HOME/benchmarks/olio/bin/fileloader.sh <load_scale> $FILESTORE
   
         This loads files for use for up to load_scale number of active users. 
   
   Eg. To load the filestore for 1000 users, you would do the following:
   # cd /filestore/artifacts
   # /opt/faban/benchmarks/OlioDriver/bin/fileloader.sh 1000 .

   This loads files for use for up to load_scale number of concurrent users. 

3. The filestore location is read from the application server as a Java environment variable, webapp.image.directory.  To set the value of the filestore, simply add an JVM argument eg. -Dwebapp.image.directory=/export/myfilestore.  If you are using GlassFish, you use the administration interface to add this JVM argument by looking at the GlassFish Installation and Configuration section, step 3.
   
   

Testing the Application

1. Check the home page (HomePage) http://<web_server>:8080/webapp/index.jsp. If there are no error messages and all images get loaded, that's a great start !

2. Click on an event (EventDetail). Make sure the whole page looks OK.

3. Click on an attendee (PersonDetail) to see a person's profile.

4. Go back to the home page and click on a tag in the tag cloud. Choose a big tag and check that we have good results and images get loaded OK.

5. Click on the Register button on the top left of the home page. Fill in the form and create a user. Make sure you find some jpeg images to upload. If not, take them from $FABAN_HOME/benchmarks/OlioDriver/resources.
   Submit the form. Make sure the form goes through. This completes the AddPerson transaction.

6. Login using your new login name you just created. The top right of the screen should show that you're logged on.

7. Select an event, go back to the EventDetail page but this time as a logged on user.
   Add yourself as an attendee. This is the EventDetail transaction with attendee added (about 8-9% of all EventDetail views).

8. Click on the add event tab and add an event. Make sure to include an image and some literature. You can also use the files from
   $FABAN_HOME/benchmarks/OlioDriver/resources. Fill in the form and submit. This is the AddEvent transaction.
   
   

Starting a benchmark run

Now that we know that the web application is running and the faban harness is up, it is time to kick off a benchmark run.

1. Kill memcached. Memcached is always started by the driver before the run to ensure a clean cache and will cause port conflicts if it is already running.

2. Point your browser at http://<driver_machine>:9980

3. Click on the Schedule Run link.

4. Under the JAVA tab, set the JAVA_HOME. You can accept the default value for JVM options. DO NOT click on the OK button yet !

5. Select the Driver tab.

   Enter a Description for the run (say 'First test run' for this case). In general, the Description field is very useful to get a quick idea of what a particular run is testing.

   Enter the name of your driver(s) machine for Host (when using more than one machine, simply separate them by a space).

   Enter 10 for 'Concurrent Users' (we want to start small).

   Enter 'vmstat 10' for Tools. This indicates the measurement tools that will be run on the driver machine. It's a good idea to keep an eye on the driver cpu utilization.

   Now enter 30, 30, 10 for the Ramp up, Steady State and Ramp down times. This is a very short test run. For normal runs, you may need a ramp up of 200 seconds and a steady state of at least 600 seconds during which measurements are made.

   For current systems, the time between client startup of 200 milliseconds is good enough. Some servers or slower systems may not be able to accept connections very frequently. In that case we may want to increase this value to 1000 milliseconds.
   

          The number of Agents needs to be set based on the number of concurrent users - we start with 1 and add an agent for every 500 users.

6. Select the Web Server tab.
   
   The Host:Port Pairs field takes the host port pairs where the web applications are running. The host and port is separated by a colon. Each pair is separated by space.  Note that you cannot use the string "localhost" - you must specify a valid hostname or IP address. 
   
   In the tools box, enter the tools you want to run. Here are the tools we typically run : vmstat 10; mpstat 10; nicstat 10; iostat -x 10.
   
   For Application Server, select "GlassFish" or leave the field blank if you're using some other server. Only the GlassFish server is supported at this time - for other servers, you will have to manually start/stop the servers before and after a run. Then provide the application server's bin, log, and config directories. For Glassfish, faban will automatically capture the server config and log file and make it available via the 'Statistics' tab of a run.
   
   The "restart" field indicates whether you want the Faban harness to automatically start and stop the GlassFish server before and after each run.  By default, it is set to true which indicates that the Faban harness will restart the GlassFish server, assuming that the path information for the server is entered correctly.
   
   

7. Select the Data Servers tab.

   For the database server, enter the Host name. Edit the hostname part in the JDBC Connection URL.  Note that you cannot use the string "localhost" - you must specify a valid hostname or IP number.
   This is used by the loader program to reload the database before a run. Set the 'Loader for Concurrent Users' to 25 (this is the minimum number of users we can load for and is good for up to 25 concurrent users). You can set the loader to run a larger number so that you don't have to edit this field every time. It is not absolutely necessary to reload the database and files every time, but you should do so for all performance runs. In that case, set the reload fields to true every time.
   

   Set the Data Storage server. For local storage this is the same host as the application server. 

   Set the memcached server instances to the servers you've configured in web.xml of the web application. The driver harness will start the memcached server instances accordingly. Note that the memcached server instances are given as host:port pairs, separated by space. If a port is not given, the default memcached port of 11211 is assumed.  
   

8. That's it. Click OK and the run should be scheduled. You can click on the View Results link on the left to monitor the run.