OJB Performance
- Introduction
- The Performance Test Suite
- How OJB compares to other O/R mapping tools?
- What are the best settings for maximal performance?
Introduction
"
There is no such thing as a free lunch."
(North American proverb)
Object/relational mapping tools hide the details of relational databases from the application developer. The developer can concentrate on implementing business logic and is liberated from caring about RDBMS related coding with JDBC and SQL.
O/R mapping tools allow to separate business logic from RDBMS
access by forming an additional software layer between business logic
and RDBMS. Introducing new software layers always eats up additional
computing resources.
In short: the price for using O/R tools is performance. But on the
other hand the biggest performance consumption is the database access itself
(database performance, network performance, jdbc driver, driver settings, ...).
So the percentage of O/R tool performance consumption isn't big.
Software architects have to take in account this tradeoff between programming comfort and performance to decide if it is appropiate to use an O/R tool for a specific software system.
This document describes the OJB Performance Test Suite which was created to lighten the decision between native JDBC, OJB (the different OJB API's) and other O/R mapper.
The Performance Test Suite
The OJB Performance Test Suite allows to compare all supported OJB API's against native single-threaded JDBC programming against your RDBMS of choice and run OJB API's in a virtual multithreaded environment. Further on it is possible to compare OJB against any O/R mapping tool using a simple performance application.
All tests are integrated in the OJB build script, you only need to perform the according ant target:
ant target
The following 'targets' exist:
- perf-test multi-threaded performance/stress test of PB/OTM/ODMG api against native JDBC
- performance single-threaded test, OJB API implementations (PB, ODMG) against native JDBC
By changing the JdbcConnectionDescriptor in the configuration files you can point to your specific RDBMS. Please refer to this document for details.
Interpreting test results
Interpreting the result of these benchmarks carefully will help to decide whether using OJB is viable for specific application scenarios or if native JDBC programming should be used for performance reasons.
Take care of compareable configuration properties when run performance tests with different O/R tools.
If the decision made to use an O/R mapping tool the comparison with other tools helps to find the best one for the thought scenario. But performance shouldn't be the only reason to take a specific O/R tool. There are many other points to consider:
- Usability of the supported API's
- Flexibility of the framework
- Scalability of the framework
- Community support
- The different licences of Open Source projects
- etcetera ...
How OJB compares to native JDBC programming - single-threaded
OJB is shipped with tests compares native JDBC with ODMG and
PB-API implementation.
This part of the test suite is integrated into the OJB build mechanism.
A single client test you can
invoke it by typing
ant performance or
ant performance.
If running OJB out of the box the tests will be performed against the Hypersonic SQL (in-memory mode) shipped with OJB. A typical console output looks like follows:
performance: [ojb] .[performance] INFO: Test for PB-api [ojb] [performance] INFO: [ojb] [performance] INFO: inserting 1500 Objects: 188 msec [ojb] [performance] INFO: updating 1500 Objects: 265 msec [ojb] [performance] INFO: querying 1500 Objects: 0 msec [ojb] [performance] INFO: querying 1500 Objects: 0 msec [ojb] [performance] INFO: fetching 1500 Objects: 16 msec [ojb] [performance] INFO: deleting 1500 Objects: 63 msec .... [ojb] Time: 5,672 [ojb] OK (1 test) [jdbc] .[performance] INFO: Test for JDBC [jdbc] [performance] INFO: [jdbc] [performance] INFO: inserting 1500 Objects: 157 msec [jdbc] [performance] INFO: updating 1500 Objects: 187 msec [jdbc] [performance] INFO: querying 1500 Objects: 94 msec [jdbc] [performance] INFO: querying 1500 Objects: 94 msec [jdbc] [performance] INFO: fetching 1500 Objects: 16 msec [jdbc] [performance] INFO: deleting 1500 Objects: 62 msec .... [jdbc] Time: 8,75 [jdbc] OK (1 test) [odmg] .[performance] INFO: Test for ODMG-api [odmg] [performance] INFO: [odmg] [performance] INFO: inserting 1500 Objects: 266 msec [odmg] [performance] INFO: updating 1500 Objects: 359 msec [odmg] [performance] INFO: querying 1500 Objects: 531 msec [odmg] [performance] INFO: querying 1500 Objects: 531 msec [odmg] [performance] INFO: fetching 1500 Objects: 47 msec [odmg] [performance] INFO: deleting 1500 Objects: 125 msec .... [odmg] Time: 13,75 [odmg] OK (1 test)
Some notes on these test results:
- You see a consistently better performance in the second and third run. this is caused by warming up effects of JVM and OJB.
- PB and native JDBC need about the same time for the three runs although JDBC performance is better for most operations. This is caused by the second run of the querying operations. In the second run OJB can load all objects from the cache, thus the time is much shorter. Hence the interesting result: if you have an application that has a lot of lookups, OJB can be faster than a native JDBC application (without caching extensions)!
- ODMG is much slower than PB or JDBC. This is due to the complex object level transaction management it is doing and the fact that ODMG doesn't have a specific method to lookup objects by it's identity. The second reason is responsible for slow querying results, because in test always a complex query is done for each object. It is possible to use the PB-api within ODMG, then the query by identity will be as fast as in PB-api.
- You can see that for HSQLDB operations like insert and update are faster with JDBC than with PB. This performance difference strongly depends on the used cache implementation and can rise up to 50% when the cache operate on object copies. This ratio is so high, because HSQLDB was running with in memory mode and is much faster than ordinary database servers. If you work against Oracle or DB2 the percentual OJB overhead is going down a lot (10 - 15 %), as the database latency is much longer than the OJB overhead.
It's easy to change target database. Please
refer to this
document for details.
Also it's possible to change the number of test objects by
editing the ant-target in build.xml.
Another test compares PB-api,ODMG-api and native JDBC you can find next section.
OJB performance in multi-threaded environments
This test was created to check the performance and stability of the
supported API's (PB-api, ODMG-api and future API's) in a single/multithreaded
environment and to compare the different api's against native JDBC calls.
Running this test out of the box (a virgin OJB version against hsql)
shouldn't cause any problems.
Per default OJB use a in-memory hsql database, by changing the JdbcConnectionDescriptor in the repository.xml file or modify the build.properties file when running OJB out of the box you can point to your specific RDBMS. Please refer to this document for details.
To run the multithreaded performance test call
ant perf-test
A typical output of this test, using OJB against in-memory hsql looks like this
[ojb] ================================================================================================ [ojb] OJB PERFORMANCE TEST SUMMARY, Thu Dec 29 23:42:20 CET 2005 [ojb] ------------------------------------------------------------------------------------------------ [ojb] 12 concurrent threads, handle 500 objects per thread [ojb] 500 INSERT operations per test instance [ojb] FETCH collection of 500 objects per test instance [ojb] Repeat FETCH collection of 500 objects per test instance [ojb] 125 get by Identity calls per test instance [ojb] 500 UPDATE operations per test instance [ojb] 500 DELETE operations per test instance [ojb] - performance mode - results per test instance (average) [ojb] ================================================================================================ [ojb] API Total Insert Fetch Fetch 2 by Id Update Delete [ojb] [%] [msec] [msec] [msec] [msec] [msec] [msec] [ojb] ------------------------------------------------------------------------------------------------ [ojb] JDBC 100 475(100%) 26(100%) 23(100%) 209(836%) 477(100%) 197(100%) [ojb] PB 203 1341(282%) 153(588%) 151(656%) 25(100%) 648(135%) 239(121%) [ojb] ODMG 250 1469(309%) 104(400%) 105(456%) 527(2108%) 800(167%) 569(288%) [ojb] ================================================================================================ [ojb] PerfTest takes 70 [sec]
This test run shows the overhead caused by the O/R layer compared to handcoded sql statements. Most overheads results in populate the two-level cache which is useless when using a in-memory database.
Below you can see the same test against MaxDB running on the same machine
.... [ojb] ================================================================================================ [ojb] API Total Insert Fetch Fetch 2 by Id Update Delete [ojb] [%] [msec] [msec] [msec] [msec] [msec] [msec] [ojb] ------------------------------------------------------------------------------------------------ [ojb] JDBC 100 5855(100%) 55(100%) 38(100%) 1628(5087%) 5588(184%) 4084(136%) [ojb] ODMG 117 12043(205%) 180(327%) 294(773%) 754(2356%) 3027(100%) 2988(100%) [ojb] PB 123 11577(197%) 94(170%) 84(221%) 32(100%) 4240(140%) 3193(106%) [ojb] ================================================================================================ [ojb] PerfTest takes 440 [sec]
To change the test properties go to target
perf-test
in the
build.xml file and change the program parameter.
The test needs five parameter:
- A comma separated list of the test implementation classes (no blanks!)
- The number of test loops
- The number of concurrent threads
- The number of managed objects per thread
- The desired test mode.
false means run in
performance mode,
true means run in stress mode
(useful only for developer to check stability).
<target name="perf-test" depends="prepare-testdb" description="Simple performance benchmark and stress test for PB- and ODMG-api"> <java fork="yes" classname="org.apache.ojb.performance.PerfMain" dir="${build.test}/ojb" taskname="ojb" failonerror="true" > <classpath refid="runtime-classpath"/> <!-- comma separated list of the PerfTest implementations --> <arg value= "org.apache.ojb.compare.OJBPerfTest$JdbcPerfTest, org.apache.ojb.compare.OJBPerfTest$PBPerfTest, org.apache.ojb.compare.OJBPerfTest$ODMGPerfTest" /> <!-- test loops, default was 6 --> <arg value="6"/> <!-- performed threads, default was 12 --> <arg value="12"/> <!-- number of managed objects per thread, default was 500 --> <arg value="500"/> <!-- if 'false' we use performance mode, 'true' we do run in stress mode --> <arg value="false"/> <!-- if 'true' all log messages will be print --> <arg value="true"/> <jvmarg value="-Xms128m"/> <jvmarg value="-Xmx256m"/> </java> <!-- do some cleanup --> <ant target="copy-testdb"/> </target>
How OJB compares to other O/R mapping tools?
Many user ask this question and there is more than one answer. But OJB was
shipped with a simple performance application (independend from OJB)
which allows a rudimentarily comparision of OJB with other (java-based) O/R mapping tools.
All involved classes can be found in dirctory [db-ojb]/src/test in package
org.apache.ojb.performance.
Call ant perf-test-jar to build the jar file contain all necessary classes to set up a test with an arbitrary O/R mapper. After the build, the db-ojb-XXX-performance.jar can be found in [db-ojb]/dist directory.
Steps to set up the test for other O/R frameworks:
- Implement a class derived from PerfTest
-
If persistent objects (used within your mapping tool) must be derived from a specific
base class or must be implement a specific interface write your own persistent
object class by implementing
PerfArticle
interface and override method #newPerfArticle()
in your PerfTest implementation class.
Otherwise a default implementation of interface PerfArticle was used.
The default implementation class is org.apache.ojb.performance.PerfArticleImpl.
That's it!
You can find a example implementation called org.apache.ojb.compare.OJBPerfTest in the test-sources directory under [db-ojb]/src/test (when using source-distribution).
This implementation class is used to compare the performance of the PB-API, ODMG-API, OTM-api and native JDBC (to bunch all API's, for each API a static inner implementation class of PerfTest was used). See more about multi-threaded performance.
Run the test
You have two possibilities to run the test:
a) Integration in the OJB build script
Add the full qualified class name of your PerfTest implementation class
to the
perf-test target of the OJB
build.xml
file, add all necessary jar files to
[db-ojb]/lib. The working
directory of the test is
[db-ojb]/target/test/ojb.
b) Run PerfMain
It's possible to run the test using
org.apache.ojb.performance.PerfMain.
java -classpath CLASSPATH org.apache.ojb.performance.PerfMain [comma separated list of PerfTest implementation classes, no blanks!] [number of test loops] [number of threads] [number of insert/fetch/delete loops per thread] [optional boolean - run in stress mode if set true, run in performance mode if set false, default false] [optional boolean - if 'true' all log messages will be print, else only a test summary, default 'true']
For example:
java -classpath CLASSPATH my.A_PerfTest,my.B_PerfTest 3 10 2000
This will use A_PerfTest and B_PerfTest and perform three loops each loop run 10 threads and each thread operate on 2000 objects. The test run in normal mode and log all messages.
Take care of compareable configuration properties when run performance tests with different O/R tools (caching, locking, sequence key generation, connection pooling, ...).
What are the best settings for maximal performance?
We don't know, that depends from the environment OJB runs (hardware, database, driver, application server, ...). But there are some settings which affect the performance:
-
The API you use, e.g. PB-api is much faster then the ODMG-api. See which API for more information.
-
The used cache implementation.
-
ConnectionFactory implementation / Connection pooling. See connection pooling for more information.
-
The autocommit setting of used connections. For best performance it's recommended to use autocommit 'false' setting in the jdbc-driver to avoid Connection.setAutoCommit(...) calls by OJB.
-
PersistentField class implementation.See OJB.properties section 'PersistentFieldClass' to change the implementation.
-
Used sequence manager implementation. See sequence manager for more information.
-
Use of batch mode (when supported by the DB). See jdbc-connection-descriptor batch-mode attribute for more information.
-
PersistenceBroker pool size. See OJB.properties for more information.
-
The JDBC driver settings (e.g. statement caching on/off).
To test the different settings use the tests of the performance test suite.
by Armin Waibel, Thomas Mahler