Skip to main content

Japex

Version 1.1.3 - Updated September 12, 2007

Santiago Pericas-Geertsen


Introduction

Japex is a simple yet powerful tool to write Java-based micro-benchmarks. Micro-benchmarks are simple benchmarks that rarely involve complicated deployments and are often used to test specific parts of an application. They are also characterized by the use of wall-clock blocks: start a clock, run the code, stop the clock and report the result. Japex is similar in spirit to [JUnit] in that it factors out most of the repetitive programming logic that is necessary to write micro-benchmarks. This logic includes loading and initializing multiple drivers, warming up the VM, forking multiple threads, timing the inner loop, etc.

The input to Japex is an XML file describing a test suite. The output is a timestamped report in XML and HTML formats (although generation of the latter can be turned off). HTML reports include one or more charts generated using [JFreeChart].

In the context of this document, the term throughput is defined to be work over time. Japex is designed to estimate an independent throughput for each of the tests in the suite. Throughput estimation is done based on some parameters defined in the input file. There are basically two ways to specify that: (i) fix the amount of work and estimate time or (ii) fix the amount of time and estimate work. In addition to each test's individual throughput, aggregate throughtputs in the form of arithmetic, geometric and harmonic means are computed for each driver.

Even though by default Japex will compute a per test and a per driver throughput in transactions per second (TPS), it is possible to customize a test suite to either calculate throughput using a different unit, such as milliseconds or KBs per second, or produce a different kind of result such as latency, memory usage or message size.

For the latest updates as well as the latest tips on how to use Japex, the reader is referred to this [Blog] which serves as a supplement to this manual.

Writing Drivers

A driver encapsulates knowledge about a specific algorithm implementation, and thus a different driver is required for each implementation under test. For example, if XML parsing is the algorithm under test and parsers X, Y and Z are being evaluated, a different driver must be provided for each implementation.

A driver is a Java class that must extend JapexDriverBase, which is a class that in turn implements the interface JapexDriver shown below.

Example 1. JapexDriver Interface

    public interface JapexDriver extends Runnable {
      
        public void initializeDriver();
      
        public void prepare(TestCase testCase);    
        public void warmup(TestCase testCase);
        public void run(TestCase testCase);
        public void finish(TestCase testCase);
      
        public void terminateDriver();
    } 
        

The method initializeDriver() is called whenever a driver class is instantiated; the method terminateDriver() is called when all the results for a driver instance are computed and the instance is no longer needed. A driver class may be instantiated multiple times, so even though these two methods are guaranteed to be called once per instance, they are not guaranteed to be called once per driver class. Multiple driver instances are created whenever the number of threads is greater than one (c.f., parameter japex.numberOfThreads) or the number of runs per driver is greater than one (c.f., japex.runsPerDriver), or both. The remainder of the other methods define the four phases carried out by Japex for the generation of the output report.

The prepare() method defines a similarly named phase in which drivers set up their initial state. For example, a driver can load its input data into memory to avoid I/O during the subsequent two phases. The prepare phase is always executed using a single thread.

The warmup() method is called during the warmup phase in which the inner loop of the computation is carried out a number of times in order to warm up the Virtual Machine (VM). The duration of this phase can be controlled by the global parameters japex.warmupIterations and japex.warmupTime, with the latter taking precedence over the former if both are specified. The number of iterations or time set using these parameters is for each test case. Thus, if for example japex.warmupTime is set to 10 seconds and there are N test cases in the test suite (see Section [Test Suites]), the overall duration of this phase will be 10 * N seconds per driver. The default implementation of the warmup() method is to call the run() method explained next. The warmup phase is carried out in as many threads as defined by the japex.numberOfThreads parameter. Executing the warmup phase using multiple threads is particularly useful for client-server benchmarks in which server-side threads need to be started and warmed.

The run() method is intended to execute the inner loop under test. For some drivers, this method will be identical to warmup(), but there are cases in which additional steps may be necessary during the run phase. The duration of this phase can be controlled by the parameters japex.runIterations and japex.runTime, with the latter taking precedence over the former if both are specified. As it was the case for the corresponding warmup parameters, these two parameters refer to the number of iterations or the amount of time on a per test case bases. Like the warmup phase, the run phase will be carried out using japex.numberOfThreads threads.

Finally, the finish() method defines the last execution phase. Japex will set the value of the test case parameter japex.actualRunTime after the run phase is completed but before the finish phase is started. Therefore, the finish() method can be used to override the default behavior which is to compute a per test throughput in Transactions Per Second (TPS), where a transaction is defined to be a single call to run(). Whatever result is computed in this method must be of numeric type and must be used to set the output parameter japex.resultValue. If the default result unit of TPS does not apply, the driver should also set the value of japex.resultUnit (since this unit is the same for all drivers, this parameter can also be set globally in Japex input file).

There are in fact two result values and two results units, one for the Y axis and one for the X axis. The parameters japex.resultValue and japex.resultUnit represent the Y axis, and the parameters japex.resultValueX and japex.resultUnitX represent the X axis. Japex uses these parameters for displaying two-dimensional data in scatter charts. Note that unlike the Y axis parameters, neither of the X axis parameters are automatically computed by Japex.

Test Suites

Test suites are described using a configuration file written in XML. This file includes information such as runtime class path, drivers and test cases. A driver encapsulates knowledge about a specific algorithm implementation, and thus a different driver is required for each implementation under test. For example, if XML parsing is the algorithm under test and parsers X, Y and Z are being evaluated, a different driver must be provided for each implementation.

A sample configuration file is shown below:

Example 2. Sample Configuration File

    <testSuite name="Sample Test Suite" xmlns="http://www.sun.com/japex/testSuite">
        <param name="libraryDir" value="lib"/>
        <param name="japex.classPath" value="${libraryDir}/../dist/classes"/>
                
        <param name="japex.warmupTime" value="10"/>
        <param name="japex.runTime" value="10"/>
                
        <driver name="XDriver">
            <param name="Description" value="Driver for X parser"/>
            <param name="japex.DriverClass" value="com.foo.XDriver"/>
        </driver>
        <driver name="YDriver">
            <param name="Description" value="Driver for Y parser"/>
            <param name="japex.driverClass" value="com.foo.YDriver"/>
        </driver>
        <driver name="ZDriver">
            <param name="Description" value="Driver for Z parser"/>
            <param name="japex.driverClass" value="com.foo.ZDriver"/>
        </driver>
                
        <testCase name="file1.xml">
            <param name="japex.inputFile" value="data/file1.xml"/>
        </testCase>
        <testCase name="file2.xml">
             <param name="japex.inputFile" value="data/file2.xml"/>
        </testCase>
    </testSuite>

A test suite configuration file is defined as a sequence of zero or more global parameters followed by a sequence of one or more drivers and then followed by a sequence of one or more test cases. Correspondingly, there are three kinds of parameters: global, driver and test case paremeters. The location of a parameter definition in a config file determines its kind and scope. Global parameters are defined as immediate children of <testSuite>, driver parameters are defined as immediate children of <driver> and test case parameters are defined as immediate children of <testCase>. Moreover, a driver parameter or a test case parameter can override a global parameter of the same name. However, test case parameters cannot override driver parameters (or vice-versa) since, being XML siblings, their scope is disjoint.

Parameters whose names start with the prefix japex. are reserved and assigned specific meanings by Japex, regardless of their kind. The reader is referred to Section [Reference] for the list of reserved parameters.

Drivers, test cases and parameters can also be defined in groups using <driverGroup>, <testCaseGroup> and <paramGroup>, respectively. For drivers and test cases, groups are useful to define a set of common parameters to be shared among the members of the group. In addition, grouping of drivers, test cases or parameters is necessary when modularizing a configuration file and using XInclude. The following example shows how to define a driver group with a set of shared parameters:

Example 3. Defining a Driver Group

    <driverGroup name="MyDriverGroup" xmlns="http://www.sun.com/japex/testSuite">
        <param name="libraryDir" value="lib"/>                   
        <param name="proxyHost" value="proxy:8080"/>                   
                    
        <driver name="XDriver">
            <param name="libraryDir" value="xlib"/>   <!-- Overrides group param -->
            ...
        </driver>                    
        <driver name="YDriver">
            ...
        </driver>                    
        ...            
    </driverGroup>

All groups can be named using the name attribute, but there is currently no use for the value of this attribute. Driver and test case groups start with a sequence of parameters (or parameter groups!) followed by a sequence of test cases or drivers, respectively. Thus, driver and test case groups define new parameter scopes whose outer scope depends on the location in which they appear in a configuration file. Wherever a driver, test case and parameter is allowed in a configuration file, so is a group of its kind. To allow modularization of a Japex configuration file, groups can be nested to any depth.

Normalizing Results

It is often convenient to normalize all the results based on those for a specific driver. This is especially useful in those cases where the difference between the results is so large that it becomes difficult to read them from a bar chart.

Defining which driver will be used as the normalizer is as simple as setting the normal attribute on the driver element as shown below:

Example 4. Setting the normal Attribute

    <testSuite name="ParsingPerformance" xmlns="http://www.sun.com/japex/testSuite">
        <param name="libraryDir" value="lib"/>                   
        ...
        <driver name="XDriver" normal="true">   <!-- Normalize results using this driver -->
            <param name="Description" value="Driver for X parser"/>
            <param name="japex.DriverClass" value="com.foo.XDriver"/>
        </driver>                    
        ...            
    </testSuite>

Note that the normalization of results only applies to the bar charts generated by Japex: the absolute values of the results are still included in the XML report and, consequently, in HTML report tables. The default value for normal is false. If more than one driver is defined as the normalizer, all but the first (in document order) are ignored.

Customizing Japex

The value of the global parameter japex.resultUnit determines the type of report generated by Japex. By default, the value of this parameter is tps (Transactions Per Second) which is, in essence, the speed of the software under test. This speed can be computed when japex.runTime is set, by counting the number of times the run() method is called, and when japex.runIterations is set, by estimating the elpased time needed to complete that number of iterations.

There are two other possible values for japex.resultUnit which are reserved and interpreted by Japex, these are, ms and mbps. The former can be used to produce a latency report as opposed to a throughput report. The latency in this case is defined to be the average time spent in one call to run(). The latter, another throughput unit, differs from tps in that the speed is calculated relative to the size of the input, which for the sake of simplicity is always a file whose path is defined by japex.inputFile. Note that an error will be reported if japex.resultUnit is set to mbps but japex.inputFile is undefined.

Although Japex reserves the units tps, ms and mbps, drivers are not limited to using one of these three: a driver can set an arbitrary value for japex.resultUnit provided it also sets a value for japex.resultValue. These calculations should take place during the finish phase, i.e., during the call to the finish() method. A number of output parameters are made available to the drivers for this purpose, see Section [Testcase Reference] for more information.

The following is an example of a driver that outputs the size of an XML file (defined by the testcase parameter "xmlfile") in kilobytes:

Example 5. Setting the japex.resultUnit and japex.resultValue parameters

    public void finish(TestCase testCase) {
        String xmlFile = testCase.getParam("xmlfile");
        if (xmlFile == null) {
            throw new RuntimeException("xmlfile not specified");
        }     
        testCase.setParam(Constants.RESULT_UNIT, "KBs");
        testCase.setDoubleParam(Constants.RESULT_VALUE, 
                                getFileSizeInBytes(xmlFile) / 1024.0);
    }
                    

An error is reported if, in a certain run, only a subset of the drivers set the value of japex.resultValue. I.e., the intention is for this parameter to be either automatically computed by Japex, or manually computed by all drivers. [1]

Class Paths and Class Loaders

The parameter japex.classPath can be set to define the location of driver classes and their dependencies. To add multiple locations to a class path, simply define this parameter multiple times (i.e., no class path separator is allowed). The following example shows how to add three locations to a class path:

Example 6. Setting the japex.classPath parameter

    <testSuite name="WSpex" xmlns="http://www.sun.com/japex/testSuite">

        <param name="japex.classPath" value="./classes/drivers"/>
        <param name="japex.classPath" value="./lib/*.jar"/>
        <param name="japex.classPath" value="./classes/generated"/>
        ...
    </testSuite>

Note the presence of the "*.jar" wildcard in the last location; this syntax can be used to add several jar files and is only allowed at the end of a path.

Although defined globally in the example above, japex.classPath is a driver parameter. Japex creates a specialized class loader to load each driver and its dependencies. This class loader is initialized with the (concatenated) value of japex.classPath as defined for each driver. This ensures drivers to be completely isolated from each other and it also allows to test mutiple versions of the same library (using similarly named Java packages) simultaneously. Moreover, it avoids conflicts between libraries in use by the Japex engine itself and those used by the drivers.

Parameter Values

It is possible to use the syntax ${param_name} to refer to the value of a parameter. If, while evaluating the expression ${param_name}, no parameter with that name is found, param_name is searched as an environment variable and an error reported if no variable of that name is found.

Setting Parameters Using System Properties

In order to reduce the number of configuration files, it is often desirable to externally define or override a global parameter. A typical example is changing the number of threads between runs without having to edit or duplicate a configuration file. This can be accomplished by using Java system properties and the -D option of the java command.

Example 7. 

java -Djapex.numberOfThreads=4 com.sun.japex.Japex config.xml

Any system property that starts with the "japex." prefix will be set as a Japex global parameter provided that its value is non-empty (i.e., different from the empty string). Ignoring a property whose value is empty is particularly useful when invoking Japex via Ant.

Parameter Reference

Input parameters are those specified in the configuration file that defines the test suite. Output parameters are those that are either computed by Japex or by any of drivers during the execution of the test. In principle, all input parameters are also output parameters since they can be overriden by a driver. However, we say that a parameter is of type input/output if it is of input type and it is commonly, but not always, overriden by a driver.

The default output axis is Y. Thus, parameters such as japex.resultUnit and japex.resultValue refer to the the unit and value of the Y axis, respectively. We use the notation [X] to indicate that the X is optional in a name. For example, japex.resultUnit[X] refers to the parameters japex.resultUnit and japex.resultUnitX.

Table 1. Global Parameters

ParameterDescription
japex.reportsDirectoryThe directory to write timestamped XML and HTML reports. DEFAULT: 'reports', TYPE: input.
japex.chartTypeThe type of chart selected for the output report. Possible values are 'barchart', 'linechart' and 'scatterchart'. Scatter charts are the only types of charts with two axes: japex.resultValue will be used for the Y axis and japex.resultValueX for the X axis in this case. DEFAULT: 'barchart', TYPE: input.
japex.plotGroupSizeThe number of test cases to plot in the same chart. To avoid very small groups, a simple algorithm is used to compute the size of the last two groups. For example, if this parameter is set to 5 and there are 21 test cases, to avoid having a group of size 1, Japex will create groups of sizes 5, 5, 5, 3 and 3. Thus, the value of this parameter only controls the size of all but the last two groups, for which it is an upper bound. To force all test cases to be plotted in the same chart, simply set the value of this parameter to a large integer. DEFAULT: 5, TYPE: input.
japex.plotDriversThis parameter is particularly useful when japex.chartType is set to 'linechart'. By default, Japex will draw a line for each test case across the set of drivers. By setting this parameter to true, a line is drawn for each driver instead each test case. This is particularly useful for testing scalability of a driver over a set of test cases. DEFAULT: false, TYPE: input.
japex.resultUnit[X]The unit of japex.resultValue. Unit values are case insensitive. There are number of reserved units that are understood by Japex. These are 'tps', 'ms' and 'mbps'. The first, 'tps' stands for Transactions Per Second and is the default. The second, 'ms' or Milliseconds can be used if the output of the test is latency instead of throughput. Finally, 'mbps' stands for Megabits Per Second and must be used together with the parameter japex.InputFile; i.e., Japex must know the length of the input file in order to compute the number of megabits per second. If Japex is unable to determine the size of the input file, and 'mbps' is the value of this parameter, an error will be reported. The value associated with this japex.resultUnit[X] can be displayed using scatter charts. DEFAULT: 'tps', TYPE: input/output. See also japex.chartType.
japex.resultAxis[X]Set to either 'normal' or 'logarithmic' to indicate the desired type of axis in the output charts. DEFAULT: undefined, TYPE: input. See also japex.resultUnit.
japex.versionVersion of Japex used in the run. TYPE: output.
japex.osNameName of the operating system. TYPE: output.
japex.osArchitectureName of the operating system architecture. TYPE: output.
japex.vmInfoVirtual machine information. TYPE: output.
japex.dateTimeTimestamp of the run. This value is also used to create a unique directory in which the reports are stored. TYPE: output.
japex.configFileName of the configuration file used for this run. TYPE: output.
japex.numberOfCpusNumber of physical CPUs or processors detected by Japex. TYPE: output.

Table 2. Driver Parameters

ParameterDescription
japex.driverClassName of the Java class that implements a driver. If absent, the driver's name is used instead. DEFAULT: undefined, TYPE: input.
japex.numberOfThreadsNumber of threads to use during the warmup and run phases of a driver. DEFAULT: 1, TYPE: input.
japex.classPathRuntime class path needed to load drivers and related libraries. Can be defined multiple times, with each new definition taken as an addition to the runtime class path. Thus, only a single class path entry is allowed per definition (i.e., path separators are not allowed). It is also possible for an entry in this path to end with '*.jar' to denote a set of jars located in a specific directory. DEFAULT: empty, TYPE: input.
japex.runsPerDriverNumber of complete runs per driver. By default Japex will carry out a single run per driver. A run includes a result for each test as well as horizontal means for the run. It is also possible to compute vertical means by setting this parameter to a value greater than one. The resulting output will include vertical means and standard deviations for each test across all runs. Moreover, the vertical means will be used to generate the output report. DEFAULT: 1, TYPE: input. See also japex.includeWarmupRun.
japex.warmupsPerDriverNumber of complete warmup runs per driver. By default Japex will carry out a warmup run per driver whenever japex.runsPerDriver is greater than 1. This parameter can be used to set the number of driver warmup runs independently of japex.runsPerDriver.DEFAULT: 1 if japex.runsPerDriver is greater than 1, 0 otherwise, TYPE: input.
japex.includeWarmupRunThis parameter is deprecated. See japex.warmupsPerDriver.
japex.resultAritMean[X]Arithmetic mean (average) for all test case results estimated for this driver. TYPE: output.
japex.resultGeomMean[X]Geometric mean of all test case results estimated for this driver. TYPE: output.
japex.resultHarmMean[X]Harmonic mean of all test case results estimated for this driver. TYPE: output.
japex.resultAritMean[X]StddevStandard deviation for the arithmetic means computed for each driver run. It only applies if there is more than one driver run. TYPE: output.
japex.resultGeomMean[X]StddevStandard deviation for the geometric means computed for each driver run. It only applies if there is more than one driver run. TYPE: output.
japex.resultHarmMean[X]StddevStandard deviation for the harmonic means computed for each driver run. It only applies if there is more than one driver run. TYPE: output.

Table 3. Test Case Parameters

ParameterDescription
japex.warmupTimeNumber of hours, minutes and seconds to warm up the VM prior to executing each test. Specified as a string in the format '((HH:)?MM:)?S?S'. Examples are '10' for 10 seconds and '02:00' for 120 seconds. Takes precedence over japex.warumpIterations if both are specified. UNIT: second, DEFAULT: undefined, TYPE: input.
japex.runTimeNumber of hours, minutes and seconds to execute each test. Specified as a string in the format '((HH:)?MM:)S?S'. Examples are '10' for 10 seconds and '02:00' for 120 seconds. Takes precedence over japex.runIterations if both are specified. UNIT: second, DEFAULT: undefined, TYPE: input.
japex.warmupIterationsNumber of iterations used to warm up the VM prior to executing each test. DEFAULT: 300, TYPE: input.
japex.runIterationsNumber of iterations used to execute each test. DEFAULT: 300, TYPE: input.
japex.runIterationDelayNumber of milliseconds to pause between iterations. If this value is greater than 0, the running thread will sleep for the specified number of milliseconds between calls to a driver's run() method. DEFAULT: 0, TYPE: input.
japex.inputFileA path to the input file used in this test. This parameter is primarily used by Japex whenever japex.resultUnit is 'mbps'. Determining the size of the input is required to compute the test's throughput in Megabits Per Second. TYPE: input/output.
japex.resultValue[X]The value of the output result for this test. If not specified, Japex will compute this value based on japex.resultUnit[X]. UNIT: japex.resultUnit[X], TYPE: output.
japex.resultValue[X]StddevStandard deviation for the result values computed for a test on each driver run. It only applies if there is more than one driver run. TYPE: output. TYPE: output.
japex.resultIterationsThe number of times this test was executed. If the japex.numberOfThreads is greater than 1, the value of this parameter will be a sum of all the iterations across all threads. If japex.runsPerDriver is greater than 1, the value of this parameter will be an average of the number of iterations across all runs. TYPE: output.
japex.resultTimeThe exact period of time used to compute the result for this test. If japex.runsPerDriver is greater than 1, the value of this parameter will be an average of the periods across all runs. TYPE: output.
japex.actualPrepareTimeActual number of milliseconds spent during the prepare phase for this test. UNIT: millisecond, TYPE: output, not serialized.
japex.actualWarmupTimeActual number of milliseconds spent during the warmup phase for this test. Whenever japex.numberOfThreads is greater than 1, this value will correspond to that of the last thread that updated this parameter. UNIT: millisecond, TYPE: output, not serialized.
japex.actualRunTimeActual number of milliseconds spent during the run phase for this test. Whenever japex.numberOfThreads is greater than 1, this value will correspond to that of the last thread that updated this parameter. UNIT: millisecond, TYPE: output, not serialized.
japex.actualWarmupIterationsActual number of iterations carried out during the warmup phase for this test. Whenever japex.numberOfThreads is greater than 1, this value will correspond to that of the last thread that updated this parameter. UNIT: iteration, TYPE: output, not serialized.
japex.actualRunIterationsActual number of iterations carried out during the run phase for this test. Whenever japex.numberOfThreads is greater than 1, this value will correspond to that of the last thread that updated this parameter. UNIT: iteration, TYPE: output, not serialized.


[1] If only a subset of the drivers compute japex.resultValue explicitly, this is most likely an indication of an error in the benchmark. However, because this observation does not apply in all cases, this behavior is subject to change in future releases.

 
 
Close
loading
Please Confirm
Close