9 Instrumenting Applications With DMS

9.1.1 Instrumenting Applications With DMS Metrics

DMS Instrumentation refers to the process of inserting DMS calls into application code. Using the DMS API is a simple and efficient way to enable your application to measure, collect, and save performance information.

To create DMS metrics developers add calls that notify DMS when events occur, when important intervals begin and end, or when pre-computed values change their state. At runtime, DMS stores metrics in memory and allows you to save or view the metrics.

Oracle Application Server includes built-in DMS metrics. By adding DMS calls to your applications you can expand the set of built-in metrics. When you instrument your applications with DMS calls, you use the same API that the built-in metrics use. In addition, to save and display your metrics, you use the same monitoring tools that you use with built-in metrics.

9.1.2 Monitoring DMS Metrics

Monitoring DMS metrics refers to the process of retrieving performance metrics. When an application runs, DMS stores metrics in memory and allows you to show metrics on the console or to view metrics using a web browser.

Oracle Application Server provides several runtime tools for viewing and saving DMS metrics, including dmstool and the AggreSpy Servlet.

Example 9-1 shows a set of metrics output using dmstool.

computeSeries.active:    0    threads
computeSeries.avg:    5931.7    msecs
computeSeries.completed:    20    ops
computeSeries.maxActive:    1    threads
computeSeries.maxTime:    57086    msecs
computeSeries.minTime:    2    msecs
computeSeries.time:    118634    msecs
lastComputed.value:    184756
loops.count:    4325    ops
timeStamp.ts:    1091035411174    milliseconds

Host:    system1
Name:    BasicBinomial
Parent:    /dmsDemo
Process:    home:OC4J:3301:6004
iasInstance:    10g2.tv.us.oracle.com
uid:    2109472775

9.1.3 Understanding DMS Terminology (Nouns and Sensors)

This section introduces the terminology you need to understand to use DMS. Figure 9-1 illustrates the organization of a set of DMS metrics corresponding to the metrics in the demo application described in this chapter and the metrics shown in Example 9-1.

This section covers the following topics:

• DMS Metrics

• DMS Sensors

• DMS Nouns

• DMS Object Relationships

Figure 9-1 Organization of Sample Metrics From dmsDemo Application

Description of "Figure 9-1 Organization of Sample Metrics From dmsDemo Application"

9.1.4 DMS Naming Conventions

Certain guidelines apply for defining DMS names. By following these guidelines, people viewing DMS metric reports can easily understand metrics across applications and across Oracle Application Server components.

This section covers the following topics:

• General DMS Naming

• General DMS Naming Conventions and Character Sets

• Noun and Noun Type Naming Conventions

• Sensor Naming Conventions

9.2.1 Including DMS Imports

To use DMS you need to add DMS imports. The following example shows the imports that the sample application BasicBinomial.java requires.

import oracle.dms.instrument.DMSConsole;
import oracle.dms.instrument.Event;
import oracle.dms.instrument.Noun;
import oracle.dms.instrument.PhaseEvent;
import oracle.dms.instrument.State;
import oracle.dms.instrument.Sensor;  

9.2.2 Organizing Performance Data

Define DMS Nouns to organize Sensors and their associated metrics. DMS Nouns organize Sensors in a tree hierarchy in a manner comparable to a directory structure in a file system, starting with a root at the top of the tree.

Example 9-2 shows a section of code using Noun.create() from the BasicBinomial.java.

In Example 9-2, MathSeries specifies the Noun type. The Noun type is a name that reflects the set of metrics being collected. For example, MathSeries represents the metrics collected for the sample application containing a Binomial series computation. AggreSpy displays Sensors using the same Noun type together.

It is good practice to only use Noun types for Nouns that directly contain Sensors. When a Noun contains only Nouns, as in the Noun dmsDemo, and does not directly contain Sensors, AggreSpy displays the Noun type as a metric table, with no metrics. Example 9-2 shows the dmsDemo Noun that includes a Noun, BasicBinomial, but no Sensors. When the Noun type is not included for such a Noun, AggreSpy does not display a metric table associated with the Noun.

private Noun binRoot;       // Container for Binomial series DMS metrics.
Noun base = Noun.create("/dmsDemo");
binRoot = Noun.create(base, "BasicBinomial", "MathSeries");

9.2.3 Defining and Using Metrics for Timing

To create metrics that measure the duration of a segment of code, define and use a PhaseEvent Sensor using the following steps:

• Defining PhaseEvent Sensors

• Using PhaseEvent Sensors

private PhaseEvent binComp; // Time to compute Binomial series. 
..
.  
binComp = PhaseEvent.create(binRoot, "computeSeries",  
                                     "Time to compute a Binomial series");
binComp.deriveMetric(Sensor.all);

long token = 0; // DMS
try {
    token = binComp.start(); // DMS
    BigInteger bins[] = bin(length);
    out.println("<H2>Binomial series for " +  length + "</H2>");
    for (int i = 0; i < length; i++)
        out.println("<br>" + bins[i]);
    }
    finally {
        binComp.stop(token); // DMS
        out.close();
    }

PhaseEvent pe = heavyPhase(param);
    long token1 = 0;
    long token2 = 0;
    boolean stopped = false;
    try {
      token1 = binComp.start(); 
      if (pe != null) token2 = pe.start();
      BigInteger bins[] = bin(length);
      out.println("<H2>ImprovedBinomial series for " +  length + "</H2>"); 
      for (int i = 0; i < length; i++)
        out.println("<br>" + bins[i]);
      if (pe != null) pe.stop(token2);
      binComp.stop(token1);
      stopped = true;
    }
    finally {
      if (!stopped) {
        if (pe != null) pe.abort(token2);
        binComp.abort(token1); 
      }

9.2.4 Defining and Using Metrics for Counting

To create metrics that count the occurrences of an event, define and use an Event Sensor as follows:

• Defining Event Sensors

• Using Event Sensors

private Event binLoop;   // Loops needed for Binomial series.
.
.
.
   binLoop = Event.create(binRoot, "loops", "Iterations to compute series");

binLoop.occurred();

9.2.5 Defining and Using Metrics for Recording Status Information (State Sensors)

DMS captures status information with State Sensors. State Sensors track the value of Java primitives or the content of a Java Object. The supported types include integer, double, long, and object, as specified in the third argument to the create() method. When a Java primitive State Sensor is updated with the wrong type, DMS attempts to convert the supplied value to the correct type. For Object type State Sensors, DMS stores a reference to the Object and by default and calls toString() on the object when the DMS value is sampled.

To create metrics that record status information, define and use a State Sensor as follows:

• Defining State Sensors

• Using State Sensors

private State binLast;   // Value of the last computed element in series.
.
.
.
binLast = State.create(binRoot, "lastComputed", State.OBJECT, "", 
                           "Value of last computed series element");

binLast.update(bins[k-1].toString());

9.3.1 Validating DMS Metrics

Use the dmstool and the other available DMS monitoring tools to verify and test new metrics.

Try to validate the following for new metrics:

• Do expected metrics appear in the display? Test this by examining the code to make sure that all the metric names added using DMS instrumentation appear in your display or saved set of metrics.

• Do unexpected metrics appear in the display? Verify that you have only added the metrics that you planned to add.

• Are the metric values you see within reasonable ranges? Usually, upper and lower bounds for metrics can be established. You then test that the reported values for metrics do not exceed the expected bounds.

  For example, a "size of pool" metric should never report a negative value.

• Make sure that new metrics are needed. For example, if you add a PhaseEvent that always measures an event of very short duration, consider changing the metric to an Event metric, or remove the metric.

• Make sure that new metrics are accurate. For most applications using DMS metrics, accuracy is more important than the performance cost of adding the DMS instrumentation. New DMS metrics should provide reliable and useful information.

  Testing for accuracy can be difficult; however, if an alternate means of measuring a particular metric is available then use it to verify metric values. For example, if you submit a known number of requests to a server and measure total time for the experiment, then you predict correct values for the relevant metrics and compare them with the actual monitored values. As another example, you can verify an Event Sensor count metric by examining records that you write to a log file or to the console.

  Check for timing inaccuracies that may apply for the metrics. Timing inaccuracies may be caused when low-resolution clocks time metrics for an interval of short duration. For example on Windows systems, the default Java clock advances only once every 15 milliseconds. DMS metrics reported for brief events on these systems must be analyzed with care. Consider using the high resolution clock to address this issue.

9.3.2 Testing DMS Metrics For Efficiency

The use of DMS metrics has some influence on application performance. When adding metrics, note the following:

• The processing required for computing and storing metrics can slow down the execution of an application. DMS is fast, but it does have some required overhead cost. In addition, DMS cannot prevent developers from using the DMS API inefficiently. Therefore, before adding DMS instrumentation, establish reasonable expectations. After completing the implementation, measure the actual costs and compare them to your expectations. Be prepared to make changes to the instrumentation to reduce overhead costs until the measurements agree with expectations.

• DMS provides the DMSConsole.getSensorWeight() method to help you control the use of metrics. The central setting is an advisory measurement level that DMS does not enforce. To control which metrics to include, at runtime, the code must test the value for SensorWeight to determine whether to make DMS calls.

• When integrating DMS instrumentation with an existing package or when implementing a new feature, you should consider insulating a previously working system. For example, you could include an option to enable and disable new DMS metrics.

• Worrying about performance too soon often leads to costly design and implementation errors. According to Donald Knuth, "Premature optimization is the root of all evil".

• You should run your performance tests with and without DMS enabled. If your tests show unacceptable results with DMS enabled, then you may want to re-design or re-implement metrics.

9.8.1 Isolating Expensive Intervals Using PhaseEvent Metrics

Carefully consider the requirements for new metrics when you add DMS instrumentation. It is important to add a sufficient number of metrics to validate that your code is behaving as desired.

Try to observe the following guidelines when you add DMS metrics:

1. Add PhaseEvent Sensors only to provide an overview of the time the system spends in your block of code or module. You do not need to collect performance data for every method call, or for every distinct phase of your code or module.

2. When your code calls external code that you do not control, and that you expect could take a significant amount of time, add a PhaseEvent Sensor to track the start and the completion of the external code.

Following these guidelines for adding PhaseEvent metrics provides the following benefits:

• Helps to limit the amount of information that DMS collects.

• Allows those analyzing the system to prove that a module gives the expected runtime performance.

• Ensures that people viewing DMS metrics can validate runtime performance without seeing an overwhelming amount of data.

• Allows those analyzing system performance to separate and track your module from other system modules that are either expensive or failure prone.

9.9.1 Configuring DMS Clocks for Reporting Time for OC4J (Java)

For Java processes, the default clock uses java.lang.System.currentTimeMillis(). Selecting the high resolution clock changes this call for all applications running on the process where the clock is changed. You set the DMS clock and the reporting units globally using the oracle.dms.clock and oracle.dms.clock.units properties, which control process startup options.

For example, to use the high resolution clock with the default units, set the following property on the Java command line for OC4J.

-Doracle.dms.clock=highres

-Doracle.dms.clock.units=msecs

Table 9-6 shows supported values for the oracle.dms.clock property.

Table 9-7 shows supported values for the oracle.dms.clock.units property.

Note the following when using the high resolution DMS clock:

• When you set the oracle.dms.clock and the oracle.dms.clock.units properties, any combination of upper and lower case characters is valid for the value that you select (case is not significant). For example, any of the following values are valid to select the high resolution clock: highres, HIGHRES, HighRes.

• DMS checks the property values at startup. When you set the clock with a value that does not match those listed in Table 9-6, then DMS uses the default clock. If the oracle.dms.clock property is not set, DMS also uses the default clock.

• If the specified clock units property value does not match those listed in Table 9-7, then DMS uses the default units for the specified clock. If the oracle.dms.clock.units property is not set, DMS uses the default units for the specified the clock.

Table 9-8 lists the platform specific environment variables settings for supported platforms. To use the high resolution DMS clock, the environment variables need to be set appropriately. The high resolution clock uses the DMS C library. On UNIX systems, this requires libdms2.so to be in the specified environment variable path. On Windows systems this requires yod.dll to be in the PATH environment. If a nanosecond clock is not available, high resolution timings use a microsecond clock.

9.9.2 Configuring DMS Clocks for Reporting Time for Oracle HTTP Server

The default clock for measuring Oracle HTTP Server performance has a resolution of microseconds (usecs). You can optionally select a higher resolution clock to monitor C processes running under Oracle HTTP Server. To use the High Resolution clock under Oracle HTTP Server, you need to set configuration options in httpd.conf, or specify environment variables on the command line.

Table 9-9 lists the environment variables that control the Oracle HTTP Server DMS clock. Table 9-10 describes the httpd.conf configuration options that control the Oracle HTTP Server DMS clock. If you set both the command line options and the httpd.conf configuration options, the configuration options override the values set on the command line.

For example, if you want to use the high resolution clock and use the same units to show times for Java processes running under OC4J and for mod_oc4j running under Oracle HTTP Server, update the Oracle HTTP Server httpd.conf file to include the following parameters and values:

DmsClock=HIGHRES
DmsClockUnits=MSECS

Also, include the following values as startup options for the OC4J process:

-Doracle.dms.clock=HIGHRES
-Doracle.dms.clock.units=MSECS

Using these options DMS uses a high resolution clock for all the Oracle HTTP Server processes that it monitors, for the Java OC4J processes that it monitors, and DMS reports values using the milliseconds units (msecs).

DmsClock=HIGHRES
DmsClockUnits=USECS