Trend analysis using iterative value increments

Question

We have configured iReport to generate the following graph:

The real data points are in blue, the trend line is green. The problems include:

• Too many data points for the trend line
• Trend line does not follow a Bezier curve (spline)

The source of the problem is with the incrementer class. The incrementer is provided with the data points iteratively. There does not appear to be a way to get the set of data. The code that calculates the trend line looks as follows:

import java.math.BigDecimal;
import net.sf.jasperreports.engine.fill.*;

/**
 * Used by an iReport variable to increment its average.
 */
public class MovingAverageIncrementer
  implements JRIncrementer {
  private BigDecimal average;

  private int incr = 0;

  /**
   * Instantiated by the MovingAverageIncrementerFactory class.
   */
  public MovingAverageIncrementer() {
  }

  /**
   * Returns the newly incremented value, which is calculated by averaging
   * the previous value from the previous call to this method.
   * 
   * @param jrFillVariable Unused.
   * @param object New data point to average.
   * @param abstractValueProvider Unused.
   * @return The newly incremented value.
   */
  public Object increment( JRFillVariable jrFillVariable, Object object, 
                           AbstractValueProvider abstractValueProvider ) {
    BigDecimal value = new BigDecimal( ( ( Number )object ).doubleValue() );

    // Average every 10 data points
    //
    if( incr % 10 == 0 ) {
      setAverage( ( value.add( getAverage() ).doubleValue() / 2.0 ) );
    }

    incr++;

    return getAverage();
  }


  /**
   * Changes the value that is the moving average.
   * @param average The new moving average value.
   */
  private void setAverage( BigDecimal average ) {
    this.average = average;
  }

  /**
   * Returns the current moving average average.
   * @return Value used for plotting on a report.
   */
  protected BigDecimal getAverage() {
    if( this.average == null ) {
      this.average = new BigDecimal( 0 );
    }

    return this.average;
  }

  /** Helper method. */    
  private void setAverage( double d ) {
    setAverage( new BigDecimal( d ) );
  }
}

How would you create a smoother and more accurate representation of the trend line?

Answer 1

This depends on the behavior of the item you are measuring. Is this something that moves (or changes) in a manner that can be modeled?

If the item is not expected to change, then your trend should be the underlying mean value of the entire sample set, not just the past two measurements. You can get this using Bayes theorem. The running average can be calculated incrementally using the simple formula

Mtn1 = (Mtn * N + x) / (N+1)

where x is the measurement at time t+1, Mtn1 is the mean a time t+1, Mtn is the mean at time t, and N is the number of measurements taken by time t.

If the item you are measuring fluctuates in a manner that can be predicted by some underlying equation, then you can use a Kalman filter to provide a best estimate of the next point based on the previous (recent) measurements and the equation that models the predicted behavior.

As a starting point, the Wikipedia entry on Bayesian estimators and Kalman Filters will be helpful.

Answer 2

Resulting Image

The result is still incomplete, however it clearly shows a better trend line than that in the question.

Calculation

There were two key components missing:

• Sliding window. A List of Double values that cannot grow beyond a given size.

• Calculation. A variation on the accept answer (one less call to getIterations()):

  ((value - previousAverage) / (getIterations() + 1)) + previousAverage

Source Code

import java.math.BigDecimal;

import java.util.ArrayList;
import java.util.List;

import net.sf.jasperreports.engine.fill.AbstractValueProvider;
import net.sf.jasperreports.engine.fill.JRFillVariable;
import net.sf.jasperreports.engine.fill.JRIncrementer;


/**
 * Used by an iReport variable to increment its average.
 */
public class RunningAverageIncrementer
  implements JRIncrementer {
  /** Default number of tallies. */
  private static final int DEFAULT_TALLIES = 128;

  /** Number of tallies within the sliding window. */
  private static final int DEFAULT_SLIDING_WINDOW_SIZE = 30;

  /** Stores a sliding window of values. */
  private List<Double> values = new ArrayList<Double>( DEFAULT_TALLIES );

  /**
   * Instantiated by the RunningAverageIncrementerFactory class.
   */
  public RunningAverageIncrementer() {
  }

  /**
   * Calculates the average of previously known values.
   * @return The average of the list of values returned by getValues().
   */
  private double calculateAverage() {
    double result = 0.0;
    List<Double> values = getValues();

    for( Double d: getValues() ) {
      result += d.doubleValue();
    }

    return result / values.size();
  }

  /**
   * Called each time a new value to be averaged is received.
   * @param value The new value to include for the average.
   */
  private void recordValue( Double value ) {
    List<Double> values = getValues();

    // Throw out old values that should no longer influence the trend.
    //
    if( values.size() > getSlidingWindowSize() ) {
      values.remove( 0 );
    }

    this.values.add( value );
  }

  private List<Double> getValues() {
    return values;
  }

  private int getIterations() {
    return getValues().size();
  }

  /**
   * Returns the newly incremented value, which is calculated by averaging
   * the previous value from the previous call to this method.
   * 
   * @param jrFillVariable Unused.
   * @param tally New data point to average.
   * @param abstractValueProvider Unused.
   * @return The newly incremented value.
   */
  public Object increment( JRFillVariable jrFillVariable, Object tally, 
                           AbstractValueProvider abstractValueProvider ) {
    double value = ((Number)tally).doubleValue();

    recordValue( value );

    double previousAverage = calculateAverage();
    double newAverage = 
      ((value - previousAverage) / (getIterations() + 1)) + previousAverage;

    return new BigDecimal( newAverage );
  }

  protected int getSlidingWindowSize() {
    return DEFAULT_SLIDING_WINDOW_SIZE;
  }
}