Modeling distribution of performance measurements

Question

How would you mathematically model the distribution of repeated real life performance measurements - \"Real life\" meaning you are not just looping over the code in question, bu

Answer 1

Not exactly answering your question, but relevant still: Mor Harchol-Balter did a very nice analysis of the size of jobs submitted to a scheduler, The effect of heavy-tailed job size distributions on computer systems design (1999). She found that the size of jobs submitted to her distributed task assignment system took a power-law distribution, which meant that certain pieces of conventional wisdom she had assumed in the construction of her task assignment system, most importantly that the jobs should be well load balanced, had awful consequences for submitters of jobs. She's done good follor-up work on this issue.

The broader point is, you need to ask such questions as:

1. What happens if reasonable-seeming assumptions about the distribution of performance, such as that they take a normal distribution, break down?
2. Are the data sets I'm looking at really representative of the problem I'm trying to solve?