What is the best way to measure how long code takes to execute?

Question

I\'m trying to determine which approach to removing a string is the fastest.

I simply get the start and end time a

Answer 1

I'll have to recommend the highly profiler included in Visual Studio Team Suite or Development Edition (or the upcoming Visual Studio 2010 Premium or Ultimate is even better) as the best way. It is highly configurable, extremely powerful, dead simple to use, very fast, and works with both native and managed code. I'm not familiar with the workflow of ANTS, but that appears to be another option. Without a doubt using a profiler is the only option for a developer that is concerned with their application's performance. There is no substitute, and you really can't take any commercial developer working on performance that would pass up a profiler seriously.

However, if you are interested in measurements for one of the following, you may not have access such a profiler, in which case the Stopwatch class can form a basis for your measurement technique:

• A student or hobbyist interested in performance of their projects (a commercial profiler may be out of reach for financial reasons)
• In a publicly released application, you may want to time sections of code that execute on the UI thread and report statistics to may sure the operations never cause noticeable delays for any of your users. The Office team used a method like this with enormous success (Outlook 2007 SP2 anyone?), and I know the Visual Studio team has this code in at least the 2010 release.

Answer 2

The trouble with all the clock-based approaches is that you are never quite certain what you are timing. You might, whether you realise it or not, include in your timing:

• delays while the o/s pre-empts your processor;
• context-switching
• stalls while the program waits for data;
• and much much more.

I'm not saying that all of these apply to this particular timing but to timings in general. So you should supplement any timing you do with some consideration of how many basic operations your alternative codes execute -- some considerations of complexity not ignoring (as we usually do) the constant terms.

In your particular case you should aim to time much longer executions; when your times are sub-second the o/s is extremely likely to mess you around. So run 10^6 iterations and use the average over enough runs to give you a meaningful calculation of average and variance. And make sure, if you take this approach, that you don't inadvertently speed up the second trial by having data already loaded after the end of the first trial. You have to make sure that each of the 10^6 trials does exactly what the 1st trial does.

Have fun

Mark

Answer 3

You should use System.Diagnostics.Stopwatch, and you might want to consider a large sample. For example, repeat this test something like 10,000 times and average the results. If you think about it scientifically, it makes sense. The larger the sample, the better. You can weed out a lot of edge cases this way and really see what the core performance is like.

Another thing to consider is that JIT compilation and object creation can definitely skew your results, so make sure that you start and stop your Stopwatch at the appropriate times, and call the methods you want to test at least once before you begin your tests. Try to segregate just the parts you want to test from the rest of your code as much as possible.

Answer 4

Generally: Don't look at the wallclock time but the CPU time consumed by your process to determine how long it ran. Especially for things that are just computation this is much more reliable because it will be unaffected by other processes running at the same time.