Performance Comparisons (a Helper Class)

Not very special but useful: A helper class for performance comparisons using a Stopwatch.

You may know the Stopwatch class which enables much more accurate performance tests than the DateTime struct.

This little helper class makes it use a little bit easier and more comfortable.

Sample Test

Let’s look at a sample test first which introduces the syntax:

static void Main(string[] args)
{
    PerformanceTestRunner runner = new PerformanceTestRunner(1000, WhileLoop, ForLoop);

    runner.Run();

    foreach (PerformanceTestResult result in runner.Results)
        Console.WriteLine("Result of {0}: Minimum: {1}; Average: {2}", result.Test.Name, result.MinimumTicks, result.AverageTicks);

    Console.ReadKey();
}

static void WhileLoop()
{
    int i = 0;
    while (i < 10000)
    {
        i++;
    }
}

static void ForLoop()
{
    for (int i = 0; i < 10000; i++)
    {

    }
}

This test answers the question which loop is faster: A while loop or a for loop.

The output on my system is:

Result of WhileLoop: Minimum: 69; Average: 71

Result of ForLoop: Minimum: 69; Average: 71

You see: The performance is equal but that isn’t our topic here.

The code

Well – here it is:

public sealed class PerformanceTest
{
    private static string GetActionName(Action action)
    {
        Contract.Requires(action != null);

        if (action.Method != null)
            return action.Method.Name;

        return string.Empty;
    }

    public Action Action
    {
        get;
        private set;
    }
    public string Name
    {
        get;
        private set;
    }
    public string Description
    {
        get;
        private set;
    }

    public PerformanceTest(Action action)
        : this(action, GetActionName(action))
    {
        Contract.Requires<ArgumentNullException>(action != null);

        Contract.Ensures(Object.Equals(this.Action, action));
        Contract.Ensures(this.Name != null);
        Contract.Ensures(this.Description != null);
    }
    public PerformanceTest(Action action, string name)
        : this(action, name, String.Empty)
    {
        Contract.Requires<ArgumentNullException>(action != null);

        Contract.Ensures(Object.Equals(this.Action, action));
        Contract.Ensures(Object.Equals(this.Name, name));
        Contract.Ensures(this.Description == String.Empty);

    }
    public PerformanceTest(Action action, string name, string description)
    {
        Contract.Requires<ArgumentNullException>(action != null);

        Contract.Ensures(Object.Equals(this.Action, action));
        Contract.Ensures(Object.Equals(this.Name, name));
        Contract.Ensures(Object.Equals(this.Description, description));

        this.Action = action;
        this.Name = name;
        this.Description = description;
    }
}

public sealed class TestResultCollection
    : ReadOnlyCollection<PerformanceTestResult>
{
    public TestResultCollection(IList<PerformanceTestResult> results)
        : base(results)
    {
    }

    [Pure]
    public bool ContainsResultForTest(PerformanceTest test)
    {
        Contract.Requires<ArgumentNullException>(test != null);

        return this.Any(result => Object.Equals(test, result.Test));
    }
    [Pure]
    public PerformanceTestResult GetResultForTest(PerformanceTest test)
    {
        Contract.Requires<ArgumentNullException>(test != null);
        Contract.Requires<ArgumentException>(this.ContainsResultForTest(test));

        return this.Where(result => Object.Equals(test, result.Test)).First();
    }
}

public sealed class PerformanceTestRunner
{
    private readonly IEnumerable<PerformanceTest> tests;
    private readonly TestResultCollection results;
    private readonly IList<PerformanceTestResult> editableResults;
    private readonly Stopwatch stopwatch;

    public TestResultCollection Results
    {
        get
        {
            Contract.Requires<InvalidOperationException>(this.results != null);
            Contract.Ensures(Contract.Result<TestResultCollection>() != null);

            return this.results;
        }
    }
    public bool HasRun
    {
        get;
        private set;
    }
    public int NumberOfRuns
    {
        get;
        private set;
    }

    public PerformanceTestRunner(int numberOfRuns, params Action[] tests)
        : this(numberOfRuns, (IEnumerable<Action>) tests)
    {
        Contract.Requires<ArgumentNullException>(tests != null);
        Contract.Requires<ArgumentException>(numberOfRuns > 0);

        Contract.Ensures(this.NumberOfRuns == numberOfRuns);
    }
    public PerformanceTestRunner(int numberOfRuns, IEnumerable<Action> tests)
        : this(numberOfRuns, tests.Select(test => new PerformanceTest(test)))
    {
        Contract.Requires<ArgumentNullException>(tests != null);
        Contract.Requires<ArgumentException>(numberOfRuns > 0);

        Contract.Ensures(this.NumberOfRuns == numberOfRuns);
    }
    public PerformanceTestRunner(int numberOfRuns, PerformanceTest[] tests)
        : this(numberOfRuns, (IEnumerable<PerformanceTest>) tests)
    {
        Contract.Requires<ArgumentNullException>(tests != null);
        Contract.Requires<ArgumentException>(numberOfRuns > 0);

        Contract.Ensures(this.NumberOfRuns == numberOfRuns);
    }
    public PerformanceTestRunner(int numberOfRuns, IEnumerable<PerformanceTest> tests)
    {
        Contract.Requires<ArgumentNullException>(tests != null);
        Contract.Requires<ArgumentException>(numberOfRuns > 0);

        Contract.Ensures(this.NumberOfRuns == numberOfRuns);

        this.tests = tests;

        this.editableResults = new List<PerformanceTestResult>();
        this.results = new TestResultCollection(this.editableResults);
        this.stopwatch = new Stopwatch();
        this.NumberOfRuns = numberOfRuns;
    }

    public void Run()
    {
        Contract.Ensures(this.HasRun);

        this.IncreaseThreadAndProcessPriority();

        foreach (PerformanceTest test in this.tests)
            test.Action();

        foreach (PerformanceTest test in this.tests)
            this.Run(test);

        this.HasRun = true;
    }

    private void Run(PerformanceTest test)
    {
        List<long> durations = new List<long>();

        for (int run = 0; run < this.NumberOfRuns; run++)
        {
            this.CollectGarbage();

            Thread.Sleep(0);

            long duration = this.Measure(test.Action);
            durations.Add(duration);
        }

        this.ProcessDurations(test, durations);
    }
    private void ProcessDurations(PerformanceTest test, IEnumerable<long> durations)
    {
        Contract.Requires(test != null);
        Contract.Requires(durations != null);
        Contract.Requires(durations.Count() > 0);

        long minimum = durations.Min();
        long maximum = durations.Max();
        long average = (long) (durations.Average() + 0.5);

        PerformanceTestResult result = new PerformanceTestResult(test, minimum, maximum, average, this.NumberOfRuns);
        this.editableResults.Add(result);
    }
    private long Measure(Action action)
    {
        this.stopwatch.Reset();
        this.stopwatch.Start();
        action();
        this.stopwatch.Stop();

        return this.stopwatch.ElapsedTicks;
    }
    private void CollectGarbage()
    {
        GC.Collect();
        GC.WaitForPendingFinalizers();
        GC.Collect();
        GC.WaitForFullGCComplete();
    }
    private void IncreaseThreadAndProcessPriority()
    {
        Thread.CurrentThread.Priority = ThreadPriority.Highest;
        Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.RealTime;
        Process.GetCurrentProcess().ProcessorAffinity = new IntPtr(1);
    }
}

public sealed class PerformanceTestResult
{
    public PerformanceTest Test
    {
        get;
        private set;
    }
    public long TicksPerMillisecond
    {
        get
        {
            return Stopwatch.Frequency / 1000;
        }
    }
    public long MinimumTicks
    {
        get;
        private set;
    }
    public double MinimumMilliseconds
    {
        get
        {
            return (double) this.MinimumTicks / (double) this.TicksPerMillisecond;
        }
    }
    public long MaximumTicks
    {
        get;
        private set;
    }
    public double MaximumMilliseconds
    {
        get
        {
            return (double) this.MaximumTicks / (double) this.TicksPerMillisecond;
        }
    }
    public int TestRuns
    {
        get;
        private set;
    }
    public long AverageTicks
    {
        get;
        private set;
    }
    public double AverageMilliseconds
    {
        get
        {
            return (double) this.AverageTicks / (double) this.TicksPerMillisecond;
        }
    }

    public PerformanceTestResult(PerformanceTest test, long minimumTicks, long maximumTicks, long averageTicks, int testRuns)
    {
        Contract.Requires<ArgumentNullException>(test != null);
        Contract.Requires<ArgumentException>(minimumTicks > 0);
        Contract.Requires<ArgumentException>(maximumTicks > 0);
        Contract.Requires<ArgumentException>(minimumTicks <= maximumTicks);
        Contract.Requires<ArgumentException>(averageTicks >= minimumTicks);
        Contract.Requires<ArgumentException>(maximumTicks >= averageTicks);
        Contract.Requires<ArgumentException>(testRuns > 0);

        this.Test = test;
        this.MinimumTicks = minimumTicks;
        this.MaximumTicks = maximumTicks;
        this.AverageTicks = averageTicks;
        this.TestRuns = testRuns;
    }
}

The classes

They are mostly self-explaining:

PerformanceTest

PerformanceTest represents a single test to run. It may have a name and a description but always has got a delegate pointing at the test logic.

PerformanceTestRunner

PerformanceTestRunner is the core class: It measures the PerformanceTests.

PerformanceTestResult

PerformanceTestResult encapsulates the result (means: The measured values) of a PerformanceTest.

Points of Interest

Calling the methods once before starting measuring

You may have noticed that PerformanceTestRunner executes all tests once before really starting to measure them.

The reason for that is the JIT-Compiler: We do not want it to interrupt our tests.

Thread.Sleep(0)

Before each test Thread.Sleep(0) is called so that we get a new timeslice for each test run. Otherwise, that possibility of other threads “stealing” our processor time during the test would be bigger.

Number of test runs

It’s always good to execute the tests more than once: There are lots of factors influencing our tests negatively. Thus, more executions mean better quality of our results.

Explicit Garbage Collection

As the CLR comes with a built in GC, our threads may be paused in order to collect garbage – so we explicitly collect it before every test.

Ticks / Milliseconds

I am using Stopwatch.ElapsedTicks to get the elapsed time.

Be careful: The “Stopwatch-ticks” are different from the “DateTime-ticks”: You have to use Stopwatch.Frequency to find out how many ticks a second takes.

How to test

One note: Build the project using the “Release” configuration and start it with “Debug – Start Without Debugging”. Otherwise, your results will not represent the actual values.

Minimum, Average or Maximum?

When interpreting the results you should only rely on the minimum time needed for execution.

This may sound strange at first, but here’s the reason why: All results worse than minimum are caused by something disturbing our tests. So the test did not run without any “problems”.

Close

However, do not forget: Today’s computers are unbelievable fast. So do not prefer code because it is some ticks faster than other code. The readability is much more important!

Did I forget anything? Do you know better ways for something in my code? Please let me know!

About these ads

4 Responses to “Performance Comparisons (a Helper Class)”

  1. DotNetShoutout Says:

    Performance Comparisons (a Helper Class)…

    Thank you for submitting this cool story – Trackback from DotNetShoutout…

  2. tobi Says:

    you put very much thought into how to make the measurement accurate. you increased the thread priority, collected garbage and did a sleep(0) between runs. very clever. here are 3 additional ideas a had:

    Process.GetCurrentProcess().ProcessorAffinity = new IntPtr(1); //i dont think this increases accuracy

    you could do 10 runs of the test and discard the 3 slowest runs. this would discard cpu-hickups, disk activity and cosmic rays. i would not discard the fastest runs.

    the third idea might be a little crazy but maybe it is useful to generate the test runner with reflection emit or codedom to eliminate the delegate overhead. this would however necessitate that the method to run is passed as a methodinfo.

  3. winsharp93 Says:

    Thanks for your comment and your interest in my component!

    >> Process.GetCurrentProcess().ProcessorAffinity = new IntPtr(1); //i dont think this increases accuracy
    Well – my intention behind this was that (on multicore CPUs) the OS might move the thread to another core. It may be useless – but I think that it does not do any harm :-)

    >> you could do 10 runs of the test and discard the 3 slowest runs. this would discard cpu-hickups, disk activity and cosmic rays. i would not discard the fastest runs.
    I am not discarding the fastest runs. As said in the article you should always look at the fastest test run. So I see no advantage in ignoring any test runs.

    >> the third idea might be a little crazy but maybe it is useful to generate the test runner with reflection emit or codedom to eliminate the delegate overhead.
    Hmm – sounds interesting but also a little bit oversized. In my opinion, this overhead is no problem as it occurs in all tests. So all results get a little bit “worse”. The absolute values usually do not express much; more important is the comparsion between them.

  4. tobi Says:

    i think you are right.


Comments are closed.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: