Visual Studio 2010–New and little-know features–Part 5–Better Unit Tests

lego_explorerTable of Contents for this series.

I occasionally give a talk about “Driving Quality through the Development Process”.. One of the things I try to drive home with developers is that you can’t test for software quality, you have to build it in from the start.  One of the ways you can do this is to write a good set of Unit Tests. 

For those not familiar with the term, here’s a thorough definition that I like:

“Unit Testing is a level of the software testing process where individual units/components of a software/system are tested. The purpose is to validate that each unit of the software performs as designed.

A unit is the smallest testable part of software. It usually has one or a few inputs and usually a single output. In procedural programming a unit may be an individual program, function, procedure, etc. In object-oriented programming, the smallest unit is a method, which may belong to a base/super class, abstract class or derived/child class. (Some treat a module of an application as a unit. This is to be discouraged as there will probably be many individual units within that module.)”    
                                    – Software Testing Fundamentals –
Unit Testing

My personal definition is a bit simpler:

“Unit tests are small pieces of code testing small pieces of code”

The really great thing about unit tests is that they pay for themselves over and over again.  There is an up-front cost to writing the tests but it saves you significant time and therefore money over their lifetime. This is described on ExtremeProgramming.org:

“The biggest resistance to dedicating this amount of time to unit tests is a fast approaching deadline. But during the life of a project an automated test can save you a hundred times the cost to create it by finding and guarding against bugs. The harder the test is to write the more you need it because the greater your savings will be. Automated unit tests offer a pay back far greater than the cost of creation.”
                                    – Extreme Programming – Unit Tests

Another great thing is that Unit Tests aren’t just for folks doing Agile development processes. They are equally applicable to teams doing Traditional software development practices.

For the rest of this article I’m going to assume that you are writing Object Oriented code.  I’m also going to assume that your definition of a Unit is the same as mine, namely that it is a Method. 

Ok, great! We write code that tests code. So that brings up a few questions…

  1. How many tests do I need to write for each “unit” (method)?
  2. How can I tell which logic paths my tests aren’t covering?
  3. How do I know what test to write if I don’t have enough?
  4. I have lots of legacy code and no tests, how do I start?

Let’s look at each of these questions in turn.


Q1: How many tests do I need to write for each “unit” (method)?

The answer here is “as many as are required to fully cover that method”. Not a really profound answer, huh?  What I mean is that there has to be a certain “minimum” number of tests to cover my method’s behavior.  Basically I want to be able to write enough tests to cover the “happy path” through my code. How many test is that?

To find that number all we have to do is run the Code Metrics tools in Visual Studio and look at the Cyclomatic Complexity metric for our method.

Cyclomatic Complexity can be defined as the number of decisions that are made within our source code. Another way to put it is to say that it is the number of distinct paths through a given piece of code.

So we have a tool that gives us a count of the “distinct paths” through our code. Guess what? Those “distinct paths” are the “happy path”. Look at the following example.

NOTE: You can click any image to display it at it’s original size.

SNAGHTML6e98e6 

The ShoppingCart class has 3 overrides for the AddItem() method.  The first 2 versions call the third version passing in additional default parameters. Since the first two only make a single call out they have a Cyclomatic Complexity of 1.  They only have a simple “happy path”.

The third AddItem() method does all of the real work.  As the diagram shows, the Code Metrics for this method show that it has a Cyclomatic Complexity of 5.  There are 5 basic paths through this code.

Important!: Cyclomatic Complexity does not show the total number of tests needed, just the minimum.  It does not account for boundary conditions and the like.  It is just a basic metric of complexity that we can use to get a handle on our testing.

This leads us nicely into the next question…


Q2: How can I tell which logic paths my tests aren’t covering?

So let’s say that you have the AddItem() method with a cyclomatic complexity of 5 and you want to start writing tests for it.  How do you know you’ve written the correct tests to cover the happy path?  I know I need a minimum of 5 unit tests, but how can I tell which paths have been tested and which haven’t? 

To answer this question we want to bring in the Code Coverage tool in Visual Studio.  Code Coverage determines how much of your code is being exercised by your unit tests.

To enable Code Coverage you need to do four things:

  • First, turn on Code Coverage in your .testsettings file.
    • Open the local.testsettings file in the Solution Items folder in your SolutionSNAGHTML7aadcd 
    • Navigate to the Data and Diagnostics tab and check the box next to Code Coverage.

SNAGHTML7e21e4 

    • While the Code Coverage entry is selected, the Configure button will be enabled.  Click on it to open the Code Coverage Detail dialog.

SNAGHTML81e86d 

    • In the Code Coverage Detail dialog, select the assemblies that you wish to collect against.  This is usually all of your custom assemblies. We generally don’t collect coverage data on the unit test assemblies, but you could if you had a need.
    • You can click Ok then Apply and Close to close the dialogs. 

 

  • Second, tell Visual Studio to use our .testsettings file.
    • In the Visual Studio menus, select Test –> Select Active Test Settings –> Local (Local.testsettings)

SNAGHTML867f6e 

  • Third, run our unit tests in Visual Studio
    • Open the Test View window to show the tests in our Solution. From the menu select Test –> Windows –> Test View

SNAGHTML8ca5c3 

    • In the Test View pane, select one or more tests and click the Run button.  This will launch the Test Results window.

SNAGHTML933832 

    • When the tests are finished running, click on the Show Code Coverage Results button in the Test View.

SNAGHTML970485 

    • This will bring up the Code Coverage Results window.  From here you can drill down to the methods that interest you. Once there you can see how much of the code is covered.

SNAGHTMLacb4e0

    • In the example above, the AdjustQuantity() method (yellow highlight) had zero blocks touched by the test run.  This shows up as a red highlight over the lines of code that were not exercised.  The ClearItems() method on the other hand have every block touched by the tests that were run.  This does not indicate that every possible test was run for this block of logic, only that all of the lines of code were hit. 

Q3: How do I know what test to write if I don’t have enough?

The Code Coverage tools have the ability to color your code to visually indicate which paths weren’t hit by your tests in that run. 

  • Turn on code coloring by clicking on the Show Code Coverage Coloring button on the Code Coverage Results toolbar.

SNAGHTML9d2001

  • It is possible to have a partially covered method. The EntityBase() constructor below was tested for a case where the passed parameter was an instantiated object, but no test was run that checked for a null object being passed.

SNAGHTMLa5e053


Q4: I have lots of legacy code and no tests, how do I start?

This is an easy one.  Just write your first test. Smile

To help with this process here are some things that I’ve used with my own code to help me get unit testing started.

Feathers-WorkingEffectivelyBook: Working Effectively with Legacy Code by Michael Feathers

“Get more out of your legacy systems: more performance, functionality, reliability, and manageability

Is your code easy to change? Can you get nearly instantaneous feedback when you do change it? Do you understand it? If the answer to any of these questions is no, you have legacy code, and it is draining time and money away from your development efforts.

In this book, Michael Feathers offers start-to-finish strategies for working more effectively with large, untested legacy code bases. This book draws on material Michael created for his renowned Object Mentor seminars: techniques Michael has used in mentoring to help hundreds of developers, technical managers, and testers bring their legacy systems under control.

The topics covered include

  • Understanding the mechanics of software change: adding features, fixing bugs, improving design, optimizing performance
  • Getting legacy code into a test harness
  • Writing tests that protect you against introducing new problems
  • Techniques that can be used with any language or platform—with examples in Java, C++, C, and C#
  • Accurately identifying where code changes need to be made
  • Coping with legacy systems that aren’t object-oriented
  • Handling applications that don’t seem to have any structure

This book also includes a catalog of twenty-four dependency-breaking techniques that help you work with program elements in isolation and make safer changes.”
                                                      – Back cover of book from Amazon.com

This is the one book that I recommend to my customers that are starting out with Unit Testing.  It has practical advice and techniques on how to make untestable-code testable. 

Tool: Pex and Moles from Microsoft Research

“Pex and Moles are Visual Studio 2010 Power Tools that help Unit Testing .NET applications.

  • Pex automatically generates test suites with high code coverage. Right from the Visual Studio code editor, Pex finds interesting input-output values of your methods, which you can save as a small test suite with high code coverage. Microsoft Pex is a Visual Studio add-in for testing .NET Framework applications.

The great thing about Pex is that it will generate test cases for all of your boundary conditions.  It is very thorough.  In fact, if your code can pass a full set of Pex-generated tests, it is robust indeed. Smile


Summary

So to wrap it all up…

  • You want to write unit tests to save yourself from injecting bugs into existing code while making changes or refactoring code. 
  • To determine the minimum number of unit tests for a given method, run the Code Metrics on your assembly or Solution and review the Cyclomatic Complexity metric for your method.
    • Cyclomatic Complexity does not give the total number of tests needed.  It does not take into account boundary conditions for a given algorithm or formula, only the decision points in the code.
  • To determine how much of your code is being exercised by your unit tests, turn on Code Coverage and review the metrics for your methods.
  • To visually determine which code paths weren’t hit in a method, turn on Code Coverage Coloring.
  • If you don’t currently have any unit tests, start by writing one, then one more, then one more.

 

 

Table of Contents for this series.