Friday, 28 August 2015

Using TestDriven.NET and NCover to benchmark and monitor unit test code coverage

I wrote this blog post about a year ago - have finally got around to putting it online. It uses a "genericised" system called UpdateManager to demonstrate how to set up code coverage analysis for .NET. Hopefully it's not too dated just yet...

What is unit test code coverage benchmarking/analysis?

Unit test code coverage analysis essentially tells us "how much of my codebase is covered (or "exercised") by my unit test suite?"
When we are writing unit tests, it is necessary to visually scan the codebase and get a feel for what degree of coverage we have and what parts of the codebase are important/relevant for unit test.
Setting up a benchmark such as "a standard .NET codebase must have at least 70% unit test coverage" can be useful, as although this is not a fail-safe way to make sure that we are testing the right things, we can at least be certain that if our metrics indicate we meet the benchmark, most of the important stuff will be tested.
There are several tools available that will provide this sort of metric on a .NET codebase - including NDepend, NCover, etc.

TestDriven.NET vs ReSharper

TestDriven.NET is a tool that is primary intended to support unit test development, but it has a lot of overlap with ReSharper.
If you're already using ReSharper currently (as many .NET development teams do) - and given the fact that ReSharper generally has superior functionality where there is functionality overlap with TestDriven.NET - there is not a lot of value for us in having an additional tool for this purpose.
TestDriven.NET however supports one function that ReShaper lacks - which is unit test code coverage analysis - TestDriven.NET has a built-in instance of NCover, which can be accessed like this:
[Getting to NCover from TestDriven.NET]
When NCover is run across a unit test library, it will run all your tests and provide a code coverage report (percentage coverage of codebase by unit tests), that looks something like this (see the Output window of VS2010):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
------ Test started: Assembly: UpdateManagerUnitTests.dll ------ 

2014-06-19 16:18:29,366 [TestRunnerThread] ERROR Logger XxxMetadata.LoadFile - ERROR:
System.ArgumentException: Empty path name is not legal.
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at UpdateManager.Logic.XxxMetadata.LoadFile() in E:\UpdateManager\Logic\XxxMetadata.cs:line 56
2014-06-19 16:18:29,423 [TestRunnerThread] ERROR Logger XxxMetadata.XxxMetadata - LoadFile FAILED

[etc...]

19 passed, 0 failed, 0 skipped, took 10.13 seconds (NUnit 2.6.1).
Following that NCover will open an interactive explorer app/window called “NCoverExplorer” – like this:

[NCoverExplorer window]
Using NCoverExplorer, you can browse the codebase (by namespace) under test that has been analysed by NCover and establish the percentage that each assembly, namespace, class and event method/property is covered by unit test.

[NCoverExplorer window - browse to assembly]

[NCoverExplorer window - browse to namespace, etc]

Summary

Setting and adhering to a code coverage benchmark (and tooling up accordingly) is a step along the way to establishing a mature, modern software development practice.
Adopting unit testing as a practice by itself is extremely important. However, code coverage benchmarking and analysis provides the on-going measurement and reporting that is required to be able to sustain the practice.
TestDriven.NET and NCover provide a way to get started with adopting and learning this practice relatively cheaply. TestDriven.NET is free for personal use.

Gotcha!

When TestDriven.NET is installed over the top of ReSharper, it can sometimes "replace" some of ReSharper's very nice unit testing automation functions (such as those neat little green dots on the left side of the text editor that will launch a test for you and indicate the most recent result).

[ReSharper - neat little green dots for running tests]
Never fear - in order to get ReSharper's functionality back, just re-install ReSharper back over the top of TestDriven.NET. It's a bit of a "sledge hammer" approach, but you only need to do it once, and it works.

References

TestDriven.NET: http://testdriven.net/


Friday, 2 January 2015

The importance of improvisation

Preamble, background, etc

Scrum is a process improvement framework that has grown up within the world of software development and/but is increasingly used not only within the world of software, but outside of it also. Scrum in-fact lends itself well to product development in general. This post focuses on Scrum (and frameworks like it); in particular how it copes (or doesn't cope) with significant organisational change and where/when improvisation becomes more important than a framework.

The following links provide some information about Scrum:

An acquaintance of mine recently asked an interesting question about Scrum

...which was essentially: 

"What do you do when an established Scrum team encounters a serious problem?" 





["What do you do...what do you do...?" - source: http://chaffeenguyen.com/win-win-situations/]

As a process improvement framework, Scrum is able to cope effectively with most of the day-to-day problems that an organisation may face. Occasionally though the problem is simply a mismatch for the framework - just too big. For example; the business has changed hands, management has changed and/or the business' strategic direction changes.


Scrum works great when business is relatively smooth; it surfaces problems to be resolved by the team/business, etc. Beyond a certain size/volume of problem(s) though Scrum seems to become impractical. What emerges then is a need for the team to lean more heavily on it's ability to self organise and improvise - outside of any framework - in order to be able to recover effectively or perhaps survive. For this reason, I have come to think that drilling a Scrum regime across an organisation is not necessarily always best for business. 

Strategy and the importance of improvisation

To reach their true potential, teams need to have the freedom to explore, develop and optimise their own capabilities within themselves and outside of any set framework. Rigorous adherence to a framework can mar the natural and relatively fragile team-gelling process. Another trade-off to rigorous adherence to a framework is that a team's improvisation skills become rusty. And the ability to improvise in particular is key not only in emergency situations but also (arguably, more-so) in day-to-day business.




[Miles Davis; exemplary improviser - source: http://www.fastcompany.com/3000340/if-miles-davis-taught-your-office-improvise]

Furthermore, in reality, business is not about trying to get things to go as smoothly as possible - to survive and be prosperous, a business must seek out and overcome difficult challenges. If we focus on "getting to smooth" then we invariably begin to steer clear of challenges (AKA opportunities). 


A successful business (or one that at least plans to succeed) will have strategic direction and in alignment with that strategy some challenges will need to be avoided. More importantly though a business' strategy ideally will generally - and openly - identify the type of challenge that it wants to line-up and engage. That way at least everyone knows what the intention and direction of the business is and can have a feel for what's right and what's not. 

Of course, engaging and avoiding challenge is a balancing act that needs to take in to account resourcing, scheduling, cashflow, etc. Shying away from challenge purely in the interest of maintaining stability though is essentially laziness or myopia. Of course, this is deadly in business - remember Kodak? MySpace? Etc. 

Where am I going with this in terms of the framework...? Well, from a strategic perspective, the framework is simply a tool that's there to help a business reach it's strategic objective. Interestingly, working within the framework it can be difficult to identify if/when the framework itself has become a problem. It can happen though, especially in times of significant organisational change - a framework can in-fact be used as a shelter; a means to avoid or delay dealing with organisational change openly. In this situation things can become fuzzy and political - so I won't take this line of thought any further. I wanted to at least identify this point as part of this post though.

Is the framework still relevant?

So, certainly not suggesting that Scrum and frameworks like it are rubbish - far from it - they can help new teams get a great kick-start and can provide a clear ramp-up for organisations new to Agile. 




[Bamboo scaffolding, Hong Kong - source: http://www.archdaily.com/tag/architectural-photography/]

What I am suggesting is that beyond a certain point - whether it's due to an organisation reaching a certain level of maturity or an unavoidable and significant organisational change - the framework may become irrelevant and perhaps even cumbersome. It's worth being aware of this as a possible scenario, and if possible maintaining a general "feel" for a framework's relevance within the organisation.

Agility/stability

In my opinion, Agile essentially still holds the key in times of challenge and uncertainty. Outside of any framework, the basic principles of the Agile Manifesto (http://agilemanifesto.org/) seem to hold true; providing a simple, value-oriented sounding board when things get awkward/ambiguous, and a clear guide back to stability - until the next time!