In my previous blog entry I wrote about how to extend Visual Studio in order to get transactional tracing for the applications that I develop in my Visual Studio Solution. This could be a console application, a WinForms or WPF Rich Client Application, an ASP.NET Web Application or a SharePoint Extension (WebPart, Custom List, …)
What about Unit Testing?
Visual Studio Team System Test Edition offers a set of different test types, e.g.: Unit-, Manual-, Web- and Load. Those tests can be executed via MSBuild and are therefore suitable to be executed by Team Foundation Server to verify the correctness of every build that comes out of your Continuous Integration Environment.
Lets focus on Unit Testing: If I – as a developer – start coding my unit tests I execute them locally before I checkin my code changes. Visual Studio allows me executing the unit tests from within the IDE showing me the status of my tests, error messages and even the execution time. Here is a sample test result view:
What are the limitations of Unit Testing in Visual Studio?
My unit test class implements 3 test cases that all call a Web Service hosted on a test machine in my development environment. The test machine runs the latest build of the application that I am working on and the unit tests make sure that the Web Service calls are working as expected – at least from a functional perspective.
With Unit Test support as it is right now I run into the following limitations:
- I don’t see how much time is actually spend in my ClassInitialize and ClassCleanup methods. I could have introduced code changes that takes several seconds to execute. Several seconds might not be long for my 3 tests that I run in my test suite. But what if my colleagues that work on the other hundreds or thousands of unit test suites do the same thing without knowing it? This would cause the overall unit test execution cycle to take very long and nobody would know why that is because the current set of tools doesn’t tell us
- Code coverage can help me to find out which methods have actually been called by my unit tests. But I run into two issues here: I don’t know which test method actually covered which code (this seems to be a feature that will be addressed in the next version of Visual Studio). The other limitation is that code coverage only works for the code that is exectuted in the unit test container (VSTestHost.exe). In my case I call a remote web service which is not automatically covered by code coverage
- With a Unit Test I can only verify the functional correctness of the components that I am testing. I cannot verify if those components would also run well outside the unit test environment or whether the components are configured correctly to use them in the most efficient way
Transactional Tracing with dynaTrace
dynaTrace offers transactional tracing with its unique PurePath technology. It allows tracing every single unit test transaction/execution from the actual run method in the unit test runner (Microsoft Unit Test or NUnit) through all relevant methods that are executed by the test and is even able to trace the transaction across runtime and technology (Java and .NET) boundaries. Along the PurePath information like execution times, method arguments, exceptions, logging messages and database statements are captured.
How to integrate transactional tracing for Visual Studio Unit Test Executions?
I extended the Add-In I created in my previous post to enable injecting the dynaTrace Agent into the VSTestHost process that is launched by Visual Studio when executing any type of test. By doing that I get PurePaths for every single test that is executed. The PurePath not only includes the test method execution itself but also includes initialization and cleanup methods, gives me timing on every single method along the execution path and it also traces the calls across runtime and technology boundaries. The following illustration shows the same 3 unit tests executed with transactional tracing enabled:
It seems that there are several things that we can see from the PurePath of my TestAccountInformation test case. Expanding the PurePath for this test case we see that there is a significant amount of time spent in the ClassInitialize method. Drilling into it we see that exceptions have been thrown because the XmlSerializers have not been deployed with the Web Service Proxies – this resulting in an in-memory compilation of the Serializers that are needed.
Furthermore I can drill into the actual web service implementations on the test server that is hosting those services and figuring out what those services are doing and why they take a certain amount of time.
The two examples above give you an indication about how much more we can get out of our unit tests. Analyzing the remoting or database behaviour of the components that we test can help us understand if the current implementation will actually scale in a production environment.
Looking at memory behavior will get us insight about memory consumption per component or use case that we test – allowing us to make predictions about the environment we need in order to host our application and run it with the predicted user load.
And finally – looking at timings – not only on the top test method level but on the individual methods that we call will help us to address performance issues from the beginning – allowing our unit test executions to complete within our continuous integration cycles as well as fixing performance issues before they make it to the test – staging or production environment.
The ability to do transactional tracing for Unit Tests executed from within the development environment enables developers to better understand what is going on in the code that they are testing, in the frameworks that they base their app on and in the services that their code relies on. Visual Studio offers these extension mechnisms and dynaTrace – with the PurePath technology – enables capturing the necessary data.