Functional Testing has always been an activity done by Test Engineers using Functional Testing tools like QTP from HP/Mercury, SilkTest from Borland/Segue, Functional Tester from IBM/Rational, …
But time is changing – so is Functional Testing – and the things you can do with it. In this article I talk about

The Past: Traditional Functional Testing

All those tools traditionally had their record/replay approach by recording the interactions with the User Interface that should be tested as well as recording the control structure of the windows/controls that were tested. With this separation it was easier to maintain scripts in case the layout of the application under test changed. You only had to update the description of the UI and not each test script that used that description.

The drawback of all these tools was that it required a special set of skills to do Functional Testing. Each tool came with its more or less proprietary testing language as well as IDE and runtime. These were some of the reasons developers usually didn’t get in-touch with functional testing. Due to the lack of exposure developers also frequently questioned the results of functional testing tools as they were presented to them by the Testing Engineers leading to built up tension between the testing and development teams.

The Present: The Functional Testing world is changing to …

Over the past couple of years – greatly driven by the Open Source Community as I believe – we could see a move towards bringing Functional Testing closer to the developers. The approach taken was to provide testing frameworks that allow writing functional tests in Java, .NET, JavaScript, … just as you would write your unit tests using frameworks like JUnit or NUnit.  The frameworks are easy to use and provide support for the major browsers, RIA frameworks and desktop UI technologies.

The list of available frameworks is growing – a lot of them are free, e.g.: Selenium, WebAii, WindMill

Benefits of Functional Testing Frameworks

I see multiple benefits of these frameworks:

  • Developers can now easily write functional tests besides only writing unit tests
  • Functional tests can cover test areas where it is hard to write unit tests for
  • Functional tests can easily be integrated in your Continuous Integration process -> execute them as you execute your unit tests
  • Developers not only think about how components can be made testable for unit testing – but also how the UI can be made testable for Functional Testing
  • BETTER TEST COVERAGE
  • Reuse tests for load-testing and production monitoring

The Future: Go beyond Functional Testing with Transactional Tracing

Functional Testing in the hands of developers promises to bring us more tests that can easily be integrated in the existing automation processes like Continuous Integration. So – is there anything missing? YES THERE IS

  • Root Cause Analysis: you still have to sit down and figure out why certain tests failed. This often requires manual re-runs of the tests with an attached debugger to track down the problem. Too often the problems cannot be reproduced on the developers machine but only on the test machine that was setup with a special environment – which might just not be available right now
  • Architecture Validation: functional testing as well as unit testing only verifies the functional correctness of the tested components. It does not verify if the underlying code is efficient and scalable when executed under load or in production like environments

The next evolutionary step for Functional Testing is to combine Functional Testing with Transactional Tracing. Analyzing the executed transactions from end-to-end down to the invoked components,  across tiers and runtime boundaries allows you to

Speed up Root Cause Analysis

  • you get the dynamic execution path of the tested code as part of the functional test result for each functional test step
  • no need to manually re-run tests as all code-level execution results have already been collected on the target machine where it happened
  • no need to debug through code as the transactional trace contains method level information including execution time, arguments, exceptions, …

Automate Architecture Validation

  • you can analyze the dynamic code behaviour in terms of e.g.: number of sql-queries, number of remoting roundtrips, memory allocations, …
  • you can analyze which components really interact with each other and if the dynamic execution matches what is on the blueprint
  • Automating the analysis of these “Architectural Rules” allows you to let test runs fail in case certain rules have been violated even though the application might be functionally correct
  • identify problems early on that would later cause problems under load or in production

How it works with dynaTrace and WebAii

ArtOfTest offers an easy to use automation framework for .NET. As part of their upcoming 2.0 version (which is currently in beta) it will be possible to register a callback for every request/response that is sent by the browser while executing the test script. This callback enables us to add the dynaTrace HTTP header to the HTTP request allowing dynaTrace to link the original test request with the resulting PurePath. Furthermore we can log the unique PurePath ID that was captured to the unit test output as the HTTP Response header contains this unique ID. This ID can then be used to open the PurePath in question in case a problem occurred.

I will write a more detailed blog entry about how the callback has been implemented and how you can use it. Here are some screenshots that show how to enable a WebAii testing script with Transactional Tracing using dynaTrace and how it can be used to improve Root Cause Analysis as well as to automate Architecture Validation.

Step 1: Writing a test script and enabling the dynaTrace Integration

WebAii test in C#. Enabling Integration with dynaTrace
WebAii testcase in C#. Enabling Integration with dynaTrace in Setup step

Step 2: After running the test – analyze the recorded transactions in dynaTrace

dynaTrace shows transactions with the names used in the test steps
dynaTrace shows transactions with the names used in the test steps

Step 3: Speed up Root Cause Analysis by by looking at the PurePath (=Transactional Trace) that captured the problematic test case

Analyze failed test steps by drilling into the execution details
Analyze failed test steps by drilling into the execution details

Step 4: Perform Architectural Analysis by looking at the PurePath data showing service-to-service interactions, roundtrips to the database, … – all for individual transactions. Automate that process by defining Alerts on Architectural Rules

Architectural Analysis of individual test steps
Architectural Analysis of individual test steps

Conclusion

We have come far with Functional Testing but there is still more that we can do with it. Adding Transactional Tracing into the mix allows you to speed up problem resolution as well as finding problems early on through Functional Testing.