One of the most difficult things in any large maintenance project is making sure that by fixing bugs or adding functionality you don’t break things in the application in places you didn’t expect. This is especially difficult when extending or replacing core components. Unit testing is supposed to help you avoid this, but in the bad old days, when men were men and C# was the new frontier, not all systems were developed with testing in mind.
At the moment I’m working, together with a couple of collegues on a such a - let’s call it “legacy rich” - system, the core of which was developed over 5 years ago in .Net 1.1 by two independent teams. It’s since been extended by several generations of developers in several stages, some as in house projects, some fixed price external projects. The result is a thinly veiled minefield for the unwary.
So, how do we deal with this? Retrofitting unit tests is an option in some cases, when the functionality isolated and well defined, but this leaves tens of thousands of lines of code just waiting to trip you up. The only feasible answer is large scale regression testing. Luckily this particular customer has already learned the value of independent testing, but still there is only so much you can get tested in every release cycle. So testing new features while having adequate coverage of the rest of the system is a rather hit-and-miss affair.
The answer we came up with, thanks in large part to my collegue Peter Notenbaert who suggested it and did a lot of the groundwork, is automated acceptance testing. The basic idea was to construct a test bench for the application that controls the UI and runs through known scenarios to check the effect each action has on the UI. Basically running tests on the entire application at once, but using nothing more than the standard unit testing framework. The scenarios we use for this are partly supplied by testers for the regression part, but at the same time we use scenarios based on the user stories to test our new features.
Once a minimum number of tests exist they can be used to verify progress in the project, as well as ensuring the continued stability of the entire system. The idea seems simple enough, but the mechanics are just a little more complicated. When we decided to test systems in this way we had to keep in mind that there are dependant systems that may not cope very well with this. Repeated transactions could cause a lot of trouble if we can’t clean up after every test.
Before diving deeper into the how, let’s just recap over the why:
- We want to black box test as much of the application as possible by only using the UI and checking for results.
- We want to be able to change core functionality in an application that has little or no unit testing in place, or for which the documentation is patchy at best, without introducing nasty side-effects in the rest of the system.
- We want to do this by constant automated regression testing known scenarios checking for any unexpected behavior, thus ensuring stability. Serious bugs are also added to the set so get solved and don’t reappear later on.
- We want to monitor the progress of projects by counting the number of new scenarios that can be completed successfully. These new scenarios will join the set of existing regression tests after the project is finished.
- We want to be able to test complex scenarios that may take several steps, potentially by different actors, in order to complete, handing the achieved result from one step to the next.
- We do not want to impact backend systems that cannot be rolled back after our tests.
- We want to be able to run the test set at regular intervals, and preferably a smaller subset nightly, using nothing more than standard test frameworks and tools.
In the next part I will go into the details of our solution.
By Nick Verschueren, .Net Solutions Architect