Improved Software Development by Unit Test Automation

Copyright © 2001, 2002 Harald Wabnig
email: wabnig@testready.net
homepage: www.testready.net

Abstract

Unit testing is in many projects a task which is performed occasionally by programmers during coding of functions. The main purpose there is to help the programmer to assure that (small) code fragments actually work as he assumes them to. The unit tests are usually not a planned or controlled task and no actual ‘deliverables’ or documents are generated by the task. Hence, only little attention is put into unit testing from the sight of the entire development process.

This paper elaborates that putting more effort into unit testing - up to unit test automation - can significantly improve the software development and maintenance process. It is described what units are worth to be considered and how unit test automation can be set up in a way that faster unit development in better quality (besides many other benefits) can be expected.

Improved Software Development by Unit Test Automation

1 Introduction

Unit testing usually only plays a small role in the software development process. Indeed, it is in practice often a task which is occasionally applied by programmers for their own internal use. Unit testing is not considered an important task in the development process; it is not planned and no resources are assigned to it; no “deliverables” are expected from this task.

Despite this unit testing can provide much more helpings. If unit testing is considered seriously many improvements in the software development process can be expected. Among others, it can make complex programming tasks much easier, can prevent errors in the early development phases, allows faster development in better quality, improves documentation in many ways and enlarges flexibility and speed of error correction - as maintainability in general - substantially.

Nevertheless, usual the effort to attain unit testing at this advanced level is often seen to be too big to be practical. It is true, that unit testing – especially when it is elaborated to become organized and finally automated – is a task which is actually very hard to attain. There are several important things to be obeyed to make unit testing a technical and economical success.

2 Automated Unit Testing

Unit testing is a quite common term. It means that a portion of program code is checked in isolation that it works properly. The term of automation in this context can be seen as the usual term for automating testing (i.e. the automatic execution of test cases and the comparison of the test results to given reference results).

So the term “automated unit testing” means the possibility to automatically execute test cases for a unit of program code to check whether the test results match given reference results.

Today successful test automation is mainly related to the testing of dialog based functionality. There the approach is to capture and replay mouse and keyboard user inputs although script coding recently became more important because of its better maintainability.

As to general test automation the efforts for test automation have to be outweighed by the efforts to automate and maintain the automated tests compared to the efforts for manually testing (considering the number of times the tests are performed).

But besides this, there are several other points which can justify additional efforts for test automation. Among others these are the time needed to perform the test cases manually, the availability of qualified testing staff, the visibility of program internal details to a manual tester, the risk of a failure and the necessity for short error correction times.

As the effort for test automation is large the definition of “unit” can be done more closely. Dialogs will not be considered units in the course of this paper. There are easier possibilities to test dialogs (for example by manual testing) and there are already viable ways for automating dialog testing as stated before. Furthermore, small units with low complexity are usually not worth the effort. So the focussed units are large portions of program code with substantial and complex functionality. Complexity can be comprised by complex underlying business processes, by difficult numerical calculations, by difficult data flow (including database accesses) or by a vast number of input and output parameters (including database accesses and online interfaces).

Furthermore, the test cases for such units should have a close relationship to business processes to allow interpretation of the test cases by field experts or the customers.

To successfully implement automated unit testing many technical topics have to be considered. As only units of the program code are tested many other important things have to be substituted in the testing environment. These are for example dialogs for editing database contents (the real dialogs can be erroneous or not yet finished; usually the real program will also not allow to define illegal values which prohibits the definition of test cases for special error situations). So the programmer has to develop additional functionality to set up a test environment for unit testing. It is very important that this additional programming can be done as easy and fast as possible. To do this, a special frontend middleware specific for the testing demands should be used. For example, for testing it is viable to use a simplified optimized frontend middleware which automates the necessary coding parts for defining data editing dialogs to a great extent. Yet it is important that the dialogs for defining test cases are comfortable to make the usage of the testing environment as easy and fast as possible.

If the application is based on a backend middleware (e.g. to standardize database accesses) it often makes also sense to substitute the real backend middleware by a special testing version. As automated testing requires also special extensions it is often too risky to extend the complexity of the production middleware by testing extensions. Also performance degrades will usually not be tolerable. Flexibility will also be a cause why a special testing (frontend and backend) middleware should be used. The testing middleware can easily be changed as it only affects the internal testing process. Changing the real middleware is risky and will only happen as seldom as possible. Testing extensions in the middleware can for example be the support for the generation of test case documentation including database contents.

Using a special testing middleware should allow to develop a testing environment for the given program unit without too big efforts in a relatively short time. That allows to perform manual unit testing with full control of input and output parameters. But to proceed from manual unit testing to automated unit testing some more effort is required. Again, to keep this effort as small as possible the testing middleware should contain special facilities. These facilities will basically consist of data(base) load and save functions to store the data(base) settings of the different test cases.

It is also preferable if the database is dedicated to testing. Testing has other requirements to the database than production has. For example database security options are only rarely needed for unit testing purposes. Instead, performance is important. When test cases are executed the database has to be loaded to the initial state, do the processing, compare the results with the stored reference test case results (again, the comparison of the stored results should be supported by middleware facilities). Therefore the database processing mainly determines the execution speed of test cases which is important because complex functionality requires many test cases. Another advantage for using a specific database is that it is possible to reduce the effort for database changes. Simplified organizational processes for database changes can be applied in the test environment which would not be allowed in a production environment. Usually a company depends on a stable database because database changes often affect many other tasks in the project. So having a dedicated test environment makes database changes much less critical as it has only local effect.

To automate testing it is always a requirement to have the test cases (and the expected results) stored in some kind of database. Furthermore, it is necessary to organize them so that it is possible to access the test cases in an easy and fast manner. Yet, this is usually hard to attain. The unit test environment has to provide functionality to add, change, delete, copy, move and group test cases. Hence, again it is important that there is support for the developer who has to create the unit test environment. Luckily, there are only small differences in this functionality among different unit test environments. So it is usually possible to use a template test environment and only adapt the database and the specific dialogs. Furthermore, one developer should be able to use a single test environment for many units (although the units should be belonging to a similar application area). The test environment template should also provide basic support for this approach.

Another important thing is the need for having more than one version of the program. This can be actually different releases of the product but also different builds of the same release. The testing environment should be able to cope with the different program versions and be able to compare the results of the different versions against each others. Again it is a great advantage to have a special testing database because it is in this case possible to (mainly) automatically generate compatibility code for allowing to compare program versions which are relying on different databases (i.e. more/less tables or table attributes or even changed attribute domains). Furthermore, the testing database can differ from the production database if attributes or tables only used in previous versions are already deleted in the production database. In this case the old program versions will not be executable with the production database.

For unit test automation - as for test automation in general - the most tedious task is to maintain the existing test cases. Changes of the test database and the actual “correct” test results will occur quite frequently. Database changes - but also error corrections and changed or new functionality - will lead to differences in the stored database contents. If many test cases must be manually adapted the unit test automation task will get too time consuming and expensive. So it is very important that most of these maintenance tasks can be done automatically using facilities of the testing middleware or special functionality included in the test environment.

It is necessary that all the differences that are found in the test case results are presented to the user in detail. So the user can decide whether the results are correct or not. The test environment should then allow the user to define result changes that are valid because of functional changes in the application. It must be possible to automatically adapt the results of the test cases.

3 Benefits from Automated Unit Testing

Automated unit testing includes the possibility to perform regression tests. So the general benefits of regression testing are also available when automated unit testing is available.

Important benefits in this context are:

Implicit learning.
If all internally or externally detected errors are covered in the test cases these errors can not occur twice without notice.

Implicit documentation.
If an important error is corrected in the test environment the related test cases can be provided to the customers to show them that and how the error has been corrected.

There are several other benefits which will be discussed in more detail in the following sections.

3.1 Testability

Usually the product is tested at the end of the development process (waterfall model) before the product is shipped to the customers. With this approach often only little concern is put into the topic of product-testability during program construction. Hence, testability of complex functions is usually only given to a limited extent. Especially for complex functionality it is often not possible to see many important internal details at the application interfaces (e.g. intermediate results, internal/online interfaces are often not directly testable). From the results that are presented it is then very hard (and therefore expensive) to detect the cause of an error or it is even not possible to see the error in the testing environment at all.

Another cause for bad testability is that complex functions usually produce so many results that it is hard to verify that the results are correct. If results are calculated by complex logic it is often also time consuming to check that the result values are right. In general, it is necessary to know the very details of the checked unit to know what the correct result must be. It will be hard to find testers that know all these required details. It is at least time consuming to instruct the testers into the functionality. After a time the knowledge will get lost either because it is forgotten or because testers leave the project. So having test cases with reference results will help substantially as the result can be checked by experts once and the knowledge is conserved further on.

With the unit testing testability is provided by the program developer who constructs the (unit) test environment. Testability as a “by-product” is so no longer a concern of its own. The unit test environment will at least exhibit the relevant unit input and output parameters and (as necessary) can be extended to also show more details (e.g. intermediate results).

3.2 Software Development Speed

Development speed is often not found satisfactory in software development projects. Time schedules are experienced so tight that many projects are either late or not finished at all. Of course, a longer development time always goes along with (much) more development costs. By the way, a fast “time to market” is in any way a important advantage against competitors.

Here the main influences which hinder fast development are listed:

(User) Requirements are not specified clearly enough. Code construction is done under false assumptions. So the development process - beginning by the corrected requirements - has to be redone.

Coding errors are only found late at the end of the development process in the product testing phase. So time consuming and expensive error corrections are necessary.

Information is not properly propagated. Missing, incomplete or wrong/old documents and hindering project organizations (persons and processes) can be the cause.

Complex functionality are hard to develop and hard to (make) test(able).

Unit test automation can help in the following ways:

Having test cases in early development phases (by building prototypes for specific requirements) allows to check (customer) requirement specifications very early. This will significantly reduces project risks. These prototypes and test cases can often be directly used as basis for the real program development.

Having business process oriented test cases helps to concentrate on the main business processes of the customers. So it is at least guaranteed that the main business processes are well tested which allows earlier shipping of the product even if there are minor bugs left in the product.

Using test cases almost eliminates the risk that adding new functionality destroys already tested program functionality.

Errors are prevented, errors are found earlier, hence less time is required for correcting errors and less time is required to perform the final product testing before shipping.

The complex functionality in the product working properly decreases also the time needed for the development of other product functionality because less time is spent searching for errors which resides in other parts of the program code.

Better documentation increases knowledge about complex functionality for other developers as well as for the customers which might need to integrate the product into their software environment (especially interfaces requires detailed documentation).

Earlier documentation based upon the common usage of the product reflected in test cases and program prototypes reduces the time to (manually) write the required user documentation.

Development based upon test cases simplifies coding significantly and hence speeds up code construction. Using existing test cases, it is immediately possible to check if new program code destroys existing functionality and which functionality it is.

Doing coding and testing together substantially increases interactivity of the coding process (like the availability of integrated development environments with editor, compiler and debugger does). Having direct response to coding and very often eliminates the need to rethink already done coding as errors are reported weeks or months later by the testing group. Seeing that a new functionality is working properly also increases motivation as project progress becomes immediately visible to the people working on the project.

3.3 Program Documentation

The writing of a program documentation usually starts too late and hence delays shipping of the product or reduces the quality of the product. The writing of a good program documentation requires people that know the details of the program and its intended usage. But this information is usually only available after a requirement specification has been refined into a programming specification. To finish the user documentation the shape of the final program dialogs and the actually shipped functionality is required.

All program changes must be reflected in the documentation. If dialogs are changed the associated user program documentation must be corrected. If this frequently happens it gets time consuming (especially in late development phases when the documentation is already printed).

In general, detailed documentation of complex program functionality is even more time consuming to construct and harder to write because it needs a deep insight into the program details. It is not easy to find the staff to do the documentation as documentation needs to be accurate and detailed but also simple and readable.

Besides the user program documentation it can be helpful or even necessary to deliver documentation of test cases to the customers. This documentation is even more hard and time consuming to write than the user program documentation. If complex functionality has to be documented many test cases have to be described each with a lot of details. Besides the tedious task of writing this documentation it is nearly impossible to manually maintain the documents if changes affect the detailed test case documentation. But also reading the test case documentation and understanding the meaning of the test cases gets hard and time consuming. Hence the usage of the newest technology - like hypertexts in internet browsers - is an absolute must.

Using unit test automation can help in the following ways:

When program development is done on the basis of test cases these test cases can be used as a basis for the user program documentation. The test cases include the typical business usage and show earlier what the program functionality will be. Hence, writing the documentation can be started earlier.

Completely finishing program functionality one by one allows to generate the associated documentation by the way if the responsible person is closely integrated into the development process. Finishing a program functionality also reduces the risk that an incomplete or false program state is taken as the basis for the documentation.

Reducing errors (especially in early development phases) also helps in writing documentation. Error corrections which requires to rewrite the documentation (e.g. dialog corrections, changes in requirements) will happen less frequently.

Generating test case documentation automatically makes test case documentation for complex functionality feasible and maintainable. Using internet browser technology allows to generate documentation which is using hypertext facilities to quickly move to the information of interest. Generation of good layouts for data helps to perceive the test case intention.

3.4 Maintenance Considerations

Developing by means of a fully elaborated set of the important test cases allows to very quickly check whether an error correction did destroy an important functionality and which one it is. On this basis the development of new functionality as the correction of detected errors gets much easier and the associated risks are significantly reduced (especially in production versions).

Having such a set of test cases conserves the knowledge about the complex functionality. So people who are not originally involved in the development of the functionality can do the maintenance task much easier.

Furthermore having defined the major business processes in detail allows to rebuild the whole complex functionality and make a cleanup of the application feasible because the functionality is fully documented in the test cases.

If it is important that a product is compatible to old program versions it is very helpful to have the relevant test cases which can be executed in the old and in the new version and its results be automatically compared.

3.5 Improvements to the Software Development Process

Development based on test cases offers many advantages. In some kind it can be seen a further step to the building of integrated development environments (which currently consists of editors, compilers and debuggers). Accordingly, the development environment is extended by a tool which allows the definition of test cases that can be automatically regression tested. Integrated environments with testing tool support then further increase interactivity as the full unit program development (concept, construction, testing) is now completed covered inside this environment. Immediate response about the proper functionality allows immediate error correction.

Especially in large projects the development based on test cases which are made testable in such an integrated unit development environment allows to build complex functions without the need to wait for other modules working properly. The necessary interaction with other developers is reduced other developers can work more concentrated on their program parts. They are not hindered by late or non functional programs which are required by their programs. Only locally tested modules are inserted into the (daily) program builds hence the program builds are more stable and integration testing can be done more quickly. Even the concept of test cases in an integrated environment increases the development process as the necessary testing on a global host system where testers interfere with each other is significantly reduced.

The development process is also improved because the test cases document the typical usage of the program units and the test case development methodology requires the people who are concerned with the program unit to work more closely together. The knowledge about the requirements and the business processes has to be used to define the test cases, the knowledge about the program and the programming skills have to be used to make the functions testable in the environment and the testing skills have to be used to systematically construct the test cases for appropriate coverage of the functionality in the test cases. Documentation skills can be required to document the test cases. Closely working together generates synergy effects because knowledge and skills are distributed.

The development of test cases as the basis for development with the gathered skills allows to detect errors much earlier and prevents errors. So the product can be shipped earlier with less errors and development speed is increased significantly and development costs can be reduced accordingly. With common development approaches it is often hard to perform testing at early development phases.

When critical requirements have to be developed it can be wise to build a (unit testable) prototype and delivering test cases to the customer. If the test cases cover the main business processes the customer can check the very detailed information contained in the test cases the project risk can be significantly reduced for both sides. Based on the prototype the required development time can be much more accurately estimated and hence the project schedule is improved.

Furthermore, the possibilities to control the development progress are much larger internally and externally. So for example the risks associated with outsourcing can be reduced significantly. But also the customer can get much more confidence in a timely product release if such controlling is provided. This may well be an argument for a customer to choose the software company that offers him such possibilities.

4 Conclusions

This paper has discussed how automated unit testing can help in the software development process. Several important aspects of getting automated unit testing to work have been elaborated.

It has also been stated that significant initial efforts for building a dedicated unit test environment are meaningful to make automated unit testing a practical approach. Having such a dedicated environment allows to overcome the severe test automation problems like test case creation and test case maintenance for complex backend programs.