The RUP for Practical Software and Systems Measurement (PSM) V3.0 and the PSM Standalone V3.0 Plug-ins

Previous articles in the series:

• Overview
• Core Implementation
• Testing Processes
• Fixture Setup/Teardown, Test Repetition, And Performance Tests

What's New?

Added a section on Presentation Layer Test Patterns.

Join The Advanced Unit Testing Project!

We're looking for experienced C# developers that can dedicate quality time to a large list of to-do features to help make Visual Test Studio the leader in unit test open-source efforts.  Are you interested in working with cutting edge technology and being a leader in a rapidly growing engineering practice?  Would you like to become known in the unit test field as helping to architect and develop Visual Test Studio?  Then sign up at http://aut.tigris.org/ and contact Marc for information on how you can contribute!

Contents

• Introduction
• Patterns
  • Pass/Fail Patterns
    • The Simple-Test Pattern
    • The Code-Path Pattern
    • The Parameter-Range Pattern
  • Data Driven Test Patterns
    • The Simple-Test-Data Pattern
    • The Data-Transformation-Test Pattern
  • Data Transaction Patterns
    • The Simple-Data-I/O Pattern
    • The Constraint-Data Pattern
    • The Rollback Pattern
  • Collection Management Patterns
    • The Collection-Order Pattern
    • The Enumeration Pattern
    • The Collection-Constraint Pattern
    • The Collection-Indexing Pattern
  • Performance Patterns
    • The Performance-Test Pattern
  • Process Patterns
    • The Process-Sequence Pattern
    • The Process-State Pattern
    • The Process-Rule Pattern
  • Simulation Patterns
    • Mock-Object Pattern
    • The Service-Simulation Pattern
    • The Bit-Error-Simulation Pattern
    • The Component-Simulation Pattern
  • Multithreading Patterns
    • The Signalled Pattern
    • The Deadlock-Resolution Pattern
  • Stress-Test Patterns
    • The Bulk-Data-Stress-Test Pattern
    • The Resource-Stress-Test Pattern
    • The Loading-Test Pattern
  • Presentation Layer Patterns
    • The View-State Test Pattern
    • The Model-State Test Pattern
• Conclusion

Introduction

The idea of unit testing seems to always evoke a strong reaction in people.  For those that buy into the concept, they have unanimously stated that good unit tests are difficult to write, and some question whether the tests they have written were really worth it while others rave about their effectiveness.  On the other hand, there is also a large community that guffaws at the idea of unit testing, especially the concept that "the code is good when it passes the unit tests".  When all the hoopla dies down, unit testing may one day be relegated to the dusty shelf of "yet another programmer tool".  If this fate is to be changed, unit testing has to be embraced by both the community and the tool developers.  The next version of Microsoft's Visual Studio will include tools to automate refactoring.  It seems obvious to me that tools that automate unit test generation would not only address some of the issues concerning maintenance and cost, but would also introduce the concept to a much wider audience.

However, to achieve this acceptance, unit testing must be formalized so that it becomes a real engineering discipline rather than an ad hoc approach that relies on the dubious capabilities of the programmer.  After all, the unit test is supposed to test the code that the programmer writes.  If the programmer writes bad code to begin with, how can you expect anything of better quality in the tests?  Of even more concern is the concept that the unit test should be written first, before the code that is to be tested.  To a certain extent, this implies that not only does the programmer have to consider what the code will do, he/she has to consider how the code is designed.  Both drive the interface.  This is why many people balk at the idea of writing the unit test first--it places them in the uncomfortable position of having to do up front design work without consciously recognizing that this what they are doing.

So, we are faced with a double edged sword.  First, there is no formal unit test engineering discipline established in the community that provides a guide to the programmer and works to ensure some level of unit test quality.  Second, the prerequisite that the design has to be somewhat formalized before any tests can be written causes difficulty for many programmers because they either don't have formal design experience or simply don't like up front design work.  Aggravating this situation is the idea that up front design work can be replaced under the guise of "refactoring".

In order to blunt this sword, two things are needed--a formalization of unit testing by establishing unit test patterns, and the early adoption of object oriented design patterns in the developing application to specifically target the needs of unit testing.  This article will paint a picture of this two pronged solution with some very large brush strokes.  The intention is to whet your appetite and hopefully begin a dialog amongst yourselves that will lead to a more formal unit test engineering process, similar to object oriented design, design patterns, and refactoring.

As you read this article, keep in mind that one of the goals is a tool suite that can be used to automatically generate unit tests, both as a reverse and forward engineering process.  With the latter, it should be possible to generate the method stubs for the code under test.  After all, one of the benefits of unit testing is that it provides the implementer with some documentation as to the expected structure and behavior of the code under test.  Also, to keep this article in the general reader category, there are no code examples.

Patterns

The patterns that I have identified so far can be loosely categorized as:

• pass/fail patterns
• collection management patterns
• data driven patterns
• performance patterns
• process patterns
• simulation patterns
• multithreading patterns
• stress test patterns

Again, let me emphasize that these are broad brush strokes.  From my research, this appears to be quite new territory.

Pass/Fail Patterns

These patterns are your first line of defense (or attack, depending on your perspective) to guarantee good code.  But be warned, they are deceptive in what they tell you about the code.

The Simple-Test Pattern

Pass/fail unit tests are the simplest pattern and the pattern that most concerns me regarding the effectiveness of a unit test.  When a unit test passes a simple test, all it does is tell me that the code under test will work if I give it exactly the same input as the unit test.  A unit test that exercises an error trap is similar--it only tells me that, given the same condition as the unit test, the code will correctly trap the error.  In both cases, I have no confidence that the code will work correctly with any other set of conditions, nor that it will correctly trap errors under any other error conditions.  This really just basic logic.  However, on these grounds you can hear a lot of people shouting "it passed!" as all the nodes on the unit test tree turn green.

The Code-Path Pattern

The Simple-Test pattern typifies what I call "black box testing".  Without inspecting the code, that's about all you can do--write educated guesses as to what the code under test might encounter, both as success cases and failure cases, and test for those guesses.  A better test ensures that at least all the code paths are exercised.  This is part of "white box testing"--knowing the inside workings of the code being tested.  Here the priority is not to set up the conditions to test for pass/fail, but rather to set up conditions that test the code paths.  The results are then compared to the expected output for the given code path.  But now we have a problem--how can you do white box testing (testing the code paths) when the code hasn't been written?  Here we are immediately faced with the "design before you code" edge of that sword.  The discipline here, and the benefit of unit testing by enforcing some up front design, is that the unit test can test for code paths that the implementer may not typically consider.  Furthermore, the unit test documents precisely what the code path is expected to do.  Conversely, discipline is needed during implementation when it is discovered that there are code paths that the unit test did not foresee--time to fix the unit test!

The Parameter-Range Pattern

Still, the above test, while improving on the Simple-Test pattern, does nothing to convince me that the code handles a variety of pass/fail conditions.  In order to do this, the code should really be tested using a range of conditions.  The Parameter-Range pattern does this by feeding the Code-Path pattern with more than a single parameter set.  Now I am finally beginning to have confidence that the code under test can actually work in a variety of environments and conditions. 

Data Driven Test Patterns

Constructing Parameter-Range unit tests is doable for certain kinds of testing, but it becomes inefficient and complicated to test at a piece of code with a complex set of permutations generated by the unit test itself.  The data driven test patterns reduce this complexity by separating the test data from the test.  The test data can now be generated (which in itself might be a time consuming task) and modified independent of the test.

The Simple-Test-Data Pattern

In the simplest case, a set of test data is iterated through to test the code and a straightforward result (either pass or fail) is expected.  Computing the result can be done in the unit test itself or can be supplied with the data set.  Variances in the result are not permitted.  Examples of this kind of of Simple-Test-Data pattern include checksum calculations, mathematical algorithms, and simple business math calculations.  More complex examples include encryption algorithms and lossless encoding or compression algorithms.

The Data-Transformation-Test Pattern

The Data-Transformation-Test pattern works with data in which a qualitative measure of the result must be performed.  This is typically applied to transformation algorithms such as lossy compression.  In this case, for example, the unit test might want to measure the performance of the algorithm with regard to the compression rate vs. the data loss.  The unit test may also need to verify that the data can be translated back into something that resembles the input data within some tolerance.  There are other applications for this kind of unit test--a rounding algorithm that favors the merchant rather than the customer is a simple example.  Another example is precision.  Precision occurs frequently in business--the calculation of taxes, interesting, percentages, etc., all of which ultimately must be rounded to the penny or dollar but can have dramatic effects on the resulting value if precision is not managed correctly throughout the calculation.

Data Transaction Patterns

Data transaction patterns are a start at embracing the issues of data persistence and communication.  More on this topic is discussed under "Simulation Patterns".  Also, these patterns intentionally omit stress testing, for example, loading on the server.  This will be discussed under "Stress-Test Patterns".

The Simple-Data-I/O Pattern

This is a simple data transaction pattern, doing little more than verifying the read/write functions of the service.  It may be coupled with the Simple-Test-Data pattern so that a set of data can be handed to the service and read back, making the transaction tests a little bit more robust.

The Constraint-Data Pattern

The Constraint-Data pattern adds robustness to the Simple-Data-I/O pattern by testing more aspects of the service and any rules that the service may incorporate.  Constraints typically include:

• can be null
• must be unique
• default value
• foreign key relationship
• cascade on update
• cascade on delete

As the diagram illustrates, these constraints are modeled after those typically found in a database service and are "write" oriented.  This unit test is really oriented in verifying the service implementation itself, whether a DB schema, web service, or other model that uses constraints to improve the integrity of the data.

The Rollback Pattern

The rollback pattern is an adjunct to the other transaction testing patterns.  While unit tests are supposed to be executed without regard to order, this poses a problem when working with a database or other persistent storage service.  One unit test may alter the dataset causing another unit test to inappropriately fail.  Most transactional unit tests should incorporate the ability to rollback the dataset to a known state.  This may also require setting the dataset into a known state at the beginning of the unit test.  For performance reasons, it is probably better to configure the dataset to a known state at the beginning of the test suite rather than in each test and use the service's rollback function to restore that state for each test (assuming the service provides rollback capability).

Collection Management Patterns

A lot of what applications do is manage collections of information.  While there are a variety of collections available to the programmer, it is important to verify (and thus document) that the code is using the correct collection.  This affects ordering and constraints.

The Collection-Order Pattern

This is a simple pattern that verifies the expected results when given an unordered list.  The test validates that the result is as expected:

• unordered
• ordered
• same sequence as input

This provides the implementer with crucial information as to how the container is expected to manage the collection.

The Enumeration Pattern

This pattern verifies issues of enumeration, or collection traversal.  For example, a collection may need to be traversed forwards and backwards.  This is an important test to perform when collections are non-linear, for example a collection of tree nodes.  Edge conditions are also important to test--what happens when the collection is enumerated past the first or last item in the collection?

The Collection-Constraint Pattern

This pattern verifies that the container handles constraint violations: null values and inserting duplicate keys.  This pattern typically applies only to key-value pair collections.

The Collection-Indexing Pattern

The indexing tests verify and document the indexing methods that the collection container must support--by index and/or by key.  In addition, they verify that update and delete transactions that utilize indexing are working properly and are protected against missing indexes.

Performance Patterns

Unit testing should not just be concerned with function but also with form.  How efficiently does the code under test perform its function?  How fast?  How much memory does it use?  Does it trade off data insertion for data retrieval effectively?  Does it free up resources correctly?  These are all things that are under the purview of unit testing.  By including performance patterns in the unit test, the implementer has a goal to reach, which results in better code, a better application, and a happier customer.

The Performance-Test Pattern

The basic types of performance that can be measured are:

• Memory usage (physical, cache, virtual)
• Resource (handle) utilization
• Disk utilization (physical, cache)
• Algorithm Performance (insertion, retrieval, indexing, and operation)

Note that some languages and operating systems make this information difficult to retrieve.  For example, the C# language with its garbage collection is rather difficult to work with in regards to measuring memory utilization.  Also, in order to achieve meaningful metrics, this pattern must often be used in conjunction with the Simple-Test-Data pattern so that the metric can measure an entire dataset.  Note that just-in-time compilation makes performance measurements difficult, as do environments that are naturally unstable, most notably networks.  I discuss the issue of performance and memory instrumentation in my fourth article in a series on advanced unit testing found at http://www.codeproject.com/csharp/autp4.asp.

Process Patterns

Unit testing is intended to test, well, units...the basic functions of the application.  It can be argued that testing processes should be relegated to the acceptance test procedures, however I don't buy into this argument.  A process is just a different type of unit.  Testing processes with a unit tester provide the same advantages as other unit testing--it documents the way the process is intended to work and the unit tester can aid the implementer by also testing the process out of sequence, rapidly identifying potential user interface issues as well.  The term "process" also includes state transitions and business rules, both of which must be validated.

The Process-Sequence Pattern

This pattern verifies the expected behavior when the code is performed in sequence, and it validates that problems when code is executed out of sequence are properly trapped.  The Process-Sequence pattern also applies to the Data-Transaction pattern--rather than forcing a rollback, resetting the dataset, or loading in a completely new dataset, a process can build on the work of the previous step, improving performance and maintainability of the unit test structure.

The Process-State Pattern

The concept of state cannot be decoupled from that of process.  The whole point of managing state is so that the process can transition smoothly from one state to another, performing any desired activity.  Especially in "stateless" systems such as web applications, the concept of state (as in the state of the session) is important to test.  To accomplish this without a complicated client-server setup and manual actions requires a unit tester that can understand states and allowable transitions and possibly also work with mock objects to simulate complicated client-server environments.

The Process-Rule Pattern

This test is similar to the Code-Path pattern--the intention is to verify each business rule in the system.  To implement such a test, business rules really need to be properly decoupled from surrounding code--they cannot be embedded in the presentation or data access layers.  As I state elsewhere, this is simply good coding, but I'm constantly amazed at how much code I come across that violates these simple guidelines, resulting in code that is very difficult to test in discrete units.  Note that here is another benefit of unit testing--it enforces a high level of modularity and decoupling.

Simulation Patterns

Data transactions are difficult to test because they often require a preset configuration, an open connection, and/or an online device (to name a few).  Mock objects can come to the rescue by simulating the database, web service, user event, connection, and/or hardware with which the code is transacting.  Mock objects also have the ability to create failure conditions that are very difficult to reproduce in the real world--a lossy connection, a slow server, a failed network hub, etc.  However, to properly use mock objects the code must make use of certain factory patterns to instantiate the correct instance--either the real thing or the simulation.  All too often I have seen code that creates a database connection and fires off an SQL statement to a database, all embedded in the presentation or business layer!  This kind of code makes it impossible to simulate without all the supporting systems--a preconfigured database, a database server, a connection to the database, etc.  Furthermore, testing the result of the data transaction requires another transaction, creating another failure point.  As much as possible, a unit test should not in itself be subject to failures outside of the code it is trying to test.

Mock-Object Pattern

In order to properly use mock objects, a factory pattern must be used to instantiate the service connection, and a base class must be used so that all interactions with the service can be managed using virtual methods.  (Yes, alternatively, Aspect Oriented Programming practices can be used to establish a pointcut, but AOP is not available in many languages).  The basic model is this:

To achieve this construct, a certain amount of foresight and discipline is needed in the coding process.  Classes need to be abstracted, objects must be constructed in factories rather than directly instantiated in code, facades and bridges need to be used to support abstraction, and data transactions need to be extracted from the presentation and business layers.  These are good programming practices to begin with and result in a more flexible and modular implementation.  The flexibility to simulate and test complicated transactions and failure conditions gains a further advantage to the programmer when mock objects are used.

The Service-Simulation Pattern

This test simulates the connection and I/O methods of a service.  In addition to simulating an existing service, this pattern is useful when developing large applications in which functional pieces are yet to be implemented.

The Bit-Error-Simulation Pattern

I have only used this pattern in limited applications such as simulating bit errors induced by rain-fade in satellite communications.  However, it is important to at least consider where errors are going to be handled in the data stream--are they handled by the transport layer or by higher level code?  If you're writing a transport layer, then this is a very relevant test pattern.

The Component-Simulation Pattern

In this pattern, the mock object simulates a component failure, such as a network cable, hub, or other device.  After a suitable time, the mock object can do a variety of things:

• throw an exception
• return incomplete or completely missing data
• return a "timeout' error

Again, this unit test documents that the code under test needs to handle these conditions.

Multithreading Patterns

Unit testing multithreaded applications is probably one of the most difficult things to do because you have to set up a condition that by its very nature is intended to be asynchronous and therefore non-deterministic.  This topic is probably a major article in itself, so I will provide only a very generic pattern here.  Furthermore, to perform many threading tests correctly, the unit tester application must itself execute tests as separate threads so that the unit tester isn't disabled when one thread ends up in a wait state.

The Signalled Pattern

This test verifies that a worker thread eventually signals the main thread or another worker thread, which then completes its task.  This may be dependent on other services (another good use of mock objects) and the data on which both threads are operating, thus involving other test patterns as well.

The Deadlock-Resolution Pattern

This test, which is probably very complicated to establish because it requires a very thorough understanding of the worker threads, verifies that deadlocks are resolved.

Stress-Test Patterns

Most applications are tested in ideal environments--the programmer is using a fast machine with little network traffic, using small datasets.  The real world is very different.  Before something completely breaks, the application may suffer degradation and respond poorly or with errors to the user.  Unit tests that verify the code's performance under stress should be met with equal fervor (if not more) than unit tests in an ideal environment.

The Bulk-Data-Stress-Test Pattern

This test is designed to validate the performance of data manipulation when working with large data sets.  These tests will often reveal inefficiencies in insertion, access, and deletion processes which are typically corrected by reviewing the indexing, constraints, and structure of the data model, including whether code is should be run on the client or the server.

The Resource-Stress-Test Pattern

Resource consumption stress testing depends on features of the operating system and may be served better by using mock objects.  If the operating system supports simulating low memory, low disk space, and other resources, then a simple test can be performed.  Otherwise, mock objects must be used to simulate the response of the operating system under a low resource condition.

The Loading-Test Pattern

This test measures the behavior of the code when another machine, application, or thread is loading the "system", for example high CPU usage or network traffic.  This is a simulation only (which does not use mock objects) and therefore is of dubious value.  Ideally, a unit test that is intended to simulate a high volume of network traffic would create a thread to do just that--inject packets onto the network.

Presentation Layer Patterns

One of the most challenging aspects of unit testing is verifying that information is getting to the user right at the presentation layer itself and that the internal workings of the application are correctly setting presentation layer state.  Often, presentation layers are entangled with business objects, data objects, and control logic.  If you're planning on unit testing the presentation layer, you have to realize that a clean separation of concerns is mandatory.  Part of the solution involves developing an appropriate Model-View-Controller (MVC) architecture.  The MVC architecture provides a means to develop good design practices when working with the presentation layer.  However, it is easily abused.  A certain amount of discipline is required to ensure that you are, in fact, implementing the MVC architecture correctly, rather than just in word alone.

Sun Microsystems has a webpage which I consider is the gospel for the MVC archicture: http://java.sun.com/blueprints/patterns/MVC-detailed.html.  To summarize the MVC pattern here:

It is vital that events are used for model notification changes and user gestures, such as clicking on a button.  If you don't use events, the model breaks because you can't easily exchange the view to adjust the presentation to the particular presentation layer requirement.  Furthermore, without events, objects become entangled.  Also, events, such as managed by an event pool, allow for instrumentation and ease debugging.  The only exception to this model that I have found in practice is that on occasion, a state change in the model might be captured by an event in the controller rather than the view.  Some model changes, such as user authorization, are view-less but end up affecting other aspects of the controller.

The View-State Test Pattern

This test verifies that for a change in the model state, the view changes state appropriately.

This test exercises only half of the MVC pattern--the model event notifications to the view, and the view management of those events in terms of affects on the presentation layer content and state.  The controller is not a factor in this test pattern.

The Model-State Test Pattern

Once we have verified application's performance with the View-State Pattern, we can progress to a more complicated one.  In this pattern, the unit test simulates user gestures by setting state and content directly in the presentation layer, and if necessary, invoking a state change event such as "KeyUp", "Click", etc.

As diagrammed above, the unit test verifies that the model state has changed appropriately and that the expected events fired.  In addition, the view's events can be hooked and verified to fire correctly for the simulated user gesture.  This test may require some setup on the model itself, and treats the controller as a black box, however model state can be inspected to determine whether the controller is managing the model state appropriately.

Conclusion

This article has described 24 test patterns that hopefully bring the technique of unit testing closer to a more formal engineering discipline.  When writing unit tests, reviewing these patterns should help in identifying the kind of unit test to write, and the usefulness of that unit test.  It also allows the developer to choose how detailed the unit tests need to be--not every piece of code needs to be stress tested, nor is it cost effective to do so.

As you can see from this article, we desperately need some tools that generate unit tests automatically.  I'm not much of an advocate of code generates, but I can see that unit testing would really benefit from code generation.  It would all but eliminate the arguments against unit testing based on cost and effectiveness, and could also be used as an application generating tool--given information on the unit test, the code generator can also create the application code classes, structure, and stubs.

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here