Automated unit tests (A.K.A. "developer tests") and functional test (A.K.A. "customer tests") are a cornerstone of many agile development methods (such as eXtreme Programming). The availability of automated, self-checking tests allows developers to be much bolder in how they modify existing software. Almost every test requires some kind of test fixture. The test fixture is everything you need to have in place to run the test. Typically, it consists of objects (in memory or on disk), records in a database, and/or the state of any other depended-on components (or even entire systems or applications). At first glance, the construction of the test fixture would appear to be a rather simple task. But as the size and complexity of the SUT grows, the complexity of managing the fixture grows in proportion. This is particularly true as we move from unit tests for simple entity objects (which have simple state models) or utility objects (which are typically stateless) to component tests for stateful service objects or functional tests for entire applications. This set of patterns describes the key techniques for addressing the issues around test fixture management.
The submitted document is an introductory narrative that refers (via live hyperlinks) to detailed writeups of some 30+ patterns specific to fixture setup (in scope) as well as some other patterns and test smells (out of scope). The author proposes to use a live wiki as the primary means to communicate detailed comments between the shepherd and the author. The author is looking for both format and content validation through the shepherding process. Ideally, the shepherd will have at least some experience writing automated tests using XUnit. (I.e. JUnit, NUnit, VbUnit, RubyUnit, etc..)
This text is the introductory narrative to the Fixture Setup part of a pattern language book on patterns of XUnit test automation. The actual material to be reviewed and shepherded is available on our website at http://testautomationpatterns.com/FixtureSetup.html; only patterns in the category http://testautomationpatterns.com/Fixture Setup Patterns.html are in scope”; all others can be ignored. Underlined phrases in normal font beginning with capital letters are patterns; lowercase are definitions while hyperlinks in italics refer to “test smells” Hyperlinks beginning and ending with ?-marks are links to items that have not yet been written.
The focus of this submission is only the patterns (not the definitions or smells).
Almost every test requires some kind of test fixture. The test fixture is everything you need to have in place to run the test. It is the "before" state of the system; the state in which the system must be before the actual behaviour we are attempting to verify can be induced. Typically, it consists of objects (in memory or on disk), records in a database, and/or the state of any other depended-on components (or even entire systems or applications).
At first glance, the construction of the test fixture would appear to be a rather simple task. Just instantiate the objects, components or records in the right state and be done with it. Right?
This may be true for very simple ?unit tests? that require just one or two objects for their test fixture. In this case, creating the fixture may truly be as simple as instantiating a single object in memory and setting its state explicitly. And if this is all you need, feel free to skip ahead to Building Clean Slate Fixtures later in this chapter.
But as the size and complexity of the SUT grows, the complexity of managing the fixture grows in proportion. This is particularly true as we move from unit tests for simple entity objects (which have simple state models) or utilty objects (which are typically stateless) to ?component tests? for stateful service objects that have complex state. In these cases, the effort to create the fixture goes up considerably and the potential for ?Test Code Duplication? starts to rise dramatically.
As soon as the SUT has persistent state (such as a database), the time it takes to set up the test fixture and execute the tests increases dramatically. This introduces a further constraint into our automated test design: test execution speed. This is particularly true of ?functional tests? for entire applications.
A common reaction to having slow tests is to reuse the same test fixture to reduce the execution time spent building and tearing down the fixture. This is typically done in one of two ways. The most common is to use a Prebuilt Fixture that is set up before any test suites are run. The second is the use of Chained Tests that depend on the order in which tests execute within the test suite to build the fixture up incrementally. Both of these approaches are fraught with perils that makes it harder to achieve robust and maintainable tests.
As the complexity of and consequences of fixture management becomes the predominant complexity factor in our tests, how you manage your test fixtures becomes the single most critical decision you will need to make as you adopt a test automation strategy. Because this is such a large and important topic, we will attack it in three phases.
·
First,
in Fixture Management Decisions,
we'll introduce the fundamental decisions we need to make about the test
fixture.
·
Second,
in Test Fixture Strategies we
will summarize the two key test fixture management strategies as well as
several hybrid strategies.
·
Finally,
in Building Test Fixtures, we
will deal with how you actually construct the test fixture in each of the
strategies.
Decisions, decisions, decisions! So many to make; so little time! The decisions you need to make can be summarized as:
·
Is
the test fixture temporary or persistent?
·
Is
the test fixture private to a single test or shared between tests?
·
Is
the code used to build test fixtures duplicated or reused?
·
Do
you set up the fixture via the SUT's API or via
the back door?
We'll go through the reasoning behind each decision one by one.
The first decision is whether the test fixture will be temporal (custom built for each test run) or more persistent (does it survive between test runs). This is important because it has far-reaching ramifications on how you construct it, whether tests can interact and how tests clean up after themselves.
The next decision is about who (actually, which tests) can see the test fixture that we create. The test fixture can be:
· a Private
Test Fixture that can only be accessed within the test method for which
it was created
· a Shared
Test Fixture that can be used by either all or a specific set of tests
run from a single Test
Runner,
· a Shared
Test Fixture that can be used by all tests run from a single computer
(i.e., several Test
Runners running on the same computer),
· a Shared
Test Fixture that can be used by all tests regardless of which computer
they are being run from, or somewhere in between.
This ?test fixture visibility? can also be some combination of the above if it consist of several parts that are constructed in different ways. A good practice that helps avoid Interacting Tests is Immutable Shared Test Fixture; you ensure that any objects or records that the test intends to modify are part of the Private Test Fixture.
When setting up the starting state of the SUT should you go in through the front door or the back door? In Front Door Fixture Setup, we use the "normal" API of the SUT to set up the fixture. In Back Door Fixture Setup, we bypass the SUT's API and set the state through the back door (usually the database.) Bypasssing the SUT's API has two benefits:
·
It
let us set things up even if the functionality is not yet available via the
SUT's API. This is particularly important when doing highly incremental
development and you want to focus on building "real business
functionality" instead of a ?CRUD? API.
·
It
can be a lot faster than setting things up through the SUT's API because it may
bypass a lot of code. And it may let you put things into states that the SUT's
API may prevent.
But with Back Door Fixture Setup we are coupling ourselves more directly with how the SUT is implemented. This can result in ?Fragile Setup? if the implementation details change (which is one of the hallmarks of ?agile software development?!)
We like to say things ?Once and Only Once?. This applies equally to test code though some would argue that "if it is not important to reuse test code, it is important not to reuse test code".
The sharing of the test fixture should not be confused with sharing of the code used to create the test fixture . Sharing of the code does not necessarily imply that the fixture that the code creates is itself shared. That depends on when the fixture is created and how tests access the fixture. For example, if the code creates some objects that are only accessed via local variables in a Test Method, the fixture would be private to the test method. But if the same objects were then persisted into a database from which they could be recovered by another test being run on a different computer, the fixture could be shared globally. We say could because there are strategies for ensuring that even if the fixture is visible globally, it is not accessed by any tests other than the owner.
Whenever the test fixture is shared between tests, the exact same things (objects, records or systems) are used by many tests. This introduces the possibility of interactions between the tests.
This is a lot to think about! The good news is that a lot of this can be boiled down to two basic strategies for test fixture creation:
·
Just
in Time (A.K.A. "Clean Slate Fixture")
·
Ahead
of Time (A.K.A. "Global ?Test Fixture?")
These strategies represent the two most common approaches and lie at either end of the spectrum. They are supported by more detailed patterns. As you gain experience with these two strategies, you can start making conscious decisions to move away from either pure strategy and adopt a hybrid that exhibits characteristics of both approaches.
As we stated earlier, the choice of test fixture strategy is probably the most far-reaching decision you will have to make as you adopt an automated testing strategy using XUnit. The right strategy could make automated testing a "piece of cake" while choosing the wrong strategy could lead to large amounts of wasted effort and frustration. Forewarned is for-armed!
In this approach, each test creates a Private Test Fixture as it runs. Any objects or records it requires are created by the test itself. Because the ?test fixture visibility? is private to the one test alone, we ensure that each test is completely independent because it cannot depend, either accidently or on purpose, on the output of any other tests that use the same fixture.
We call this "clean slate" because each test starts with a clean slate and builds from there. It does not "inherit" or "reuse" any part of the fixture from other tests or from a Prebuilt Fixture.
The main disadvantage of Clean Slate Fixture is the additional CPU cycles it takes to create all the objects for each test. This can make the tests run slower than an Prebuilt Fixture approach, especially if the test fixture is constructed within a database.
We feel this is best addressed by a two pronged approach:
1.
Construct
a Minimal
Test Fixture (the smallest possible fixture you possibly can) and
2.
Speed
up the construction by using ?Stub Out
Slow? to bypass anything that takes too long to construct.
We have found that, on average, our tests run 50 (yes,
fifty!) times faster when we use ?Stub
Out Slow Component? to replace the entire database with a set of
HashTables so we can do ?In-Memory
Test Execution?. This is because a test doesn't just access the
database when it exercises the system, but it also writes records during
fixture setup and deletes them during fixture teardown. That's a lot of disk
accesses in each test.
There is a lot to be said for minimizing the size and complexity of the test fixture. A Minimal Test Fixture is much easier to understand and helps highlight the "cause and effect" relationship between the fixture and the expected outcome. In this regard, it is a major enabler of Tests as Specification. In some cases, you can make the test fixture much smaller by using Entity Chain Snipping to eliminate the need to instantiate objects on which your test depends only indirectly. This will certainly speed up the instantiation of your test fixture.
There will be times when you cannot or choose not to use a Clean Slate Fixture test fixture strategy. In these cases, you can use Prebuilt Fixture. This approach involves prebuilding the test fixture and then using it over and over again when you run your tests.
The most common example of Prebuilt Fixture is the ?Test Database?. In this approach, a database is populated with some (possibly standardized) test data and all tests are written assuming this set of records. This approach has the advantage that it executes much faster because there is little or no test fixture creation overhead associated with individual tests. The main disadvantage is that it opens the door to a number of smells, both ?code smells? and ?behaviour smells?. Examples of ?code smells? include ?Test Code Duplication?, ?Obscurity Through Verbosity?, ?Mystery Guest? and ?Complex Teardown?. Examples of ?behaviour smells? this approach breeds includes Unrepeatable Test, Interacting Tests and ?Test Run War?.
The ?Mystery Guest? problem is caused by the use of magic values that refer to particular objects in the pre-built test fixture. The test reader cannot tell by merely reading the test how the referenced object affects the outcome of the test without consulting the database. This can be partially alleviated by using a variable with an ?Intent Revealing Name? to hold the magic value so that the test reader is given some inkling about why the test uses this particular object. This doesn't avoid Unrepeatable Tests, Interacting Tests and ?Test Run Wars? but at least we can understand what the test was intended to do when it fails (Emoticon: smiley) . To avoid a ?Test Run War?, each test runner must have it's own copy of the test fixture. This is most commonly done by using a ?Private Database?; either a ?Private Real Database? or ?Private Virtual Database? . The ?Private Virtual Database? can be implemented using built-in database support (?DB Schema per TestRunner?) or by using a ?Database Partitioning Scheme? (i.e., giving each potential test runner it's own set of customers, accounts, etc..)
All these approaches solve the problem by ensuring that tests run from different test runners find different objects even though they are using the same identifier (i.e., the key). To avoid having Unrepeatable Tests, you must ensure that each test leaves the pre-built test fixture in the same state as it found it (i.e., treat it as an Immutable Shared Test Fixture) This is easier said than done because it must do so regardless of the outcome of the test. It must understand what the SUT would do to the state of the fixture when the test succeeds and undo it upon successful completion. If the test fails, it must accurately assess what changes the SUT did make and undo those. This can very easily lead to ?Complex Teardown? which will make the tests very hard to write. ?Data Leaks? are inevitable and just as hard to debug as ?memory leaks? because the tests that fail as a result are not the tests that are leaving behind the leftover test fixture data.
One way to avoid the ?Complex Teardown? is to use Automated Teardown. In this technique, anything that is newly created or modified by the test is registered with the Automated Teardown mechanism so that it can automatically clean it up in the teardown method (which is run regardless of whether the test passes or fails.) Because the Automated Teardown mechanism can be a separate component (well, at least a separate method) and not embedded in each Test Method, it can be tested using automated Self-Checking Tests.
Another problem associated with Prebuilt Fixture is ?Fragile Fixture?. This occurs when tests fail merely because we've added additional objects/rows to an existing test fixture. This will happen if the tests assume they know what is already in the database (including the assumption that the database is empty) and code their assertions based on that knowledge. As soon as someone extends the fixture to support new tests, any tests that "know" what used to be in that part of the fixture will likely fail. A common strategy for dealing with this is to use ?Before and After Delta Assertions? for verifying the expected results. First, the test queries the results before adding its own objects. Then it adds the objects and queries again. Then it removes the original results from the new results to cancel the impact of any objects/records that already existed. This should handle the straightforward cases but it does increase the complexity of the tests by adding an extra step before the ?test fixture setup? phase of the test. It works best when the newly added test objects are created from scratch because adding existing objects could be affected by changes to the predefined test fixture.
The two strategies we've just described are obviously stereotypical; not all projects will be able to use just one or just the other. What kinds of hybrid strategies make sense? So far, we've focused on two extremes, building the test fixture from scratch in each test, or pre-building the test fixture well ahead of time.
A third option is to share the test fixture between a number of tests, but only create it when the first of a group of tests is executed. This reduces the amount of effort spent building the test fixture while reducing the likelihood of Unrepeatable Test and maybe even ?Test Run War? by rebuilding the test fixture before each run. Unfortunately, it doesn't do anything to avoid Interacting Tests but we can't have everything!
The key decision we need to make is how we will trigger the construction of the test fixture. Let's assume that we are using a Fixture Holding Class Variable to access the fixture from a Test Method; when is the class variable initialized? The most obvious choice to the experienced object-oriented developer is to use Lazy Fixture Initialization to build the test fixture the first time any test wants to use it. This works fine for creating the test fixture. But what if our fixture setup involves something which needs to be undone when the test run is finished? How do we know all the tests have run so that we can tear the fixture down? Even the teardown method doesn't help us because it gets called after each Test Method in the TestCase class so using it would result in an unintended Clean Slate Fixture approach. We could defer the teardown until the next time the test is run ?Lazy Teardown? but this only works if we can still find all the objects that were part of the test fixture. If we've lost the references to the objects and they won't be garbage collected by ?Builtin Garbage Collection? (e.g., connections to a database), we need to ensure that they get cleaned up at the end of the current test run before the variable(s) holding the references go out of scope. What we really want to do is to place "book ends" around the entire test suite. Luckily for us, there is a ?design pattern? that achieves just this purpose: ?Decorator?. We simply create a Test Suite Factory that creates our test suite and installs it into the appropriate Fixture Setup Decorator. In the decorator's setup method we execute the code to construct the test fixture; in the teardown method we tear it down. Each method is guaranteed to be run once and only once for each we run time the entire test suite it decorates. Of course, if we have multiple Testcase Classes in our test suite, they won't be able to access the same class variable unless we've created a Test Fixture Singleton to provide access to the test fixture (and a ?Test Database? is just one such example.)
So far so good. But what happens if we want to run only a subset (or even just one) of our tests? If we don't need to clean up the fixture, we can use Lazy Fixture Initialization within the tests. But if we need to tear down the fixture after each run, we need to decorate it with the same Fixture Setup Decorator to ensure that the test fixture is set up. In most variants of XUnit this will require a separate Test Suite Factory for each such set of tests just to instantiate the test suite and wrap it in the Fixture Setup Decorator. One other alternative is to always run the tests from within the ?Decorated Test Suite? but to use the "drill down" capability offered by the Graphical Test Runner in JUnit and some other XUnit variants to chose the test to run from the decorated test suite. The challenge here is to ensure that the Fixture Setup Decorator is still called right after the "Run" button is pressed. We've solved this problem in the past by using ?Pushdown Decorator? to ensure that each test in the suite is wrapped in it's own copy of the Fixture Setup Decorator.
Since the main problem with the Shared Test Fixture approach is the possibility of Interacting Tests and ?Test Run Wars?, why not just agree that certain objects in the test fixture are immutable and have tests use the Clean Slate Fixture approach for any objects they need to modify? This gives us most of the speed of the Shared Test Fixture with at least some of the robustness of the Clean Slate Fixture approach.
The main problems lies with understanding what we mean by ?immutable?. A practical definition should be based on why we have chosen to use an Immutable Shared Test Fixture in the first place: tests should not be affected by the presence or absence (whether it depends on the passing or failing) of other tests. This means that if we do something to the Immutable Shared Test Fixture and another test fails (or passes) unexpectedly, we have modified that test fixture. Similarly, if we make a change to it within a test and no other tests fail, that is not considered a mutation. Of course, we could write a test tomorrow that inadvertently depends on this change and will fail if the first test isn't run.
The bigger problem lies with the definition of ?change?. For example, if we add several objects (say Flights) that refer to an ?immutable? object (say, Airport), are we violating it's immutability? Let's apply our little test:
Assume we have a test that verifies the results of the findAllFlightsFromAirport method. It creates some new Flights, associates them with some Airports in our Immutable Shared Test Fixture and calls this method. It then verifies that the returned collection only contains the flights it had associated with the airport. But we've added several flights to this airport already so the collection will contain those flights also. The test will fail because the method returns more airports than expected. This tells us that even just associating objects with an immutable object is a no-no.
Now, this might all seem pretty obvious when we are dealing with objects that hold collections of other objects. But when the fixture is stored in a relational database, we don't even have to modify the Airport table to associate additional flights with it. The rows in the Flights table would hold the Id's of the Airports they connect. By adding more rows to the Flight table, we are modifying the Airport! This is not to say that this hybrid approach is not workable; you just need to make sure everyone understands what constitutes a change (or mutation) to an Immutable Shared Test Fixture and work very diligently to avoid them.
So you've decided which fixture setup strategy you want to use. Congratulations! Now what?
Next come the decisions around how to go about actually constructing the test fixture. The patterns we use to do this depend somewhat on the strategy we have chosen. But possibly a bigger consideration is the avoidance of ?Test Code Duplication?.
In general, the test fixture can be constructed using one of several coding techniques: Inline Setup, Delegated Setup, Implicit Setup or some combination of the three..
In Inline Setup, the test does all the work within the body of the Test Method either by calling the appropriate SUT methods to construct the objects (Front Door Fixture Setup) or by inserting the necessary records directly into the database (Back Door Fixture Setup). Think of Inline Setup as being the do-it-yourself approach to fixture creation. The main drawback of Inline Setup is that it tends to lead to a lot of ?Test Code Duplication? which in turn leads to ?Fragile Tests?. If the work to create the fixture is complex, it can also lead to ?Complex Setup? and all that test fixture construction logic can obscure the intent of the test (?Obscurity Through Verbosity? ).
In the Implicit Setup approach, all the fixture creation logic is put into the setup method of the ?TestCase Class?. As a result, every Test Method in the test suite shares this fixture setup logic; this has the down-side that you can only do the lowest common denominator part of the setup here. That is, only the setup which won't cause problems in any of the tests can be put into the setup method. Another issue is that it can make the tests harder to understand because you cannot see how the preconditions of the test (the test fixture) relate to the expected outcome. Having said this, some people like to deliberately group their Test Methods based on a common test fixture; hopefully, they use ?Intent Revealing Names? for these Testcase Classs.
A third issue is that you are forced to use instance variables (rather than ?local variables?) to hold references to any objects that are constructed in the setup method and which are needed either when exercising the SUT or when verifying the expected outcome. These instance variables act as ?global variables? between the parts of the test and come with all the problems that global variables introduce. But as long as you use instance variables rather than class variables, the test fixture will be newly constructed for each test case in the test class. (This can lead to ?Slow Tests? if you build a ?General Test Fixture? (the opposite of a Minimal Test Fixture). If you use a class variable to hold the object and you use ?Lazy Initialization? to populate the variable, you have crossed over the line into the world of Shared Test Fixtures because later tests within the test suite may reuse the object(s) created in earlier tests and thus may become dependent on the changes the other test (should have) made to it.
Our preferred approach fixture setup is Delegated Setup because we find it strikes a nice balance between allowing each test in the Testcase Class to set up the fixture as it sees fit by calling the appropriate Creation Methods without the tests becoming bloated with fixture creation code. It also leaves us with more choices for how to organize our tests; we don't have to group tests based on fixture requirements. A common application of this approach is when writing input validation tests for SUT methods that are expected to validate the attributes of an object argument. We build objects with One Bad Attribute by first calling the Creation Method and then replacing one attribute with an invalid value that should be rejected by the SUT. Similarly, we might create an object in the correct state by using a Named State Reaching Method.
Creation Methods with ?Intent Revealing Names? make the test's preconditions readily apparent to the reader while avoiding unnecessary ?Test Code Duplication?. We achieve this by creating a set of test fixture setup building blocks in the form of Creation Methods that can be called from tests. These methods are named using an ?Intent Revealing Name? style thus allowing the test reader (and writer) to focus on what is being created without being distracted by how it is done.
One of the goals of these Creation Methods is to remove the need for every test to know the details of how the objects it requires are created. When a test doesn't care about the specific identity of the objects it is creating, we use Anonymous Creation Methods. They generate any unique keys required by the object being created. This Distinct Value Generation guarantees that no other test instance that requires a similar object will end up using the same object as this test. This prevents many of the behaviour smells including Unrepeatable Test and ?Test Run War?.
When a test does care about some of the attributes of the object being created, we use a Parameterized Anonymous Creation Method; the method is passed any attributes that the test cares about (that are important to the test outcome) leaving all other attributes to be defaulted by the implementation of the Creation Method.
The various Creation Methods can be kept as private methods on the Testcase Class, promoted to a ?Testcase Superclass? or moved to a ?Test Helper?. The "mother of all creation methods" is Object Mother. This strategy-level pattern describes a family of approaches that centers around the use of Creation Methods on one or more ?Test Helpers? and may include Automated Teardown as well.
Some people prefer to ?Reuse Tests For Fixture Setup?. That is, they call other tests directly within the setup part of their test. This is not an unreasonable thing to do as long as it is readily apparent to the test reader what the other test is setting up for this test. Unfortunately, very few tests are named is such a way as to convey this intention. So, if you value ?Tests as Specification?, you may want to consider wrapping the called test with a Creation Method with an ?Intent Revealing Name? so that the intent would be much more clearly expressed to the reader of the test calling the Creation Method.
For a test fixture to be shared, it must be built before a Test Method needs it. This could be as late as right before the Test Method's logic is run, when the test suite is run, or at some time earlier. This leads us to the three basic patterns of Shared Test Fixture creation.
Some people prefer to ?Reuse Tests For Fixture Setup?. That is, they call other tests directly within the setup part of their test. But this does not necessarily result in a shared test fixture if a Clean Slate Fixture approach is being used by the called tests. An approach that definitely does result in a shared fixture is Chained Tests because each test relies on the tests that run before it in the test suite to set up their fixture. Of course, the order of test execution becomes very important in this approach and the tests cannot be run by themselves (i.e. they are ?Lonely Tests?.)
Another variation on this same theme is to include a ?Fixture Setup TestCase? at the beginnining of a test suite. The "Test Methods" set up the fixture that is then used by the other tests in the suite but because it looks just like any other test, the Test Runner doesn't need to be aware of it.
If we are happy with creating the test fixture the first time any test needs it, we can use Lazy Fixture Initialization to create it as part of running the first test. Subsequent tests see that the fixture already exists and reuse it. But because there is no obvious signal that the last test in a test suite (or Suite of Suites) has been run, we won't know when to tear down the fixture after each test run.
If we need to be able to teardown the test fixture after running a test suite, we can use a Fixture Setup Decorator to bracket the running of a test suite with execution of fixture setup and teardown logic. We can then guarantee that a new fixture is built for each test run. This ensures that the fixture is not left over to cause problems with subsequent test runs.
The third option is to build the fixture well before the tests are run -- a Prebuilt Fixture. The fixture itself can be built a number of different ways including running a Data Loader (separate fixture setup program), running a ?Fixture Setup TestCase?, another application (either manually or via remote control) or even a database population script.
Regardless of which way you choose to build the Shared Test Fixture, the tests need a way to find the test fixture they are to reuse. Common techniques include the use of Fixture Holding Class Variables or Finder Methods on the Testcase Class itself, on a Test Fixture Singleton, or on a ?Test Helper?.
Copyright © 2004
Gerard Meszaros