A better approach is to write assertion code which automatically checks program behaviour; this code is often based on test frameworks such as JUnit. However, writing assertion code can require a lot of effort, especially for data-intensive programs. It can also be difficult to believe that assertion code is really working without additional printing or debugging.

Tracetest combines the ease and feedback of printing output with the automation of assertion checking.

Instead of writing assertion code you use Tracetest’s simple API to print the program output. The first time you run the test, Tracetest captures the output in a “baseline” file. On subsequent runs, Tracetest captures the new output in a “trace” file and reports any differences with the baseline. If the differences are intentional, you can update the baseline. If not, you’ve found a bug!

Tracetest is easy to use and can dramatically reduce the amount of test code you need to write. It can scale from the simplest unit test to the most complex regression test; all you need to do is output a representation of your program's behaviour.

Tracetest requires Java 1.4 or higher.

Two minute introduction

See here for a simple example of how Tracetest can be used with JUnit and the Eclipse IDE. Please read this even if you don't use Eclipse (or JUnit) since it introduces some important concepts.

Downloads

Using the Tracetest API

Javadocs for the Tracetest API can be found here. This section gives an overview of how the API is used.

The main entry point to the Tracetest API is the net.pllu.tracetest.Tracetest class. This contains two public methods:

• public static TraceOutputStream getOutputStream() throws TracetestException

• public static TracePrintStream getPrintStream() throws TracetestException

When you call the methods above, Tracetest needs to tell the TraceOutputStream/TracePrintStream instance where to look for trace, baseline and filter files. To do this, it determines the name of the calling class and method. It then tries to find the source code of the calling class as follows:

• Determine the directory containing the class file (classdir)

• Convert the fully qualified name of the calling class into a path fragment (pathfragment), e.g. com/mycompany/MyClass.java

• Recurse up the parent directory hierarchy of the classdir. If parentdir/pathfragment exists, the source file has been found. Note that this works when the source and class files are in the same directory.

• Recurse up the parent directory hierarchy of the classdir, looking for an Eclipse .classpath file. If found, use this to determine the project's source folders. Iterate over these in the order they appear in the .classpath file; if sourcefolder/pathfragment exists, the source file has been found.

• Determine if the SOURCEPATH system property has been set (see here). If so, iterate over the sourcepath entries; if sourcepathentry/pathfragment exists, the source file has been found.

If the source file cannot be found, a TracetestException is thrown by the Tracetest.get*() methods.

net.pllu.tracetest.TraceOutputStream is a subclass of java.io.OutputStream and net.pllu.tracetest.TracePrintStream is a subclass of java.io.PrintStream. You can write output to them as you would to any other OutputStream or PrintStream.

Both of these classes implement the interface net.pllu.tracetest.DiffOutputStream. We will refer to them as DiffOutputStreams.

It is important to remember that each DiffOutputStream instance writes to a trace/baseline file which is specific to the class and method that created the instance. If this method calls another method, and you want to continue writing to the same file, you must pass the instance. For example:

public void testBlah() throws TracetestException {
    ...
    TracePrintStream out = Tracetest.getPrintStream();

    createData(out);
    ...  
}

public void createData(TracePrintStream out) throws TracetestException {
    ...
    out.println("Created data " + data);
    ...

}

Trace and baseline files use the following naming convention:

<classname>.<methodname>.trace
<classname>.<methodname>.baseline

Where classname and methodname are the names of the class and method which called the Tracetest.get*() methods.

Baseline and trace files are created and deleted as follows.

• If a test is run and no baseline file exists, a new baseline file is automatically created.

• To update a baseline, manually delete the baseline file and run the test again.

• If a baseline file exists, a trace file will be automatically created.

• If there are no differences between the baseline and trace files when close() is called, the trace file is automatically deleted.

If you call the Tracetest API from a non-public top level class, or an inner class, see here.

The net.pllu.tracetest.TracetestUtils class contains some useful methods for writing output, for example to write a file's contents or to log a "banner".

DiffOutputStreams must be closed with the close() method before you can determine if there were any differences between the trace and the baseline.

You can then detect differences by using the differences() method. For example, in JUnit,

public void testBlah() throws TracetestException {
    ...
    TracePrintStream out = Tracetest.getPrintStream();
    ...
  out.flush();
    out.close();
    boolean diffs = out.differences();
    assertTrue(!diffs, "Differences between the trace and the baseline!");
}

Note that differences() will throw an IllegalStateException if called before close().

It it often more convenient to to use the closeAndCheck() method, which flushes the stream, closes it, then throws a TracetestException if there are any differences.

public void testBlah() throws TracetestException {
    ...
    TracePrintStream out = Tracetest.getPrintStream();
    ...
  out.closeAndCheck();
}

The exception contains the name of the trace and baseline file.

Running Tracetest

Tracetest requires Java 1.4 or later. The classpath must include tracetest.jar.

Tracetest must be able to find the source code of the classes which use its API. It can do this automatically if the source code is in the same directory or if the source code is packaged in an Eclipse project. Otherwise the SOURCEPATH system property must be set on the JVM command line (using the -D switch), for example:

java -DSOURCEPATH=c:\dev\java\myproject\src;c:\dev\other\src com.myproject.MyClass

Note that entries in the sourcepath must be seperated by semicolons on Windows and colons on *nix.

If running from the java task in Ant:

<java classname="com.myproject.MyClass">
    ...
       <sysproperty key="SOURCEPATH" value="c:\dev\java\myproject\src;c:\dev\other\src"/>
    ...
</java>

Similarly, if running from the junit task in Ant:

<junit>
    ...
       <sysproperty key="SOURCEPATH" value="c:\dev\java\myproject\src;c:\dev\other\src"/>
    ...
</junit>

See above for full details of the source lookup strategy.

Using filters

Sometimes a trace will include output which varies between different test runs, or between different machines. For example:

Test complete at 15.31 19/01/2005
Mapping took 2.1 seconds
Home directory is C:\Documents and Settings\Martin

Tracetest allows you to filter this kind of output by including a .filter file in the source directory. See here for an example.

A filter file can use the following naming conventions:

(1) <classname>.<methodname>.filter
(2) <classname>.filter

Where classname and methodname are the names of the calling class and method, for example

MyClass.myMethod.filter
MyClass.filter

If the source directory contains files in both formats, the method-specific file is used in preference.

A filter file contains lines with the format token=pattern, where pattern is a regular expression which matches the text that you want to replace, and token is the string which will replace it. Lines beginning with # are comments.

For example:

# Absolute paths in Windows, e.g. C:\foo, D:\foo\bar
WINDOWS_PATH=.:\\.*
# Date and time, e.g. 27/01/2005 15:44:04
DATE=DATE=\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}

See https://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/Pattern.html for documentation of the regular expression syntax.

Note that in the trace file, tokens are delimited by ##. For example:

Today's date is ##DATE##

Note that tracetest applies filters to a trace after close() is called on the DiffOutputStream. The output written to console (which appears immediately when written) is not filtered. Only the trace and baseline files are filtered.

By default, the . pattern operator does not match line termination characters such as \n and \r. This means that patterns will match single lines only. However, sometimes you may want to filter multiple lines, for example:

useful stuff
times
8989ms
1.2s
4092ms
1m23s
end
more useful stuff

You can do this by including the following line in the filter file:

Pattern.DOTALL=true

Now all instances of the . operator will match line termination characters and your patterns can match multiple lines. For example:

TIMES=times.*end

produces

useful stuff
##TIMES##
more useful stuff

Alternatively, you can specify the "dotall" mode by using the (?s) modifier within a particular pattern, e.g.

TIMES=(?s)times.*end

Performance considerations

Tracetest is very memory and CPU efficient when filters are not used. As each character of output it written to a DiffOutputStream, a character is read from the baseline file (if it exists). If any pair of read and written characters are different, or if the baseline and trace streams are of different length, differences() will return true.

When filters is used, the memory and CPU requirements are higher. When the DiffOutputStream is closed, and a filter file is detected for the test, the entire baseline and trace files are read into memory. Filtering is then performed on the trace string, and the two strings are compared. If there are any differences, differences() will return true.

If you are dealing with very large traces, consider avoiding the use of filters. You can do this by filtering the output yourself before writing it to the DiffOutputStream. For example, you could refactor your application avoid writing timestamps when running in "test" mode.

Tips for using Tracetest

• Use assertions in your code when appropriate, e.g. if you had to ensure that 1000 numbers generated by your program were all divisible by 7. This would  be difficult to check by eyeballing the program output!

• Use assertions if you want to check values that have to be filtered out, e.g. checking that two timestamps are the same.

• Override the toString() method in your application classes so that you can easily print the state of your objects. This is a good habit to get into, as Josh Bloch points out in Effective Java.

• If a test fails and you decide that some of the differences are deliberate and some are bugs, you can update the baseline with just the deliberate differences. Use the Compare Editor's navigation controls and "copy current change" controls (see below) to do this.

• You should add .baseline files to version control along with your test code. This allows other people to run the test. Usually you would not add .trace files since you shouldn't be checking failing tests in!

Troubleshooting

If the FAQs below don't hep, please use the discussion forum.

Files aren't being created, deleted or updated as expected

• Check that you are closing your DiffOutputStream, either with close() or closeAndCheck().

• If using Eclipse, set the workspace to refresh automatically, or select the containing project and press F5.

"No source file found for com.myproject.MyClass"

• Ensure that Tracetest is able to find the source code for your class as described here.