Measuring JUnit Code Coverage


Measuring JUnit Code Coverage

Elliotte Rusty Harold

Wednesday, February 8, 2006

elharo@metalab.unc.edu

http://www.cafeaulait.org/


Who's writing unit tests now?


Why cover code with tests?


How good is your testing?


Code Coverage Tools


How code coverage works


A Code Coverage Report

Sample code coverage report

http://www.cafeconleche.org/cobertura-jaxen/


Package Level Coverage


Line Level Coverage


Commons Math


The Code Coverage Loop

  1. Measure your coverage to identify untested code

  2. Write tests

  3. Run tests

  4. Fix bugs

  5. Repeat


Strategy


Cobertura via Ant

<taskdef classpath="cobertura.jar" resource="tasks.properties" />

<target name="instrument">
  <cobertura-instrument todir="target/instrumented-classes">
    <fileset dir="target/classes">
      <include name="**/*.class"/>
    </fileset>
  </cobertura-instrument>
</target>

<target name="cover-test" depends="instrument">
  <mkdir dir="${testreportdir}" />
  <junit dir="./" failureproperty="test.failure" printSummary="yes" 
         fork="true" haltonerror="true">
    <!-- Normally you can create this task by copying your existing JUnit
         target, changing its name, and adding these next two lines.
         You may need to change the locations to point to wherever 
         you've put the cobertura.jar file and the instrumented classes. -->
    <classpath location="cobertura.jar"/>
    <classpath location="target/instrumented-classes"/>
    <classpath>
      <fileset dir="${libdir}">
        <include name="*.jar" />
      </fileset>
      <pathelement path="${testclassesdir}" />
      <pathelement path="${classesdir}" />
    </classpath>
    <batchtest todir="${testreportdir}">
      <fileset dir="src/java/test">
        <include name="**/*Test.java" />
        <include name="org/jaxen/javabean/*Test.java" />
      </fileset>
    </batchtest>
  </junit>
</target>

Running Cobertura via Ant

% ant instrument
% ant cover-test
% ant coverage-report

Cobertura via Maven

  <reports>
     <report>maven-jdepend-plugin</report>
     <report>maven-checkstyle-plugin</report>
     <report>maven-cobertura-plugin</report>
     <report>maven-changes-plugin</report>
     <report>maven-changelog-plugin</report>
     <report>maven-developer-activity-plugin</report>
     <report>maven-file-activity-plugin</report>
     <report>maven-license-plugin</report>
     <report>maven-javadoc-plugin</report>
     <report>maven-jxr-plugin</report>
     <report>maven-junit-report-plugin</report>
     <report>maven-linkcheck-plugin</report>
     <report>maven-tasklist-plugin</report>
     <report>maven-pmd-plugin</report> 
  </reports>
  
  ...
  
    <dependencies>

    <dependency>
      <groupId>xerces</groupId>
      <artifactId>xmlParserAPIs</artifactId>
      <version>2.6.2</version>
    </dependency>

   ...

    <dependency>
      <groupId>maven-plugins</groupId>
      <artifactId>maven-cobertura-plugin</artifactId>
      <version>1.1.1</version>
      <type>plugin</type>
    </dependency>

  </dependencies>

Different Kinds of Coverage


The Importance of a Minimal API


Why You Only Test the Published API


Don't neglect the test packages


Getting to 100% Tested


Hard Tests


What can't you cover?


Private constructors in utility classes


Uncalled methods in private implementations of public interfaces

package nu.xom.xslt;

/**
 * 
 * <p>
 * This is just for XSLTransform, and implements only the functionality
 * that class requires. Other classes should not use this.
 * It is far from a conformant implementation of XMLReader. 
 * </p>
 *
 */
class XOMReader implements XMLReader {

    private SAXConverter converter;
        
    public boolean getFeature(String uri) 
      throws SAXNotRecognizedException, SAXNotSupportedException {
        
        if ("http://xml.org/sax/features/namespace-prefixes".equals(uri)
          || "http://xml.org/sax/features/namespaces".equals(uri)) {
            return true;   
        }
        throw new SAXNotRecognizedException("XOMReader doesn't support features");
        
    }

    
    public void setFeature(String uri, boolean value)
      throws SAXNotRecognizedException, SAXNotSupportedException {

    }

    
    public Object getProperty(String uri) throws SAXNotRecognizedException,
            SAXNotSupportedException {
        
        if ("http://xml.org/sax/properties/lexical-handler".equals(uri)) {
            return converter.getLexicalHandler();
        }
        else {
            throw new SAXNotRecognizedException("XOMReader doesn't support features");
        }
        
    }

    
    public void setProperty(String uri, Object value)
      throws SAXNotRecognizedException, SAXNotSupportedException {

        if ("http://xml.org/sax/properties/lexical-handler".equals(uri)) {
            LexicalHandler handler = (LexicalHandler) value;
            converter.setLexicalHandler(handler);
        }
        else {
            throw new SAXNotRecognizedException(
              "XOMReader doesn't support " + uri);
        }

    }

    
    public void setEntityResolver(EntityResolver resolver) {
        throw new UnsupportedOperationException();
    }

    
    public EntityResolver getEntityResolver() {
        return null;
    }

    
    public void setDTDHandler(DTDHandler handler) {
        // throw new UnsupportedOperationException();
    }

    
    public DTDHandler getDTDHandler() {
        return null;
    }

    
    public void setContentHandler(ContentHandler handler) {
        converter = new SAXConverter(handler);
        // warn the SAXConverter that we're using this for XSLT
        converter.setContentHandler(new XSLTHandler(null));
    }

    
    public ContentHandler getContentHandler() {
        return null;
    }

    
    public void setErrorHandler(ErrorHandler handler) {
    }

    
    public ErrorHandler getErrorHandler() {
        return null;
    }

    
    public void parse(InputSource source) 
      throws IOException, SAXException {
        
        XOMInputSource xis = (XOMInputSource) source;
        converter.convert(xis.getNodes());
        
    }

    
    public void parse(String url) throws IOException, SAXException {
        throw new UnsupportedOperationException();
    }

    
}

fail() statements in test cases

fail()
    public void testGetChild() {
        try {
            a1.getChild(0);
            fail("Didn't throw IndexOutofBoundsException");
        }
        catch (IndexOutOfBoundsException success) {
            // success   
        }
    }

Platform/VM-specific code


Excluding lines from coverage measurements


Alternative Techniques


Jester: a different kind of code coverage tool


Prerequisites

$ java -classpath ../jester/jester.jar:lib/junit.jar:build/classes:lib/xalan.jar:lib/xercesImpl.jar:build/xom-1.1b5.jar junit.textui.TestRunner nu.xom.tests.AttributeTest
...................................
Time: 0.196

OK (35 tests)

Running Jester

$ java -classpath ../jester/bin:lib/junit.jar:lib/xalan.jar:lib/xercesImpl.jar:src:build/classes:build/xom-1.1b5.jar jester.TestTester nu.xom.tests.AttributeTest src/nu/xom/Attribute.java

Be sure instrumented clases in src show up in classpath before the regular classes


Jester Output

~/projects/xom$ java -classpath ../jester/bin:lib/junit.jar:lib/xalan.jar:lib/xercesImpl.jar:src:build/classes:build/xom-1.1b5.jar jester.TestTester nu.xom.tests.AttributeTest src/nu/xom/Attribute.java
Use classpath: ../jester/bin:lib/junit.jar:lib/xalan.jar:lib/xercesImpl.jar:src:build/classes:build/xom-1.1b5.jar
Warning - could not find jester.cfg so using default configuration values.
Warning - could not find ignorelist.cfg so using default ignore list.
Warning - could not find mutations.cfg so using default mutations.
For File src/nu/xom/Attribute.java: 54 mutations survived out of 96 changes. Score = 44
src/nu/xom/Attribute.java - changed source on line 149 (char index=5618) from 0 to 1
      if (name.indexOf(':') > >>>0) {
            prefix = name.substring(0, n
src/nu/xom/Attribute.java - changed source on line 162 (char index=5995) from if ( to if (false &&
    _setValue(value);
        >>>if ("xml".equals(this.prefix) && "id".equals(
src/nu/xom/Attribute.java - changed source on line 198 (char index=6904) from if ( to if (true ||
  String prefix = "";
        >>>if (qualifiedName.indexOf(':') >= 0) {
      
src/nu/xom/Attribute.java - changed source on line 198 (char index=6904) from if ( to if (false &&
  String prefix = "";
        >>>if (qualifiedName.indexOf(':') >= 0) {
      
src/nu/xom/Attribute.java - changed source on line 198 (char index=6938) from 0 to 1
qualifiedName.indexOf(':') >= >>>0) {
            prefix = qualifiedName.subst
src/nu/xom/Attribute.java - changed source on line 199 (char index=6988) from 0 to 1
fix = qualifiedName.substring(>>>0, qualifiedName.indexOf(':'));
            i
src/nu/xom/Attribute.java - changed source on line 200 (char index=7032) from if ( to if (true ||
me.indexOf(':'));
            >>>if ("xml:id".equals(qualifiedName)) {
       
src/nu/xom/Attribute.java - changed source on line 200 (char index=7032) from if ( to if (false &&
me.indexOf(':'));
            >>>if ("xml:id".equals(qualifiedName)) {
       
src/nu/xom/Attribute.java - changed source on line 219 (char index=7607) from 0 to 1
s.length();
        int pos = >>>0;
        while (pos < length && s.charAt(po
src/nu/xom/Attribute.java - changed source on line 220 (char index=7655) from == to !=
pos < length && s.charAt(pos) >>>== ' ') pos++;
        s = s.substring(pos);

src/nu/xom/Attribute.java - changed source on line 222 (char index=7729) from 1 to 2

        int end = s.length()->>>1;
        while (end > 0 && s.charAt(end) ==
src/nu/xom/Attribute.java - changed source on line 223 (char index=7753) from 0 to 1
gth()-1;
        while (end > >>>0 && s.charAt(end) == ' ') end--;
        s =
src/nu/xom/Attribute.java - changed source on line 223 (char index=7772) from == to !=
ile (end > 0 && s.charAt(end) >>>== ' ') end--;
        s = s.substring(0, end
src/nu/xom/Attribute.java - changed source on line 224 (char index=7811) from 0 to 1
nd--;
        s = s.substring(>>>0, end+1);
        
        length = s.length
src/nu/xom/Attribute.java - changed source on line 224 (char index=7818) from 1 to 2
       s = s.substring(0, end+>>>1);
        
        length = s.length();
   
src/nu/xom/Attribute.java - changed source on line 228 (char index=7939) from false to true
);
        boolean wasSpace = >>>false;
        for (int i = 0; i < length; i+
src/nu/xom/Attribute.java - changed source on line 229 (char index=7967) from 0 to 1
= false;
        for (int i = >>>0; i < length; i++) {
            char c = s.
src/nu/xom/Attribute.java - changed source on line 231 (char index=8035) from if ( to if (true ||
 c = s.charAt(i);
            >>>if (c == ' ') {
                if (wasSpace)
src/nu/xom/Attribute.java - changed source on line 231 (char index=8035) from if ( to if (false &&
 c = s.charAt(i);
            >>>if (c == ' ') {
                if (wasSpace)
src/nu/xom/Attribute.java - changed source on line 231 (char index=8041) from == to !=
.charAt(i);
            if (c >>>== ' ') {
                if (wasSpace) conti
src/nu/xom/Attribute.java - changed source on line 232 (char index=8067) from if ( to if (true ||
 (c == ' ') {
                >>>if (wasSpace) continue;
                sb.ap
src/nu/xom/Attribute.java - changed source on line 232 (char index=8067) from if ( to if (false &&
 (c == ' ') {
                >>>if (wasSpace) continue;
                sb.ap
src/nu/xom/Attribute.java - changed source on line 234 (char index=8150) from true to false
);
                wasSpace = >>>true;
            }
            else {
      
src/nu/xom/Attribute.java - changed source on line 238 (char index=8246) from false to true
);
                wasSpace = >>>false;
            }
        }
        return
src/nu/xom/Attribute.java - changed source on line 267 (char index=8892) from if ( to if (true ||
Type type) {
        
        >>>if (isXMLID() && ! Type.ID.equals(type)) {
  
src/nu/xom/Attribute.java - changed source on line 267 (char index=8892) from if ( to if (false &&
Type type) {
        
        >>>if (isXMLID() && ! Type.ID.equals(type)) {
  
src/nu/xom/Attribute.java - changed source on line 320 (char index=10485) from if ( to if (false &&
ing value) {
        
        >>>if ("xml".equals(this.prefix) && "id".equals(
src/nu/xom/Attribute.java - changed source on line 361 (char index=11601) from if ( to if (true ||
LocalName(localName);
        >>>if (isXMLID()) {
            this.setType(Att
src/nu/xom/Attribute.java - changed source on line 361 (char index=11601) from if ( to if (false &&
LocalName(localName);
        >>>if (isXMLID()) {
            this.setType(Att
src/nu/xom/Attribute.java - changed source on line 444 (char index=14067) from if ( to if (false &&
tring URI) {
        
        >>>if ("xml".equals(prefix) && "id".equals(this.
src/nu/xom/Attribute.java - changed source on line 492 (char index=15759) from if ( to if (false &&
t = this.getParent();
        >>>if (parent != null) {
           // test for 
src/nu/xom/Attribute.java - changed source on line 496 (char index=15953) from if ( to if (false &&
espaceURI(prefix);
           >>>if (currentURI != null && !currentURI.equals(
src/nu/xom/Attribute.java - changed source on line 496 (char index=15968) from != to ==
x);
           if (currentURI >>>!= null && !currentURI.equals(URI)) {
       
src/nu/xom/Attribute.java - changed source on line 597 (char index=18965) from 1 to 2
ult = new StringBuffer(length+>>>12);
        for (int i = 0; i < length; i++)
src/nu/xom/Attribute.java - changed source on line 719 (char index=22649) from 0 to 1
               result.append('>>>0');
                    break;
             
src/nu/xom/Attribute.java - changed source on line 722 (char index=22742) from 1 to 2
               result.append('>>>1');
                    break;
             
src/nu/xom/Attribute.java - changed source on line 725 (char index=22835) from 2 to 3
               result.append('>>>2');
                    break;
             
src/nu/xom/Attribute.java - changed source on line 728 (char index=22928) from 3 to 4
               result.append('>>>3');
                    break;
             
src/nu/xom/Attribute.java - changed source on line 731 (char index=23021) from 4 to 5
               result.append('>>>4');
                    break;
             
src/nu/xom/Attribute.java - changed source on line 734 (char index=23114) from 5 to 6
               result.append('>>>5');
                    break;
             
src/nu/xom/Attribute.java - changed source on line 737 (char index=23207) from 6 to 7
               result.append('>>>6');
                    break;
             
src/nu/xom/Attribute.java - changed source on line 740 (char index=23300) from 7 to 8
               result.append('>>>7');
                    break;
             
src/nu/xom/Attribute.java - changed source on line 743 (char index=23393) from 8 to 9
               result.append('>>>8');
                    break;
             
src/nu/xom/Attribute.java - changed source on line 746 (char index=23486) from 9 to 0
               result.append('>>>9');
                    break;
             
src/nu/xom/Attribute.java - changed source on line 772 (char index=24175) from true to false
sAttribute() {
        return >>>true;   
    } 
    
    
    /**
     * <p>

src/nu/xom/Attribute.java - changed source on line 805 (char index=25185) from 1 to 2
c final Type CDATA = new Type(>>>1);

        /**
         * <p>
         *   
src/nu/xom/Attribute.java - changed source on line 817 (char index=25707) from 2 to 3
atic final Type ID = new Type(>>>2);
        
        /**
         * <p>
     
src/nu/xom/Attribute.java - changed source on line 829 (char index=26173) from 3 to 4
c final Type IDREF = new Type(>>>3);

        /**
         * <p>
         *   
src/nu/xom/Attribute.java - changed source on line 841 (char index=26686) from 4 to 5
 final Type IDREFS = new Type(>>>4);

        /**
         * <p>
         *   
src/nu/xom/Attribute.java - changed source on line 851 (char index=27070) from 5 to 6
final Type NMTOKEN = new Type(>>>5);

        /**
         * <p>
         *   
src/nu/xom/Attribute.java - changed source on line 862 (char index=27495) from 6 to 7
inal Type NMTOKENS = new Type(>>>6);


        /**
         * <p>
         *  
src/nu/xom/Attribute.java - changed source on line 873 (char index=27918) from 7 to 8
inal Type NOTATION = new Type(>>>7);

        /**
         * <p>
         *   
src/nu/xom/Attribute.java - changed source on line 884 (char index=28341) from 8 to 9
 final Type ENTITY = new Type(>>>8);

        /**
         * <p>
         *   
src/nu/xom/Attribute.java - changed source on line 895 (char index=28799) from 9 to 0
inal Type ENTITIES = new Type(>>>9);

        /**
         * <p>
         *   

54 mutations survived out of 96 changes. Score = 44
took 3 minutes
~/projects/xom$

Jester Optimization Techniques


War Stories


Learning from Code Coverage


The package that couldn't work


Another untested package


JavaDoc Problems

From Jaxen's BaseXPath class:

Are these methods too simple to fail?

    /** Return the original expression text.
     *
     *  @return the normalized XPath expression string
     */
    public String toString()
    {
        return this.exprText;
    }

    /** Returns the string version of this xpath.
     *
     *  @return the normalized XPath expression string
     *
     *  @see #toString
     */
    public String debug()
    {
        return this.xpath.toString();
    }

The second one wasn't tested. Testing revealed that it didn't return what the JavaDoc said it did. In this case, the JavaDoc was incorrect and needed to be repaired.


    /** Returns a string representation of the parse tree.
     *
     *  @return a string representation of the parse tree.
     */
    public String debug()
    {
        return this.xpath.toString();
    }
    

Further testing revealed a problem in the toString() method in DefaultNameStep that was implicitly invoked by this method.

http://cvs.jaxen.codehaus.org/viewrep/jaxen/jaxen/src/java/main/org/jaxen/expr/DefaultNameStep.java?r1=1.46&r2=1.47

Code Coverage as Profiling


Why is this Method called so much?


Reducing Algorithms

URIUtil

Too Far?

UnicodeUtil

What You Can't Find

Program testing can be quite effective for showing the presence of bugs, but is hopelessly inadequate for showing their absence.

--Edsger W. Dijkstra, 1976


An Infinite Loop

This code from XOM's Canonicalizer class was 100% covered. However code coverage did not show that this was an infinite loop if the loop executed more than twice:

ParentNode root = (ParentNode) node;
while (root.getParent() != null) root = node.getParent(); 

try-finally

Also from XOM's Canonicalizer class, Code coverage can notice an exception is or is not thrown into a catch block. However, it can't determine if both cases reach into a finally block:

            try {
                write(node.query(".//. | .//@* | .//namespace::*"));
            }
            finally {
                if (pseudoRoot != null) pseudoRoot.removeChild(0);
            }

Final Thoughts


To Learn More


Index | Cafe con Leche

Copyright 2005, 2006 Elliotte Rusty Harold
elharo@metalab.unc.edu
Last Modified February 15, 2006