Tweaking automated checks - part four

Useful error messages

 

Have you come across a check that fails with the message "expected true but was false"? Did you know what was wrong without having to investigate the check first and then the code? I have seen many of those checks and find that they usually cost me a lot of time searching the issue when they fail. 

 

Imagine a fire brigade, someone calls in: "my house burns" but they wouldn't tell where. Yes, you can say "but I'm giving the test a real good name". That might help. But sometimes it's just not enough to tell the whole story and sometimes the name of a check isn't in the same place as the error message. It's as if the caller says "My name is Smith and my house burns". With luck you have only one Family Smith in your village. Oh, and what if your phone book with the adresses is out of date?

 

I want my checks to speak clearly. In some languages/frameworks that is easier than in others. I am really happy with Clojure and its expectations framework that allows me to write something like this 

 

(expect up-to-date? measurements)

 

Even though it doesn't read "expect measurements up-to-date?" as the expected value (or the predicate) comes first, I like this a lot, because the error message in case of a failure reads

 

measurements is not up-to-date?

 

Not perfect English but better than "true is not false". Unfortunately, I haven't seen other frameworks yet where I can do that this comfortably. However, of all languages, Java was the one which helped me the most here. From there I knew matchers like the Hamcrest ones. Both in Java and in Scala, Matchers are available and I am using them when I consider a clear error message more important than my lazyness of not wanting to write extra code. (Therefore, I use them less often in Java.)

 

In Scala you can write traits that you can use to extend your test classes. And one of those can contain your matchers. Here an example:

 

trait DataObjectMatchers {

 

  def beUpToDate = new UpToDateMatcher()

 

  class UpToDateMatcher() extends Matcher[DataObject] {

 

    def apply(data: DataObject) = {

      val startDate = Utils.parse(data.startDate)

      val midnightToday = now.withTimeAtStartOfDay

      val endDate = startDate.plusDays(data.length)

 

      MatchResult(

        after(endDate, midnightToday),

        s"""Last data is from "$endDate" and not yesterday: "$data"""",

        s"""Last data is from yesterday, "$endDate": "$data""""

      )

    }

  }

}

 

Within the MatchResult you can see the actual matching and the error messages that you'll get when a test fails on a positive check and a check with "not". In both you can easily give any clues you find useful for about on which exact data a check fails. I find that utterly useful when writing higher level checks where I cannot control the inputs or when working with randomized data for the checks, especially when I sample.

 

You might have noticed the weird name within the first statement in the trait. This allows me to write the following plain English line within a check:

 

data should beUpToDate

 

As the author, I'm the one to be blamed if it's not understandable when that check fails. As you can see, the downside is a bit of extra code. One thing that I like a lot about it though is that I can store the additional predicates which I need only for testing to an object apart from the production code. As I want the actual check to stay uncluttered I prefer something as easy to read as a predicate of the object itself like data.isUpToDate.

 

Btw, I usually write my matchers test-driven through property-based checks. Often I can reuse the generators I write for them in the checks that are not on a meta level.

 

 

Wrap up

 

All in all I described four tweaks that I find useful when I write checks: 

 

 

In combination they unveil their full potential but there are situations when I make use of but some of them. Use your context to guide you to what you really need. And if you have more tweaks that seem simple but have a significant effect on your automation, please tell me about it.

 

Feel free to leave comments! And if you want to go back to the last post of this series, click here.