Good programmers check, Great ones test as well
I wrote a comment on Janet Gregorys post about Programmers as testers, and still feel I have to elaborate my comment a little more with the concept of testing vs checking and how these differ. While Janet already cleared out in a more recent comment that we are talking about two different things, I think it is of great importance to have these differences in mind when talking about testing, especially in the Agile development context where the word testing has become a word with many different meanings in various discussions.
So what is the difference between testing and checking?
Michael Bolton wrote a really good (and quite long) blog post series about testing vs checking, followed by a lot of feedback and elaboration by others (links below). Since then, I have taken every opportunity I got to discuss this with other developers. And when laying out the thinking behind this whole discussion in a simple and pedagogical manner, there has really been less resistance towards these thoughts than I thought.
Let me explain
Often when programmers are asked if they test, they hopefully say they do. But when asked about the type of testing carried out, it is mostly about writing and running automated unit or integration tests as they call it. Considering the discussions in the linked blog posts, these would really be checks. This is how Michael defines a check in the comments of Aarons post (original post):
- Is there an observation being made?
- Is there a decision rule linked to that observation?
- Could the determination be made by a machine?
If the answer to all three questions is Yes, then you’re looking at a check; if not, it’s a test, or neither a test nor a check.
This is the approach I have had in my discussions with developers. The answers to these will always be yes when talking about the unit/integration checking code that my developer friends call tests. The most important thing to add here is that the evaluation of the observation really has to be made by a thinking human being for the whole process to be called testing.
Does this distinction matter?
As a tester myself, it matters a lot that the software that is delivered for testing is of some level of quality. And by experience there are many things that can happen to a software product when its modules are put together in its context. If the modules have only been checked in isolation and integration against the predefined and expected criteria which are built into an automated check suite, there are usually quite a few issues left to be easily discovered. Programmers that are aware of this would not deliver the product to test without running it themselves and explore the functionality on a product level first (Running from IDE doesn’t always count).
Programmers tip: Do some real testing yourself and discover those /easily found/-/easily fixed/ bugs after checking is done, but before declaring implementation completely Done.
As an example of good practice, my colleague Davor Crnomat introduced DET, Developers Exploratory Testing, in his Scrum team. Not only did the programmers perform testing, but by using proper coaching they were able to perform good testing as well. And they enjoyed it, this is Ester.
Some more blog posts in this valuable concept discussion:
Markus Gärtner speaks of active vs passive testing, which is also another way of looking at it. However, I see a disadvantage using that wording for my developer discussions when I want to make the point that having a unit check suite is not testing at all.
James Bach brings out the need to go through a debate/discussion in order to understand or acknowledge the difference between words on this level. He also brings up the important aspect of being able to differentiate automated testing and automated checking, where the distinction between testing and checking makes it easier to talk about tool-aided testing vs tool-executed checking.
Scott Barber writes about being able to grasp and accept when new discussions like this come up, not just dismissing them because they don’t fit your context. About the value of these kind of discussions. He also has some good comments on Aarons post.
Aaron Evans uses another set of words, verification and exploration, for distinguishing the meanings of them. I like the use of them when used in a discussion between testers. However, in my perspective when speaking to programmers about these things, these words would not really suffice for my purpose of clarifying the distinction.
Btw1, I wonder if we will ever see a first unit check framework, using the word ‘check’ instead of ‘test’ as the trigger for checking input against predefined output. That would be really something.=)
I found Anders Dinsens post on why acceptance tests (automated checks on higher level) are not enough.