Working with expectations – Software quality characteristics game

When working with our mission of creating an organization wide test strategy I was thinking about quality related problems we have in some projects. I realized that the expected quality is not explicitly stated anywhere. This means neither customer nor development team are aligned about which level of quality is expected to be delivered when done. I needed something to achieve a better awareness of quality requirements and a light weight way of stating them explicitly.

I wanted

•      a solution that would fit into all of our different projects in my organization
•      something that when applied works in a proactive way with quality
•      an easy to use tool for development teams to work with customers about quality

Working with already existing work

I already knew about James Bachs Heuristic Test Strategy Model, and thought I would use that as a starting point. Or actually, my starting point was the Software Quality Characteristics created by TheTestEye which are originally inspired by James’ model. When I later came back to HTSM I realized thats what I was working with as well. Let me explain.

I want to align expectations on quality before development, thus before any quality can be perceived. What I actually want to do within HTSM is to optimize Quality Criteria work for successful delivery. Project environment will briefly be the same in our different projects, thus these parts will not be a part of this exercise. Product elements are usually discussed about just before being developed, and later on I will show how these will be affected by our game. The derived test techniques is something that the team will be able to make decisions on when knowing about the inputs, where Quality Criteria is an important part.

Playing games with Software quality criteria

Quality is a complex thing. To ease the conversation we will talk about the quality criteria, as referred to by James as:
A quality criterion is some requirement that defines what the product should be. By considering different criteria, you will be better able to plan tests that discover important problems fast.

Thus, I take the Software Quality Characteristics and work on them as they would have been requirements. In agile and in my organization we call them user stories or just stories. I print them separately on bigger index cards with the main title and context question a little bigger than the sub-characteristics. Find the index cards here.

Then I play the 20/20 vision innovation game (by Luke Hohmann) with the customer, regarding the quality of the product. I explain to them that we all of course value quality highly, but as with anything there needs to be some sort of prioritization. This will enable the team even more , since they will know what type of quality matters when delivering their stories. The result will be a prioritized list which explicitly states which types of quality matter the most to the business stakeholders.

Quality characteristics index cards

Quality characteristics prioritization

Value gained

Now I have a common ground to build on when I want to work with quality in our projects. The abstract word of quality has been over bridged with the acknowledged set of quality characteristics that explain themselves with the understandable questions attached.

The tool works in our projects in a proactive way, since it is possible to apply regardless of the project/product context before any development work is done. However it needs to be applied after some common understanding of scope has been reached.

Nothing extra is needed to use this tool, except for the things that is anyway required in our projects like an open minded and active customer and development team.

Real value

The real value in this is not really the prioritized list of quality characteristics. Its the gained discussion on what is important for the customer, that matters. For example, imagine this exercise done when having a project scope of derived user stories but before much effort has been done in development. When speaking about security as quality, it really sounds differently in a customers mind than when talking about the “Use https” story. Then at least there was such a story, but maybe not prioritized high enough. If there is no such story, then maybe it needs to be added before doing anything else. What we are doing is actually taking the discussion about scope from a completely different angle than people are used to, which will give interesting results.

Different viewpoints

Of course I already have some ways in mind to alter this approach.

Prioritization strategy

The 20/20 vision suits well when there is not very many people working on the same set. A colleague of mine pointed this out as a weakness of the tool. If there are more people that should be involved then maybe a partitioning or dotting game would be more suitable. However, since we want to play this game with the people that matter it really has to be a small group of people. If not only THE one person that matters. But of course, if there is a better way to get the prioritization done with the gained discussion being constructive.

Product contexts

There could be certain areas in the product that you are developing that are very different in context, for example if there is an enterprise back-end integration and then having mobile clients. Then you would want to differentiate these by creating separate lists for these, one for the back-end and one for the iOS and Android apps respectively. Why would you separate the two client apps? Logic behind business decisions like this is not trivial, but an example would be that the business strategy suggests the iOS app needs to be launched first with an astounding success while the Android app needs to fill the needs for the early adopters that dont have iOS devices. This app maybe does not need the same type of stability focus as the iOS app, but with more options for compatibility in the fragmented Android domain.

Story or feature level quality characteristics

Some stories or features are more important or bigger (or whatever more) than others. If this is the case and there are uncertainties regarding the quality expected at delivery it might be useful to take out the quality characteristics index cards to facilitate a customer discussion around the quality. It does not have to take very long for either side to realize where the solution needs to be cut.

Breaking quality stories into pieces

People will have a hard time prioritizing, that is to be expected. Some parts of a certain quality may be more important than the rest of it and some other quality. I think it should be preceded with awareness, but you might in some cases want to break away a sub-quality from its index card to enable the discussion to go forward to completeness instead of a stale.

Internal and external quality characteristics

Depending on the context it could be important to blend internal and external quality. However, I think it might be important to have them separated in the discussions. I suspect that in our projects we will probably most of the time need to have them separate.

Discussion

It should be noted that I have not tried the approach on many projects just yet. My colleagues that I shared the idea with have first been suspicious to it, but then mostly come to the conclusion that it might be an easy way of getting to the right discussion with our clients regarding quality.
I tried it with two different stakeholders in my current project, although not /the one that matters most/. They both had their views on the quality needed, but the prioritization game actually got them to talk to me about quality in a more understandable way. Or actually, by getting them to express their needs in their own words, I could translate the general wordings of quality characteristics to the project context. That way we got on the same page of expectations on quality.

This post was also published on my company blog.

UPDATE: John Stevenson was kind enough to share his poster with the Software Quality Characteristics. I might just cut out those clouds the next time I am playing the game to make if even more visual. And John, I was unable to comment on your blog for some reason. The “Comment as…profile” did not work. When I had written a comment and pressed publish it just disappeared.