To ask the (right) questions

Last week I had a really interesting and motivating dinner with James Bach, who told me some about his recent ideas and research in the testing domain. One of them was about how to visualize your product and its state of quality by letting it be represented by three different circles, the imagined product, the specified product and the actual product. We had some loose discussions about how these three circles were connected and what kind of ecosystem the were in, but that is another much greater story. However, the cool thing about this representation is that the last week I have been putting it into different contexts and found some interesting mappings.

This weekend I attended the Startup weekend in Lund, which gathered a lot of entrepreneurs combined with a mix of business and development people to a 54 hour work weekend. The goals were about creating the best business ideas from the business pitches made on the friday. The rest of the weekend was all about developing products, business plans and key selling points to be able to care for realization of the ideas. I was at this event as a part of my company Jayway being a sponsor. For my part, I was there as a mentor being able to put my professional perspective as software consultant and tester to the different teams. I tweeted about my participation as I was going to give the right questions. Of course I got a spot on reply from Olacsc about how I knew what was the right questions. It got me thinking and I found out that it was worth much more than a 140 char reply (this post).

With the circles in context from earlier in the week, it really got me thinking about some common contexts. Of course I had to adapt my questions as a mentor to the specific project at hand. All of the project ideas where different in both domain and their progress before the weekend, so any context could be taken into account. I am going to tell you about some of them.

We had the Momo project, where newcomers to Sweden were supposed to find the most relevant information as fast and smooth as possible, which evolved from being completely imagined as a product to becoming more and more specified and actual during the whole weekend. In the end, there was an actual business plan and specified sketch-up of the technical solution, whereas there still were some loose imagined parts of the complex solution still in the minds of the project group. On the first day, my questions were a lot about specifying the imagined product onto paper and in the discussions that led to consensus. There was a session where the initial design of the website was created. When I put some perspective on the design with some questions about different personas using the website, the effort was focused into a more specific setting.

We have the search engine group Geistr, where the product was almost finished. A real-time semantic search engine, that produced quality data from tweets and presenting them in an order of relevance and re-tweets. My questions here was some about the search itself and also about the business case. When asked about spam handling, I got the answer that this had not actually been thought of. So how do you test spam in a real-time semantic search engine? Well, a search for the word “sex” actually gave realistic and useful results. Another search for “Viagra” did however conclude that no one was actually tweeting anything useful about Viagra at the time.

I got into a real user role when questioning Stella Star, a platform for user-generated design that is put into production if enough people bound themselves to buy the finished product. How does a user really feel about crediting their credit card knowing that the product is not even realized yet? They got a headache thinking about this use case, and actually had a better solution with an alternative way when presenting their case.

There was the Locate Me group with an iPhone app idea. By actually performing a user testing session with a paper UI, I questioned the functionality of the app, making it visual that some extra functionality was needed and even some important aspects of the original app-idea.

As you can understand, the imagined product in the different teams could be quite diffuse, with not very clean borders, while others further in their progress were not as diffuse. Since many of the ideas were new to some team members, in some cases apart especially, since the idea originator had been thinking about for some time, I would say that the imagined product circle could have any shapes; small/big/diffuse etc.

If I would treat the specified product circle as I imagine it (*smile*), it is all about what has been agreed on and documented in some way. Written words and conversations where the consensus has been agreed on and narrowed down to a realistic view that is actually possible to document. It is important to know that something that is specified can have more dimensions on it that are actually imagined.

The actual product circle was usually very small or diffuse. An exception would be the search engine where the product in itself was quite clear, but if you widen the span of the product to include the business idea as well, the product circle is bigger and more diffuse.

For me, the testers’ role in any project is to question the product and provide useful information about its state to whoever wants or needs it. In the different projects this weekend, my questions to the products were distributed depending on the context and state of the product. Any questions asked will move the product circles in some way.

About the desired state, I would say that it is highly dependent on the context, but there are some I have some theories on it. When the actual product circle is covered by the specified and imagined circles in a satisfying way, the product could be shippable. If however there is too much gap between actual and imagined, the problem it is supposed to solve might not be solved.

In my perspective, any question that would drive the circles just a tiny bit closer to its perfect ending state, is a right question. I will not dig deeper into what is the perfect state, but it actually means that as long as the questions asked lead to moving the circles in the right directions in their contexts, no question is actually wrong. Especially if you consider that any questions asked will change the shape of the imagined product because of the learning done by both tester and project team. In that perspective, a good tester should never run out of questions.

About this post, I’ll have to add, I know that I am making a lot of assumptions and some parts of this is not really facts or something that is finished. But any questions will move these product circles and make mine and maybe James’ knowledge about this expand. So do question me, even if I might not have the answers right away.