Home > testing > To ask the (right) questions

To ask the (right) questions

Last week I had a really interesting and motivating dinner with James Bach, who told me some about his recent ideas and research in the testing domain. One of them was about how to visualize your product and its state of quality by letting it be represented by three different circles, the imagined product, the specified product and the actual product. We had some loose discussions about how these three circles were connected and what kind of ecosystem the were in, but that is another much greater story. However, the cool thing about this representation is that the last week I have been putting it into different contexts and found some interesting mappings.

This weekend I attended the Startup weekend in Lund, which gathered a lot of entrepreneurs combined with a mix of business and development people to a 54 hour work weekend. The goals were about creating the best business ideas from the business pitches made on the friday. The rest of the weekend was all about developing products, business plans and key selling points to be able to care for realization of the ideas. I was at this event as a part of my company Jayway being a sponsor. For my part, I was there as a mentor being able to put my professional perspective as software consultant and tester to the different teams. I tweeted about my participation as I was going to give the right questions. Of course I got a spot on reply from Olacsc about how I knew what was the right questions. It got me thinking and I found out that it was worth much more than a 140 char reply (this post).

With the circles in context from earlier in the week, it really got me thinking about some common contexts. Of course I had to adapt my questions as a mentor to the specific project at hand. All of the project ideas where different in both domain and their progress before the weekend, so any context could be taken into account. I am going to tell you about some of them.

We had the Momo project, where newcomers to Sweden were supposed to find the most relevant information as fast and smooth as possible, which evolved from being completely imagined as a product to becoming more and more specified and actual during the whole weekend. In the end, there was an actual business plan and specified sketch-up of the technical solution, whereas there still were some loose imagined parts of the complex solution still in the minds of the project group. On the first day, my questions were a lot about specifying the imagined product onto paper and in the discussions that led to consensus. There was a session where the initial design of the website was created. When I put some perspective on the design with some questions about different personas using the website, the effort was focused into a more specific setting.

We have the search engine group Geistr, where the product was almost finished. A real-time semantic search engine, that produced quality data from tweets and presenting them in an order of relevance and re-tweets. My questions here was some about the search itself and also about the business case. When asked about spam handling, I got the answer that this had not actually been thought of. So how do you test spam in a real-time semantic search engine? Well, a search for the word “sex” actually gave realistic and useful results. Another search for “Viagra” did however conclude that no one was actually tweeting anything useful about Viagra at the time.

I got into a real user role when questioning Stella Star, a platform for user-generated design that is put into production if enough people bound themselves to buy the finished product. How does a user really feel about crediting their credit card knowing that the product is not even realized yet? They got a headache thinking about this use case, and actually had a better solution with an alternative way when presenting their case.

There was the Locate Me group with an iPhone app idea. By actually performing a user testing session with a paper UI, I questioned the functionality of the app, making it visual that some extra functionality was needed and even some important aspects of the original app-idea.

As you can understand, the imagined product in the different teams could be quite diffuse, with not very clean borders, while others further in their progress were not as diffuse. Since many of the ideas were new to some team members, in some cases apart especially, since the idea originator had been thinking about for some time, I would say that the imagined product circle could have any shapes; small/big/diffuse etc.

If I would treat the specified product circle as I imagine it (*smile*), it is all about what has been agreed on and documented in some way. Written words and conversations where the consensus has been agreed on and narrowed down to a realistic view that is actually possible to document. It is important to know that something that is specified can have more dimensions on it that are actually imagined.

The actual product circle was usually very small or diffuse. An exception would be the search engine where the product in itself was quite clear, but if you widen the span of the product to include the business idea as well, the product circle is bigger and more diffuse.

For me, the testers’ role in any project is to question the product and provide useful information about its state to whoever wants or needs it. In the different projects this weekend, my questions to the products were distributed depending on the context and state of the product. Any questions asked will move the product circles in some way.

About the desired state, I would say that it is highly dependent on the context, but there are some I have some theories on it. When the actual product circle is covered by the specified and imagined circles in a satisfying way, the product could be shippable. If however there is too much gap between actual and imagined, the problem it is supposed to solve might not be solved.

In my perspective, any question that would drive the circles just a tiny bit closer to its perfect ending state, is a right question. I will not dig deeper into what is the perfect state, but it actually means that as long as the questions asked lead to moving the circles in the right directions in their contexts, no question is actually wrong. Especially if you consider that any questions asked will change the shape of the imagined product because of the learning done by both tester and project team. In that perspective, a good tester should never run out of questions.

About this post, I’ll have to add, I know that I am making a lot of assumptions and some parts of this is not really facts or something that is finished. But any questions will move these product circles and make mine and maybe James’ knowledge about this expand. So do question me, even if I might not have the answers right away.

  1. April 22, 2010 at 19:07

    Great post, Sigge!

    I really think you hit it here. It’s fun to read it and find that you are arguing exactly the point what was on my mind when I commented on your tweet. Knowing what the right question to ask is is not as easy as it first might appear. The learning comment is great. A good question, in my mind, is a question that both parties learn from. I guess I’m basically just trying to say that this was a good post and I am humbled by the fact that my challenge got you thinking and producing a good blog post.

    • Sigge
      June 15, 2010 at 21:12

      Ola, thank you very much for the trigger you gave me, that actually made me think about what I was tweeting. Its so easy to get away with 140 char sentences that don’t mean much. I didn’t get away with this one, but your replying question probably made us both learning from it. =)
      Thank you for appreciating the read, it makes me want to put even more effort in my writing.

  2. Isabel Evans
    June 14, 2010 at 21:05

    Hi Sigge
    very interesting post – questions can have different purposes in different situations – but I agree with Ola that a good question (and its answer) provides useful learning for both parties. In the situation you are describing it is completely reasonable that know knows the answer – may be it is not possible to know the answer (yet) – may be the answer will be unexpected. I like the overlapping circles idea – and the idea of size and diffuseness as an indicator of level of certainty – does the specification for a brand new product need to be vague and diffuse, while its inventors prototype it… but when they are ready to make it an industrial product, that circle become less diffuse and more defined? Keep posting – I look forward to the next installment!

    • Sigge
      June 15, 2010 at 21:15

      Hi Isabel. Thank you for taking your time to read this, when looking back at it, its a really long and heavy posting. I have to practice on that I think.

      “does the specification for a brand new product need to be vague and diffuse, while its inventors prototype it… but when they are ready to make it an industrial product, that circle become less diffuse and more defined?”

      I would say that this is the most common scenario when inventors prototype something. The specification is in their head, which makes me put that into the imagined circle and not the specified. After a prototype is made, that probably becomes the best specification for the product, which gives a more clear and defined circle. There could of course be other contexts, like if you need to specify things of the product to authorities, then the specification is not even allowed to be too diffuse either.

      With these circles, I think the most important thing is the moving of the three in the context of the others, so talking about them separately like this kind of takes the essence away a little. For your inventors, they start moving the specified circle (prototype) towards the imagined. The prototype is done when they are satisfied with the overlap they see. Then it is time to move the actual circle. This is how I would describe the waterfall model as well, and in software development all circles move at the same time.

  1. June 15, 2010 at 15:09

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: