About the title, I was inspired by Alexis Ohanians talk at this years Öredev, Only your mom wants to use your website. And just to point out, the story below is not strictly chronological.
My grandma is not very much of a technician. Yet she uses her computer and Internet every day, all day long. To do what? Listen to the Icelandic web radio of course, sharing her digital photos through email and play hearts, spider and winmine amongst other things. Any of these tasks must be enabled in the least amount of scripted steps to achieve as there is no room for exploration when grandma uses her computer. I have to do the exploration. Read more…
Some 1-1,5 years back, when the iPhone 3G had made such a success in Sweden, to develop apps for it got to be a hot potato in the software development business. Everyone wanted an iPhone app for their product/company, and we were not late to adopt this need and make it happen. Now we are pretty good at developing cocoa apps.
From the testing perspective, I started to look at if testing on iPhone was different from any other software testing. I started off asking a senior app developer about his view on this, and got this answer in an email:
Iphone has very many complete components, which means that the developer doesn’t have to invent the wheel all the time. These complete components are well tested by Apple.
Focus on iPhone app testing should be on the levels of
1. Application logic
2. Verifying that the Apple interface guidelines is followed.
3. Input validation
As a tester, this “easy to develop, easy to test” mentality should in my perspective trigger all bells and whistles as warning signs.
James Bach brings up the logging in a system as a tool for the exploratory tester. I really like this, and am going to see if I can straighten out his list to fit my specific project and situation. I have been thinking about this some time now, but mostly about the information within the log entries, and not the logging mechanism itself.
The system I am testing at the moment has extensive logging features, or at least it logs very many things at multiple levels. I use the logging sort of like he mentions, but not relying as extensively on it as it sounds in his post. The system is pretty straightforward relying on one input that creates one single output. Apart from that, there is only logging and status management to keep track of events.
In our case, when testing a part of the system and keeping my eyes on the logging, it struck me that there was a line “check that x has not failed”. This line was very common to see, since it was logged for every event created successfully. But thinking about it, was this entry a useful entry? NO! In my case it was so obvious that none of the developers had been thinking about it, but it was an horrible entry. Why? One of our statuses for events was “FAILED”. So if an event would fail, the obvious thing is to grep for this word in the logs. But having every successful event instance log the word failed would be devastating for any operator trying to find the cause of a failet event.
So what I would like to extend in James’ list, is that of bullet 5″- Event type description: short, unique human readable label” and “- Event information: any data associated with the event that may be useful for customer service or assessing test coverage, this data may be formatted in ways specific to that event type.”
I want to point out the value and importance of having these string fields not interfere between different types of log entries. Like my example above, keeping the word ‘Failed’ away from any successful events at all times is crucial. The problem is that it is so easy to get it wrong. Developers need to think about this when coding, and not only log what the code does, but create the human readable log entry consistent from the user/operator perspective. If something is successful, it does not matter that the code is “checkJobNotFailed(job)”, but the logged entry should verify the success.
Other types of risks with the logging content is of course that of usernames and passwords. If this type of information is logged in a readable way at any logging level, the log files must be handled according to the security policy of this information.
I have made two examples to point out less good logging events. So, what more examples of “good vs bad” log entries are there? Remember that I am not talking about the logging mechanism and its’ good and bad practices like the ones presented by James, but the content of the event entry messages.
Please enlighten me with more input.
How often do you find a really serious bug the first thing you do on monday mornings? Or any mornings for that matter, when you just started to look at it.
How often is it the other way around, that the first thing you do is trying to remember what you did on friday, to cause that serious issue you found at 4.30 before you were leaving for home and for the weekend?
For me, it is always the latter. Not only fridays, but any day of the week, the most common time when I find bugs is really late in the afternoon, preferably if you are having an appointment or anything right after work. =) But why is that?
I have three possible explanations, that I have been thinking about:
1. You could be testing a really hard-to-set-up system, that takes half of the day setting up before it is possible to test even a little bit. This is the reason for me every now and then. Recently we have been setting up load tests during night, which have failed a couple of times, so the mornings have been devoted to investigate logs for failure explanations and then it has taken some time to reset databases and getting up and running again. Usually with a bug fix or configuration tweak that also takes some time to verify.
2. Exploring the product and its features and areas all day has made you get more sensitive to finding the weaknesses, and poking the system in these spots make the really tricky bugs appear. These might be the most severe afternoon bugs, or the opposite, the ones that are too complex to be a threat to the product and just are hard to fix, with no value added if fixed.
3. You are unintentionally defocusing from the tasks at hand since it is getting late in the day and the lack of energy is carrying your senses in completely other directions from what you have been doing all day. The defocusing strategy is actually something that is brought up in rapid testing contexes for finding other types of bugs than you normally do. Lack of energy just makes you get in this mode automatically during afternoons. These bugs are more too often hard to recreate for investigation, since your lack of energy might miss out on observing the reason behind the bug.
Have you experienced the afternoon bug heuristic? What have been causing you to find them? and specifically, why do you find them at that time?
I have written before about what I think of pair testing with developers. This post is going to give you some more examples of the benefits.
Pair testing is like pair programming, in the way that you get each other to concentrate on the task at hand. Otherwise it is easy to step into sidetracks and staying there for too long. Sidetracks are possible also in a pair, but the time it takes to get back on track is shorter.
This time, I am actually performing the tests together with both of the developers of the system. It is another system from last time, but one of the developers is the same.
Just like the other system, I have learned alot about how the developers think and master the tools useful for this testing, giving me all sorts of perspectives on the product. They are both different in how they think of things, but their knowledge of the system is the same, but anyway they have different ways of expressing how the system works. This is a very exciting setting for me as tester, to gather and filter relevant information to the risks that we are facing.
Another aspect of this product, is that we have a very narrow deadline for production. This has lead to an enormous interest with the product owner, which is spending sometimes hours in a day following our progress of tuning the product for release. This gives also exciting discussions, since he has an even broader view of the users and functionality. He is technical, but sometimes I still have to translate the meaning of words between developers and him, in any direction. This is why I think of communication as a very important aspect in testing. More about that later.
Getting to the point, sitting next to developers when solving a bug, gave me the chance of getting a glance at the workflow in the product code. Usually I try to avoid this deep digging, but since he was holding the weel, I could just watch the exploration of the code. If I dont follow, I can just ask about explanation to see if there are risks to be considered. Yesterday, this actually revealed a bug. When trying to find another bug, I was observant enough about a discrepancy in another part of the code. Highlighting this just made my partner express the words I really like to hear from any developer: “Thats a bug!”
This was followed by an explanation about it, and the most important thing, it was a bug that would have been really hard to find if it had been ruining things later on.
The thing in this case, would be that if this bug had been seen by myself reviewing the code, I would not have been sure about it, and the feedback loop to any developer would have been a very long one, for this type of issue. Typo in a string, would that ruin your day? Well, in this case, the string value was very important, but as a tester do you think about that at all times?
We saved alot of money on the hours we would have been throwing away of finding this bug later on, and I think that is worth the money spent on two persons testing together, may it have been a developer and a tester.
This morning I was in a hurry. Not only did I have to get to my train on time, I also had to refill my train pass with another 30 days, before I got on the train. And of course I wasn’t the only one in the same situation when I came to the ticket office, so I had to wait for my turn.
As always when facing these kind of obstacles in my way, the natural thing for me is to find a better solution. This time I had to get a quicker way to refill my card. Looking at the small vending machine, where you normally buy ordinary tickets for a cheaper price with a card filled with money, I saw a sticker that I have never seen before. Apparently they upgraded the machine to be able to refill my kind of card as well with credit cards. YAYE! The last time I remember this machine getting an upgrade was when you started to be able to refill the money cards with money from credit cards.
So, happy with my new knowledge, I followed the instructions for refilling.
“Swipe credit card”
“Insert train card”
Observing the fact that the machine understands what kind of card it is
*Machine spits out card*
“Your purchase exceeds 850 kr” (Yes, of course it does, this card costs 1060 kr.)
Not so happy anymore, I have to try it again, to see if it was actually happening what I saw.
I start to think about how this could happen in an upgrade of software. From before when these machines were new, you were only able to buy tickets for a train ride. You could buy family tickets etc, but the fact is that a regular one-way/one-person ticket never exceeded 78 kr (or something), which used to be the maximum price for in-region travel, pretty much did not get you above 850 kr. One question would be, when did they put this limit? Was it when they started to take credit cards, or was it in there from the beginning, not allowing you to use the refill cards for more than the limit?
Some other relevant questions would be:
Who decided on the limit 850 kr? Legal issue?
And why this amount?
Wouldn’t it have been easier to have 1000? This would of course also have been a wrong amount for my monthly card, but at least it would have made sense as a maximum value.
Then we got the business value when they realized that the customers could refill their monthly cards in these machines. The cost of having employed people doing this must be soo much higher than doing the development for upgrading the machines. But did it cost too much to test it properly? Did noone even consider old business rules (ex 850 kr limit) as a risk when implementing new business rules for the refills?
I returned to another machine to take a photo of it later today. Since I was not in the same hurry, I realized the sign above it stating the obvious bug as a technical malfunction. But the thing is that this sign was not present at the first machine where I was this morning. If I would have seen it this morning, I would probably not have gotten frustrated like I did.
So I guess they realized the bug in production by the customers, but they could have the documentation for the bug at all machines. Like the company needs worse reputation than they already have.