Quality Ramblings

This is my favourite question when meeting people for the first time.

For some people it is about the software doing what it is meant to do. It might be robustness & reliability or adherence to specifications. Other people spoke about the value and solving customer problems. Is it useful? Some people referred to the swishness & polish plus the experience of it.

I like to represent these as the three pillars of quality.

I think this is really interesting to share and convey because within software we often overly focus on correctness as a measure of quality.

Let’s consider this with an example other than software… food!

When I am going out for a meal, what I want is to be fed & getting nutritional value. If I come away from a meal out feeling hungry, I really won’t feel like my user needs have been met. It was lacking value. This has happened with a fine dining meal where sure, it was fancy but I got a snack as soon as I got home. The meal wasn’t quality.

However if I go to a fast food place, maybe it will fill me up, but it is hardly the greatest experience as there’s possibly screaming kids, the food is… fine… and . This is why despite being correctly cooked and filling me up, I wouldn’t describe it as quality.

Finally if I eat a meal but it hasn’t been correctly cooked then I am highly unlikely to regard it as quality. For example when a thai meal had way too much 5 spice, it wasn’t quality. When my dirty fries made my body crash (vomit), it wasn’t quality.

However it isn’t essential to achieve high levels in all of these.

I will still enjoy my fast food because it fills my needs and I can have confidence it won’t make me ill. I will sometimes eat food knowing it won’t meet my nutritional needs… but it will be gooooood too eat. However my red line would be correctness. Or at least aspects of it (don’t make me feel ill!).

It is important to know what matters to you and your customers. For the past 14+ years I’ve worked in an industry when we didn’t care about excellence. Our customers wanted value and correctness.

Similarly some customers will be happy with a lack of excellence and correctness because they are so happy that their problems are being solved.

However a word of caution. Building off one core pillar isn’t without risks. Even two pillars isn’t without risks. When a core pillar crumbles or the focus changes for the customer, suddenly your perceived quality will tumble.

So in summary:

• Quality is more than correctness. Excellence and Value are essential to think about.
• Think about what matters most to your customers.
• But don’t forget about the other pillars.

Side note: I originally started using this referring to it as Dan Ashby’s Three Pillars of Quality… but he never mentioned it. I’m now thinking that I took a concept from one of his (many amazing) talks and adjusted it into the pillars. If I stole this from someone, I’m sorry and please let me know!

One of the first things I did when joining my previous organisation was pitch testing throughout the SDLC. I wanted to share the importance of testing everything that we do.

However not everyone shares that idea of continuous testing throughout the SDLC. For some continuous testing is continuously running the same tests over and over. If you don’t have them fully automated E2E, repeat them manually over and over.

Skipping over how expensive it is to focus on E2E tests, this really misses the entire point of continuous testing. Instead of continuously trying to discover more about our application and challenging everything that we are doing, all we are doing is repeating that same check… and getting the same answers.

Whilst there can definitely be value in repeating tests when you have fear of a risk of regressions (and automation makes that so much easier), the true value in testing is to be trying new things, new ideas and devising new tests.

This is partly why I think people wedded to writing traditional manual test cases are doomed to struggle with test cases. It is a slow, tedious and usually fruitless endeavour.

Testing is more than checking. We need to be continuously experimenting, learning and discovering behaviours. When looking to regression test, don’t ask “what test cases do we have?” but instead “what are the risks and concerns that I have?” Ask “what information do we need to uncover?”. Then devise tests, ideally exploratory or at least looser than rigid test cases in order to uncover the information that you need and the risks associated with that.

In a future post I intend to give a case study or two on some interesting approaches that I’ve looked at.

I originally meant to post something around this about 2 years ago – better late than never! What I won’t state is whether it was still an issue at time of writing…

I repeatedly hear the notion that developers can’t test. I’m not sure if this is because we’re protective of our role (testing has been a dying role for decades) or to excuse developers who can’t be bothered to test.

So let me be clear…

Devs can test.

In fact if I had a high risk feature that I needed someone to look at, there’s a few developers that I’d trust more than a tester I didn’t know (and some I do).

I worked as a developer for 5 years. I was alright at it. However, in that time I never forgot how to test. In fact I never stopped testing. Instead of functional test and system testing, I was unit testing and dev testing my code. I was writing documents where I’d identify edge cases and what the behaviour should be… then implementing it. When our tester was on holiday or overworked, we chipped in. The quality didn’t tumble. The world didn’t collapse.

In one of my testing roles, going back over a decade, I was bemoaning that our processes lent towards following test specs when the pass rate was ridiculously high. Instead I’d find the bugs when I wandered away from the defined test cases that I was meant to be executing and started performing what I called destructive testing (ad hoc). I was shocked to learn that the reason why these tests (nearly) always passed was because the devs try and test them first. There were possibly some automated tests in the mix as well.

What I’ve learnt is that when developers, or testers, say that devs can’t test, what they mean is developers would rather focus on writing code. Perhaps they’ve been tainted from only running the boring test specs that drive me mad. Perhaps they won’t invest time in learning about exploratory testing. However, they have to test. They will. They can.

In the coming weeks I’ll share some more detailed case studies on examples of how I’ve seen (& helped) developers perform excellent testing

As quality / testing specialists we should endeavor to include developers in the conversation. We can remain the specialists but with a little bit of support/encouragement, developers can do a solid job of testing. This all helps us build better software and for the dedicated testers, the opportunity to focus on areas where your elevated skillset can add value.

And the next time someone says developers can’t test, ask them if that means quality / testing specialists can’t write code?

This topic could touch on many areas. I rate myself as an excellent tester, but I’m no expert in the field (I’m not “book smart”). I used to have great domain knowledge, but lately I’ve struggled after moving project and finding myself not understanding a word of what people are talking about (not fun when people look to you for answers).

This post is on security testing. I am not an expert.

I find the field very interesting and also the more I learn, the more I know how important it is. This is why I’ve spoken about security testing at TestBash UK, along with talks, a blog post and created a card game all on threat modelling (something I enjoy, even if I’m not that great at it!).

But I’m not an expert.

I recently attended a work cyber security conference. There we had people who have years, even decades, of experience in various aspects of security. They live and breathe it. Pen testers are doing incredibly clever and technical things to get into systems that I just wouldn’t have a clue on. I am years away from their level and even with the best will in the world, I may never reach that.

But that is fine. I don’t have to be an expert.

What I have wanted to stress to people through my posts, activities and talks is that even if we’re not experts, we can find security issues. Whether this is through threat modelling, asking the right questions in planning or adding some injection test data into our usual set of inputs, if we can find these problems earlier, we can fix them earlier and ultimately build better and more secure software.

The other side of me wanting to reject the expert label is that it makes it seem like the level I’ve reached is somehow further to achieve for the average tester. That isn’t the case. I’ve just been fortunate enough to get a small amount of training, maybe a few hours a month, for a couple of years (and much of that wasn’t relevant to testing).

So I want to end by saying, please don’t view me as an expert. I’ve just dabbled a little more than most folk. Join me on this journey into becoming security newbie – it is great fun!