Categories
Ramblings

Non-technical testers are the gatekeepers of quality

Apologies for the intentionally ridiculous title, which is fitting in two of the most frustrating terms in testing.

Non-technical testers is a term often used to describe a test specialist who doesn’t do automation. As I don’t write automated tests in my current role (and haven’t used Selenium), I guess that includes me then?

No. This is nonsense.

  • I use Wireshark to analyse network traffic, comparing against protocol documents to understand what is going on.
  • I look at the contents of crash dumps to help me understand why it crashed and to get better reproduction steps.
  • I occasionally pick up development tasks.
  • I can set up & work with complex system tests and environments.
  • And perhaps most important and applicable to many of us “non-technical testers”, I understand our software & technologies. A good tester will use their domain knowledge to find the edge cases and risks in a complex system.

Non-technical…

I wanted to bring this up because I strongly believe this label is not just insulting to those who don’t write automated tests but belittles the profession. Also a topic for another day but I also think developers are better placed to be writing the automated tests anyway.

Let’s move on.

I’ve often seen testers described as the gatekeeper of quality but I have never agreed with it for two reasons.

1. I am, have never and don’t expect to be the gatekeeper on release

At a simple level, I am not in that position of authority. I don’t see why I should be as a test engineer or QA role (be that junior, senior or lead etc). It is is deeper than who calls the shots though.

Ahead of a release the decision on whether the quality is there and whether the product is sufficient quality needs to be a collaborative effort between the teams and roles.

I certainly prefer to take the approach that my role is to ensure that the decision on whether the quality level is high enough is adequately informed. As a test engineer I have knowledge of using the product. I’ve looked to put myself in the position of a customer and I look understand the implications of known defects. Consequently I believe that my opinion on the quality level is important and typically valued, whether that is attending key meetings or simply informing the team’s representatives and providing quality data/reports etc.

2. Quality is subjective and needs balancing

What are the implications of not achieving a deadline? Could this be a loss of a deal or legal implications? Further to this, is any delay to improve quality of sufficient value to the customer?

As a customer I may be OK with the app having significant alignment issues on Edge given that I typically use Chrome and the predominate page that I use isn’t affected. However I probably won’t be OK with my own projects missing their deadlines because our software was late.

In a similar line, I wouldn’t be happy if functionality that I regularly use has a poor locking design meaning that I have to keep hitting retry. Sure, the code might work as intended but are my needs being met?

Further to this, as a customer I would certainly take an application that provides all of the functionality that I want but needs restarting periodically over having to use half a dozen meticulously implemented applications with conflicting workflows.

Within this thought process, as test engineers we need to consider the customer but with perhaps more balance and realism. I will try and provide my advice on what I think of the quality, trying to think of a customer.

Bugs happen and especially in larger and more complex products, there will be known bugs in a release. Quality of software in terms of stability, bugs and functionality is a balancing act.

No one person or team should be regarding themselves as gatekeepers of quality. Instead everyone across the SDLC should be working to help enable quality.

Categories
Experience Reports

“Just run what was done before”

My biggest challenge when switching back to test

Before I start, a bit of background. I started at my current company as a Software Test Engineer. I didn’t really enjoy the very rigid processes that we had in place and felt it didn’t make the most of my creative ability to find bugs. I ended up switching to a couple of other roles before returning to test some 6 years later. In that time a lot of changes have been made.

Whilst not explicitly stated, my team now favour a Context Driven Testing approach to both features and user stories. How we write tests, share them and execute them varies project to project and even user story to user story so that.

I do like this freedom to choose and adapt my approach, meaning that if I think that more detail & structure is required, I can use that. For user stories where tests are very niche & specialist and won’t ever be re-ran, I can dramatically cut down on documentation, allowing more time for bugs.

However when I joined I was replacing the existing sole tester in the team, having been a developer in a different team. I found it quite challenging to use my limited knowledge/experience in several scenarios.

For example when picking up a fairly regular testing around appliance updates, it would seem logical to use a similar strategy as before. However I don’t fully know the context of those tests. Were they shortened due to a time scale or extended beyond the normal coverage because of risks specific to that update? Was it only manual testing because we don’t have automated tests or is looking at making this our first automated project a blind alley that has been pursued before? Why was this strategy used?

In the end, after consulting with the team I repeated the same test cases on a selection of platforms plus one extra due to a specific risky package update. We suspect that was the logic used previously. As per usual I provided my list of tests, equipment to use in a test plan for the story, which can be looked back on.

However this problem is going to continue to resurface as we try to decide the same the best strategy for this context next time. It is possible that the next person to perform these tests might not be me, or I’ve simply forgotten the thought process and time will be needlessly spent trying to figure out the ideal strategy for this context. Likely the same tests will be ran again, including the extra one that I added. This is a problem we have in a few areas.

To try to resolve the problem we discussed that to help people repeating this are of testing down the line that we should provide a rigid set of instructions on what to run etc. We’d break from CDT for these tests. Not ideal, but saves effort for whoever picks up the testing next time. A user story was created to do this.

Thinking on this matter more, this is a blunt way to “resolve it” and papers over the re-occuring mistake. What I really needed to understand previously was the logic in the previous strategy. Why these test cases? Why these platforms? Why were others excluded?

I still want to provide guidance for testers picking this up down the line, detailing the core tests that you need to run every time. However the solution is to not just have the strategy written on the user story, but to ensure that the logic and reasoning is provided for historical use. This should help the next person to pick up this testing to devise the best strategy for that context.

Categories
Ramblings

Behaviour Driven Development as a manual test engineer

I’ve heard a lot of enthusiam for Behaviour Driven Development and seen a number of talks on the matter. However every time I dive into the subject, it leads into creating automated tests and my work is primarily with manual testing.

Nonetheless I am fascinated by the topic and am keen to explore how we can take some of the concepts of BDD and apply them to manual testing.

When I first started learning about BDD is was as part of a talk on Gherkin. Given I’m not overly fond of very rigid language structures in documentation (or at least no scope to deviate when applicable), I didn’t initially show a lot of interest, however over time I’ve changed my thoughts.

To start with, we are defining both requirements and the tests at the same time. Whilst all the material I’ve seen ends in tying this to automated tests, why can’t this also be manual testing?

Speaking with another test engineer recently, they commented on how they found existing test specifications a better source of describing behaviour than our UCMs (if the docs exist for that product). It did strike me a little – we’re writing requirements in User Stories, then updating our UCMs and then our test specs. Our unit tests also define the behaviour, although as a test engineer I am not exposed to them in our usual working processes.

BDD seems to solve this by defining the behaviour and using that as the definition for your test. Whilst sometimes a Gherkin scenario won’t fully explain what to do, in many (or most) test cases it does. I do question whether a test entitled “Log in with valid user credentials” needs 5 steps explaining how to log in with pre-conditions of not being logged in. If the behaviour is clearly explained (especially when using more “real world” examples), that is what I’ll test doing. These are my test cases.

A common exercise in BDD, Example Mapping, is also very interesting. Often testing for a User Story might take the acceptance criteria and map it to a test task, with some test cases created as a result. However using Example Mapping and the knowledge in the User Story and any other requirements docs can help us define great test cases and/or data sets.

I’m keen to give this a shot when I return to work. Whilst clearly it is an exercise that is intended as a planning exercise for a team, I can see the usage as an individual test engineer. I’ll even use the output of this to create a load of TestLink test cases, possibly using Gherkin a shot within the description, only including steps if actually essential.

Categories
Uncategorized

Hello World

Like so many people around the world, I am currently not working due to the COVID-19 pandemic. However I hope to take advantage of my spare time by gaining more knowledge on software testing and general good practice for software development.

Having seen quite a few interesting talks, with my notes jotted on notepads or my work’s wiki pages. However during this period I’m going to record a few thoughts on a blog. I don’t expect anyone to ever read this but if you do, I hope both of us are finding it useful!

By the way – if you are a work colleague and find a topic interesting, let me know and we can explore it in a meeting 🙂