Tag: Hands on testing

Categories
Ramblings

Exploring my testing

When I first started testing within the games industry we would perform general “destructive testing”. This basically meant there was no specific work so we went off to find bugs (or slack off). I liked to pick on a particular area and would explore that and the behaviours, looking for any little nuances.

Over the years I’ve tried using this destructive testing within my day-to-day testing of user stories, going beyond the remit of what my assigned test cases would say and trying to break the feature. I especially liked doing this when I was picking up something new or unfamiliar – my time to shine – and also to my shame, in a bit of a grumpy mood. The buzz of finding a bug, something that has previously slipped through the net, always cheered me up. Even if I didn’t find a bug, it was often enjoyable and could also be informative.

The one downside of saying that you will do this is that it has a very negative name. I am going to break all of your toys. I didn’t like using ad hoc though. That felt like telling my team “I’m going to go do stuff”.

In more recent times as I’ve learnt more about testing, and perhaps matured (that is debatable), I’ve started using the term “exploratory testing” instead. However I am aware that I’m not really doing it correctly as I was never writing charters, just bulleted lists as a reminder of areas to cover.

I still struggle with them a little. Part of my problem is that I often have an idea of the sort of thing I’m looking for but I often feel like I’m shoehorning thato into the Explore … With … To Discover … format and can end up being “Explore feature with what I always use to discover any regressions”. Very meaningless.

  • Explore changing AD config with existing <redacted> users to discover if they have a seamless experience.
  • Explore alarm ownership combined with features like protection, procedure and escalation to discover if <redacted> users are handled the same as <redacted> users. [~90mins]
  • Explore video lockout with <redacted> to discover if it now works.

These are some of my charters for a large user story. Terms internal to the team have been redacted.

Interestingly when having a read of the Exploratory testing APIs section of Mark Winteringham’s “Testing Web APIs” book I learnt about an alternative template that might suit me better. Going forwards I am going to try writing my charters using “Look at … To test for …”.

The definite positive that I’ve found from my time using more structured/formal exploratory testing is when I’ve had a report to produce. Whilst I usually just have a rolling comment in a test task on Jira (or whatever tool I’m using) to keep my notes, occasionally using Google Docs/Sheets, for some larger testing I used a tool that I had built myself. Whilst obviously I can’t share the reports publicly, my team responded very positively to them. Typically I’d only expect people to look at my final comment on the story. To my surprise after attaching my first exploratory report as a PDF, we were reading them as a team and chatting about some of the findings. No doubt having pictures helped, as opposed to a wall of test that might have seen been in my “test task rolling comment” approach.

Here’s an example report that my tool can create, based on my tool:

ExampleDetectiveVisionReportDownload

I have just started using Xray Exploratory App and it seems pretty promising. I haven’t used it for any functional testing but when I took part in a Capture the Flag style tournament, it was a great way to keep notes on things I learnt about the system and commands that works.

I am starting to feel like I can genuinely call myself an exploratory tester.

Next step – getting my team using it!

Categories
Ramblings

What is the future for a manual tester?

I started as a manual QA tester for a games company back in 2008 as part of a massive team, very disconnected from developers. Since then I’ve worked in smaller test teams, within feature teams (i.e. server software) and within a scrum team. This includes a chunk of time where I was the developer, with a tester in our small team.

This time next year I am expecting that my team will join a larger organisation within my company, where I would be the only dedicated tester within a scrum team. It does raise the question of what my role will be. Unsurprisingly this is often on my mind.

It also got me thinking more about what might be the future for someone who most enjoys manual exploratory testing. What roles might there be?

No doubt that for several years to come there will still be jobs out there for manual testers, either in dedicated teams or working more closely with the developers. I hope that we see more of the latter and fewer companies still having dedicated test teams. However it is clear that most people recruiting right now are most interested in automated testing. An idea that I am not fond of and have previously written about (twice in fact).

I recall listening to an AMA session where testing guru Alan Page suggests that developers will be responsible for writing automated tests, probably with the use of record and playback tooling. From my experience working as both a developer and test engineer, I definitely agree that having developers writing the automated tests is the way forward (although convincing some of my colleagues to use click and record seems ambitious).

Developers are obviously skilled in coding and as part of good engineering practice, should be thinking about edge cases and writing testing. It can help force them into thinking about writing code for testability as well. I’ve heard the argument that developers might miss edge cases that a tester writing automated tests would get, which I don’t buy. There may be developers who are more reluctant to be writing these tests or say they test, but I’m not accepting that. Not trusting your teams to do a good job isn’t a reason to hire automated test engineers. If you have a weakness, you need to develop it. Perhaps be coached somehow?

Over the past couple of years I’ve heard of more and more people who’ve become test specialists/coaches who work across multiple teams in a coaching capacity. This is intended to ensure that developers are capable of doing their own testing.

I do like this as a concept and can definitely see this being the way forward. I think it works well with methodologies like scrum and ensures that the whole team is responsible for quality.

Techniques like ATDD / BDD mean that those with a business viewpoint can get involved with testing and quality by defining the tests using gherkin. As tooling continues to improve here, it will get easier to collaborate to define behaviour and tests together. This sounds like a great time to get your test specialists involved and shifts that bug “know how” left.

This does however sound like it is suited for people who like changing company every year or two. That isn’t me. My other concern with this is that I really enjoy exploratory testing. It is why I switched back to test from dev. If I was to become a coach then would I be doing the job I love?

It possibly also leans into one weakness in having scrum teams being solely responsible for their own testing. Teams can easily wind up being in their own bubble, working on their features. Quite often what a tester brings is not just the “knack” of finding a bug but a wider product knowledge.

If I was able to pitch my perfect role, one that best leverages my skillset as a test engineer and brings most value to my company it would be to see test specialists that are more akin to POs/BAs/Scrum Masters in an organisation than someone who comes in to help teams solve their problems, then possibly move on.

Developers should write their own tests and functionally test what they are developing. However having a test/quality focused role that works with a handful of teams seems like a really intriguing prospect to me.

A typical work day could be joining a couple of stand ups then attending a refinement session, where I use knowledge of the products and knack for edge cases to help teams spot the gotchas early. I then might meet up to pair with a developer who is a bit unsure on how to test a complex problem or pick up testing of a story for a team who are short handed at the moment. I later have a look at a sprint review that I couldn’t make and see that a team have just got their feature across the line. I know its been a challenge for them so lets get the build installed and do some exploratory testing. Finally I might put together some training material on a new extension that I’ve found for our E2E automated testing framework.

This sounds awesome as a job to me. Being an almost free-spirit who helps the teams. Sadly it is most likely a fantasy although who knows? Perhaps there may be a role out there?

The future is rather uncertain and, if I’m being honest, a little unsettling. Will my role continue? Will I be able to find a new manual testing role should I want/need to? Will I be forced into automated testing, or (more preferably) a return to development? Will I move to coaching?

Curiously I am also thinking that with my interest and training in cyber security, I may end up being a pen tester. From my experience so far, this seems very much like classic testing – trying to find bugs in the system. To move forward in my career, will I find myself working in the same ways as all those years ago? …

/Rich

Categories
Experience Reports Ramblings

Why I believe that manual testing is a great job

I’ve had an unusual journey to my current role (Senior Test Engineer, doing primarily manual testing).

My career started as a QA tester in games as a “foot in the door” to be a games developer. This was very common in the industry. However after establishing myself and becoming a Senior, I moved to Games Design rather than development. Being games, I was eventually redundant and with the desire to get paid again, I took a role as a Software Test Engineer.

I was good at it. I learnt new techniques and skills. I was using Wireshark to see communications between devices and understand why things may be behaving incorrectly. However I was also bored. Most of our testing was running test cases that had been written (and often already executed) by the developers. I then moved into an “Engineering Support” role where I’d take on all support cases passed to Engineering, taking the load off our senior & lead developers. I loved trying to analyse the system and using my “tester brain”, but constantly handling escalated cases with no useful information was miserable.

This is when I made the leap to development. After 5 solid years, working on a variety of different products, I was at the stage where I really ought to be taking on the responsibility to become a senior software engineer but I had very little appetite for it. Instead of taking the lead on new development technologies and emerging languages I found myself more interested in improving our testing. When the opportunity for a senior manual test engineer role came up, I went for it.

A few people have asked me “why?” and treat it as a step down (and even a waste of my talents), however I believe that it has made me more valuable to the company

I like to feel that I am a fairly creative person and am also good at problem solving & analysing data. This lends well to both professions. There’s common ground like being involved in the planning phase, breaking down a feature and identifying the risks and challenges that are there. The “tester brain” is really handy here. Developers then get to flex their brain in designing the code to solve the problem whilst testers will be performing exploratory testing and identifying things that were hard to see when the feature was conceptual. Whilst developers get the thrill of seeing the code they’ve written become a feature that customers use, I certainly enjoy the buzz of finding a bug. Finally there’s debugging. I can really hunting through logs, network traces and code to understand a “weird bug”. This applies to both roles (and is something I’ll touch on in a later blog).

Testing can be boring and laborious, especially when you are mainly doing “checking”. Being given a bunch of things to check, following a load of steps then providing the result is rubbish. It is just as bad as writing what seemed like endless documentation during my time in development.

During my time in development I was always undone by build infrastructure. Particularly with C++ and Apple-based applications, I had a torrid time getting things built for the first time and often my projects were light on feature work and about pulling in latest dependencies etc. I didn’t understand most of the failures or why it wouldn’t just work. Words cannot describe how happy I am that this is a rare occurrence for me nowadays (although newer technologies do seem to have alleviated a lot of the pain here).

Ultimately I prefer manual testing to development. I find that I get to spend more time doing the interesting bit (finding bugs vs writing feature code) and that because (I believe that) I am a great tester and an decent developer, I add a lot more value to the company in helping us deliver quality features in my current role than as a tester.

But what about automation testing?

What I loved about development is seeing something work. Knowing that it will be deployed for customers to use. I felt like I was making a difference in delivering the product. Automation includes the same enjoyment of writing code but ultimately it lacks that feature delivery buzz. As a role it feels inferior to being a developer. You’re doing the less interesting development tasks. Similarly if I’m spending my time writing automated tests, I am not doing exploratory testing. I am not digging through logs and code to see if I can understand the behaviour.

I believe that writing scripts, tools and on occasion tests to reduce my effort and time spent doing boring work is a valuable use of my time. Automation can be great here but to check that the ACs are in place, it is usually quicker to fire up the software and check. Then I can focus on experimenting and exploration, the best bit of being a test engineer.

So the next time someone asks as to whether I’d want to become an automation test engineer, perhaps I should ask “why would I want to do that?”.