Exploring my testing

When I first started testing within the games industry we would perform general “destructive testing”. This basically meant there was no specific work so we went off to find bugs (or slack off). I liked to pick on a particular area and would explore that and the behaviours, looking for any little nuances.

Over the years I’ve tried using this destructive testing within my day-to-day testing of user stories, going beyond the remit of what my assigned test cases would say and trying to break the feature. I especially liked doing this when I was picking up something new or unfamiliar – my time to shine – and also to my shame, in a bit of a grumpy mood. The buzz of finding a bug, something that has previously slipped through the net, always cheered me up. Even if I didn’t find a bug, it was often enjoyable and could also be informative.

The one downside of saying that you will do this is that it has a very negative name. I am going to break all of your toys. I didn’t like using ad hoc though. That felt like telling my team “I’m going to go do stuff”.

In more recent times as I’ve learnt more about testing, and perhaps matured (that is debatable), I’ve started using the term “exploratory testing” instead. However I am aware that I’m not really doing it correctly as I was never writing charters, just bulleted lists as a reminder of areas to cover.

I still struggle with them a little. Part of my problem is that I often have an idea of the sort of thing I’m looking for but I often feel like I’m shoehorning thato into the Explore … With … To Discover … format and can end up being “Explore feature with what I always use to discover any regressions”. Very meaningless.

  • Explore changing AD config with existing <redacted> users to discover if they have a seamless experience.
  • Explore alarm ownership combined with features like protection, procedure and escalation to discover if <redacted> users are handled the same as <redacted> users. [~90mins]
  • Explore video lockout with <redacted> to discover if it now works.

These are some of my charters for a large user story. Terms internal to the team have been redacted.

Interestingly when having a read of the Exploratory testing APIs section of Mark Winteringham’s “Testing Web APIs” book I learnt about an alternative template that might suit me better. Going forwards I am going to try writing my charters using “Look at … To test for …”.

The definite positive that I’ve found from my time using more structured/formal exploratory testing is when I’ve had a report to produce. Whilst I usually just have a rolling comment in a test task on Jira (or whatever tool I’m using) to keep my notes, occasionally using Google Docs/Sheets, for some larger testing I used a tool that I had built myself. Whilst obviously I can’t share the reports publicly, my team responded very positively to them. Typically I’d only expect people to look at my final comment on the story. To my surprise after attaching my first exploratory report as a PDF, we were reading them as a team and chatting about some of the findings. No doubt having pictures helped, as opposed to a wall of test that might have seen been in my “test task rolling comment” approach.

Here’s an example report that my tool can create, based on my tool:

I have just started using Xray Exploratory App and it seems pretty promising. I haven’t used it for any functional testing but when I took part in a Capture the Flag style tournament, it was a great way to keep notes on things I learnt about the system and commands that works.

I am starting to feel like I can genuinely call myself an exploratory tester.

Next step – getting my team using it!