Categories
Experience Reports Ramblings

Effective testing

Many people view testing as a checkbox exercise where you spend a chunk of time writing what you plan to test to verify the ACs (basically re-wording the ACs) and what you will regression test. This may be in test cases or tasks/comments on a story. Once reviewed/shared, the tester of the story would then go through these one at a time and check them off.

One advantage of this is that you are less likely to miss something for forget to check that a certain behaviour works. The other advantages are, well actually I don’t have a scooby. You can give them to anyone that can follow instructions I guess?

It probably isn’t a surprise to hear that I’m not fond of this approach.

This is where I babble on about exploratory testing right? Well no.

My preference isn’t quite exploratory testing as per the text books. I will have an idea of what I need to test and I may jot a few bullet points to remind myself. These aren’t as detailed as charters, lacking the clear scope and purpose in them. Instead these bullet points will serve as a reminder for if/when I get side tracked.

As I use the software and a new feature, it can often quickly become clear what it pretty bullet proof and what could be a little bit more flakey. Based on the story, my experience of the software (and software in general) plus knowledge of previous bugs I can identify the risks as I go. If I’m being effective in my testing, I can quickly adapt and refocus my efforts so that most of my time is spent trying out behaviour that is most likely to be broken.

Bringing this back to how some colleagues thing testing should work, when doing the more rote testing you might spend 50% of your time testing what the developers have already tested. On top of that you’ll probably be testing behaviour that through usage of the software you’ll see anywhere.

I feel like there’s a lot more value in just using the software. You can use heuristics, notes or charters to guide you but don’t give equal weight to code that never yields bugs compared to an area where there’s more likely to be bugs. I try to avoid having to explicitly test the more basic / core ACs. In most cases I will hit these as part of my exploration and testing.

After all, most interesting functional bugs tend to be a result of behaviour that wasn’t called out in the ACs or planning.

(This assumes the tester is involved throughout the SDLC, as they should be)

The other thing of course that any good tester will do is learn.

Software can be very complicated and it can be quite rare that everyone is an expert on using every single feature, protocol and so forth within the solution. Quite often I will be learning something new about the software, Windows or tooling. As we experiment we may find that the operating system has a nuance or configuration option that you previously weren’t aware of or find a piece of related functionality that hasn’t received much love in a while and no one thought about how it responds to the changes in your user story.

There is also the behaviour that may not be captured in the ACs. I wouldn’t expect the ACs to detail the expected behaviour every single possible scenario that could occur (you could be writing ACs until the end of time). Through your exploration you will hopefully be asking yourself questions that weren’t asked back during planning (especially before you’ve seen it for real). Just because no one asked the question, finding out the answer can still be valuable and insightful. It may be that the behaviour in some circumstances “doesn’t matter” as far as the Product Owner is concerned for the feature to ship, however it can still be worth sharing with them.

A learning isn’t necessarily just “this is odd” or “I have a new skill”. It may be that your new button correctly handles OS themes, even if your app doesn’t have it as a requirement.

In my opinion a good tester, after a day of testing will have more to say than “ACs all passed”.

As a bit of a silly example of what I mean in all this waffle, let’s say I was testing a video game that has added the ability to walk around in a user story. Do I need to study the ACs? A lot of it is hopefully obvious. Perhaps I’ll make a quick skim read and jot a few notes. The second I get in I might use the left thumbstick to move around. I’ll no doubt go in a bunch of different directions. What about other buttons/sticks? As I use it, can I notice speed differences depending on how much I tilt the stick? What happens if I spin it? If I bring up the Inventory added previously, does my character run around as I am picking my items?

After a little while I’ll have a good feel for it. I’ve possibly found a few issues with it and have a few notes. I can then go back and properly read through the ACs. No doubt I can tick off most of the ACs already and can then look at verifying the remaining ACs as well as exploring the areas that aren’t working as expected – or aren’t defined in the ACs. For example if I found that the inventory has issues, what more can I do?

So hopefully that explains how I think a skilled manual tester is most effective and useful when they are using their knowledge, experience and heuristics to define their testing and letting it evolve based on what they find, not dictated by a fixed list.

As a short note, recently I was involved in a nasty support case. After finding the bug, I was tasked with giving an estimate in getting a fix out. In doing this I called out the sorts of testing that would be required. A day or two later I was quizzed as to why I hadn’t written up a proper test plan yet. My response – I wasn’t planning on writing one. I never do. I think we were equally horrified with each other with the manager thinking I was lazy & not treating it seriously with me wondering why I was stuck back in the dark ages.

Categories
Experience Reports

Using unit tests to unlock quality (Pt II)

In a previous blog entry I talked about unit testing and how I’ve learnt from my (many) mistakes when writing unit tests and practices that I’ve seen that wind me up.

Today I’d like to talk about how I’ve been writing unit tests recently, employing the ideas of TDD (test driven development), and some of the pros and cons of using this approach.

When I first learnt of TDD and was strongly encouraged to use it, I thought it was about writing tests then code. This is kind of true but it also a gross simplification and one that others that I’ve spoken with also have. At the time I really didn’t like it and rejected the idea but having learnt more, I think it is actually kind of swell.

TDD is more iterative and helps you design the code.

  1. Write a “single” unit test describing an aspect of the program
  2. Run the test, which should fail because the program lacks that feature
  3. Write “just enough” code, the simplest possible, to make the test pass
  4. “Refactor” the code until it conforms to the simplicity criteria
  5. Repeat, “accumulating” unit tests over time

Here’s a basic example of TDD for a method to take two strings and adds them:

  1. Start with the most basic case:
    • Assert.Eq(myThing.Add(“1”, “2”), 3)
  2. Write code to make that pass.
  3. Tidy up the code you’ve written
  4. Repeat the process as you build up functionality
  5. What’s next? Error handling with string parsing:
    • Assert.Null(myThing.Add(“cat”, “2”))
  6. After writing the test, see the result and fix if necessary (seems likely at this point).
  7. Okay, time to do the tricky bit. Again, write a new test, see the result and iterate:
    • Assert.Eq(myThing.Add(“one”, “2”), 3)
  8. Some edge cases:
    • Assert.Eq(myThing.Add(“-3”, “four”)
    • or: Assert.Eq(myThing.Add(input1, input2), expectedOutput)
  9. What’s next? Error handling:
    • StrToInt.returns(null) / StrToInt.Throws(ex)
  10. And so on…

One thing I quite liked was when I’m testing my interface for the new class within other classes. It had me thinking “how do I want to handle these situations?”. Previously I would have written a wad of code, handling errors as I see potential to bump into them etc then knowing what I intended the code to do, I’d write the test to ensure it passed. TDD got me more focused on desirable behaviour.

The other benefit that I found was that if I found adding an extra bit of functionality required touching other unit tests that weren’t interested in that change, I knew that my code design was wrong. I was building much more independent tests and therefore, I hope, more maintainable code. If we decide to change how to handle one bit of a method, I won’t be having to update every sodding test like we’d done in the past.

Of course the benefit of better and more maintainable code could just be because I’m more experienced (even if I barely write code since returning to a test role). However I remember feeling especially chuffed with the code.

I’ve heard that TDD can help reduce manual testing required. Personally I’m not sure if that is the case for me given that historically I’ve had very good coverage – even they were written in an overly complicated manner. Anyway, I’d be very apprehensive about reducing the functional testing on the basis of code being unit testing. However I was at least happier than I wouldn’t need to repeat manual dev testing.

There are of course drawbacks. I would have a torrid time if I tried doing this in an area that has really badly written code and tests. It was definitely easier to embrace when I was adding new features.

Also thinking back to some of my previous projects, I may have started work on a changeset with a less defined idea of what I wanted to do. We all know (hopefully) of exploratory testing but I’ve often embraced “exploratory coding”, where I’m exploring ideas of how to put together a class or how an API works through the code.

You can probably still use TDD with this early doors by using behaviour driven tests with little thought on implementation. However my problem here is that if I’m not confident on how something will work, I can find myself adding/removing parameters and changing my design of the code quite a bit until I get a “feel” for it.

I’ve found that if it isn’t a clear area that I’m working on, I might do my exploration of the code, see how it works, understand what I want to be doing and importantly, know that my code is like my exploration notes and not get attached. Then when I have an understanding, I’ll switch to TDD and write it “for real”. However I’ve only limited experience of doing this so I’m not sure how practical it is.

Finally in my experience so far I’ve found that it was definitely slower than some of my similar sized user stories in the past. In the short term it may negatively impact velocity and leave a bad impression but if you’re writing tests that are easier to maintain then this should benefit you in the long run.

Yes, it took me longer to write each changeset, but I wasn’t re-writing unit tests every time my next changeset built upon my previous code. The next time I work on this feature I expect to be quicker than I would have in the past.

In the long run, TDD seems like it will not only help me write better code and tests but whoever picks up working on that area will hopefully thank me for the effort. I’d certainly be grateful if the previous developer in an area has written maintainable and testable code.

Categories
Experience Reports

Using unit tests to unlock quality (Pt I)

When I started working as a developer my mentor taught me to write unit tests with each changeset, so I did. After switching team, my new lead & mentor had us doing the same and I learnt new techniques to write more complex unit tests. When a couple of newer members joined the team, getting unit tests written was something I pushed hard. After all, it was good practice that all good software engineers do.

One of my strengths, or so I thought, was writing unit tests for any and every method. No matter how ugly to code that the test was for was, as a (very small) team we had great coverage… even if it became a running joke that maintaining the tests was often most of a user story.

In hindsight I realise that I was wrong on two accounts.

Not all developers are writing unit tests anywhere to the level that I thought.

It surprised me when we kept having regressions in sections of code. I asked why unit tests weren’t catching them. The simple answer was the code was too hard to unit test.

Now in the developer’s defence here, this is a very old code base that they were building upon and there was no existing coverage but I want to talk about the idea what code can be too hard or “not possible”.

One of the most common challenges that I’ve seen is with calling APIs (Windows / first or 3rd party) or where your method relies on an external entity. Some examples might be using DirectX, accessing the file system or calling an API for a third party system.

The solution is, in theory, pretty simple. Mocking. Rather than calling DirectX directly, have a wrapper and call that. Keep your logic separate from the API calls and you can test it. This is good for developing maintainable code as well as good for your testing. There may be the odd exception where your wrapper might complicate things too much, but that should be a rarity not a norm.

The other reason for not using testing is where timing issue make the tests flakey. Now this is a good reason to not automate something as I believe that a flakey test is worse than no test. However again in most cases I have found the mocking is again the solution. In projects where I’ve been a developer we always have wrappers for our timers so that if we want to test the behaviour in response a timer elapsing, we just invoke the timer.

I’ve found dependency injection to be really useful in making my code testable. We’ve also used reflection as well where you can insert your mock into a created object. You can also set certain properties so that if you’ve got a private member for “isAlive” then you can test “personUnderTest.PokeWith(stick)” with different values for “isAlive”, without having to include steps like “personUnderTest.ThrowOffBridge()” in your setup (meaning changes to ThrowOffBridge can affect PokeWith).

Another thing that I’ve found a little unsettling is “it’s all pushed, I’ve just some unit tests to write.”

No, no no.

There’s a few big issues here:

  • It assumes that your code would pass unit testing before trying.
  • It assumes that your code is testable.
  • If either of those are not true then you will have to re-write the functional code, dev test it again then get it through review again.
  • It can lead you to write unit tests to pass, rather than to test.

My other learning is how bad my tests and code were.

Some of the methods that we wrote were massive and complicated. This meant that in order to unit test one part of the code, I needed to mock and setup absolutely loads of other code. The worst part was making changes. Because we decided that one small part of the business logic needed changing, I was fixing up dozens of unit tests. It was nasty.

I really have learnt the value in keeping things small and ensuring that your methods are serving one function, not “go do everything”.
Some words on how keeping things small is better.

The other major mistake that I made was being what I thought was clever in creating tests that I could set a bunch of inputs on different parameters then the expected output. For example changing how some of my mocks would be setup based on logic in my unit test. Only needing one unit test to cover a bunch of different business logic is genius right?

No. No it is not. It meant that I had tests that were very hard to debug when they failed. It also made it really awkward when we made a tweak or extension to the behaviour.

Lesson learnt: Keep your code and tests simple!

In my next post I will explore more on the technique(s) that I’ve been using to improve my unit tests.

Categories
Experience Reports Guide

Using dump files to guide testing

You don’t need to understand code to make use of dump files.

One tool that I’ve frequently used throughout my testing career (and also development) is WinDbg. I was a little surprised when I realised that very few other people use it so I thought that I’d share a little about why I use it and how to get going.

What can you do?

  • See the code path in a crash dump
  • View data in memory
  • View threads that are running when software is in a hang
  • Many more things that I’ve yet to try

This can be especially useful if you’re tasked with reproducing a crash reported by a customer and (as is unfortunately common) they say “I wasn’t doing anything”.

What you’ll need:

  • WinDbg or WinDbg Preview
  • Access to symbols files for your software (developers can probably help set you up).
  • A dump file that you want to look at (more on this later)

This is part of Debugging Tools for Windows. You can download it for free from Microsoft. There’s a newer “preview” version that is quite neat plus the older one that I’m more accustomed to using as part of the Windows 10 SDK. Both are linked from here:

https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools

If the link is broken, Google for WinDbg and you should find it.

Obtaining dump files

Hopefully your software outputs crash dumps but if not, you can add some registry keys to ensure that they are generated in a known location. Even if your software does create minidumps, you may value full dumps more:

  1. Open regedit and access: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDump
  2. Create the following:
    1. DumpFolder – REG_EXPAND_SZ – %LOCALAPPDATA%\CrashDumps
    2. DumpCount – REG_DWORD – 5
    3. DumpType – REG_DWORD – 2

For more see: https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps

If you want to create a dump file of running software, for example to inspect memory or dig into a hang then you can use task manager. Just right click on the process and click “Create dump file”.

Note that if you are using a 32bit application then you’ll need the 32bit task manager, e.g. c:\windows\syswow64\Taskmgr.exe

There’s also tools out there that will generate dump files on demand.

Symbols

The next thing that you’ll need to do is setup symbols path. These will help turn the 0s and 1s in a minidump into more readable strings. Create yourself a folder for the symbols, for example c:\symbols. Then in WinDbg you’ll need to set the path. In the old version open the File menu and you should see an option. For the Preview version go to Settings then Debugging.

SRV*c:\symbols\*https://msdl.microsoft.com/download/symbols;SRV*c:\symbols\*<YOURPATHHERE>

You may have to add a few paths in there but hopefully you get the idea.

Analysing crash dumps to get a call stack

Now on to the juicy part, analysing a crash dump. You can open it from the File menu.

From the View menu, you should be able to see the option for a stack / call stack. Bring that up whilst the dump is being loaded.

Now run the following commands (they take a few minutes):

.ecxr

!analyze -v

This should tell you a chunk of information about the crash. Based on this you can

An example of using this information

When looking for an example I found a crash dump from a game that I made many years ago. I have absolutely no idea what might of caused it so hopefully now I can figure out why.

My WinDbg analysis included the following:

System.NullReferenceException

This tells me that it tried using an object that didn’t exist. Either it hasn’t been set or has been deleted but is still in use.

007edc0c 081abfb6 X_Orbtek_II!X_Orbtek_360.XOrbtek.UnloadContent+0x6
007edc10 081abe73 Microsoft_Xna_Framework_Game!Microsoft.Xna.Framework.Game.DeviceDisposing+0x1b
007edc20 081abe43 Microsoft_Xna_Framework_Game!Microsoft.Xna.Framework.GraphicsDeviceManager.OnDeviceDisposing+0x13
007edc28 081abe15 Microsoft_Xna_Framework_Game!Microsoft.Xna.Framework.GraphicsDeviceManager.HandleDisposing+0x15
007edc34 081ab7cf Microsoft_Xna_Framework_Graphics!Microsoft.Xna.Framework.Graphics.GraphicsDevice.~GraphicsDevice+0x2f
007edc40 081ab72f Microsoft_Xna_Framework_Graphics!Microsoft.Xna.Framework.Graphics.GraphicsDevice.Dispose+0x1f
007edc60 081ab543 Microsoft_Xna_Framework_Game!Microsoft.Xna.Framework.GraphicsDeviceManager.Dispose+0x133
007edc74 081ab3f5 Microsoft_Xna_Framework_Game!Microsoft.Xna.Framework.GraphicsDeviceManager.System.IDisposable.Dispose+0x15
007edc80 081ab0e3 Microsoft_Xna_Framework_Game!Microsoft.Xna.Framework.Game.Dispose+0xb3
007edcb4 081ab015 Microsoft_Xna_Framework_Game!Microsoft.Xna.Framework.Game.Dispose+0x15

From this I can tell that the game was being closed. It has happened when unloading content so likely its tried to . It may be possible dig deeper. As you gain more skill with the software it is possible to learn more about what was in memory to understand at exactly what point it crashed.

And here’s the key part

If I was trying to reproduce this crash, I can take the knowledge learnt from the crash dump to guide how I will try and get to the bottom of it. Unfortunately the example dump I picked is a tricky one but I could maybe come up with something like “Explore exiting the game with different assets loaded to discover the source of the crash”.

A 10-20 snoop in the dump file might save me a huge chunk of time in trying to reproduce a crash. Obviously I can’t share real examples from my professional life in any detail but knowing that mouse over a control caused a “random” crash or that the software crashed after hitting “Save” and the top of the call stack was “MyApp!MyApp.FileIo.SaveFile.ApplyTextOverlay” then I can focus on that area.

There’s loads more that you can do but hopefully this has been useful!

Categories
Experience Reports Ramblings

Why I believe that manual testing is a great job

I’ve had an unusual journey to my current role (Senior Test Engineer, doing primarily manual testing).

My career started as a QA tester in games as a “foot in the door” to be a games developer. This was very common in the industry. However after establishing myself and becoming a Senior, I moved to Games Design rather than development. Being games, I was eventually redundant and with the desire to get paid again, I took a role as a Software Test Engineer.

I was good at it. I learnt new techniques and skills. I was using Wireshark to see communications between devices and understand why things may be behaving incorrectly. However I was also bored. Most of our testing was running test cases that had been written (and often already executed) by the developers. I then moved into an “Engineering Support” role where I’d take on all support cases passed to Engineering, taking the load off our senior & lead developers. I loved trying to analyse the system and using my “tester brain”, but constantly handling escalated cases with no useful information was miserable.

This is when I made the leap to development. After 5 solid years, working on a variety of different products, I was at the stage where I really needed to be taking on the responsibility to become a senior software engineer but I have very little appetite for it. Instead of taking the lead on new development technologies and emerging languages I found myself more interested in improving our testing. When the opportunity for a senior manual test engineer role came up, I went for it.

A few people have asked me “why?” and treat it as a step down (and even a waste of my talents), however I believe that it has made me more valuable to the company

I like to feel that I am a fairly creative person and am also good at problem solving & analysing data. This lends well to both professions. There’s common ground like being involved in the planning phase, breaking down a feature and identifying the risks and challenges that are there. The “tester brain” is really handy here. Developers then get to flex their brain in designing the code to solve the problem whilst testers will be performing exploratory testing and identifying things that were hard to see when the feature was conceptual. Whilst developers get the thrill of seeing the code they’ve written become a feature that customers use, I certainly enjoy the buzz of finding a bug. Finally there’s debugging. I can really hunting through logs, network traces and code to understand a “weird bug”. This applies to both roles (and is something I’ll touch on in a later blog).

Testing can be boring and laborious, especially when you are mainly doing “checking”. Being given a bunch of things to check, following a load of steps then providing the result is rubbish. It is just as bad as writing what seemed like endless documentation during my time in development.

During my time in development I was always undone by build infrastructure. Particularly with C++ and Apple-based applications, I had a torrid time getting things built for the first time and often my projects were light on feature work and about pulling in latest dependencies etc. I didn’t understand most of the failures or why it wouldn’t just work. Words cannot describe how happy I am that this is a rare occurrence for me nowadays (although newer technologies do seem to have alleviated a lot of the pain here).

Ultimately I prefer manual testing to development. I find that I get to spend more time doing the interesting bit (finding bugs vs writing feature code) and that because (I believe that) I am a great tester and an decent developer, I add a lot more value to the company in helping us deliver quality features in my current role than as a tester.

But what about automation testing?

What I loved about development is seeing something work. Knowing that it will be deployed for customers to use. I felt like I was making a difference in delivering the product. Automation includes the same enjoyment of writing code but ultimately it lacks that feature delivery buzz. As a role it feels inferior to being a developer. You’re doing the less interesting development tasks. Similarly if I’m spending my time writing automated tests, I am not doing exploratory testing. I am not digging through logs and code to see if I can understand the behaviour.

I believe that writing scripts, tools and on occasion tests to reduce my effort and time spent doing boring work is a valuable use of my time. Automation can be great here but to check that the ACs are in place, it is usually quicker to fire up the software and check. Then I can focus on experimenting and exploration, the best bit of being a test engineer.

So the next time someone asks as to whether I’d want to become an automation test engineer, perhaps I should ask “why would I want to do that?”.

Categories
Experience Reports Ramblings

2021 – A year in review

I believe this is a fairly common thing to do and hopefully useful for myself. Lets have a look back at the bizarre year that was 2021.

Key events:

  • Officially became an employee of Motorola Solutions, following a takeover last year.
  • Moved to a new office, which I visited a handful of times.
  • Started a new “secondary role” as a Cyber Champion.

I had a few achievements that I ought to be pleased with:

  • Became involved in cyber security, which involved a chunk of learning theory, running a variety of different types of scans and finally I ran a load of talks for my office during cyber awareness month.
  • I created a new card games called Threat agents and got myself a single copy manufactured.
  • As it has been historically agreed that automating our older, long running programs would be too involving and I was going mad with regressions testing, I wrote my own automation tool. It wasn’t great at reporting results but it did test for stability and found some interesting issues.
  • Changed how we do release testing. It probably doesn’t seem like much but I had been pushing to revamp things for a couple of years and eventually managed to get it implemented.

Whilst there’s plenty positives from the year, ultimately it wasn’t a massive success. I had a couple of big objectives for the year that failed to accomplish:

  • Develop my exploratory testing. I’ve always had the “knack” for finding bugs but I want to learn more about how people do it more professionally. I just need to take the time.
  • Use an off the shelf automation framework for automated UI and/or API testing. I am very confident that this is something I can do, however until I’ve had some proper experience I can’t add it to my CV (even if I’m not a fan of test engineers writing all these tests).

I don’t think I should beat myself up for not managing the above and I certainly don’t want to be giving myself objectives to judge myself by, however it is good to consider what I can hope to achieve in the coming year. For 2022 I want to:

  • Sell threat agents. I’ve had positive feedback for the concept and it seems like selling it might be an easier way to get it out there than sharing for free!
  • Related to this I want to become more experienced in threat modelling. Especially if I am able to give talks on the subject.
  • Improve my knowledge and technique for exploratory testing.
  • Develop my coaching skills. In particular trying to get some of my colleagues on board with some of my ideas and to feel like I’m having an impact.
  • Discover where I fit in the world. Given that automation is all the rage in the jobs market but I prefer manual exploratory testing to try and break the software, what roles are available to me going forwards? Do I have to give up on my ideals? What might lie ahead for me?

If I don’t meet these and have success elsewhere, well so long as I’m going forwards that is all that matters right?

Categories
Experience Reports Ramblings

Threat modelling: Don’t forget your test engineer

I am a test engineer at my current work. After watching a number of talks at Ministry of Testing I also signed up for a secondary role; Cyber Champion. Through this role I’ve been learning about many aspects of cyber security and then running brown bags for our office to help people learn more about the various aspects of cyber security and I’ve also been doing vulnerability scanning. However what I most want to talk about is threat modelling.

If you’ve not heard of it, Threat modelling, at least within the context of software, is an exercise to identify vulnerabilities within your solution. I’ve written some words about it on my Threat Agents site (will explain “Threat Agents” shortly) so I won’t go into too much detail. In short, you put together a data flow diagram then look for vulnerabilities in it. Most people use a mnemonic called STRIDE to achieve this.

If this isn’t familiar then I’d recommend checking out my Threat Modelling write up on my threat Agents site to learn more, or have a look at Ministry of Testing, OWASP or have a quick Google.

Now to the point. Many teams may approach threat modelling by pulling in their senior software engineers, those with the most experience developing the software. However this is a poor idea. Bringing less experienced people to the table could lead to attacks that are “known but unsaid” and therefore easily forgotten or other blind spots that have been learnt throughout the years.

But there’s someone else that you really should bring along. Someone who spends most of their day trying to identify the risks in a feature. Someone who has the nack of finding holesand flaws. Someone who has probably has the widest knowledge of your solution.

Your test engineer.

Next time you are threat modelling, be sure to invite your test engineers. They don’t need to have any security experience or programming background. If they have the ability to spot that “X + Y – Z = Crash”, they are likely to also spot that “R + T – U = Vulnerability”.

If you’ve not done threat modelling before then it can seem quite daunting. Certainly when I was about to have my first sessions I felt pretty anxious that I’d be out of my depth, despite having read and understood plenty on it, including STRIDE. However after completing my first session, I loved it. Not only was it a useful exercise for the business but I really enjoyed threat modelling. As a test engineer I was in my element.

To help people get over that initial hurdle and avoid the risk of setting around a table, looking at a threat model going “errrr” (what my first session would have been without a great coach), I have created a card game called “Threat Agents“.

This takes the elements of STRIDE, adds my quirkiness to them and some structure to help you get going. The game is free to download and get to print off your own copies.

Categories
Experience Reports

“Just run what was done before”

My biggest challenge when switching back to test

Before I start, a bit of background. I started at my current company as a Software Test Engineer. I didn’t really enjoy the very rigid processes that we had in place and felt it didn’t make the most of my creative ability to find bugs. I ended up switching to a couple of other roles before returning to test some 6 years later. In that time a lot of changes have been made.

Whilst not explicitly stated, my team now favour a Context Driven Testing approach to both features and user stories. How we write tests, share them and execute them varies project to project and even user story to user story so that.

I do like this freedom to choose and adapt my approach, meaning that if I think that more detail & structure is required, I can use that. For user stories where tests are very niche & specialist and won’t ever be re-ran, I can dramatically cut down on documentation, allowing more time for bugs.

However when I joined I was replacing the existing sole tester in the team, having been a developer in a different team. I found it quite challenging to use my limited knowledge/experience in several scenarios.

For example when picking up a fairly regular testing around appliance updates, it would seem logical to use a similar strategy as before. However I don’t fully know the context of those tests. Were they shortened due to a time scale or extended beyond the normal coverage because of risks specific to that update? Was it only manual testing because we don’t have automated tests or is looking at making this our first automated project a blind alley that has been pursued before? Why was this strategy used?

In the end, after consulting with the team I repeated the same test cases on a selection of platforms plus one extra due to a specific risky package update. We suspect that was the logic used previously. As per usual I provided my list of tests, equipment to use in a test plan for the story, which can be looked back on.

However this problem is going to continue to resurface as we try to decide the same the best strategy for this context next time. It is possible that the next person to perform these tests might not be me, or I’ve simply forgotten the thought process and time will be needlessly spent trying to figure out the ideal strategy for this context. Likely the same tests will be ran again, including the extra one that I added. This is a problem we have in a few areas.

To try to resolve the problem we discussed that to help people repeating this are of testing down the line that we should provide a rigid set of instructions on what to run etc. We’d break from CDT for these tests. Not ideal, but saves effort for whoever picks up the testing next time. A user story was created to do this.

Thinking on this matter more, this is a blunt way to “resolve it” and papers over the re-occuring mistake. What I really needed to understand previously was the logic in the previous strategy. Why these test cases? Why these platforms? Why were others excluded?

I still want to provide guidance for testers picking this up down the line, detailing the core tests that you need to run every time. However the solution is to not just have the strategy written on the user story, but to ensure that the logic and reasoning is provided for historical use. This should help the next person to pick up this testing to devise the best strategy for that context.