As Philippa Jennings articulated in her talk at MoTaCon, quality is the goal. Automation is a means. Unfortunately some people do get that muddled. Even worse is mistaking using a tool with quality. Eek!
I love automation for three reasons.
First is unit testing. That super fast feedback when I’m writing code is handy and I love unit tests when it comes to understanding code and PR reviews. I’ve also dabbled with TDD and like that too.
Second is to enable me to do soak or load testing. Clearly I’m not going to be interested in pressing a button over and over 24/7, or manually creating 1,000,000 database entries. I’ll automate setup and execution. These aren’t automated tests as they don’t run automatically with that binary pass/fail but it is a great use of automation in testing.
Finally is the big topic. Automated regression tests. Whether it is E2E or integration, these help test behaviours. Despite many models having E2E as the minority of tests compared to unit or integration, these are the ones that we fuss over the most.
This leads to my point. If I had to manually regression test a feature then I’d be asking about risk to test what was necessary (i.e. not “everything”). The same should apply to automated E2E and to some extent, integration tests. Some testing is still best manual. Exploratory testing offers different insights and can find edge cases that you’d never think of just by staring at Jira or an IDE. Balance is essential.
My advice is to prioritise based upon:
- Whether you’d bother manually testing that scenario. For me this is:
- History of defects
- Areas constantly under change
- Areas that you’ve just changed (ideally tests written to test a story)
- Balance of effort to automate Vs run manually:
- Some tests are expensive to run manually
- Some tests are expensive to automate.
- Likelihood of that scenario
So sorta risk x effort x likelihood.
I just want to reiterate that first point. We don’t need to test everything. If we can be adding in automated regression tests to avoid ever doing hands-on regression testing, amazing. But don’t spend hours adding automated tests for the sake of it. On top of that, running the tests aren’t free. The most obvious cost is financial but as with everything we do in computing, there is a financial and environmental cost to running operations.
My absolute ideal is a BDD or ATDD style approach where we’re identifying test cases from the requirements then these tests are written before the story can be closed. As I’ve discussed previously, it is barely an improvement on old fashioned manual QA teams if you’re writing the automated tests in a separate team, weeks later.
Final parting thought. If you are wanting to improve automation for your business, don’t just immediately go to E2E tests. What sort of issues are you looking for? What is the most effective way to catch those issues as early as possible? What is your maintenance budget?