Test Automation Insight

This is the first in a series of Visual Connections blogs focused on exploring test automation.
Our goal is to share industry best practices, help you reduce costs and testing efforts, and increase your return on investment (ROI) potential.

Three Things Everybody Should Know about Test Automation

By: Bas Dijkstra
Visual Connections Testing Engineer & Test Automation Expert

With ever shorter development and release cycles and a need to continuously deliver high quality software in an increasingly competitive market, fast feedback on software quality has become something that most software-producing organizations just can't do without. Back in 'the days', automated tests written to obtain this fast feedback used to consist of either unit tests written and executed by developers, or end-to-end test scripts created and maintained by test automation engineers - all too often using the maintainability and reusability horror that is record and playback.

Since then, test automation has evolved from something considered an 'extra' into an indispensable factor in the software development and testing process. However, far though the field has come, there are still a number of fundamental concepts that are unknown to those involved in or relying on test automation. In this post, I'd like to try and shed a little light on some of them.

It's check automation, not test automation

First of all, the term 'test automation' is inherently flawed. As people like James Bach and Michael Bolton have explained in detail in their blogs, testing is a process of learning through exploration and experimentation. This is a process that cannot be automated. Checking, on the other hand, is the process of applying algorithmic decision rules to specific observations made on a product (in this case, a piece of software). This process can be automated and this is usually done through what most people refer to as 'test automation' (myself included, I even named my blog OnTestAutomation).

I don't think that using the 'test automation' label in itself is wrong though, as long as people are aware of what is being automated (checks) and what is not (tests). This difference between testing and checking also provides an argument as to why manual testing as an activity will not cease to exist, at least not for the foreseeable future: testing activities cannot be automated!

Test automation is software development and it should be treated as such

Most software developers I meet, at least those worth their salt, actively apply and promote the use of design patterns and good programming practices. Since implementing test automation generally involves designing and writing software - software that is used to perform checks on other pieces of software - it makes sense to use relevant patterns and practices when writing test code as well.

Example practices that can and should be applied include KISS (Keep It Simple, Stupid - your test code should be as simple as possible, but no simpler) and DRY (Don't Repeat Yourself - try and keep code doubles to an absolute minimum). Another good practice you should apply with regards to your test code is to periodically check your checks. Test code is software and therefore it should be tested properly. How else can you be sure that your automated checks will actually fail when they should?

On 100% automated test coverage and other common misconceptions

The true added value of test automation lies in enabling organizations to receive fast feedback on whether certain actions or processes can be successfully executed by the software under test. A key example is executing automated regression checks to make sure that that well-intended refactoring effort did not result in any unexpected negative side effects. Unfortunately, this added value is hard to express in numbers. To make up for this and to support and justify test automation investments, I see a lot of misconceptions being applied:

We can reduce testing head count by X % when we do test automation!

This is a short-sighted and possibly even dangerous misconception. As you've read above, testing and checking are two different activities, and only one of them can be automated. Test automation does not replace testers! Instead, it helps them do their work better by:

Enabling fast feedback on product quality characteristics
Removing the need to manually execute basic and often tedious checks, freeing time for the interesting tests that require experimentation and creative thinking

Using the percentage of tests being automated as an argument to reduce the number of testers needed is not a good way to think about test automation or your testers.

Let's automate all the tests!

Deciding which checks to automate and which to leave for manual testing depends on a lot of factors, including but not limited to: the effort needed to automate the check, the frequency with which the check is to be executed and the risk and damage associated with defects related to the check. Therefore, simply stating that all tests (or any fixed percentage of all tests, for that matter) should be automated is generally not a good idea. It's easy to automate hundreds or thousands of checks in order to achieve a certain percentage of test automation coverage, but this doesn't say anything about the value of these checks and of having them automated. Three well-chosen automated checks can be far more valuable than a hundred trivial or otherwise useless ones.

Once the tests are automated, we're done!

Again, test automation is software development. Just like all software development efforts, automated tests require maintenance. Test automation that is not properly maintained will quickly lose its effectiveness, because checks will start to fail where they shouldn't, or even worse, pass when they shouldn't. Test automation maintenance is an activity that should be planned for, for example during sprint planning in Agile software development processes. Applying the previously mentioned good programming practices can turn out to be very helpful in minimizing the effort required to keep your test automation running smoothly.

Blog 1 of 4 in this Series

Bas Dijkstra

I help teams and organizations improve their testing efforts through smart application of tools.

In my 13+ years in the testing profession I have successfully designed and implemented test automation solutions for a wide variety of clients. I’m currently mostly active as a trainer, teaching people and teams how to make their first or their next move in the world of test automation.

I love to share and expand my knowledge of the test automation field by delivering talks, workshops and training courses, both at conferences as well as on-site with my clients.

You can contact me on LinkedIn via https://www.linkedin.com/in/basdijkstra or through my blog OnTestAutomation.