A Context-Driven Approach To Automation In Testing

Outline: "To help ourselves test better, Context-Driven testers use tools. But, there is no such thing as Automation”

While reading the James Bach's blog, I found an interesting view about “Automation” in testing from Michael Bolton and James Bach.

Quotes:

The governing view of test automation can be summarized as “automate testing by automating the user.” We see at least three big problems here that belittle testing:

1. The word “Automation” is deceptive. We cannot automate users. We can only automate some of the actions they perform, but users do so much more than that.

2. Output checking can be automated but testers can do much more than that.

3. Automated output checking is fascinating but tools do so much more than that.

Testing is a part of the critical and creative work that happens in the design studio but “automation” inspires people to think of mechanized assembly-line work done on the factory floor. The term test automation is also unclear.

Firstly, test automation is not human at all. Since you don’t pay the computer, it is incredibly inexpensive and fast too. Secondly, test automation is a skillful activity executed by humans who write and operate software over hours, days or weeks and those people must be paid for their time.

We say that testing is about assessing a product by learning about it through experimentation and exploration which comprises of inference, questioning, observation, modeling and study etc. to some degree. We pick our words carefully. Testing is certainly a human process. Only humans can learn. Only humans can decide value.

Value is a social judgement and different people assess things differently. Technologists may believe that they can automate the assessment of requirements by encoding them into a script, but the assessment is incomplete and temporary until it has been reviewed by a human. There are situations where a manager says that a bug has been reported by a tool while this is not a problem in this case.

# Difference between Checking and Testing:

We find it essential to differentiate between checking and testing. Checking is the process of making assessments by putting on algorithmic decision rules to specific interpretations of a product. This is different from the rest of testing in one vigorous way: it can be completely automated.

# Checking is a suitable place to use the word Automation

In testing, we perform and design experiments that help us improve our understanding of the product’s status.

This understanding is an explanation and not a fact. Quality is a working hypothesis. When you exercise software and fail to spot a specific problem, you have not proven or demonstrated that it works. We only know that a failure has not been recognized yet and we only demonstrate that the product can work. The product may have failed in a delicate way you did not or cannot yet detect, maybe it works fine now but won’t work ten minutes from now. So does it actually, truthfully, totally work?

No output check can tell you that. No collection of output checks can tell you that.

# Automating actions is a method. It should not be a sacrament.

We reject sacraments in the Context-Driven world. We hold problem-solving. But this approach is only valid if problem-solving matters. Frequently, automation is followed as an unquestioned good stuff that astonishes people even when it achieves little. If your testing doesn’t really matter, except as a display for public relations purposes, then maybe sacraments are acceptable— but that cannot be so if your intent is to find main bugs before it is too late. To fulfill that mission, you must develop an appreciation of the full spectrum of tools and their applications to your work. Context-Driven testers smear tools in powerful ways to get testing done.