Rebuild confidence in your test automation

These days, development teams depend heavily on feedback from automated tests to assess the quality of the system they’re working on.

Depending on the type of software testing and the development process used, automated tests every evening or after every execution provide teams with information about the impact of changes made to the code on the overall quality. With great power, however, comes great responsibility. The more you rely on feedback from your automated tests to decide whether to allow a build to move on to the next step in your development pipeline, the more you should be able to rely on the software quality and defect detection efficiency of these highly automated tests. .

In other words, confidence in the quality of the system is essential, so if the system is tested in an automated process at any stage, you must also trust the integrity of these automated tests. Unfortunately, this is where test automation all too often falls short of its potential. Instead of being the strong and reliable defenders of product quality that they should be, automated tests are often a source of deception, anxiety, and ambiguity. They end up damaging the very trust they were supposed to provide in the first place.

How can we restore trust in our automated tests? Let’s look at two ways automated tests can destroy trust, rather than build it, and then consider what you can do to fix the situation, or better yet, prevent it in the first place.

False positives

Let’s start with a brief explanation of false positives.

A false positive is a binary classification error where a test result incorrectly indicates the presence of a condition (such as a disease when the disease is not present), while a false negative is the opposite error where a test result incorrectly indicates the presence of a condition; the absence of a condition when it actually exists; These are two types of errors in a binary test, as opposed to two types of correct results (true positive and true negative). They are also known in medicine as a false positive (or false negative) diagnosis, and in statistical classification as a false positive (or false negative) error.[1] – wiki

This type of destructive automated testing occurs more often in user interface-based testing, mainly because these tests have the highest probability of failure (synchronization and timeout issues, exception handling, uncommunicated product changes, etc.). If your timeout and exception handling are not designed correctly, false positives can be quite time-consuming in terms of root cause analysis and preventing them in the future. When these false positives occur on an irregular basis, it can be even more difficult to determine what is causing the problem.

When your team or business uses a continuous integration or continuous deployment strategy for software development, false positives can be very frustrating. When your build pipeline contains tests that sometimes generate false positives, your build can sometimes fail because your tests fail to properly handle exceptions and timeouts. This may eventually lead to you removing tests from your pipeline entirely. While this may temporarily alleviate your problem with broken buildings, it is not a long-term approach. There’s a reason you’re putting in the effort to create these tests (right?), so they should be included in your automated testing and delivery process. As a result, you should investigate the underlying cause of these false positives as soon as they occur and correct them as soon as possible.

To avoid false positives in the first place, you should make an effort to develop robust and reliable tests, including adequate exception handling and synchronization procedures. This will certainly take time, effort and experience up front, but a solid foundation will pay off in the long run by eliminating these incredibly annoying false positives.

False negatives

Let’s start with a brief explanation of false negatives.

A false negative error or false negative is a test result that incorrectly indicates that a condition is not met. – Vicky

While false positives can be quite annoying, at least they make themselves known through error warnings or failed CI builds. A significant threat of loss of confidence in test automation is in negative cases. tests that pass when they shouldn’t. The more dependent you are on the results of your automated tests when making procedural decisions, such as deployment to production, the more important it is that you can trust that your test results accurately reflect the quality of your application under test, rather than hollow tests that pass, but do not perform the checks they should.

False negatives are particularly difficult to identify and manage for two main reasons.

  1. As previously stated, they do not actively demonstrate their presence, such as by displaying an error notification, as true positives and false positives do.
  2. Although some tests may initially be false negatives, most false negatives develop over time and after continuous review of the product being tested.

Along with creating new tests and upgrading old ones, an important component of your test automation approach should be frequent evaluation of the fault-finding effectiveness of your existing test cases. This is especially true for tests that ran smoothly and successfully in their early days. Are you sure they still make the right claims? Or would they also pass if the app being tested malfunctioned? The following is a sensible plan to continually assess that your tests are still worthy of the trust you have placed in them:

  1. As you develop your tests, test them. Depending on the type of test and assertion, this could be as simple as rejecting the specified assertion at the end of the test and checking for test failure. When you use test-driven development, you automatically do something like this because your tests won’t pass until you add real production code.
  2. A pass test that is pointless might be great for statistical data, but it adds maintenance work that you can do without. Periodically review your tests to ensure they still have their original defect detection power and are not redundant due to changes to your application since the test was developed.

I suggest looking at mutation testing as a technique for creating and maintaining a high-quality, robust test suite for unit tests. Here’s what it means.

Mutation testing (or mutation analysis or program mutation) is used to develop new software tests and evaluate the quality of existing software tests. Mutation testing involves making small changes to a program.[1] Each mutated version is called a mutant, and tests detect and reject mutants by causing the original version to behave differently from the mutant. This is called killing the mutant. Test kits are measured by the percentage of mutants they kill. New tests can be developed to kill additional mutants. Mutants are based on well-defined mutation operators that either mimic typical programming errors (such as using the wrong operator or variable name) or force valuable tests (such as dividing every expression by zero). The goal is to help the tester develop effective tests or identify weaknesses in the test data used for the program or in parts of the code that are rarely or never accessed during execution. Mutation testing is a form of white box testing – Wiki

While mutation testing can be time-consuming, especially for large systems, it can give you significant insight into the quality and defect detection capabilities of your unit test suite. Building reliable automated tests requires programming skills, but it may be even more important to have the ability to decide whether to automate the test at all. Determining the most effective automation technique for a specific test also requires skill. I proposed and found the simplest answer to the test automation problem: a clear plan. If the way I’m going doesn’t lead to a simple answer, then this option probably isn’t the most efficient either.

Have questions Leave them in the LambdaTest community? Visit now!

Leave a Reply