How we shorten our test cycle against all odds

We can’t do it! Are you crazy? Seriously?


This is just an example of my team’s responses when I asked if we could shorten the system testing period to a week or two. We had a seven-month delivery cycle on a particular project, with two months dedicated to system testing that tracked the code completion milestone. We have been working evenings and weekends for the past two weeks. The meeting where I discussed taking a week off with the team came up with many reasons why we needed more time, not less. So we chose a different approach. We spent some time wondering where the time went. We started by writing down everything we did in those two months, and the result was:

Trial duration (Program X) = RC * (TCE / TV + D * DLC) / PT

  • RC: = Number of duplicate tests because code changes have been made since we previously tested this area.
  • TCE: = number of test cases executed in each cycle
  • Television = Number of test cases executed during the given time.
  • D: = number of defects found and fixed during testing (Ignoring defects closed as Invalid, duplicate, etc.).
  • DLC: = Defect life cycle from opening to closing of each defect.
  • PA: = Number of people who tested the product during this period.

Now that we have this model to use, we can ask more specific questions like:

  • How can we speed up the testing process?
  • Can we be more effective in dealing with defects?
  • Where can we find additional testers and how can we increase their productivity?
  • What can we do to reduce the number of retests?
  • Do all test cases need to be executed?
  • How can we limit the number of defects found during testing?

Only after spending time breaking down a big, nasty problem into more manageable questions did we realize we could really make a difference. We were not as powerless as we thought. We restarted the meetings by presenting only one of these questions, after which we looked at how to improve that one variable. After months of tweaking, we were eventually able to reduce the system testing time to three weeks. We didn’t believe we could take a day off the original test cycle when we first started talking about it, but we ended up cutting the duration against all odds.

Let’s examine some of the discussions and solutions we came up with in our brainstorming sessions, because they are easier said than done.

Expand your testing capabilities

The first thing the team asked for was a bigger team. I was hesitant at first to ask for a higher budget because I didn’t want to seem like I was doing the same. However, after making some of the above changes and showing their progress, our management asked what we could do next. Making significant improvements requires effort from your team. We were able to hire more people and demonstrate the exact benefits we expected from the new funding. Finally, we found an offshore partner to support the majority of the regression testing. This gave the current team more time to implement improvements by creating a feedback loop.

Before collaborating with the offshore partner, we had some help from other employees in the organization who helped with some testing. The product management, developers, and technical documentation teams were all deeply committed to building a better product and volunteered their time to support testing. We also had several “test sessions” where we brought the whole team together for a day to test different aspects of the product. Everyone contributed, including engineers and management, and everyone tested for a day. We awarded prizes to those who found the most important and serious bugs. The test sessions were satisfying and helpful for team bonding.

Focusing mainly on the most important test cases

When we first thought about reducing the number of tests we run, we encountered some pushback. However, when we started approaching it as risk-based testing, putting maximum testing effort into the areas with the greatest risk, we started to improve. We evaluated each test suite on two metrics: the likelihood that those tests would reveal a defect or malfunction, and the severity of the customer impact if a defect were found in that part of the product. We used a chart to select our method after rating each criterion as high, medium, or low;

Note: You can adjust the score provided so that each calculation determines your priority.

Probability
RiskHighAverage:Low
HighPriority 1Priority 1Priority 2
Average:Priority 1Priority 2Priority 3
LowPriority 2Priority 3Priority 3

We reviewed this analysis with the development team, asking them which areas they thought were most vulnerable. They were able to spot a few issues right away, and they also looked through the changelogs for code that had regular bug fixes. We analyzed this data with the product management team to determine the customer’s impact on severity. Likewise, they had some early issues and also did follow-up analysis based on user analysis.

The P1 Priority Test Packs were the most important to us. We took care to test them early and often throughout the cycle, and then later to make sure there was no regression testing. The P2 test suites followed the second, and we gave ourselves a bit more leeway with post-cycle regression testing. We thoroughly analyzed the P3 test suites and reduced them by using samples and simply running them once in a system test.

Click here for more information on risk-based testing

Increased testing speed

Adding test automation coverage seemed like a natural way to increase testing speed, and the automation was quite useful. But we discovered other factors that could increase the speed. We provided tools to help automatically populate test data after deployment so that testers come to work in the morning with a build already in place and the necessary test data in place. We’ve also created a list of “most requested bug fixes” and prioritized these issues. The most requested defects were those that prevented tests from being run, so we tied developer priority to tester productivity. This reduced the length of time testers had to wait for a fix.

Reducing errors in system testing

We began to address the number of defects found during system testing, as we identified and fixed many issues, showing that there was room for improvement. But more importantly, minimizing defects was critical to our overall goal of producing high-quality software. We didn’t track the root cause of the bugs we found in system testing until now, so we had to use some judgment and work with the development team. We examined a sample of problems found in the previous test cycle for trends. We found some minor coding issues as well as a number of issues.

To reduce the number of coding errors, we spent some time making sure we were getting the most out of our code reviews. We conducted code review training, tracked code reviews, and reported the results to the team. We also started using certain techniques designed to detect memory leaks. These two improvements began to reduce the work required to troubleshoot problems during system testing. Finally, we started documenting the main source of defects and conducted frequent analysis to identify more opportunities for optimization.

Shortening the Defect Life Cycle (DLC)

When we looked at our list of defects, I was embarrassed to see that 60% of the defects we submitted were closed without a fix. Two of the most important reasons contributing to this were that the engineers were unable to reproduce the problem and that the defect was a duplicate of an existing one in the system. This required only one simple change. we instructed the test team to check the defect tracking system before submitting a new bug. If they found a comparable problem, they would consider updating the original bug report with the revised facts or consulting with the developer assigned to the defect.

We have researched the defects that cannot be reproduced. Instead of documenting the defect and moving on, the tester invites a “defect batch” to show the defect to the development team. This vice gathering was often held at the end of the day. After that dialog, the tester will create the defect report. This resulted in significantly faster repairs, as developers often exclaimed: “Ah, I understand what’s going on.” Demonstration of the defect helped to eliminate any uncertainty in methods of replicating the defect. After making these adjustments, we found that more than 50% of reported bugs were fixed, and we had fewer ping-pong games.

Since this project, I have used this strategy several times to speed up test cycles. Teams enjoy the process of breaking down time into specific elements and identifying opportunities for immediate improvement.

Leave a Reply