Digging into the Data: Master Your Testing Approach

Your organization has decided on a new technology platform. If it’s a TMS or any other large scale IT application to help run and facilitate your business, testing probably has one of the most significant potential impacts in determining whether your implementation will being successful or not. You have gone through the RFP’s, Design and Build, you are ready to go live, right? In the words of longtime college football analyst Lee Corso, “Not so fast my friends!”

 

A Testing History 

When preparing a test plan for your deployments, there are a couple of formalized methods that can be considered in the review of your requirements to help you determine the best path forward.

The FMEA—or Failure Modes and Effects Analysis—was adopted by the U.S. Military in the 1940s and later by NASA as a form of evaluating the impact of the failure of any individual part or component that was involved in a machine. These machines were typically responsible for keeping people alive in adverse conditions. Every nut, bolt, and screw is considered, and contingencies developed for every part deemed critical to survival. Though effective, the duration and resources required for this type of analysis are severe and should be taken on, only when the risk of failure involves loss of life or economic ruin.  

As a more practical approach for business needs, after your Design has been completed, typically during the Build phase, consider using an FRA or Frequency Risk Assessment.

Scores are assigned on a 1-3-5 scale for assumed Frequency, Severity and Detectability of a defect for each of your requirements. Add the scores together and force rank from Highest to Lowest. Establishing a common criteria or operational definition at the beginning of the evaluation is critical so that everyone’s understanding of the scoring is aligned. 

This scoring can be done collaboratively in meetings, or individually. However, doing it collaboratively prompts a lot of good conversation about the quality of your design and overall risks that can get lost if everyone goes through the analysis separately. This helps establish a priority for testing, particularly if there isn’t enough time to test (and retest!) for every requirement.  

Samples of scoring criteria are displayed below, in addition to a general requirement scoring to display priority rank: 

Time to Have a Plan 

Your requirements have now been reviewed and a priority set in terms of your test plan. The next step is to create your Test Scripts and document your Test Plan. As an artifact of your testing efforts, scripts, and test logs can serve as a great indicator in postmortem evaluations to understand where you succeeded—and where you can improve—during your deployment.

How Test Scripts are documented varies widely from project to project. It can be based on everything from available resources and corporate cultures to government regulations and legal requirements. It can be as simple as an Excel sheet, detailing each test and which requirements and criteria are being tested for each sample. Or in some cases, it can be as complex as individual step-by-step PDF documents for each test, requiring input and signatures from stakeholders and quality organizations.

Whichever Test Plan approach you take, the most important thing is to make sure that there is alignment across both the project team and key stakeholders on the expectations and the execution.

Be sure to plan adequate time and resources for testing. During the early part of your testing efforts, you should be able to estimate how long each test will take your team to execute. If you only have twenty requirements to test against, this may not be that dramatic of an issue. If you have several hundred to several thousand requirements, using the FRA will be invaluable in understanding the necessary testing resources and tracking your progress.

Don’t Ignore the Masses—of Data 

As you complete your testing and document the results, the Go-Live date is rapidly approaching. Functionally, you have made sure the Design and Build meet the expectations set out at the beginning of the project. However, there is one final testing element that can sometimes be easily overlooked, and that is Volume Testing. 

Development systems typically don’t have the same level of “horsepower” present in a Quality-type environment. If you have a three-tiered IT environment, wait until you have proven most of your requirements work functionally, and then move to an environment that can simulate the processing speed you expect in Production. Be sure to ask the following questions: 

• What is the expected cycle time to process an order? 
• How long can the business wait for optimization or reporting jobs to complete? 
• How frequently are reports and scripts going to run? And when? 

Data preparation for Volume Testing can present challenges, so consider this concept during the build of your Test Plan to give your teams sufficient time to complete your core data load. The intent is to simulate a “day in the life” of the new system. 

 

For transportation systems, this would include location loading, functional tariffs, jobs automation, and for warehousing and ERP systems, SKUs, inventory levels, pick locations, everything you need to simulate real life execution.

It’s easy to get caught up in executing a specific test for your requirements, but pouring all your volume through a new system shouldn’t be simulated with a few repetitive samples, assuming it will cover everything in the real world—be sure to consider the average and peak volumes for your testing!

Max load during specified periods of the day should be tested with at least 20% overage to provide confidence that the system will survive and grow as your business continues to develop and expand.

More Than “Make Sure It Works” 

On the surface, the concept of testing seems relatively simple. In today’s complex business world, there are many considerations and exceptions to the rules. Customer requirements, system dependencies, budget, and resource availability play a vital role in charting the path forward for what needs to be executed during testing—and how long it will take.

With the high visibility, impact, and significant expenses associated with today’s modern IT solutions, the impact and need of thorough testing programs cannot be overlooked.