- According to PlanAxion, a large software package project generally includes 15 categories of tests, grouped into two classes: solution tests and technical certification tests.
- PlanAxion recommends setting up eight test environments in addition to the production environment to execute tests under controlled conditions.
- According to the 2022 CISQ report, poor software quality cost the U.S. economy at least $2.41 trillion USD in 2022.
- Test scenarios are a reusable asset for upgrades, business process documentation, and user training.
A testing strategy is a critical step in every ERP solution design. This approach promotes better risk management: testing software before implementation allows it to be improved and optimized before anomalies reach production. The stakes are significant.
"In 2022, poor software quality cost the U.S. economy at least $2.41 trillion." Source: Consortium for Information and Software Quality (CISQ), 2022 report
Why develop a testing strategy for software?
A software testing strategy reduces implementation risks by focusing testing efforts on the most complex parts of the solution—those where defects would have the greatest impact. For a large project, there are generally 15 categories of tests. These categories fall into two main classes: solution tests, which validate business processes and configuration, and technical certification tests, which confirm infrastructure reliability, performance, and security.
Every project is different, and so is every testing strategy. It is developed after analyzing all potential issues with the solution, and the depth of testing is then adjusted based on the complexity and impact of each component.
Throughout the execution of the strategy, business process experts and software experts work together to prepare and execute test scenarios. This sharing of knowledge is essential: it leads to optimal software configuration.
How should software test execution be structured?
Test execution relies on three pillars: specialized test management tools, controlled IT environments, and dedicated technical support. Test management tools allow you to control the activities of each category, measure progress, manage scenarios and results, track the resolution of anomalies, and prepare accountability reports.
Tests are executed by the project team in controlled IT environments. PlanAxion recommends setting up a total of eight environments in addition to the production environment.
You must also provide an adequate level of technical support throughout the testing period. A team of experts in infrastructure, operating systems, databases, and software is dedicated to the project: they handle installations, ensure environment administration and performance, apply necessary patches, manage backups, and refresh environments as needed.
How should test scenarios be managed?
Test scenarios are a lasting asset: they are reused during subsequent upgrade projects or when adding features, they inform business process documentation, and they serve as the foundation for user training materials. It is therefore essential to preserve and keep them up to date.
It is possible to accelerate scenario preparation by using a library of predefined scenarios. Some software vendors and consulting firms include sets of test scenarios in their intellectual capital that they make available to their clients.
As for test automation, the investment is not always worthwhile. The decision depends on three factors:
- the number of potential regression test cycles;
- the number of annual go-lives;
- the nature of the tests involved.
Who should control test configurations and variables?
A central group, independent of the testing teams, should manage changes to the software's configuration and parameters. This control ensures that changes are made in a controlled manner and replicated uniformly and concertedly across all environments. It also makes it possible to document differences between test environments and minimize situations where it is impossible to explain why software behaves differently from one environment to another.
The goal of the testing strategy is to control all variables introduced during the various test categories in order to isolate, diagnose, and resolve anomalies more easily. The five variables to control are:
- Infrastructure : ensuring stable, high-performance, and properly managed environments.
- Software packages : tracking versions and patches applied in each environment.
- Parameterization : replicating configurations uniformly and in a coordinated manner by a central group.
- Data : working with representative datasets and refreshed environments.
- Interfaces : verifying exchanges with other systems without extraneous variables.
What are the keys to a successful testing strategy?
A successful testing strategy combines test categories tailored to the project, controlled environments, reusable scenarios, and central configuration control. By focusing efforts where complexity and impact are greatest, and by fostering collaboration between business experts and software package experts, the organization improves and optimizes its solution before go-live, rather than dealing with anomalies in production.
Frequently asked questions
What is a testing strategy for a software package?
A testing strategy is a structured approach that plans all tests to be performed before implementing a software package. It promotes better risk management, targets the most complex parts of the solution, and allows for the improvement and optimization of the software package before it goes live.
How many test categories are there in a large software package project?
For a large project, there are generally 15 categories of tests. They are grouped into two main classes. Solution tests validate business processes and parameterization. Technical certification tests confirm the reliability of the infrastructure, performance, and environment security.
How many environments are needed to test a software package?
PlanAxion recommends setting up eight test environments in addition to the production environment. Controlled and regularly refreshed environments allow each test category to be executed under stable conditions, isolate anomalies, and explain differences in software package behavior from one environment to another.
Should software package tests be automated?
Not always. The investment in test automation depends on the number of potential regression test cycles, the number of annual go-lives, and the nature of the tests involved. A project with few planned regression cycles rarely justifies the initial effort of automation.