Project Management

How to create a software testing strategy?

An ERP testing strategy connects critical business processes with scenarios, data, owners and release criteria. It covers functions, interfaces, migration, access and exceptions. Management authorizes deployment using evidence accepted by business owners, with residual risks documented. Test execution alone is insufficient: corrected defects must be retested and critical processes must work.
•
image of an innovation lab (for an AI developer tools business)
Key takeaways
  • Link every test to a risk and business process.
  • Retain results, defects, corrections and approvals.
  • Adapt categories and environments to the actual project.
  • Authorize deployment using evidence and accepted residual risks.

A testing strategy is a critical step in every ERP solution design. This approach promotes better risk management: testing software before implementation allows it to be improved and optimized before anomalies reach production. The stakes are significant.

"In 2022, poor software quality cost the U.S. economy at least $2.41 trillion." Source: Consortium for Information and Software Quality (CISQ), 2022 report

Why develop a testing strategy for software?

A software testing strategy reduces implementation risks by focusing testing efforts on the most complex parts of the solution—those where defects would have the greatest impact. For a large project, there are generally 15 categories of tests. These categories fall into two main classes: solution tests, which validate business processes and configuration, and technical certification tests, which confirm infrastructure reliability, performance, and security.

Every project is different, and so is every testing strategy. It is developed after analyzing all potential issues with the solution, and the depth of testing is then adjusted based on the complexity and impact of each component.

Throughout the execution of the strategy, business process experts and software experts work together to prepare and execute test scenarios. This sharing of knowledge is essential: it leads to optimal software configuration.

How should software test execution be structured?

Test execution relies on three pillars: specialized test management tools, controlled IT environments, and dedicated technical support. Test management tools allow you to control the activities of each category, measure progress, manage scenarios and results, track the resolution of anomalies, and prepare accountability reports.

Tests are executed by the project team in controlled IT environments. The historical PlanAxion article describes a large-project model with eight environments in addition to the production environment.

You must also provide an adequate level of technical support throughout the testing period. A team of experts in infrastructure, operating systems, databases, and software is dedicated to the project: they handle installations, ensure environment administration and performance, apply necessary patches, manage backups, and refresh environments as needed.

How should test scenarios be managed?

Test scenarios are a lasting asset: they are reused during subsequent upgrade projects or when adding features, they inform business process documentation, and they serve as the foundation for user training materials. It is therefore essential to preserve and keep them up to date.

It is possible to accelerate scenario preparation by using a library of predefined scenarios. Some software vendors and consulting firms include sets of test scenarios in their intellectual capital that they make available to their clients.

As for test automation, the investment is not always worthwhile. The decision depends on three factors:

  • the number of potential regression test cycles;
  • the number of annual go-lives;
  • the nature of the tests involved.
Analysts validating ERP software test results at a workstation.

Who should control test configurations and variables?

A central group, independent of the testing teams, should manage changes to the software's configuration and parameters. This control ensures that changes are made in a controlled manner and replicated uniformly and concertedly across all environments. It also makes it possible to document differences between test environments and minimize situations where it is impossible to explain why software behaves differently from one environment to another.

The goal of the testing strategy is to control all variables introduced during the various test categories in order to isolate, diagnose, and resolve anomalies more easily. The five variables to control are:

  • Infrastructure : ensuring stable, high-performance, and properly managed environments.
  • Software packages : tracking versions and patches applied in each environment.
  • Parameterization : replicating configurations uniformly and in a coordinated manner by a central group.
  • Data : working with representative datasets and refreshed environments.
  • Interfaces : verifying exchanges with other systems without extraneous variables.

What are the keys to a successful testing strategy?

A successful testing strategy combines test categories tailored to the project, controlled environments, reusable scenarios, and central configuration control. By focusing efforts where complexity and impact are greatest, and by fostering collaboration between business experts and software package experts, the organization improves and optimizes its solution before go-live, rather than dealing with anomalies in production.

Which ERP tests should a Quebec SME prioritize?

Prioritize the consequence of failure rather than the number of screens checked. Adapt these examples to your business processes and organizational rules. Confirm expected results before execution.

ERP tests: business scenarios and acceptance evidence
ProcessScenario to runProposed success criterion
Distribution: order to cashOrder, partial shipment, invoice, return and paymentReconciled quantities, amounts and entries; no duplicate after replay
Manufacturing: production and traceabilityProduction order, consumption, rejected lot and returnLot origin and destination retrievable; differences explained
Services: project to billingTime, expense, approval, invoice and correctionAllocation, rates and approval match approved rules
Finance: first closeOpening balances, entries and reconciliationsDifferences explained and finance-owner acceptance recorded
Security and continuityDenied access, failed interface and recoveryAccess rules enforced, alert received and controlled recovery demonstrated

How do you write a usable acceptance scenario?

Fictional Quebec distribution example. An order contains two items but only one is available. Define the tester’s role, stock preconditions, expected partial shipment, invoice and remaining quantity. Add a return and interface replay to cover exceptions. Retain transaction identifiers as evidence.

Each test record includes the linked requirement, preconditions, data, steps, expected result, observed result, defect, owner and acceptance evidence. Executed does not mean passed. Retest defects after correction.

What criteria authorize production deployment?

Agree thresholds with the steering committee before testing. Proposed contractual criteria: all critical scenarios have passed; no blocking defect remains; data is reconciled; access is validated; the mock cutover and continuity plan are accepted. Assign an owner and due date to each exception.

Use risk rather than an overall pass rate. A 95% result would be insufficient if the missing scenarios prevent billing or production. This percentage is illustrative, not a standard.

Microsoft recommends defined scope, cycles, entry and exit criteria, and business approval. The grids and scenarios above are editorial proposals to adapt, not Microsoft certification.

Frequently asked questions

What is a testing strategy for a software package?

A testing strategy is a structured approach that plans all tests to be performed before implementing a software package. It promotes better risk management, targets the most complex parts of the solution, and allows for the improvement and optimization of the software package before it goes live.

How many test categories are there in a large software package project?

For a large project, there are generally 15 categories of tests. They are grouped into two main classes. Solution tests validate business processes and parameterization. Technical certification tests confirm the reliability of the infrastructure, performance, and environment security.

How many environments are needed to test a software package?

Eight test environments can be an organizational reference for a large project. Adapt the number to the product, contract and concurrent needs of the SME. Controlled and regularly refreshed environments allow each test category to be executed under stable conditions, isolate anomalies, and explain differences in software package behavior from one environment to another.

Should software package tests be automated?

Not always. The investment in test automation depends on the number of potential regression test cycles, the number of annual go-lives, and the nature of the tests involved. A project with few planned regression cycles rarely justifies the initial effort of automation.

Does every ERP project need eight test environments?

No. Eight environments describe one possible arrangement for large projects, not a requirement for every SME. The number depends on concurrent activities, isolation needs, product constraints and contract terms. Document conflicts between configuration, migration, training and acceptance testing before deciding.

Tests primarily reveal the quality of your governance. See how to structure client-side ERP governance to protect your project’s gate criteria and decision-making.
  • CISQ : The Cost of Poor Software Quality in the US, A 2022 Report (cost of at least $2.41 trillion USD in 2022)
  • PlanAxion : Internal control vs. application risks
  • PlanAxion : Project environments (eight environments in addition to production)

Microsoft Learn, Test your Dynamics 365 solution before deployment : scope, criteria and business acceptance; product guidance, not Quebec SME statistics.