1. technology and computing
2. software

DEPARTMENT of
INFRASTRUCTURE,
ENERGY and RESOURCES
“Acceptance Testing – How to do it!”
The Acceptance Testing Kit
Part 2
for
Projects with an IT Enhancement
or Procurement Component
A Guide
for
Project and Acceptance Test Managers
Version B
August 2001
Information Management Branch
Corporate Services
The Acceptance Testing Kit – Part 2
Acknowledgments
This Kit is the result of an initiative of the Information Management Branch of the
Department of Infrastructure, Energy and Resources.
Input was obtained from many individuals with experience in the field of
Acceptance Testing, both internal and external to the Branch, and with particular
knowledge gained from involvement with Projects with an IT enhancement or
procurement component.
In particular, the contribution of the following individuals in preparing this
document is gratefully acknowledged:
John Cuthbertson
Information Systems Analyst
DIER
Gavin Plunkett
Senior Consultant
Prologic
Allegra Huxtable
Senior Business Analyst
DIER
2
The Acceptance Testing Kit – Part 2
Table of Contents
ACKNOWLEDGMENTS ............................................................................. 2
INTRODUCTION ................................................................................... 6
FOREWORD ..........................................................................................................................................................6
PURPOSE ..............................................................................................................................................................7
HOW TO USE THIS KIT .....................................................................................................................................7
PHASE 1 INITIATION AND PLANNING........................................................ 8
SMALL OR LARGE PROJECT?...............................................................................................................................8
CREATING AN ACCEPTANCE TEST PLAN ............................................................................................................8
BEFORE YOU PREPARE THE PLAN .......................................................................................................................9
WHO DOES WHAT? ...........................................................................................................................................9
PHASE 2 PREPARING TESTS, TEST DATA, AND TRAINING ...............................15
WHAT TEST DOCUMENTATION SHOULD WE MAKE?....................................................................................15
HOW DO WE DEFINE ACCEPTANCE CRITERIA? .............................................................................................16
WHAT TEST PROCEDURES SHOULD WE FOLLOW? ........................................................................................17
WHAT TESTS CAN WE DO?............................................................................................................................17
HOW DO WE DEVELOP ACCEPTANCE TESTS? ................................................................................................21
WHAT TEST DATA SHOULD WE USE?.......................................................................................................... 22
WHAT TRAINING IS NEEDED? ...................................................................................................................... 23
WHERE IS THE BEST PLACE TO CONDUCT TESTING?................................................................................... 23
APPROVING THE ACCEPTANCE TEST PLAN ..................................................................................................... 23
PREPARING TO CONDUCT THE ACCEPTANCE TEST ......................................................................................... 25
PHASE 3 EXECUTION AND CONTROL .........................................................25
WHAT YOU NEED BEFORE YOU START TESTING ......................................................................................... 25
MANAGING THE TEST ENVIRONMENT ........................................................................................................... 26
MANAGING TEST DATA .................................................................................................................................. 26
RECORDING THE RESULTS............................................................................................................................... 29
FIXING PROBLEMS AND RETESTING .............................................................................................................. 30
SUSPENDING OR CANCELLING TESTING .........................................................................................................31
PHASE 4 CLOSURE ...............................................................................32
FORMAL ACCEPTANCE ...................................................................................................................................... 32
TEST RECORDS MANAGEMENT AND RETENTION .......................................................................................... 32
OTHER ACCEPTANCE TESTING ISSUES.......................................................33
THE TENDER PROCESS .................................................................................................................................... 33
CONTRACT MANAGEMENT............................................................................................................................... 33
DOCUMENTATION ........................................................................................................................................... 34
SYSTEM DESIGN ............................................................................................................................................. 34
PACKAGE SYSTEMS .......................................................................................................................................... 34
ACCEPTANCE TESTING A LARGE SYSTEM ...................................................................................................... 35
ACCEPTANCE TESTING A SMALL SYSTEM ...................................................................................................... 36
SYSTEM ENHANCEMENTS AND MAINTENANCE............................................................................................. 37
SOME CAUTIONARY NOTES ....................................................................38
4
The Acceptance Testing Kit – Part 2
TEST DATA CONFIDENTIALITY ...............................................................38
INTERPRETATION OF BUSINESS REQUIREMENTS ........................................................................................ 38
SYSTEM PROTOTYPING ................................................................................................................................... 38
PARALLEL RUNNING......................................................................................................................................... 38
MINOR SYSTEM DEFICIENCY DELAYS ........................................................................................................... 39
5
The Acceptance Testing Kit – Part 2
Introduction
Foreword
Acceptance Testing is a process of evaluating a new or revised system undertaken by
Business Unit Managers and end-users of the system to make sure it meets their
business needs.
With Acceptance Testing, the main objective to aim for is happy end-users and
stakeholders.
Whilst it does take time and resources to do Acceptance Testing, the rewards for the
Department, end-users and stakeholders far outweigh the effort needed to undertake
this activity. Acceptance Testing ensures that the new or improved system you want
is the system you get, at the price you were prepared to pay. Acceptance Testing
helps keep a Project budget under control by clearly stating and measuring the
delivered product and outputs against the business requirements. The Service
Provider also knows exactly what is required of them under the contract
arrangements for the delivery of the product.
The implications of overlooking or understating the importance of Acceptance
Testing are considerable. Those accountable for the Project may find the system
costing more than what was initially budgeted for because the ‘errors’ were not
properly identified and resolved before putting the product into production.
Therefore the effects of lost productivity from a product that does not meet quality
standards or functionality could have major long-term effects on Business Units.
However, Acceptance Testing must be resourced effectively and appropriate time
must be allocated to thoroughly test the product.
While this Kit can help you achieve the above, it is not a step-by-step recipe for
Acceptance Testing success. It is designed as a guide to alert and inform those
involved with a Project, and the Acceptance Testing task, of the benefits, issues, and
pitfalls that can come hand-in-hand with the Acceptance Testing process. Every
project is unique and as such, this Kit will enable this unique nature to be
successfully managed and accounted for. Above all, talk with others about the
Acceptance Testing process and benefit from their knowledge and experience they
may bring to the activity. In particular, discuss the process with stakeholders and
end-users!
In line with its Service Charter, IMB makes available this Acceptance Testing Kit to
assist Departmental staff.
6
The Acceptance Testing Kit – Part 2
Purpose
The purpose of this Kit is to help Departmental staff plan, conduct, and manage the
Acceptance Testing process for a business initiative.
How to Use this Kit
This Kit provides an overview of the essential components of Acceptance Testing. It
is divided into two Parts.
Part 1 Acceptance Testing – Why do it? introduces Acceptance Testing and outlines the
reasons for making it an essential element of any Project. It provides the motivations
for Project Managers and those involved in Testing to undertake the activity and the
benefits that can be expected from conducting the Testing process thoroughly. There
is also a brief outline of the steps required to execute Acceptance Testing from start to
finish.
Part 2 Acceptance Testing – How to do it! provides the person responsible for
Acceptance Testing with a guide for moving through the process. Included are the
types of individual tests that may be considered, how they can be managed, and a
series of templates and checklists that minimise the need to ‘reinvent the wheel’.
These resources should prove invaluable in assisting those involved in Testing.
Use of this Kit enables a consistent approach to Acceptance Testing within different
types of projects, for example from a small upgrade to an existing system, to
implementation of a new, large, software package. By referring to the Kit, the person
responsible for Acceptance Testing should be able to determine which components
or key elements need to be included within their Acceptance Testing process.
To err is human, but to really foul things up requires a computer.
Anon.
7
The Acceptance Testing Kit – Part 2
Phase 1 Initiation and Planning
Small or Large Project?
Whilst every Project has its own unique characteristics, they will often take on a
number of very similar features. As a general rule, the less expensive the Project, the
less resources will be required to Acceptance Test. This will not always be the case
but as a guide, a system developed for less than $50000 can be considered a small
system, and over $50000, a large system. If you have any uncertainty as to the nature
of the system you are involved with, consult with the Information Management
Branch in your Division.
Creating an Acceptance Test Plan
The first step in Acceptance Testing is preparation of a Test Plan.
Whenever Acceptance Testing is undertaken, there needs to be a clear plan and welldefined objectives.
The planning stage includes the following activities:
A good start to
preparing an
Acceptance Test Plan
is to first write a Table
of Contents for review
and approval.
►
Agreement with management and stakeholders on
the Table of Contents;
►
Documentation of the Acceptance Test operating
environment, Acceptance Criteria, and testing
responsibilities;
►
Preparation of an Acceptance Test
schedule;
►
Definition of the Acceptance Test
processes
including
security
procedures;
►
Determination and planning
training for Test participants; and
►
Document arrangements for review and sign-off of the Acceptance Test Plan.
of
Where a formal contract exists with a
Service Provider, there may be clauses that
require Acceptance Testing to be completed
in an agreed timeframe. The Plan should be
prepared according to these conditions.
The task of developing the Acceptance Test Plan is
the responsibility of the Manager of the Business
Unit that ‘owns’ the system. However, the task
may well be delegated, for example, to the Project
Manager. There is an Acceptance Test Plan
template included as part of this Kit.
Operating environment includes
the
hardware, software, and
network within which the system will
run. It is important to define the
operating environment because the
system
may
work
in
one
environment but not in another.
8
The Acceptance Testing Kit – Part 2
How to Prepare the Plan
Gather together the following items, information, and documentation:
►
The Acceptance Test Plan template and checklists;
►
The requirements specification (either
business or functional) for the system as this
includes the requirements to be tested,
and/or any additional Acceptance Criteria
that may exist in the Contract or Service
Level Agreement;
Template – Acceptance Test Plan
Checklist – Acceptance Testing
Checklist – Acceptance Test Planning
►
The environment the system will operate under as detailed in the Technical
Environment – Hardware and Software document (obtainable from your
Information Management Branch or on the intranet); and
►
A list of the documentation to be delivered with the system including system
release notes if this is a system upgrade.
Once all this information has been obtained, work through the Acceptance Testing
checklist and Acceptance Test Plan template, completing all the relevant sections.
Who Does What?
The roles and responsibilities for Acceptance Testing should be outlined in the
Acceptance Test Plan. The roles required will depend upon the size of the Project,
and consequently, the magnitude of the Testing task. There are four groups that
need to be involved. Remember, one person may fill more than one role.
Acceptance Testing Team
Acceptance Test Manager
9
►
Manage the Acceptance Test and coordinate Acceptance Testing activities;
►
Prepare the Acceptance Test Plan and Acceptance Test procedures and
request/obtain the necessary Acceptance Test resources;
►
Advise on establishment and training of the Acceptance Test team;
►
Assign Acceptance Test tasks;
►
Coordinate compilation of, and access to, Acceptance Test data;
►
Liaise with the Service Provider Project Manager;
►
Oversee development of Test Cases/Scripts, based upon business requirements
as detailed in the Contract, RFQ/RFT or other documentation upon which
system development will be based;
►
Formally report to the Project Sponsor and Business Unit Manager on the
status of Acceptance Testing;
The Acceptance Testing Kit – Part 2
►
Formally manage, record, and authorise modifications to the Acceptance Test
system, the documentation, the Test data, and the Acceptance Test
environment;
►
Manage and liaise/communicate with stakeholders regarding change requests
and Test problems;
►
Ensure that Tests are completed to the agreed schedule;
►
Review Test results;
►
Administer Test problem reports and recommend priorities for resolution;
►
Ensure Tests are repeated where necessary;
►
In the event of serious problems, determine whether to recommend suspension
or cancellation of Testing; and
►
Recommend formal acceptance of the system to the Project Sponsor.
Business System Administrator
It is desirable that the person who will serve as System Administrator once the
system is in production, participates in Acceptance Testing as Business System
Administrator.
►
Administer and initialise the system configuration data for:
• End-users and end-user system privileges;
• Printers and stationery;
• Work stations;
• System codes and settings;
• Access to the Acceptance Test data; and
• Batch processing;
►
Verify production support facilities;
►
Test system administration functions;
►
Coordinate generation of Test data;
►
Test system documentation;
►
Run routine batch processes and reports including audit trail reports; and
►
Run data integrity checking routines and monitor data integrity.
Senior User Representative
►
Liaise with the Acceptance Test Manager;
►
Assist in development of Test Cases/Scripts;
►
Coordinate the testing activities of Business Unit officers;
10
The Acceptance Testing Kit – Part 2
►
Coordinate the use of Test data;
►
Undertake Tests as requested;
►
Record Test Cases and conditions; and
►
Record and report successful completion of Tests and document problems
encountered.
Business Unit Officers/End-Users
►
Assist with the Acceptance Test Plan as assigned;
►
Prepare and provide Test data, and write Test Cases/Scripts, with expected
results;
►
Undertake tests as requested; and
►
Record and report successful completion of tests and document problems
encountered.
Service Provider Team
Service Provider Project Manager
►
Ensure that delivery responsibilities in relation to Acceptance Testing, as
outlined in the contract, are met;
►
Oversee establishment of the Acceptance Test environment in consultation
with IMB;
►
Liaise with the Acceptance Test Manager and assist with Plan preparation and
procedure definition as requested;
►
Ensure Development Testing of the system is complete prior to hand-over of
the system to the Acceptance Test Manager; and
►
Supervise necessary amendments to the system and manage new system
releases.
Service Provider Team Members
11
►
Establish and maintain the Acceptance Test environment.
►
Assist end-users with testing as required;
►
Conduct all necessary technical Testing prior to system hand-over;
►
Resolve system problems and update system versions as required; and
►
Migrate programs, database changes, and data to the Acceptance Test
environment.
The Acceptance Testing Kit – Part 2
IMB Representatives
Database Administrator
►
Migrate application to the appropriate test/pre-production environment;
►
Initialise the database (and re-initialise as required). This may include the
loading of appropriate security-related data;
►
Ensure database integrity, both in content and security;
►
Ensure establishment of the application environment parameters;
►
Ensure ancillary and support software is loaded and configured;
►
Provide database support services as required;
►
Undertake any specialised Testing requirements;
►
Ensure requested database modifications are appropriate;
►
Coordinate database backup procedures as outlined in the Acceptance Test
Plan;
►
Establish all end-of-day and batch processes; and
►
Support standard application release procedures and verify their operation.
Infrastructure and Support Services
►
Set up and maintain the application environment;
►
Provide the required hardware components for the Testing environment;
►
Provide and support desktop, communications, and server environments in
accordance with the Acceptance Test Plan (which in turn will nominate the
environments required for Testing, Training, and Pre-Production); and
►
Assist with stress, performance, and integrity testing as required in the Test
Plan.
Business Analyst
►
QA the Acceptance Test Plan.
IMB Management
►
Act as a contact point for any resource/staff requirements as outlined in the
Acceptance Test Plan; and
►
Oversee the hardware and environment requirements and budget accordingly.
12
The Acceptance Testing Kit – Part 2
Management Representatives
Project Steering Committee
►
Provide policy and direction for Acceptance Testing;
►
Review and verify the incorporation of business objectives and requirements
into any Contract, RFQ/RFT etc.;
►
Resolve contractual issues and issues relating to system scope;
►
Review and approve or reject submitted change requests; and
►
Endorse acceptance of the system and authorise its implementation.
Project Sponsor/Business Unit Manager
13
►
Develop business objectives and requirements and ensure they are clearly
detailed in any Contract, RFQ/RFT etc. in order to base Acceptance Criteria
upon them;
►
Appoint and direct the Acceptance Test Manager;
►
Approve the Acceptance Test Plan;
►
Manage the provision of requested Acceptance Test resources, including
Business Unit officers to participate in the Acceptance Tests;
►
Review Test results;
►
Accept each major sub-component of the system;
►
Participate in, and report to the Project Steering Committee; and
►
Formally accept the new system and recommend system implementation.
The Acceptance Testing Kit – Part 2
Roles for Large Projects
The recommendation for role assignments for a large project are:
►
Acceptance Testing Team;
►
Service Provider Team;
►
IMB Representatives; and
►
Management Representatives.
Roles for Small Projects
The recommendation for role assignments for a small project are:
►
Acceptance Test Manager (this role may be undertaken by the Project Manager
or Business System Administrator);
►
End-User(s)
►
Business Systems Administrator
►
Service Provider Team
►
IMB Representatives
►
Business Unit Manager (or other appropriate Management representative)
14
The Acceptance Testing Kit – Part 2
Phase 2 Preparing Tests, Test Data, and Training
Acceptance Testing is a project in itself, depending on the scale and scope of the
system or system changes. There is a defined body of work, a distinct start and end,
and a well-defined objective. As such, Acceptance Testing is best managed using
‘conventional’ project management processes. There needs to be:
►
An overall guiding plan;
►
Well defined
communication;
►
A detailed schedule of tasks with dates and
assignments;
►
Well defined processes and tasks;
►
A controlled environment, documentation, and product;
►
Processes to capture and record details of progress and to monitor them
against the plan; and
►
Regular review meetings.
responsibilities
and
lines
of
Don’t forget to report:
• Overall progress;
• Testing status;
• Problem status;
and
• Change status.
What Test Documentation Should We Make?
Depending on the scale of the Acceptance Test, for maximum benefit and minimum
effort, establishing and maintaining the following records and documentation can be
very helpful. The Acceptance Test Strategy is generally only necessary for a large
project, whilst all other documentation is needed for both large and small projects,
with documentation “scaled down” for the smaller projects.
►
An Acceptance Test Strategy - outlining the approach and understandings for
Testing activities during the Project. This is best formulated soon after Project
inception and can be included in the Project Execution Plan (should one exist).
A template can be found on the DPAC Project Management web site;
►
The Acceptance Test Plan - outlining the Test approach, objectives, activities,
responsibilities, and schedule. This includes:
• Test Procedures - documenting the processes to be followed during
Acceptance Testing. Procedures;
• Authorisation
cancellation;
requirements
for
commencement,
• The conduct of Tests and recording of results;
• The recording and resolution of Test problems;
• Signing-off and formal acceptance;
15
suspension,
and
The Acceptance Testing Kit – Part 2
• Authorisation requirements for changes to the Test system and/or
environment; and
• The release of a new version of the system;
►
Test Cases - showing Tests to be performed, usually measured against business
requirements (with any specific data or conditions), and the expected
outcomes. There may be many Test Cases for a single business requirement.
These include recording the outcome of each Test Case and/or Script as Test
Results;
►
Test Scripts - documenting individual testing processes as a series of steps,
many of which will be series’ of interrelated Test Cases and encompass an
individual business requirement/process; and
►
Problem Review Sheets and Problem Review Register - documenting problems
encountered and indexing the status of each problem.
In addition, maintain an:
►
Acceptance Test Log - recording significant events and changes to the test
environment and test data.
Templates for these documents form part of this Kit.
Samples and templates of these documents form part of this Kit.
How Do We Define Acceptance Criteria?
There needs to be clearly defined criteria on
which to base system acceptance. Acceptance
Criteria should be clearly documented so they
are understood by all parties. Ideally, they Even though Acceptance Criteria are often
should be decided ”up front” at the Project specified in the relevant contract or
agreement with the system supplier, they
Planning stage, prior to system development. should still be included in the Acceptance
They should then be incorporated into any Test Plan.
Contract to be fulfilled by the Service Provider. Acceptance Criteria will then be
based upon the requirements as specified in the Contract (or any other less formal
agreement) with a Service Provider, or in the case of an internal system development,
upon requirements as specified in the business requirements documentation. If
specified in a Contract, these requirements will in turn, hopefully be based upon the
business requirements as detailed in the relevant Business Systems Requirements
Specification.
It is not the responsibility of the Acceptance Test Manager to define or “invent”
Acceptance Criteria at the Testing stage. This point is too late in the development
process! If Acceptance Criteria are either sufficiently unclear or difficult to obtain,
refer the issue to the Project Manager for direction and resolution.
16
The Acceptance Testing Kit – Part 2
Often, Acceptance Criteria will also cover a number of “standard” items, that may
include:
►
system and user documentation;
►
system useability;
►
system performance and capacity;
►
data integrity;
►
system security; and/or
►
system reliability.
What Test Procedures Should We Follow?
The Acceptance Test Plan details the overall
process
of
initiating,
conducting,
and
concluding Acceptance Testing.
More
specifically, there needs to be a clear outline of Determine testing procedures at the
planning stage as they may require advice
the actual testing process used for confirmation from outside the business unit. It may be
of the Acceptance Criteria. Document these necessary for IMB in your Division to assist
you with planning and execution.
processes in the Acceptance Test Plan and
translate them into Test Cases and Test Scripts that define the Testing steps.
What Tests Can We Do?
If new business processes are intended and Template – Test Case
Template – Test Script Cover Sheet
contracted to be introduced in conjunction with a Checklist – Test Case or Script
new system, then it is necessary to prove these
new processes during Acceptance Testing. Aspects of any new system
relevant to business processes may prove unsatisfactory for end-user purposes. If
this is the case, the Business needs to correlate the unacceptable system behaviour
with the business requirements or specifications as detailed in the supply contract.
If the number and degree of business process
changes are significant, formal business change
management processes can be applied in
conjunction with system development and
implementation.
Responsibility for defining and proving new
processes rests with the business unit,
rather than the system supplier or developer.
The Business Unit is obliged to accept the system ‘as is’, should it satisfy all
contractual obligations of the Service Provider.
Therefore, it is imperative that business
requirements are precisely identified at the
Project Planning stage and included in the For each business or system requirement
Contract. Should these contractual obligations and for each Acceptance Criterion, there
should be at least one Test Case.
be met, and the system still fails to meet one or
17
The Acceptance Testing Kit – Part 2
more business requirements, steps may then be authorised (possibly at additional
expense) to improve the system with alterations. However, system changes at such a
late stage of the development life cycle are less than ideal.
As well as basic function testing, Tests may also evaluate:
►
Day, week,
processing;
period,
month,
and
year-end
►
Interfaces;
►
System sociability;
►
System performance and reliability in a range of
environments and under any special conditions;
►
Stress capabilities;
►
Backup and recovery procedures;
►
System access and security;
►
Concurrent updating; and
►
Disaster Recovery.
You may be able to use a
stop watch to measure
system response times. For
more accurate readings and
for internal processes (such
as database performance),
system monitors and
recorders may be necessary.
Checklist – Report Testing
Checklist – Web Page Usability Testing
Checklist – User Interface Testing
There may also be specific tests reviewing the supplied system
documentation (on-line help system, user guide, operations guide, maintenance
documentation etc.).
Generic Tests
For many Tests there will be common features e.g. all screens need to be tested for
ease of use and appearance. These generic tests are better managed with checklists
or standard forms rather than Test Cases. For each screen utilised during the
running of each Test, the Tester should review the screen using the generic User
Interface checklist contained in this Kit. Similarly, for each report generated during
the running of a Test, the tester should check the report using the Report checklist.
An additional checklist for Web Page Useability is also included.
Stress and Performance Testing
This form of testing addresses both specific
contractual criteria and general system performance.
By conducting both stress and performance testing in
a controlled manner, you can be sure of accurate
response and performance measurements where
system loads are precisely known. These loads may
include network traffic or database activity, as well
as normal system functions.
Stress testing is where specific
system functions (usually critical
ones) are subjected to high loads to
ensure that the system will still cope.
Performance testing is where
specific system functions (usually
critical ones) are timed under
various system loads to ensure that
the
times
meet
contractual
performance acceptance criteria.
18
The Acceptance Testing Kit – Part 2
If requirements for system throughput are specified in the Contract, then stress
testing is necessary, and Test Cases should be clearly defined. The functions to be
tested may be either on-line or batch. Stress tests tend to focus on a particular aspect
of the system’s architecture such as database updating, for example, to ensure that
the database server’s capacity meets expectations.
In performance testing, system loads may
include network traffic, database activity as well
as normal system functions. Controlling these
loads is not always straightforward and the
assistance of IT personnel is often needed with
the setup and execution. The steps and tasks to
be undertaken should already be itemised in the
Acceptance Test Plan.
Regression tests are most successful if the
test data, test environment, system dates,
and any other test inputs are either kept
constant or only changed in a controlled
manner so the impact of any changes can
be anticipated and/or explained.
Regression Testing
Regression testing is an important method of
validating system changes that occur during
Acceptance Testing. It ensures, where a change or
changes have been made to the system, the test
environment, or the test data, that these changes
do not impact the correct operation of functions
not obviously or directly related to the changes.
Regression testing is where a
series of tests (or indeed all tests)
are systematically repeated and
where expected results are known.
The results of the repeated test are
compared with the results of the
original (successful) tests.
Regression tests are usually done manually and can be time consuming and tedious.
By minimising the number of times that changes are made to the system, the Test
environment, and the Test data, effort can be significantly reduced. It is most
practical to regression test only critical functions and those known or suspected to be
affected by changes.
For any system change, the Acceptance Test Manager
should determine which Test Cases or Test Scripts
need to be repeated, and schedule these
accordingly.
System crashes can logically or
physically corrupt the database.
To ensure this does not happen,
comprehensive database
integrity checking routines
should be available.
Over time, and with the assistance of
experienced end-users of the system, a matrix of
impact can be compiled i.e. for each system
function there is a cross-referenced list of other
functions that may be potentially affected by changes to the first
function.
While tools to support regression testing are available, their use is not always cost
effective. However, using electronic means to compare the results of repeated Tests
against the results of the original (successful) Tests is a possibility.
19
The Acceptance Testing Kit – Part 2
Disaster Recovery Testing
Disaster recovery testing needs to be conducted in a controlled manner to allow the
system to be “crashed” at a precise point, with the status of active system functions
accurately known, to test the system’s disaster recovery process.
Crashes are arranged to occur at strategic points such as in the middle of a multirecord database update. Recovery from simulated events such as complete loss of a
database or server can also be tested.
Two processes are used to determine the system’s ability to recover from a disaster.
Firstly, where the whole system has crashed, the database is restored from a previous
backup, the system transaction log is applied to the restored database (rolled
forward), and the logical and physical integrity, and the completeness of the
database is checked.
The second process is where multi-database update functions are crashed and the
system is allowed to “roll back” i.e. the partial updates are undone automatically by
the database software and the logical integrity and completeness of the database is
checked (especially for the data involved in the crashed functions).
Concurrent Update Testing
If the system you are testing is an on-line, shared
system, you will need to conduct concurrent
update testing.
Concurrent update tests can fail without any
apparent impact on the user. However, the
integrity of the database must remain intact
and can be confirmed using integrity
checking routines.
Concurrent update testing assesses a system’s
capability to deal with multiple system functions
simultaneously attempting to access or update
the same database record(s). How the system handles these situations is based on
the system’s technical design. The system needs to cope with the situation where an
update transaction tries to re-read records that were read earlier but that have now
become locked or changed.
Irrespective of the internal processes, from the end-user
point of view the system should provide a
meaningful message and terminate the transaction in
a controlled manner, allowing the user to restart the
process.
The logical and physical integrity of the database
should remain intact and this should be confirmed
using the appropriate integrity checking routines.
The potential for problems in
large batch processes is often
high. If something goes wrong,
it may not affect just one Case.
20
The Acceptance Testing Kit – Part 2
Large Reports, Batch Processes and Interfaces
The amount of effort in preparing for tests of large reports, batch processes and
interfaces is often underestimated. Most attention tends to be directed to the on-line
system components. However, these items must be thoroughly tested as well.
How Do We Develop Acceptance Test Scripts?
It is important to always maintain the relationship between any Test Case and the
requirement or acceptance criterion it is verifying. There are two steps to follow
when developing Test Cases:
►
Step 1 is to describe what is needed for each Test, what is to be done, and the
expected outcome. A Test Case template is included as part of this Kit.
►
Step 2 is to document the required Test Data and conditions that enable the
Test to be undertaken.
To help you in developing your Tests, a Test Case and Script checklist has been
included as part of this Kit.
For each Test, determine:
►
The features to be tested;
►
The source database(s) that will be required;
►
The source of data required specifically for the
A Test Script is a documented
process involving a series of
steps, many of which will be
interrelated Tests. It may also
describe a set of Test conditions.
Test;
►
Any other resource requirements;
►
The task that will be performed in the Test Case;
and
►
The expected outcome.
A Test Case is a single record
or group of records used in a
Test Script.
Once Test Cases are prepared, collate related Test Cases into Test Scripts. Each Test
Script usually equates with a business process. The Script brings all related Tests
together into a logical sequence.
For example, in a financial system there may be a
series of tests to record different types of on-line
financial transactions and there may be other tests to
create and post a batch of transactions to the
system and to print a batch report. It makes
sense to relate the on-line transaction entry to the
subsequent batch posting and printing.
Remember to write Test Cases
for reports, offline processes,
interfaces, and data migration.
These are often overlooked yet
most likely to cause headaches
after the system has been
implemented!
If new business processes are involved, describe in
the Test Script the overall approach required to test them,
21
The Acceptance Testing Kit – Part 2
including the normal and alternative paths, and key exceptions.
For each Test Script, identify resource requirements, including any resources outside
the organisation. A Test Script template is available as part of this Kit.
What Test Data Should We Use?
Preparing Test Data can be a time consuming process. To
minimise effort, first validate Test Data and then preserve it
for subsequent re-use should it become corrupt through
failed tests.
To minimise data
contention between
tests, discrete test
data may be assigned
to one or several
(related) Test Scripts.
It may be possible to create data via direct data entry or by
electronic transfer (such as exporting from a spreadsheet and
direct loading into the test database). Test
Data may also be taken from standard input
forms used within the Business Unit. In some
instances where the data is not of great End-users familiar with the application area
significance, the Tester will be required to and system are best for creating test data.
“make some up”.
A useful source of data (especially where large volumes are required) is data
migrated from an existing system. However, the data migration and load processes
require thorough testing themselves and often migrated data may be incomplete and
of questionable quality.
A good way to organise Test Data is to match it with Test Scripts and maintain this
link throughout the Testing process.
Protecting Confidential Test Data
By conducting Acceptance Testing in an environment and with data that closely
emulates live operation, the chance of problems arising once the system is
implemented is decreased considerably. Unfortunately, using “real” data often
means a greater risk of accidental distribution of confidential information (for
example, being printed, emailed, or transferred over a public or wide-area network).
To minimise this risk, design security procedures into the Acceptance Test Plan and
make sure all testing personnel are trained and alerted to the dangers. Other ways of
maintaining data security include:
►
Establishing a physically separate network with separate printers;
►
Encrypting or modifying data such as names and addresses, so they are
unrecognisable;
►
Creating server and workstation directories specifically for the use of
Acceptance Testing;
►
Establishing specific Acceptance Test email accounts;
22
The Acceptance Testing Kit – Part 2
►
Regularly and routinely cleaning up Acceptance Test directories;
►
Routinely shredding any printed reports; and
►
Using special stationery that indicates the printed data is for test purposes
only.
What Training is Needed?
Training needs should be outlined in the
Acceptance Test Plan. This allows for the
scheduling of training and assignment of
trainers. This will usually be arranged by the
Acceptance Test Manager.
User Training is normally covered in the
overall
Project
Plan.
For
end-users
participating in Acceptance Testing, training
must occur prior to Testing commencement.
Participants in the Acceptance Test may also need training in the Test procedures for
undertaking Tests and recording results. This may take the form of a walk-through
or dry run of some Tests, as well as a formal training session. These sessions need to
be allowed for in the Acceptance Test and Project Plans.
Where is the Best Place to Conduct Testing?
To help personnel focus on the job at hand, Acceptance Testing
is best done in a separate work area where participants are
not distracted by their normal work activities. A dedicated,
laboratory-style facility with plenty of work space is ideal.
This allows informal communication between participants
and the Acceptance Test Manager.
Use a white board and
notice board to
communicate
Acceptance Test
“announcements” and to
display testing progress.
A secure area also provides a good location for storing
Acceptance Test records and documents and provides a
focus for testing activities.
Approving the Acceptance Test Plan
It’s important that management formally approves the Test Plan. The Manager of the
Business Unit that is responsible for the system should sign-off the Plan. This helps
ensure personnel and resources are committed to Testing when it occurs.
The Acceptance Test Plan should be reviewed by:
23
►
The Business Unit Manager;
►
Those with key responsibilities in the Test Plan – the
Acceptance Test participants should be offered the
opportunity to review and comment. It will help
them understand the nature of their involvement and
allow them to voice any concerns;
Remember, the Plan
needs formal approval
by the manager of the
Business Unit
responsible for the
system.
The Acceptance Testing Kit – Part 2
►
The Service Provider – this may be a contractual requirement (if it is not,
allowing them to review the Plan will clarify any issues they may have). They
should understand end-user expectations for the system and the support (if
any) they are required to provide;
►
Any other key stakeholders; and
►
An independent quality inspector - they may pick up items that seem obvious
to the author and participants, but may be open to misinterpretation.
24
The Acceptance Testing Kit – Part 2
Phase 3 Execution and Control
Preparing to Conduct the Acceptance Test
If the Acceptance Test Manager takes on responsibility for scheduling and assigning
Testing tasks e.g. assigning individual Test Scripts to team members, workflow will
run more smoothly and efficiently. Try and avoid any individual member of the
Testing team shouldering too great a workload. It will also help in tracking work
progress as sometimes, when people from different Business Units are involved, the
same Test Script may require running by more than one Tester.
The Acceptance Test Manager should ensure that each participant understands their
assignment, is aware of the Test procedures, and knows the timetable expectations
for each task. By previewing the Test Script and individual Tests, a tester can clarify
any concerns with the Acceptance Test Manger before undertaking a Test Script.
What You Need Before You Start Testing
Ideally, Acceptance Testing should run according to the Acceptance Test Plan from
start to finish without disruption and without the need to change either the system or
the Testing environment.
In reality, things do go wrong, and continuity is often difficult to achieve. To
minimise the likelihood of disruption, Acceptance Testing should only proceed if the
following have occurred:
1. The Acceptance Test Plan, including all Acceptance Criteria, has been signed-off
by the Business Unit Manager;
2. There has been formal management approval to proceed with Acceptance
Testing;
3. Test Data has been made available;
4. The Acceptance Test environment has been established, stabilised, and protected
against unauthorised or inadvertent change;
5. The system has been thoroughly system-tested by the suppliers or developers, the
system test has been formally signed-off, and the system (including all
documentation) formally handed over for Acceptance; and
6. Testing participants have received the necessary training for using the system and
for following Test procedures.
25
The Acceptance Testing Kit – Part 2
Managing the Test Environment
Managing Changes
Before you make changes to the Acceptance Test environment, make sure they are
planned, authorised, controlled, recorded, and communicated to Test participants.
Otherwise:
►
even minor changes may impact the validity of tests already completed; and
►
participants can become confused if the status and versions of the Test
environment are uncertain.
Keeping Track of Testing Progress
Establish an Acceptance Test Log to record all significant events occurring within the
Acceptance Test environment. The Log should be readily accessible and should be
used (mainly by the Acceptance Test Manager and System Administrator) to log
changes to:
►
The system software;
►
System documentation;
►
System parameters, code tables, and
reference data;
►
Baseline Test Data;
►
Migrated Test Data;
►
External systems and/or data;
►
Database software, configuration and scripts; and/or
►
Network, server, and workstation software, and configuration.
A Test Log helps when problems arise with
the test system or test data. If you know
precisely what took place prior to the
problem, the Log can be referenced to
prevent the problem recurring.
The information you could log includes:
►
The date and time of an event;
►
The name of the person executing an event;
►
Details of the event;
►
The signature of the authorising person; and
►
Any comments about the event and its outcome.
Template – Acceptance Test Log
Managing Test Data
Initialising Test Data
Initialisation usually involves loading or entering:
►
The necessary system codes and system parameters;
26
The Acceptance Testing Kit – Part 2
►
Either manual or automated Test Scripts and Test Data required for the Tests;
and
►
Migrated data (usually used where large volumes of data are required, and
often from a separate database).
When testing an upgrade to an existing system, the Test
database will normally be populated with data extracted
or copied from the existing live database and there may
be little or no need to add additional data.
Once you’ve prepared your
test data, make a copy
should the data become
corrupted in the future. It’s
useful for when further
system changes are made
and another round of
testing is required.
When testing a new system, the Test database will
probably be empty and need setting up with an initial Test
dataset. Some data may be loaded automatically from
spreadsheets, text files, or other databases using database
scripts. However, data may need to entered directly using the system’s standard online functions.
Backing up the Acceptance Test Database
Regularly backing up the Acceptance Test database allows recovery of the database if
the need arises to re-test, or to eliminate corrupt, spurious, or unnecessary data.
Usually, backups (and any necessary restores) need to be authorised by either the
Acceptance Test Manager or the System Administrator, and should be recorded in
the Acceptance Test Log.
Managing Changes to Test Data
Another useful and important tool is a record of changes to
the Test Data. These changes can affect the outcome of
Tests and the effectiveness of regression tests. It may be
necessary at times to restore the Test Data from an earlier
backup to eliminate unwanted or unsuccessful changes to
Test Data.
Record any important changes in the
Acceptance Test Log.
Backing up at strategic
points will enable you to
more easily return to an
historical data position in
the testing process, with
less time and effort.
The Acceptance Test Manager or the System Administrator should coordinate the
use of Test data between the various Testers and maintain a record of changes in the
Acceptance Test Log. You may also find it useful if summaries of Test data entered
during Acceptance Testing are logged as Tests are undertaken.
Checking Database Integrity
It is important that database integrity remains intact. Sometimes, Tests can affect
database integrity, particularly by programmes that don’t operate correctly. For
example, Tests such as crashing the system to Test recovery processes can cause
physical corruption of the database where whole records are lost or corrupted.
Processes that check the integrity of the test database may be provided by the system
27
The Acceptance Testing Kit – Part 2
supplier (they may be included in the system
requirements) or they may have to be provided by
database administrators in the form of database scripts.
When run, these routines scan the database tables and
relationships and report any logical data integrity
problems. The Information Management Branch in
your Division will be able to provide assistance with
these.
Database integrity is when the
data retains internal consistency or
lack of corruption.
Database
integrity checking routines are
processes (programs or scripts)
that scan the database tables and
links and report any logical data
integrity problems such a missing
relationships, undefined codes etc.
They may also check the physical
integrity of the database
The System Administrator routinely runs these scripts
during Acceptance Testing, either at the completion
of each Test Script or at the end of each Test session. They should also be run
following any significant event or activity that affects the database (such as the
loading of migration data).
Results should be reviewed to detect any problems generated by the Test undertaken
since the previous (clean) integrity check report. Keep the results with the
Acceptance Test Log for future reference.
Checking Audit Trails
When Tests are run (and if the system provides an audit trail of changes) then
records written to the audit trail should confirm changes made by the Tests.
The System Administrator should routinely run reports that allow the System’s audit
trails to be monitored. These will be run at the completion of each Test Script or at
the end of a Test session, and also following any significant event or activity that
affects the database.
These reports should be reviewed with the Testers to ensure all significant updates
have been correctly recorded in the audit trail.
Results of these runs should be retained with Acceptance Test records.
Migrated Data and the Migration Process
Where data needs to be moved or relocated from an old system to a new system, it
becomes necessary to test the migration process and the migrated data. If some
Acceptance Testing steps are dependant on using migrated data (for example, if large
volumes of migration data are to be used for stress and performance testing), then
migration process testing may need to be completed first. Consider loading the
migration data into a separate (full size) Test database.
As migrated data is often incomplete and may include generated default values, it is
necessary to repeat some functional Test Scripts using migrated data both with and
without other Test data i.e. it is necessary to ensure that the system functions
correctly with the migration data.
28
The Acceptance Testing Kit – Part 2
Reconciliation of control totals for data extracted from existing systems and loaded
into the new system is an important validation check. These steps should be
included in the migration Test Script or Scripts.
Recording the Results
For every individual Test, the result will be either ‘Successful
Completion’ or ‘Failed’. The assigned Tester records the
result of each Test Case within a Test Script in the Test
result area on the Test Case sheet. The Tester then signs
Remember, as the
Acceptance Test Manager,
the Test Case sheet alongside each Test Case and also
check and endorse the Test
the Test Script sheet once the full Test Script has been
Script sheets and file them
in the Test records.
completed. Testers must remember to verify the version
Maintain a register of
of the system tested, the date and time of the testing, and
completed Tests and Test
Scripts with their status.
any Test data identifiers.
The Test Script result (bearing in mind this covers
several Test Cases) will then be either ‘Successful Completion’, ‘Partial Completion’,
or ‘Failed’.
How Do We Manage Problems?
For each Testing issue encountered, the Tester should first seek assistance to
confirm that the issue is a valid one and not a temporary configuration or data
related problem. At least one experienced system
user should be on hand. Ready access to the
necessary technical support should also have been Template – Problem Review Sheet
of Problem Review
pre-arranged. This may be either personnel from Template – Register
Sheets
IMB or Service Provider staff.
The Tester should complete an Acceptance Test Problem Review Sheet for any
‘Partial Completion’ or ‘Failed’ Test results and the Acceptance Test Manager should
record these in the Problem Review Register. Also, remind Testing personnel to
record any documentation and on-line help inconsistencies and errors, and any other
issues, even if they are not directly related to the current Test Script.
Record and Categorise Problems
To keep track of Testing and Test problems, the Acceptance Test Manager needs to
monitor and report on Acceptance Testing progress and the status of outstanding
problems.
Each recorded problem will reference the relevant Test Case, Test Script, and
business requirement (or requirements as detailed in the Contract). Problems will be
graded by priority (severity and impact) and categorised by type.
29
The Acceptance Testing Kit – Part 2
Problems that can only be resolved by re-specifying business requirements
(compared to those in the Contract) should still be recorded but categorised so as to
be clearly identifiable as a possible future change request or Contract variation.
Attempt Resolving Problems With the Supplier
Management of any contractual issues arising from Acceptance Testing is best left to
the Acceptance Test Manager and Project Manager who will try to resolve the
problems with the Service Provider Project Manager. The outcome of any
discussions needs to be recorded against individual problems as they appear in the
Problem Review Register.
The Acceptance Test Manager and the Service Provider Project Manager will
endeavour to agree on the definition of the problem and a possible solution.
If agreement can’t be reached as to the cause of or responsibility for any problem,
refer the matter to the Project Sponsor or Project Steering Committee.
Fixing Problems and Retesting
Change Control
When Acceptance Testing occurs, it is in a defined and agreed environment, against
an agreed Plan and timetable, with agreed Test data, against agreed Acceptance
Criteria and with a specific delivered version of the system and documentation.
Any changes to these items should be managed and assessed by the Project Team,
the change impact defined, measured and costed, and due consideration given to the
change before it is approved and applied.
Where changes to the Test environment, the system, or the Test data are made, they
may impact the integrity of Testing already completed.
System Upgrades and Retesting
The Acceptance Test Manager will need to manage and authorise any upgrades or
changes to the Acceptance Test environment required to resolve Testing problems.
All changes to the system and Acceptance Test environment should be logged.
The system supplier should provide details of changes and the problems that have
been resolved with each new Release. Where applicable, regression testing may need
to be undertaken to confirm that new problems have not been inadvertently created
by changes that resolve other issues.
The number of new releases should be as few as possible and should be stipulated in
the Contract. Ideally, there should be only one subsequent release of the system.
Too many releases will result in extra work and delay the Acceptance Test schedule,
bringing into question the overall quality of the delivered system. Where there is an
30
The Acceptance Testing Kit – Part 2
external system supplier, the consequences of this should be determined by the
contractual conditions.
The Acceptance Test Manager and the Project Manager, with advice from the Service
Provider Project Manager if required, will determine the degree and extent of
retesting necessary.
‘Patching’ The System
While a “quick fix” may seem like a good idea at the time, don’t rush into any
process of making software or system “patches”.
It can often result in
undocumented changes together with problems reconciling subsequent system
releases with any patches and can lead to loss of control of the system and the status
of the Test Environment.
The Acceptance Test Manager can authorise patches, but only after considering all
the options and preferably, allowing for the ability to reverse any changes.
Remember to keep accurate documentation!
Suspending or Cancelling Testing
Where problems exist that inhibit the continuation of effective Testing, the
Acceptance Test Manager (under advice from the Project Manager) may suspend
Acceptance Testing until the relevant problem or problems are resolved. If no
resolution is possible in the immediate future, there may be grounds for cancelling
any further testing. The criteria for suspending or cancelling Acceptance Testing
should be documented in the Acceptance Test Plan and in the Contract.
31
The Acceptance Testing Kit – Part 2
Phase 4 Closure
Formal Acceptance
Once all Acceptance Criteria have been met, the Acceptance Test Manager is in a
position to recommend formal acceptance of the system. Upon this recommendation
to the Project Manager, the Project Sponsor or Project Steering Committee will
formally accept the system.
Acceptance is possible even if there are still specific (yet minor) outstanding
problems, providing agreement has been reached on problem resolution following
system implementation. The system may be accepted with the qualification that
these non-critical items will be corrected to an agreed schedule under Warranty or
Contract variation. In this way, acceptance can be achieved without unnecessary
delay for minor items of non-compliance.
The decision to allow a “qualified acceptance” and the items deemed “minor” is the
prerogative of the Acceptance Test and Project Manager, and the Manager of the
Business Unit that owns the system. Ideally, there should not be undue delay from
completion of Acceptance Testing to system implementation. However, this may
depend on overall Project and implementation planning.
Test Records Management and Retention
As far as is practicable, all test records should be retained at least for the duration of
the Warranty period. Ideally the electronic components can be archived to off-line
media.
Test Cases, Test Scripts, Test Data, and expected results should be retained (as far is
practicable) for testing subsequent releases of the system.
These records should be formally handed over by the Acceptance Test Manager to
the Business Unit Manager for safekeeping. You will usually find what corporate
information management guidelines are relevant from IMB in your Division.
32
The Acceptance Testing Kit – Part 2
Other Acceptance Testing Issues
The Tender Process
It is important to assess the ability of a Service Provider to satisfy system Acceptance
Criteria. Consequently, it is necessary to include these in any Request for Tender or
Quotation and to consider them in assessment of submitted proposals.
Possible Acceptance Criteria are outlined in the section How Do We Define Acceptance
Criteria? on page 16.
Contract Management
Where a Contract is involved with delivery of a system, it is essential that Acceptance
Criteria be included in the Contract. The standard GITC allows for the inclusion of
Acceptance Test Criteria. These provide the Service Provider with a clear
understanding of what is to be achieved to meet Acceptance. It is quite common for
the business requirements to be a formal attachment to the Contract.
Including Acceptance Criteria in the Contract also means that Acceptance and
Acceptance Testing is more likely to be fully recognised at inception of the Project.
As well as including Acceptance Criteria, the Contract should:
►
Define the broad responsibilities for Acceptance Testing and for the provision
of facilities and resources;
►
Define the Operating Environment for the system (and by implication, the
Acceptance Testing Environment);
►
Define the duration (and possibly the schedule) for Acceptance Testing;
►
Contain conditions to cater for delays in Acceptance Testing caused by either
party.
If there are clauses that require Acceptance Testing to be completed in an agreed time
frame, then to ensure continuity and progress, it is important that personnel assigned
to Acceptance Testing will be relieved of any conflicting responsibilities. This will
allow them to be available in accordance with the Test Plan. Allowances should also
be made for overruns and delays in Acceptance Testing.
The timetable for Acceptance Testing in the contract and Test Plan should allow for a
maximum of two sets of changes to the system (following the completion of the first
pass of Acceptance Testing) and retesting for those corrections. The need for
additional Releases may call in to question the quality of the system and the quality
of Testing undertaken by the Service Provider.
33
The Acceptance Testing Kit – Part 2
Documentation
End-user and operational documentation for a newly delivered system also needs to
be Acceptance Tested.
Appropriate end-users should be assigned to Acceptance Testing the documentation
by following it through and evaluating it for clarity and completeness. For each
system function tested there should be a Test to check the documentation and any
on-line or context-sensitive Help component.
Those ultimately responsible for operating the system in production also need to
review the documentation and follow through the operation instructions. This
should be done in a Test environment that emulates the final production
environment as closely as possible.
Once again, documentation should be subject to systematic document management
and change control procedures.
System Design
Where there are complexities in testing aspects of a system, it may be necessary to
design features into a system that facilitate testing of the system e.g. for a voiceactivated payment facility, the practicalities of testing this across a telephone
network may require the system to have built-in features to support Testing.
Package Systems
Where a solution is to be based on a package, selection of the package is often made
based on the package’s perceived ability to satisfy the business requirements. In this
case, it may be practical (or even contractually agreed) to test the package against its
documented capabilities.
Where a package is delivered with modifications, specifications for those
modifications should be documented and agreed upon, and system documentation
should be updated accordingly. These specifications can be used as the basis for
Acceptance Tests.
A second aspect of a package implementation is that a key factor in its selection will
have been the cost-effectiveness of the solution i.e. as a rule of thumb, it’s cheaper to
implement a package if it meets 80% or more of the business requirements, than to
custom-build a system. However, selection of a package will often involve
compromise of at least some business requirements. Apart from these issues,
Acceptance Testing a package system should not differ in approach to that for a
custom-built system.
34
The Acceptance Testing Kit – Part 2
Acceptance Testing a Large System
Planning
It is to be expected that the extent of a Test Plan for a large system will be greater and
consequently, the time necessary to prepare the Plan will be more significant.
Similarly, the number of stakeholders may be greater and more time may be needed
to allow adequate review of the Plan.
It may also be a contractual requirement that the system supplier also approves the
Test Plan. If this is the case, some allowance should be made for negotiation. The
Plan may imply certain functionality that the supplier believes is not part of the
system requirements.
Preparing Tests, Test Data, and Training
As the number of Test Cases increases, so does the complexity of coordinating and
ordering the Tests into Test Scripts. This is likely to involve more than one person to
execute and will require suitable management.
Similarly, the complexity of providing Test Data will increase and its preparation
and maintenance will need to be managed.
There is likely to a larger group of testers and the range of skills, knowledge, and
capability will vary. Preparation of a Skills Transfer and Training Plan is needed to
detail delivery and management of training needs.
For the Acceptance Testing to be (contractually) valid, the Test environment needs to
mirror the final operating environment, as defined in the contract.
The Acceptance Test Manager will need to oversee this step and the system suppliers
may request an audit of the Test environment.
The degree of coordination and planning will be higher. There may be a wide range
of resources and personnel to be confirmed.
Execution and Control
Its important to use the Register of Problem Reviews and plan several runs of
Testing, in particular, regression testing.
Closure
An allowance to accept the system subject to subsequent resolution of documented
outstanding (minor) problems should be made. If this is the case, the Problem
Review Register should be part of the Acceptance Document.
The records for Acceptance Testing of a large system can be quite extensive. It’s
important that these are retained safely.
35
The Acceptance Testing Kit – Part 2
Acceptance Testing a Small System
A small project is defined as one where the external cost does not exceed $50,000.
With a smaller project or system, the overall complexity of Acceptance Testing is
reduced. There are fewer people involved, there are fewer types of data and the
number of tests and problems is lower.
Nevertheless, the basic steps listed still need to be executed.
Planning
►
Prepare a Test Plan
►
Review the Test Plan
►
Arrange sign off
Preparing Tests, Test Data, and Training
►
Prepare the Test Cases
►
Prepare the Test Scripts
►
Prepare the Test Data
• Conduct training
• Set up test environment
• Confirm people and resources
Execution and Control
►
Run Test Scripts and Test Cases
►
Record the results
►
Log and manage problems
►
Fix problems and re-test
The relatively small number of problems from testing a small system should mean a
Problem Review Register is not required.
Closure
►
Formal acceptance
►
Hand over the system
36
The Acceptance Testing Kit – Part 2
System Enhancements and Maintenance
Usually changes to an existing production system are grouped together into a
‘Release’. The source of these changes may be requested functionality enhancements
and/or resolution of reported problems.
Planning
Existing DIER applications should already have processes established for system
upgrades, maintenance, and enhancements. It may not be necessary to establish a
formal Acceptance Test Plan, strategy, and procedures. However there is a
minimum set of details that should be defined in order for the Acceptance Testing
process to work successfully with established systems.
There should be an agreed schedule and an understanding of who is conducting
Tests. The responsibilities for providing a test version of the changed system also
need to be understood.
For systems with established support and testing processes it may be useful to go
through a process of confirming any testing requirements and confirming agreement
on the testing process.
Sign-off need not be a significant undertaking.
Preparing Tests, Test Data, and Training
Apart from any new system functions, Test Cases and Scripts should already exist
and these can be reused.
A Test database may already be available. It may be necessary to ‘refresh’ this by
extracting data from the production system. Processes to do this should already
exist.
Training is probably unnecessary as experienced users should be available. There
may be a need to ‘run through’ any new functions with the developer.
Given the minimal workload, it is quite likely that a single person can undertake all
tasks.
Closure
Upon formal acceptance of the revisions, it will be necessary to arrange installation of
the new version of the system with the Information Management Branch in your
Division.
37
The Acceptance Testing Kit – Part 2
Some Cautionary Notes
Test Data Confidentiality
Acceptance Testing often involves the use of confidential data and the production of
real documents, real interface files, generated e-mails, etc.
It is important that during preparation of the Acceptance Test Plan, procedures to
protect the security and confidentiality of data are included.
Interpretation of Business Requirements
Business system requirements may be open to misinterpretation. A system supplier
(either internal or external) may contest the manner in which a system meets the
business requirement. Therefore, the more accurately and specifically that business
requirements are specified, the less open to misinterpretation they become.
Resolution of these issues is a common part of negotiating acceptance of a system.
If the system supplier accepts or endorses the Acceptance Test Plan, then these issues
should arise early in the process, minimising any conflict at a later stage.
System Prototyping
System prototypes are sometimes built during system design and development as
part of an iterative approach to system development. The prototype may be a
working model of a system that will eventually be used and tested and it enables
end-users to experience first-hand the proposed interaction with the system. The
prototype is not a full working model and merely simulates the proposed system’s
behaviour. Any feedback and requested changes are incorporated into the system
and the system trialed again. However, this iterative trialing of a system is strictly
part of the development process of satisfying the system requirements and does not
replace formal Acceptance Testing.
Parallel Running
Parallel running is normally conducted after implementation prior to a full cut-over
to the new system. However, parallel running may also occur during Acceptance
Testing.
Parallel running is where an existing system continues to operate as normal, with the
new system duplicating the processes and activities. The results of the old and new
systems can then be compared.
Caution should be taken when planning parallel running as it can place additional
and excessive demands on end-users’ time. It may be necessary to consider
increased personnel numbers.
38
The Acceptance Testing Kit – Part 2
Minor System Deficiency Delays
Even minor system modifications, implemented at a late stage of development to
overcome deficiencies in the business requirements specification, can cause final
delivery of the system to be significantly delayed.
Acceptance of the system, subject to any minor modifications being done after the
system has been fully implemented, is at the discretion of the Project Manager and
Steering Committee.
Generally, these issues will be handled under change management or contract
variation procedures.
39