Testing Framework

Introduction

Testing is a cognitive, critical and often creative process. It builds upon experience, discipline and persistence, but also benefits from thinking out of the box. It benefits from the wise application of standards, tools, best practices, adapted to a specific case, goal, available resources, and local culture.

This framework outlines a structured approach to testing, which is aligned with the major standards in the field, such as ISO/IEC/IEEE 29119, IEEE 829 and BS 7925, complements ITIL recommendations and processes, but is at the same time in line with the common language and practices in testing. It currently meant to support the GÉANT SA4 software service validation and testing process, but may grow into a tool for strengthening of the common ground between software developers and testers and the computer networking community. Such a tool can be used in consolidation of evaluations of various products, solutions and solutions developed and applied within the GÉANT project.

Tailoring Approach to Testing

Streamlining of the testing process for software product has been the subject of significant standardisation effort in the last two decades. As software becomes an intrinsic part of almost every system, a testing effort has to align its validation with needs, practices and experiences from many industries and different fields of engineering. However, this field is still rapidly evolving and there is currently a strong opposition to its formalisation. There is no guarantee that any specification can ensure fully repeatable testing practice that is applicable in all possible contexts. Often, the inherent fluidity of the environment and goals, combined with complexity of tested systems, is likely to preclude any exhaustive codification.

This material should be seen as a highly customisable toolkit. It is intended to prevent the reinventing of the wheel, and to digest the existing expertise and practices. An over-administrative approach must be avoided, as there is no single size that fits all. Schematic and formal application and implementation of the standards, procedures or templates “by the book” (this document included) may petrify, bureaucratise, and over-proscribe validation and testing process and impede innovation. What matters is the quality. Actual testing, even with automated execution, is designed and supervised by humans. On the other hand, running of production-level services does require formalised processes, traceability or even auditability of validation, and unification of control procedures.

General Testing and Validation Governance

Although this document can be seen as a contribution to the overall GÉANT approach to validation and testing, it is beyond its scope to discuss the general validation and testing policy and strategy for implementation and management of services. Such policy and strategy may be further discussed and refined by the project management and stakeholders.

Both strategy and supporting guidelines should be periodically reviewed and updated on the base the experience from the actual evaluation, organizational and environmental changes and evolution of general policies and goals.

As a technical approach to validation and testing, this material supports the practical organisation and documentation of individual evaluations. They consist of two groups of processes: test management and test enactment.

Here proposed descriptions and outlines of documentation artifacts and related comments are of informative and guiding nature. This material does not provide any guidelines for organisational or strategic level of testing related documentation. However, the top level policy and strategy may define the overall approach to validation and testing on the base of documents such as this one.

Test Management

Test management primarily deals with the evaluation level documentation. The evaluation level artifacts are related to testing of as a specific service, product or solution that are intended for communication between those responsible for test management processes (planning, monitoring and control, and completion assessment) and those who actually design, implement, execute and document the planned tests. The evaluation level documentation consists of:

• Test Plan
• Test Status Report
• Test Completion Report

The actual testing is initially specified through the development of the test plan. In order to produce the test plan, its authors need to familiarise with the context in order to define the scope, organise the development of the test plan, identify and estimate risks, establish approach towards them, design the strategy for the particular testing, determine staffing profile and schedule, and after the plan is drafted, to establish consensus or get approval for it, and distribute it to all interested parties.

The stakeholders monitor the progress through test status and completion reports on in order to appraise test status and assess related measures and metrics. They issue directives for corrective and adaptive actions on test design, environment and execution, which results in changes in test level documentation. Ultimately, these measures may lead to modification of the test plan.

The test enactment may also provide some suggestions or inputs for updates of the general approach and strategy on testing and validation. It is on those responsible for test management to further articulate and propagate such initiatives.

Test Enactment

The enactment consists of test design and preparation and execution of all planned tests. Its processes produce and update the test level documentation.

Test design includes specification of requirements with conditions for their fulfilment. The conditions may also be contextual and include repeatability of tests or ability to present the results. It is followed by specification of individual tests cases and procedure for their execution. The subsequent steps are test environment set-up, test execution and related reporting. The feedback from environment set-up and execution may lead to an update of the original design. For example, the constraints in capabilities of the test environment may lead to the update of test cases. Alternatively, actual preparation of test cases and environment may require additional elaboration and refinement of test cases. On the other hand, significance or priority of some requirements may lead to modification of the test environment in order to enable execution of corresponding test cases. Both may lead to modification of the test procedure.

Figure: Test Enactment Processes Diagram

The test level documentation is produced by the testing team. It refines the practical details of the design and execution of the work specified in the test plan and captures or relevant information gathered during testing in order to support the evaluation level reports. It consists:

• Test Design Specification
• Test Case Specification
• Test Data Report
• Test Environment Report
• Test Execution Log
• Detailed Test Results
• Test Incident Report

Traceability is the key characteristic of both test management and test enactment processes and artifacts. Repeatability is highly desirable in tests related to core requirements and functionalities. For less important features and requirements, there is no need to maintain repeatability of tests that were already done, unless it is possible to fully automate the test execution, or if it is required for compliance or periodic audits.

Test Documentation

Below described documentation provides an hierarchy of artifacts that is aligned with common industry practices and is generally compatible with the IEEE 829 standard and its sequel/update ISO/IEC/IEEE 29119-3, which even provides detailed templates for both traditional (sequential and iterative) and agile approaches.

Metadata

All documents should start with the following common elements (metadata):

• Organizational numeric identifier of the document - it may be omitted if the document name is used as the identifier
• Descriptive and identifying name of the document
• Name of testing (sub)project or use case, if not clear from document name
• Document date
• Version number
• Author or authors and their contact information
• Version history (table with version numbers, dates, contributors and change descriptions)

Version histories, date(s) and authors are not needed if the master documents are kept up to date in already highly versioned environment, such as a CMS system or Wiki. However, the main or corresponding author and document date need to be visible in self-contained standalone snapshots that are as such published on the web or shared by email.

References/Supporting Documents/Literature

The list of documents with their identifiers, names, version numbers and hyperlinks to individual documents should be provided.

At least the reference to the project plan should be present in its subordinate documents. There is no need to reference other documents that represent the common background that is already listed in the project plan. Only the references that are crucial for understanding need to be provided in lower level documents, as well as those that point to the relevant external documentation that is not already referenced in higher level documents.

Test Plan

The test plan outlines the operational aspects of execution of the test strategy for the particular testing effort. It provides an overview of the requirements the system needs to meet in order to satisfy its intended use, user needs, or scope of the intended testing effort, and how the actual validation is to be conducted. This plan outlines the objectives, scope, approach, resources (including people, equipment, facilities and tools) and their allocation amounts, methodologies and schedule of the testing effort. It usually also describes the team composition, training needs, entry and exit criteria, risks, contingencies, test cycle details, quality expectations and tracking and reporting processes. The test plan may account for one or several test suites, but it does not detail individual test cases.

As a pivotal document, the test plan is an instrument of mutual understanding between the testing and development teams and the management. In case of major impediments or changes, it should be updated as needed and communicated to all concerned. Such updates may lead to further changes in documents that specify test design, test cases, and data and environment requirements. Given the multifaceted and overarching nature of the test plan and in order to avoid unnecessary backpropagation of changes, it should not over-specify the implementation details that are to be articulated in subordinate test level documents.

The recommended structure of the test plan is as follows.

Metadata

The descriptive name should briefly, in three to six words, state which system is tested, the target aspects, features of components, and level, type or purpose of conducted testing.

It should be immediately followed by separate description of the testing level (unit, integration, system and acceptance) and/or type or subtype (functional, non-functional, alpha, beta, performance, load, stress, usability, security, conformance, compatibility, resilience, scalability, volume, regression…).

References/Supporting Documents

All documents that support the test plan should be listed. They may include:

• Project plan
• Product plan
• Related test plans
• Requirements specifications
• High level design document
• Detailed design document
• Development and testing standards
• Methodology guidelines and examples
• Organisational standards and guidelines
• Source code, documentation, user guides, implementation records

Glossary

Key terms and acronyms used in the document, target domain and testing are described here. The glossary facilitates communication and helps in avoiding confusion.

Introduction

This is the executive summary part of the plan which summarises its purpose, level, scope, effort, costs, timing, relation to other activities and deadlines, expected effects and collateral benefits or drawbacks. This section should be brief and to the point.

Features to be Tested

The purpose of the section is to list individual features, their significance and risks from the user perspective. It is a listing of what is to be tested from the users’ viewpoint in terms of what the system does. The individual features may be operations, scenarios, and functionalities that are to be tested across all or within individual tested sub-systems. Features may be rated according to their importance or risk.

An additional list of features that will not be tested may be included, along with the reasons. For example, it may be explained that a feature will not be available or completely implemented at the time of testing. This may prevent possible misunderstandings and waste of effort in tracking the defects that are not related to the plan.

Together with the list of test items, this section describes the scope of testing.

Test Items

This is the description, from the technical point of view, of the items to be tested, as hardware, software and their combinations. Version numbers and configuration requirements may be included where needed, as well as delivery schedule for critical items.

This section should be aligned with the level of the test plan, so it may itemise applications or functional areas, or systems, components, units, modules or builds.

For some items, critical areas or associated risks may be highlighted, such as those related to origin, history, recent changes, novelty, complexity, known problems, documentation inadequacies, failures, complaints or change requests. Probably there had been some general concerns and issues that triggered the testing process, such as history of defects, poor performance, changes in the team, etc., that could be directly associated with some specific items. Other concerns that may need to be mentioned can be related to safety, importance and impact on users or clients, regulatory requirements, etc. The key concern may be general misalignment of the system with the intended purpose, or vague, inadequately captured or understood requirements.

The items that should not be tested may be also listed.

Approach

This section describes the strategy that is appropriate for the plan level and in agreement with other related plans. It may extend the background and contextual information provided in the introduction.

Rules and processes that should be described include:

• Detailed and prioritised objectives
• Scope (if not fully defined by lists of items and features)
• Tools that will be used
• Needs for specialised trainings (on testing, used tools or the system)
• Metrics to be collected and granularity of their collection
• How the results will be evaluated
• Resources and assets to be used, such as people, hardware, software, and facilities
• Amounts of different types of testing at all included levels
• Other assumptions, requirements and constrains
• Overall organisation and schedule of the internal processes, phases, activities and deliverables
• Internal and external communication and organisation of the meetings
• Number and kinds of test environment configurations (for example, for different testing levels/types)
• Configuration management for the tested system, used tools and test environment
• Change management

For example, the objectives may be to determine whether the delivered functionalities work in the usage or user scenarios or use cases, whether all functionalities required the work are present, whether all predefined requirements are met, or even whether the requirements are adequate.

Besides testing tools that interact with the tested system, other tools may be need, like those used to match and track scenarios, requirements, test cases, test results, issues and acceptance criteria. They may be manually maintained documents and tables, or specialised testing support tools.

Some assumptions and requirements must be satisfied before testing. Any special requirements or constrains of the testing in terms of the testing process, environment, features or components need to be noted. They may include a special hardware, supporting software, test data to be provided, or restrictions in use of the system during the testing.

Testing can be organised as periodic or continuous until all pass criteria are met, with passing of identified issues to the development team. This requires defining the approach to modification of test items, in terms of regression testing.

The discussion of change management should define how to manage the changes of the testing process that may be caused by the feedback from the actual testing or by external factors. This includes the handling of the consequences of defects that affect further testing, dealing with requirements or elements that cannot be tested, and dealing with parts of testing process that may be recognised as useless or impractical.

Some elements of the approach are further detailed in subsequent sections.

Item (and Phase) Criteria

This section describes the process and overall standards for evaluating the test results, not detailed criteria pass for each individual item, feature or requirement.

The final decisions may be made by a dedicated evaluation team comprised of various stakeholders and representatives of testers and developers. The team evaluates and discusses the data from the testing process to make a pass/fail decision that may be based on the benefits, utility, detected problems, their impact and risks.

The exit criteria for the testing are also defined, and may be based on achieved level of completion of tests, number and severity of defects sufficient for the abortion of testing, or code coverage. Some exit criteria may be bound to a specific critical functionality, component or test case. The evaluation team may also decide to end the testing on the base of available functionality, detected or cleared defects, produced or updated documentation and reports, or progress of testing.

If testing is organised into phases or parallel or sequential activities, the transitions between them may be gated by corresponding exit/entry criteria.

If the testing runs out of time or resources before the completion or is aborted by stakeholders or the evaluation team, the conclusions about the quality of the system may be rather limited, and this may be an indication of the quality of the testing itself.

Suspension Criteria and Resumption Requirements

These criteria are used to determine in advance whether the testing should be prematurely suspended or ended before the plan has been completely executed, and when the testing can be resumed or started, that is, when the problems that caused the suspension have been resolved.

The reason for the suspension can be the failure of the test item (for example, a software build that is the subject of the test) to work properly due to critical defects which seriously prevent or limit testing progress, accumulation of non-critical defect to the point where the continuation of testing has no value, client’s changes of the requirements, system or environment downtime, inability to provide some critical component or resource at the time indicated in the project schedule.

It is may be required to perform a smoke test before the full resumption of the tests.

Issues noticed during testing are often consequence of other previously noticed defects, so continuation of testing after certain is number of identified defects in the affected functionality or item is wasting of resources, particularly if it is obvious that the system or the item cannot satisfy the pass criteria.

Deliverables

This section describes what is produced by the testing process. The deliverables may be the subject of quality assessment before their final approval or acceptance, and besides all the elements of test documentation that are described here, may also include test data used during testing, test scripts, code for execution of tests in testing frameworks and outputs from test tools.

Activities/Tasks

This section outlines the testing activities and tasks, dependencies and estimates their duration and required resources.

Staffing and Training Needs

This is the specification of staff profiles and skills needed to deliver the plan. Depending on the profile of the personnel, it should also detail needed trainings on the tested system, elements of the test environment and test tools.

Responsibilities

This section specifies personal responsibilities for approvals, processes, activities and deliverables described by the plan. It may also detail responsibilities in development and modification of the elements of the test plan.

Schedule

The schedule of phases should be detailed to the level which ensures certainty that is attainable with information available at the moment of planning. It should define the timing of individual testing phases, milestones, activities and deliverables and be based on realistic and validated estimates, particularly as the testing is often interweaved with the development. Testing is the most likely victim of slippage in the upstream activities, so it is a good idea to tie all test dates directly to the completion dates of their related developments. This section should also define when the test status reports are to be produced (continuously, periodically, or on demand).

Risks and Contingencies

This section, which complements "Suspension Criteria and Resumption Requirements", defines all risk events, their likelihood, impact and counter measures to overcome them. Some risks may be testing related manifestations of the overall project risks. The exemplary risks are lack or loss of personnel at the beginning or during testing, unavailability or late delivery of required hardware, software, data or tools, delays in training, or changes to the original requirements or designs.

The risks can be described using the usual format of the risk register, with attributes such as:

• Category
• Risk name
• Responsible (tracker)
• Associated phase/process/activity
• Likelihood (low/medium/high)
• Impact (low/medium/high)
• Mitigation strategy (avoid/reduce/accept/share or transfer)
• Response action
• Actionee
• Response time

Although some contingency events may actually be opportunities, it is, due to the limited scope of testing within the wider project or service delivery context, quite unlikely that the opportunities related to the very testing process will occur. However, the outcome of testing or some of its results may offer opportunities related to the subject of testing that may be enhanced, exploited or shared.

The approach for dealing with schedule slippages should be described in responses to associated risks. Possible actions include simplification or reduction of non-crucial activities, relaxation of the scope or coverage, elimination of some test cases, engagement of additional resources, or extension of testing duration.

Test Status Report

Test status report is a one-time interim summary of the results of execution of tests. It may describe the status of the all testing activities or be limited to a single test suite. This report, as well as the test summary report, sublimes the information obtained during test execution and recorded in test logs and incident reports. It must be highly informative and concise and should not elaborate minor operational details.

Depending on the approach defined in the test plan, it may be produced periodically, on completion of milestones or phases, or on demand. If periodic, it may be a base for continuous tracking through progress charts. Individual status reports are also sources for the test completion report.

The report should start with metadata in the condensed form, but without version history, since the document itself is considered to be a one-time snapshot.

Summary

The document can start with totals of passed, failed and pending test cases, scenarios or tests, and identified defects (if individually tracked). It may also show the coverage of test cases or code, consumption of resources and other established progress metrics, in numbers and, if possible, charts. Close to this summary, a comprehensive assessment should be provided.

The details of the progress are expressed in a form of a table describing the outcome of execution of individual test cases or scenarios, cumulatively since the start of testing. Typical columns to be used are:

• Test case ID
• Test case name
• Date and time of the last execution
• The last execution status (not run, failed, partial, passed) or counts of failed and passed test executions
• Number of associated defects
• Brief comment

If there are if more than dozen items, consider grouping and sub-totaling the table rows according to the test suites, test items or scenarios (as listed in the project plan), test types or areas. If a single report addresses several suites, each should have a separate test status report, or at least its own totals and details tables.

Observations and Highlights

If needed, the document provides evaluations and recommendations based on the interim results and incidents encountered since the previous status report. It may also signal the red flags that deserve recipients’ attention. It should report the resolution of issues that were highlighted in the previous status report.

Activities

The summary of activities conduced since the previous status report is optional. If present, is should exist in all status reports.

Test Completion Report

The test completion or summary report is a management report that brings together the key information uncovered by the accomplished tests. It recapitulates the results of the testing activities and indicates whether the tested system is fit for purpose according to whether it has met the acceptance criteria defined in the project plan. This is the key document in deciding whether the quality of the system and the performed testing are sufficient for the decision that follows the testing. Although the completion report provides a working assessment on success or failure of the system under test, the final decision is made by the evaluation team.

This document reports all pertinent information about the testing, including an assessment about how well the testing has been done, the number of incidents raised and outstanding events. It must describe all deviations from the original test plan, their justifications and impacts. Provided data should be sufficient for the assessment of the quality of the testing effort.

The provided narrative should be more elaborated than in the test status reports.

Summary

It recapitulates the evaluation of the test items. Although the test completion report can reflect the structure of the test status report, the details that were only temporarily significant can be omitted. The table rows or subsections should correspond to the test items or scenarios listed in the test design specification. The summary should indicate the versions of the items that were tested, as well as the used testing environment. For each item it should be briefly explained what was tested and what was the outcome.

Test Assessment

This is a comprehensive assessment of the conducted testing. It should also indicate the areas that may require further investigation and testing.

Variances

All variations and discrepancies from the original test plan should be noted here. This section can also provide an assessment of differences between the test environment and the operational environment and their effect on the test results.

Test Results

This is a comprehensive interpretation of the test results. It includes description of issues or defects discovered during the testing. It should also describe the unexpected results and problems that occurred during the testing. For resolved incidents, their resolutions should be summarised. For unresolved test incidents, an approach to their resolution should be proposed.

Evaluation and Recommendations

Propose the decisions regarding the tested system and suggest further actions on the base of the acceptance criteria, quality of the test process, test results and outcomes for individual test items. Provide recommendations for improvement of the system or future testing that result from the conducted testing.

Activities

The summary of activities conducted during testing should record what testing was done and how long it took. In order to facilitate improvement of future test planning it should be in a form that does not require later thorough investigation of the plans and records of the conducted testing.

Test Design Specification

Test design specification addresses the test objectives by refining the features to be tested, testing approach, test cases, procedures and pass criteria. This document also establishes groups of related test cases.

Features to Be Tested

This section describes the features or requirements and combinations of features that the subject of testing and, if some of features are closely related to test items, expresses these associations. Each feature is elaborated through its characteristics and attributes, with references to the original documentation where the feature is detailed. The requirement descriptors include ID/short name, type, description and risks. The references lead to the associated requirements or feature specifications in the system/item requirement specification or design description (in the original development documentation). If the test design specification covers several levels or types of testing, the associated levels or types are noted for each individual requirement, feature or test item.

The features may be grouped into a few key applications, use cases or scenarios. If such groping is made, a single feature or requirement may be present in several groups.

The use case is a high level description of a specific system usage, or set of system behaviours or functionalities. It should not be mistaken for UML use case. It implies, from the end-user perspective, a set of tests that need to be conducted in order to consider the system as operational for particular use. It therefore usually describes the system usage, features and related requirements that are necessary for utilization of the system by end users on regular basis.

The requirement is a description of necessary capability, feature, functionality, characteristic or constraint that the system must meet or be able to perform. It is a statement that identifies a necessary quality of a system for it to have value and utility to a user, customer, organization, or other stakeholder. It is necessary for the fulfilment of one or several use cases or usage scenarios (in scenario testing).

The high level requirements include business, architectural and stakeholder/user requirements. There are also some transitional requirements that are only relevant during the implementation of the system. The detailed requirements are defined on the base of identified high-level features or requirements. Some of them are consequences of system's functions, services and operational constraints, while others are pertaining to the application domain.

Functional requirement defines a specific behaviour or function. It describes what the system does, but not how it is being done in terms of implementation, quality, or performance.

Non-functional requirement specifies the quality criteria used to assess the characteristics or properties the system should possess (what it is). Typical non-functional requirements include:

• Performance, availability, stability, load capacity, efficiency, effectiveness, scalability, response time;
• Reliability, robustness, fault tolerance, recoverability, resilience;
• Privacy, security, safety;
• Configuratability, supportability, operability, maintainability, modifiability, extensibility;
• Testability, compliance, certification;
• Usability, accessibility, localization, internationalization, documentation;
• Compatibility, interoperability, portability, deployability, reusability.

In the classical engineering and waterfall software engineering, requirements are inputs into the design stages of development. The requirements specification is an explicit set of requirements to be met by the system, and therefore is usually produced quite early in its development. However, when iterative or agile methods of software development are used, the system requirements are incrementally developed in parallel with design and implementation.

The requirements specification is an important input into the testing process, as it lays out all requirements that were, hopefully, addressed during the system development, so the tests to be performed could trace back to them. Without the access to the requirements from the development, the requirements that are directly associated with testing should be formulated during its planning. If an agile methodology was used for development, these requirements can reflect the completed Scrum epics, user stories and product backlog features or "done" Kanban board user stories and features cards.

The individual requirements need to be mutually consistent, i.e. consistent with the external documentation, verifiable, and traceable towards high-level requirements or stakeholder needs, but also towards the test cases.

The requirements are the base for development of test cases.

Scenario testing is a higher level approach to testing of complex systems that is not based on test cases, but on working through realistic and complex stories reflecting user activities. These stories may consist of one or several user stories, which capture what user does or should do as a part of his/her job function, expressed through one or more sentences in the everyday or domain language. The tester who follows the scenario must interpret the results and judge whether they can be considered as a pass or failure. This interpretation may require backing by domain experts. This term should be distinguished from test procedure and test case scenario.

Approach Refinements

This section refines the approach described in the test plan. The details of the included test levels are provided and how the individual features are addressed at those levels.

Specific test techniques to be used are selected and justified. Particular test management, configuration management and incident management tools may be mandated. Code reviews, static and dynamic code analysis or unit testing tools may support the testing work. Test automation software tools may be used to generate, prepare and inject data, set up test preconditions, control the execution of tests, and capture outputs. The method for the inspection and analysis of test results should be also identified. The evaluation can be done based on visual inspection of behaviours and outputs, or on the usage of instruments, monitors, assertions, log scanners, pattern matching programs, output comparators, or coverage measurement and performance testing tools.

In order to avoid redundancy, common information related to several test cases or procedures is provided here. It may include details of the test environment or environmental needs, system setup and recovery or reset, and dependencies between the test cases.

If the test design includes some deviations from the test plan, they have to be described here.

Test Cases

Individual test cases are identified here. After the identifier, a brief description of the test case and associated test procedure is provided. Associated test levels or other important test case attributes may also be recorded.

The test case refines criteria that need to be met in order to consider some system feature, set of features or the entire use case as working. It is the smallest unit of testing and is sometimes colloquially referred as test. A single test case may be included into several test suites or related to a requirement associated with several use cases. If different test levels have separate test design specification, a single test case may be present in several design specifications.

The selection of test cases may be the result of an analysis that provides a rationale for a particular battery of test cases associated with a single requirement. For example, the same feature may be tested with distinct test cases that cover valid and invalid inputs and subsequent successful or negative system responses. The logic behind the selection of test cases should be described here.

A feature from the test design specification may be tested in more than one test case, and a test case may test more than one feature. The test cases should cover all features, that is, each feature should be tested at least once. The relationship between the requirements/features and test cases is summarised in the Requirements/Test Cases Traceability Matrix, which is usually placed in a separate document that is updated with the evolution of the requirements and test design specification document, but also with refinement of individual test cases. It enables both forward and backward traceability, as it simplifies modification of test cases after the change of requirements, and vice versa. It is used to verify whether all the requirements have corresponding test cases, and to identify for which requirement(s) a particular test case has been written for. The Requirements/Test Cases Traceability Matrix is a table where requirements and test cases are paired, thus ensuring their mutual association and coverage. Since there are always more test cases than requirements, the requirements are placed in columns, and tests cases in rows. The requirements are identified by their IDs or short names and can be grouped by their types, while the test cases can be grouped into sections according to levels: unit, integration, system and acceptance.

Feature Pass/Fail Criteria

This specifies the criteria to be used to determine whether the feature or a group of features has passed or failed, on the base of results of individual test cases.

Test Case Specification

The test case specifications are produced after the test design specification has been prepared. The test case specification is a detailed elaboration of a test case identified in the test design specification and includes a description of the functionality to be tested and the preparation required to ensure that the test can be conducted. A single test case is sometimes associated with several requirements. It may be partially or fully automated.

The test case specification is provided within the use case or test suite specification, a document that details all test cases, or even separate document dedicated to a single test case. A formal written test case is characterised by a known preconditions, input and expected output and postconditions, which are worked out before the execution.

For a system without preexisting formal requirements, the test cases can be written based on system’s desired or usual operation, or operation of similar systems. The test cases are a result of decomposition of a high-level scenario. This scenario provides a story and description of the setting  that explain the system and its operation to the tester. Alternatively, test cases may be omitted altogether and replaced with scenario testing, which substitutes a sequence or group of test cases, as they may be hard to precisely formulate and maintain with the evolution of the system.

The typical attributes or segments of the test case specification are:

• Test case ID or short identifying name
• Related requirement(s)
• Requirement type(s)
• Test level
• Author
• Test case description
• Test beds to be used (if there are several)
• Environment information
• Preconditions, prerequisites, states or initial persistent data
• Inputs (test data)
• Execution procedure or scenario
• Expected postconditions or system states
• Expected outputs
• Evaluation parameters/criteria
• Relationship with other use cases
• Whether the test can be or has been automated
• Other remarks

The test case typically comprises of several steps that are necessary to assess the tested functionality. The explained steps should include all necessary actions, including those assumed to be a part of common knowledge.

The test suite is a collection of test cases that are related to the same testing work in terms of goals and associated testing process. There may be several test suites for a particular system, each one grouping together many test cases based on corresponding goal and functionality or shared preconditions, system configuration, associated common actions, execution sequence of actions or test cases, or reporting requirements. An individual test suite may validate whether the system complies with the desired set of behaviours or fulfils the envisioned purpose or associated use cases, or be associated with different phases of system lifecycle (such as identification of regressions, build verification, or validation of individual components). A test case can be included into several test suites. If test cases descriptions are organised along test suites, the overlapping cases should be documented within their primary test suites and referenced elsewhere.

The test procedure defines detailed instructions and sequence of steps to be followed while executing a group of test cases (such as a test suite) or single test case. It can give information on how to create the needed test setup, perform execution, evaluate results and restore the environment. The test procedures are developed on the base of the test design specification and in parallel or as parts of test case specifications. Having a deatiled test procedure is very helpful when a diverse set of people is involved in performing the same tests at different times and situations, as this supports consistency of test execution and result evaluation. Test procedure can combine test cases based on a certain logical reason, like executing an end-to-end situation, when the order in which the test cases are to be run is also fixed.

The test script is a sequence for instructions that need to be carried out on the tested system in order to execute a test case, or test a part of system functionality. These instructions may be given in the form suitable for manual testing or, in automated testing, as short programs written in a scripting or general purpose programming language. For software systems or applications, there are test tools and frameworks that allow specification and continuous or repeatable execution of prepared automated tests.

Test Data Report

This document describes the data used in testing. It is typically produced in two stages.

First, the data requirements implied by the test plan and test design are put together. They include requirements related to type, range, representativeness, quality, amount, validity, consistency and coherency of test data. Additional concerns may be related to the sharing of test data with the development team or even end users.

The set test levels and use cases determine the tools and means that are used for collection, generation and preparation of test data. If data are not entirely fabricated, but are extracted from the existing databases or services and can be associated with the real services, processes, business entities or persons, then the policies and technical procedures for its depersonalisation, obfuscation or protection may need to be established. Otherwise, the approach to creation of adequate input data should be established, and validity of test results obtained with such inputs assessed.

Test case specifications provide more elaborate details about data needed and should be sufficient to support the actual collection or generation of adequate test data. During the second stage, the selected tools are used to prepare these data for execution of all use cases, including their injection into the system or interactions during the test procedure steps. At the same time, the corresponding expected test outputs are defined, and, if possible, automated methods for comparing the baseline test data against actual results. The limitations of test data and supporting tools are identified and it is explained what to do in order to mitigate them during the test execution and in interpretation and evaluation of the results. Finally, the measures that ensure usability and relevancy of test data throughout testing process are specified. They include data maintenance to support changes in the system, but also data versioning and backup. The decisions and knowledge produced during the preparation of the test data are captured.

Test Environment Report

This document describes the test environment. It is typically produced in two stages.

The requirements for the test bed implied by the test plan, test design specification and individual test cases are put together, and the initial test environment setup is designed. The test bed requirement related to the test level, system features and requirements, test items, test data, testing scenarios and procedures, chosen support, measurement and monitoring tools are put together. Security, safety and regulatory concerns are also considered. Policies and arrangements for sharing of the test bed and allocated resources with other teams or users are established.

The initial test bed design can be a simple deployment diagram or test bed implementation project, but it should cover all elements of the setup, including hardware, software, network topology and configuration of hardware, external equipment, system software, other required software, test tools, system under test and individual test items. If some components are not immediately available, the staged implementation schedule or workarounds need to be devised. A walkthrough through at least the most important requirements and test cases needs to be performed in order to validate the proposed design.

The report is updated after the test environment is set up. A smoke test can be performed. The limitations of the test environment are identified and explained what to do in order to mitigate them during the test execution and in interpretation and evaluation of the results. The maintenance plans, responsibilities and arrangements are established. If envisioned in the test plan, this includes upgrades of current versions of test items, upgrading of used resources and network topology, and updating of corresponding configurations. The initial design is updated to reflect the test environment as it was built. The decisions and knowledge produced during the implementation of the test bed are captured.

Test Execution Log

The test execution log is the record of test cases executions and obtained results, in the order of their running. Along with test incident reports, the test execution log is a base for test status and completion reports. These documents allow direct checking of the progress of the testing and provide valuable information for solving what caused an incident.

This log provides a chronological record of relevant details about the execution of tests, by recording which tests cases were run, who ran them, in what order, and whether the test were passed or failed. The test passed if the actual and expected results were identical; it failed if there was a discrepancy. If the test design specification permits to classify the execution status as "partial", it must also clarify how to treat such outcomes within the feature pass/fail criteria and acceptance criteria.

Each test execution should start with a standardised header. The executions should be ordered from the oldest to the newest. Optionally, the recorded executions may grouped by test cases, but, if this is done, it is important to maintain ability to trace the actual execution sequence of all test cases in order to detect possible interference between them. For each test execution, the versions of the system under test, its components, test environment, and specifics of input data must be recorded. These are the recommended descriptors:

• Test case ID or short identifying name - may be placed at the top of the group
• Order of execution number - useful for cross-referencing
• Date and time
• Testers - person who run the test, may also include observers
• Test bed/facility - if several test beds are used in testing
• Environment information - versions of test items, configuration(s) or other specifics
• Specific presets - initial states or persistent data, if any
• Specific inputs - inputs/test parameters or data that are varied across executions, if any
• Specific results - outputs, postconditions or final states, if different from the expected; may refer to detailed test results
• Execution status - passed, failed, partial (if permitted)
• Incident reports - if one or several test incident reports are associated with the execution, they are referenced here
• Comments - notes about the significant test procedure steps, impressions, suspicions, and other observations, if any

If the test execution log is natively maintained or presented in a human readable format, and there are series of many repeated executions of the same test case adorned with same values of an attribute (tester, test bed, environment, presets, input...), these common values can be documented in the heading of the group, along with the test case ID.

If there are several typical versions of the environment, presets or inputs, they may be described in the test case or elsewhere in the test execution log and referenced in the test case executions that use them. This reduces the clutter. However, any particular variances in the configuration, input data, and results need to be documented. The actual outputs may be separately captured in the detailed test results, especially if an in-depth discussion of the alignment of actual and expected outcomes is needed.

In case of a deviation or partial success or failure, a more detailed description of the execution status should be provided.

The data captured in the test execution log, along with other test level specifications and reports, should be sufficient to reproduce individual tests, that is, to recreate the needed setup, execute the test and produce same or similar results.

If the testing is organised around scenarios instead of test cases, the general structure of the log is unchanged, except that inputs, states and outputs are replaced with interpretation of the interactions and results for each segment of the scenario, supported by key excerpts or snapshots of characteristic inputs and outputs.

Detailed Test Results

Detailed test results are the actual outputs, assertions and system and monitoring logs produced during the execution of tests. They should be at least paired with the corresponding test execution log records. Their format may depend on test tools that are used to capture them. In addition, the detailed results may encompass the reports produced by test automation tools that compare the baseline test data against actual results, which highlight all noted deviations. Such reports are valuable traces of how obtained results measure up with expected postconditions, states and outputs and can be used in assessment of the execution status and writing of the test execution log and test incident reports.

Test Incident Report

The test incident report is used to document any event that occurs during the testing process that requires investigation. A discrepancy between expected and actual results can occur because the expected results are wrong, the test was wrongly run, or due to inconsistent or unclear requirements, fault or defect in the system or problem with the test environment. It should provide all details of the incident such as actual and expected results, when it failed, and any supporting evidence that will help in its resolution. All other related activities, observations, and deviations the standard test procedure should be included, as they may also help to identify and correct the cause of the incident. The report also includes an assessment of the impact of an incident upon testing, if possible.

The test incident report needs to be a standalone document, so it provides pieces of information that are already recorded in the corresponding test case and test execution log record.

A failed test may raise more than one incident, while an incident may occur in more than one test failure. The testers should (according to their knowledge and understanding) try to identify unique incidents and associate them with the tested features or originating test items. This will provide a good indication of the quality of the system and its components, and allow monitoring of their improvement.

If an incident is the consequence of a fault or bug, the causing error may be not in the failed execution, but in the previous one. It the case of apparently random incidents, the earlier executions and incidents should be checked in an attempt to recognise the pattern of tests that leads to them.

Metadata

The test incident report ID allows the report to be referenced in the test execution log and issue tracking system. If one or several test incident reports are raised or updated during the single test execution, and of them need to be recorded in the test execution log.

Summary

It briefly recapitulates the incident.

Description

The following elements should be provided.

• Test case ID or short identifying name*
• Order of execution number*
• Date and time
• Testers
• Associated requirement/feature/test items
• Test procedure step - where the event occurred*
• Test bed/facility
• Environment information
• Presets*
• Inputs*
• Expected results*
• Actual results*
• Anomalies - discrepancies, errors or faults that occurred
• Attempts to repeat - whether they were made, how, and what was the outcome
  

Test case related details (marked with *) will be omitted if the incident in not linked to a specific test case or scenario. In such situations, a detailed narrative description should be provided.

Impact

If known, indicate the impact of the incident on test plans, test design specifications, test case specifications or test procedures.