Testing Framework

Tailoring Approach to Testing

Testing as a cognitive, critical and often creative process.

Wise application of standards, tools, best practices, adapted to a specific case, goal, available resources, and local culture. This document as a support for establishing the common ground and customisable toolbox, and supporting guide. Relationship to the existing GN documents, testing standards, and software testing practices. Acknowledge that the recent standardization efforts try to build upon practices and experiences from different fields of engineering and industries. Do not reinvent the wheel, use the existing expertise, tools, practices, and processes. Actual testing, even with automated execution is designed and performed by humans.

No guarantee that any specification can ensure fully repeatable testing practice that is applicable in all possible contexts. Often, inherent fluidity of the environment and goals and complexity of tested system is likely to preclude any exhaustive formalization.

Avoid over-administrative approach; no one-size-fits-all. Schematic and formal application and implementation of the standards, or procedures templates “by the book”, this document included, endangers to petrify, bureaucratise, and over-proscribe validation and testing process and impede innovation. What matters is the quality. However, running production-level services may require formalized processes, traceability or even auditability of validation, and particularly of check procedures.

Generic Testing Process Diagram

General Testing and Validation Governance

This document provides GN and, more specifically, GN4.1 SA4 with a generalised technical specification for testing and validation. It can be seen as a a contribution to the general specification of GN approach to validation and testing, on the base of its the overall testing policy (for implementation and management of services - any sources/refs?), and as an (tentative?) technical expression of the shared testing strategy.

The overall policy and strategy, as well as this technical document, must be further discussed and agreed by stakeholders and published.

Should be periodically reviewed and updated on the base the experience from the actual performed tests, organizational and environmental changes and evolution of overall policies and goals. It is (one of / input for) general GN-level testing related policy and strategy documents.

Evaluation Planning

Need to familiarize with the context in order to define the scope, organize development of the test plan, identify and estimate risks, establish approach towards risks, design test strategy, determine staffing profile and schedule, draft test plan, establish consensus/get approval for the plan, publish the plan.

Traceability is the key characteristic of both Evaluation and Test Enactment related artifacts. Repeatability for tests related to core requirements and functionality. No need to repeat and maintain repeatability of tests that were already done unless possible to fully automate or required for compliance or periodic audits.

Test Enactment

Test design includes specification of requirements with conditions for their fulfillment. The conditions may also be contextual and include repeatability or presentability. It is followed by specification of individual tests cases and procedure for their execution. The subsequent steps are test environment set-up, test execution and related reporting. The feedback from environment set-up and execution may lead to an update of the original design. For example, the constraints in capabilities of the test environment may lead to the update of test cases. Alternatively, actual preparation of test cases and environment may require additional elaboration and refinement of test cases. On the other hand, significance/priority of some requirements may lead to modification of the test environment in order to enable execution of corresponding test cases. Both will lead to modification of the test procedure.

In parallel: Monitor test progress on the base of test status and related measures/metrics/reports), report test status to stakeholders, issue control/correction directives and perform corrective/adaptive actions for/on the test design, environment, execution, and perhaps even evaluation level test plan. Can be done in a single flow or iteratively.

Possibly provide suggestions or inputs for the update of the general Testing and Validation Process and Strategy.

Test Documentation

On the common industry practices and with the strong influence from IEEE 829, and its sequel/update ISO/IEC/IEEE 29119-3, which even provides even templates for both traditional (sequential and iterative) and agile approaches.

Organisational/Strategic Level

Policy and Strategy, also mandating the overall approach on validation and testing on base of documents such as this one.

Evaluation Level Documentation

For a specific service, product or solution

Test Plan
Test Status Report
Test Completion Report

Test Level Documentation

Test Design Specification
Test Case Specification
Test Data Requirements
Test Data Report
Test Environment Requirements
Test Environment Report
Detailed test results (base for status and completion reports on the base of individual tests execution logs and anomalies/incidents reports)

Test Plan

Test Environment or Test Bed is an execution environment configured for testing. It may consist of specific hardware, OS, network topology, configuration of the product under test, other application or system software, test tools, etc. The Test Plan for a project should enumerated the test beds(s) to be used.

Test Status Report

Test Completion Report

Test Design Specification

The Use Case is a high level description of a specific system usage, or set of system behaviours or functionalities. It should not be mistaken for UML use case. It implies, from the end-user perspective, a set of tests that need to be conducted in order to consider the system as operational for particular type of use. It therefore usually describes the system usage, features and related requirements and that are necessary for utilization of the system by end users on regular basis.

Requirement is a high level description of necessary capability, feature, functionality, characteristic or constraint that the system must meet or be able to perform. It is a statement that identifies a necessary quality of a system for it to have value and utility to a user, customer, organization, or other stakeholder. It is necessary for the fulfillment of one or several Use Cases or usage scenarios (in Scenario Testing).

The requirements specification is an explicit set of requirements to be satisfied by the system which is to be developed, and is therefore produced quite early in its development. Such a specification may be a direct input for testing process, as it lays out all requirements that were, hopefully, addressed during the system development.

Functional

Non-functional

Scenario Testing is a higher level approach to testing of complex systems that is not based on Test Cases, but on working through realistic and complex stories reflecting user activities. The tester who follows the scenario must interpret the results and evaluate whether they can be considered as a pass or failure. This interpretation may require backing by domain experts. This term should be distinguished from Test Procedure and Test Case scenario.

Test Case Specification

The Test Case is a specification of criteria that need to be met in order to consider some system feature, set of features or Use Case as working. It is the smallest unit of testing and is sometimes colloquially referred as Test. It should include a description of the functionality to be tested, and the preparation required to ensure that the test can be conducted. A single test case is sometimes associated with several requirements. It may be partially or fully automated.

A formal written test case is characterised by a known preconditions, input and expected output and postconditions, which are worked out before the execution. The test case may comprise of a single or several steps that are necessary to assess the tested functionality.

For a system without preexisting formal requirements, the Test Cases can be written based on system’s desired or usual operation, or operation of similar systems. In this case, they may be a result of decomposition of a high-level scenario, which is a story or setting description used to explain the system and its operation to the tester. Alternatively, Test Cases may be omitted altogether and replaced with scenario testing, which substitutes a sequence or group of Test Cases, as they may be hard to precisely formulate and maintain with the evolution of the system.

The Test Suite is a collection of Test Cases that are related to the same testing work in terms of goals and associated testing process. There may be several Test Suites for a particular system, each one grouping together many Test Cases based on corresponding goal and functionality or shared preconditions, system configuration, associated common actions, execution sequence of actions or Test Cases, or reporting requirements.

An individual Test Suite may validate whether the system complies with the desired set of behaviors or fulfills the envisioned purpose or associated Use Cases, or be associated with different phases of system lifecycle, such as identification of regressions, build verification, or validation of individual components.

A Test Case can be included into several Test Suites. If Test Cases descriptions are organised along Test Suites, the overlapping cases should be documented within their primary Test Suites and referenced elsewhere.

A Test Procedure defines detailed instructions and sequence of steps to be followed while executing a group of Test Cases (such as a Test Suite) or single Test Case. It can give information on how to create the needed test setup, perform execution and evaluate results for a given test case. Having a formalised test procedure is very helpful when a diverse set of people is involved in performing the same tests at different times and situations, as this supports consistency of test execution and result evaluation. Test Procedure is a combination of Test Cases based on a certain logical reason, like executing an end-to end situation. The order in which the Test Cases are to be run is fixed.

The Test Script is a sequence for instructions that need to be carried out on the tested system in order to execute a test case, or test a part of system functionality. These instructions may be given in the form suitable for manual testing or, in automated testing, as short programs written in a scripting or general purpose programming language. For software systems or applications, there are test tools and frameworks that allow specification and continuous or repeatable execution of prepared automated tests.

Space shortcuts

Page tree