...

Here proposed descriptions and outlines of documentation artifacts and related comments are informative and guiding nature. This material does not provide any guidelines for organisational or strategic level of testing related documentation. However, the top level policy and strategy may mandating the overall approach to validation and testing on the base of documents such as this one.

Test Management

Test management

...

primarily deals with the evaluation level documentation.These artifacts are related to testing of as a specific service, product or solution that are intended for communication between those responsible for test management processes (planning, monitoring and control, and completion assessment) and those who actually design, implement, execute and document the planned tests.  The evaluation level documentation consists of:

• Test Plan
• Test Status Report
• Test Completion Report

The actual testing is initially specified through the development of the test plan. In order to produce the test plan, its creators need to familiarise with the context in order to define the scope, organise its development of the test plan, identify and estimate risks, establish approach towards them, design the strategy for the particular testing, determine staffing profile and schedule, and after the plan is drafted, establish consensus or get approval for it, and distribute it to all involved.

The assessment relies on stakeholders monitor the progress through test status reports and test completion reports to monitor the progress, introduce corrective measures and appraise testing results. The monitoring of test progress is performed on the base of test status and related measures/metrics/reports. The stakeholders receive reports on test status, and issue control/correction directives are issued and corrective/adaptive actions are made for/on the test design, environment, execution, and perhaps even evaluation level on in order to appraise test status and assess related measures and metrics. They issue directives for corrective and adaptive actions on test design, environment and execution, which results in changes in test level documentation. Ultimately, these measures may lead to modification of the test plan.

The test enactment may also provide some suggestions or inputs for updates of the general approach and strategy on testing and validation. It is on those responsible for test management to further articulate and propagate such initiatives.

Traceability is the key characteristic of both evaluation planning and test enactment processes and artifacts. Repeatability is highly desirable in tests related to core requirements and functionalities. For less importan or involving features and requirements, there is no maintain repeatability of tests that were already done unless it is possible to fully automate the test execution, or if it is required for compliance or periodic audits.

Test Enactment

Test Enactment

The enactment consists of test design and preparation and execution of all planned tests. its processes The enactment consists of test design and preparation and execution of all planned tests. its processes produce and update test level documentation.

...

Figure: Test Enactment Processes Diagram

Test Documentation

The test level documentation is produced by the testing team. It refines the practical details of the design and execution of the work specified in the test plan and captures or relevant information gathered during testing in order to support the evaluation level reports.

• Test Design Specification
• Test Case Specification
• Test Data Report
• Test Environment Report
• Test Execution Log
• Detailed Test Results
• Test Incident Report

Traceability is the key characteristic of both test management and test enactment processes and artifacts. Repeatability is highly desirable in tests related to core requirements and functionalities. For less important or involving features and requirements, there is no maintain repeatability of tests that were already done unless it is possible to fully automate the test execution, or if it is required for compliance or periodic audits.

Test Documentation

Below described documentation provides an hierarchy of artifacts that is aligned with common industry Below described documentation provides an hierarchy of artifacts that is aligned with common industry practices and is generally compatible with the IEEE 829 standard and its sequel/update ISO/IEC/IEEE 29119-3, which even provides detailed. templates for both traditional (sequential and iterative) and agile approaches.

...

• Organizational numeric identifier of the document, may be ommitted if the document name is used as the identifier
• Descriptive and identifying name of the document
• Name of testing (sub)project or use case, if not clear from document name
• Document date
• Version number
• Author or authors and their contacts
• Version history (table with version numbers, dates, contributors and descriptions of changes)

Version history, date(s) and authors are not needed for documents with masters that are kept up to date in already highly versioned environment, such as an CMS system or Wiki. However, the main or corresponding author and document date need to be visible in self-contained standalone snapshots that are as such published on the web or shared by email.

References/Supporting Documents/Literature

The list of documents with their identifiers, names, version numbers and hyperlinks to individual document should be provided.

At least the reference to the project plan should be provided in its subordinate documents. There is no need to reference other documents that represent the common background that is already listed in the project plan. In lower level documents, only the references that are crucial for its understanding need to be provided, and in order to point to the relevant external documentation that is not already referenced in higher level documents.

Evaluation Level Documentation

These are the elements of documentation related to testing of as a specific service, product or solution that are intended for communication between those responsible for test management processes (planning, monitoring and control, and completion assessment) and those who actually design, implement, execute and document the planned tests.On the base of of status and completion reports, control and correction directives are issued which lead to corrective or adaptive actions and changes in test level documentation or even the test plan.

• Test Plan
• Test Status Report
• Test Completion Report

Test Level Documentation

These documents are produced by the testing team. They refine the practical details of the design and execution of the work specified in the test plan and capture or relevant information gathered during testing in order to support the evaluation level reports.

• Test Design Specification
• Test Case Specification
• Test Data Report
• Test Environment Report
• Test Execution Log
• Detailed Test Results
• Test Incident Report

...

• Version history (table with version numbers, dates, contributors and descriptions of changes)

Version history, date(s) and authors are not needed for documents with masters that are kept up to date in already highly versioned environment, such as an CMS system or Wiki. However, the main or corresponding author and document date need to be visible in self-contained standalone snapshots that are as such published on the web or shared by email.

References/Supporting Documents/Literature

The list of documents with their identifiers, names, version numbers and hyperlinks to individual document should be provided.

At least the reference to the project plan should be provided in its subordinate documents. There is no need to reference other documents that represent the common background that is already listed in the project plan. In lower level documents, only the references that are crucial for its understanding need to be provided, and in order to point to the relevant external documentation that is not already referenced in higher level documents.

Test Plan

The test plan outlines the operational aspects of executing the test strategy for the particular testing effort. It provides an overview of what the system needs to meet in order to satisfy its intended use, user needs, or scope of the intended testing effort, and how the actual validation is to be conducted. This plan outlines the objectives, scope, approach, resources (including people, equipment, facilities and tools) and amount of their allocation, methodologies and schedule of the testing effort. It usually also describes the team composition, training needs, entry and exit criteria, risks, contingencies, test cycle details, quality expectations and tracking and reporting process. The test plan may account for one or several test suites, but it does not detail individual test cases.

...

The approach for dealing with schedule slippages should be described in responses to associated risks. Possible actions include simplification or reduction of non-crucial activities, relaxation of the scope or coverage, elimination of some test cases, engagement of additional resources, or extension of testing duration.

Test Status Report

Test status report is a one-time interim summary of the results of the execution of testing activities. It may describe the status of the all testing activities or be limited on a single test suite. This report, as well as the test summary report, is sublimes the information obtained during test execution and recorded in test logs and incident reports. It needs to be highly informative and concise and should not elaborate minor operational details.

...

The summary of activities conduced since the previous status report is optional. If present, is should exist in all status reports.

Test Completion Report

The test completion or summary report is a management report that brings together the key information uncovered by the accomplished tests. It recapitulates the results of the testing activities and indicates whether the tested system is fit for purpose according to whether it has met the acceptance criteria defined in the project plan. This is the key document in deciding whether the quality of the system and the performed are sufficient for to allow taking of the decision or step that was the reason for the testing effort. Although the completion report provides a working assessment on success or failure of the system under test, the final decision is made by the evaluation team.

...

The summary of activities conduced during testing should record what testing was done and how long it took. In order to facilitate improvement of future test planning, it should be in a form that does not require later thorough investigation of the plans and records of the conducted testing.

Test Design Specification

Test design specification addresses the test objectives by refining the features to be tested, testing approach, test cases, procedures and pass criteria. This document also establishes groups of related test cases.

...

This specifies the criteria to be used to determine whether the feature or a group of features has passed or failed, on the base of results of individual test cases.

Test Case Specification

The test case specifications are produced after the test design specification is prepared. The test case specification is a detailed elaboration of a test case identified in the test design specification and includes a description of the functionality to be tested and the preparation required to ensure that the test can be conducted. A single test case is sometimes associated with several requirements. It may be partially or fully automated.

...

The test script is a sequence for instructions that need to be carried out on the tested system in order to execute a test case, or test a part of system functionality. These instructions may be given in the form suitable for manual testing or, in automated testing, as short programs written in a scripting or general purpose programming language. For software systems or applications, there are test tools and frameworks that allow specification and continuous or repeatable execution of prepared automated tests.

Test Data Report

This document describes the data used in testing. It is typically produced in two stages.

...

Test case specifications provide more elaborate details about data needed and should be sufficient to support the actual collection or generation of adequate test data. During the second stage, the selected tools are used to prepare these data for execution of all use cases, including their injection into the system or interactions during the test procedure steps. At the same time, the corresponding expected test outputs are defined, and, if possible, automated methods for comparing the baseline test data against actual results. The limitations of test data and supporting tools are identified and explained what to do in order to mitigate them during the test execution and in interpretation and evaluation of the results. Finally, the measures that ensure usability and relevancy of test data throughout testing process need to be conducted. They include data maintenance, for example to support the changes in the system, but also data versioning and backup. The decisions and knowledge produced during the preparation of the test data are captured.

Test Environment Report

This document describes the test environment. It is typically produced in two stages.

...

The report is updated after the test environment is set up. A smoke test can be performed. The limitations of the test environment are identified and explained what to do in order to mitigate them during the test execution and in interpretation and evaluation of the results. The maintenance plans, responsibilities and arrangements are established. If envisioned in the test plan, this includes upgrades of current versions of test items, upgrading of used resources and network topology, and updating of corresponding configurations. The initial design is updated to reflect the test environment as it was built. The decisions and knowledge produced during the implementation of the test bed are captured..

Test Execution Log

The test execution log is the record of test cases executions and obtained results, in the order of their running. Along with test incident reports, the test execution log is a base for test status and completion reports.These documents allow direct checking of the progress of the testing and provides valuable information for finding out what caused an incident.

...

If the testing is organised around scenarios instead of test cases, the general structure of the log is unchanged, except that inputs, states and outputs are replaced with interpretation of the interactions and results for each segment of the scenario, supported by key excerpts or snapshots of characteristic inputs and outputs.

Detailed Test Results

Detailed test results, are the actual outputs, assertions, and system and monitoring logs produced during the execution of tests. They should be at least paired with the corresponding test execution log records. Their format may depend on test tools that are used to capture them. In addition, the detailed results may encompass the reports produced by test automation tools that compare the baseline test data against actual results, which highlight all noted deviations. Such reports are valuable traces of how obtained results measure up with expected postconditions, states and outputs and can be used in assessment of the the execution status and writing of the test execution log and test incident reports.

Test Incident Report

The test incident report is used to document any event that occurs during the testing process that requires investigation. A discrepancy between expected and actual results can occur because the expected results are wrong, the test was wrongly run, or due to inconsistent or unclear requirements, fault or defect in the system or problem with the test environment. It should provide all details of the incident such as actual and expected results, when it failed, and any supporting evidence that will help in its resolution. All other related activities, observations, and deviations the standard test procedure should be included, as they may also help to identify and correct the cause of the incident. The report also includes, if possible, an assessment of the impact of an incident upon testing.

...