...
Testing as a cognitive, critical and often creative process. Wise It builds upon experience, discipline and persistence, but also benefits from thinking out of the box. It benefits from the wise application of standards, tools, best practices, adapted to a specific case, goal, available resources, and local culture. This document as a support for establishing the common ground and customisable toolbox, and supporting guide. Relationship to the existing GN documents, testing standards, and software testing practices. Acknowledge that the recent standardization efforts try to build upon practices and experiences from different fields of engineering and industries. Do not reinvent the wheel, use the existing expertise, tools, practices, and processes. Actual testing, even with automated execution, is designed and supervised by humans.
This "framework" outlines a structured approach to testing, which is aligned with the major standards in the field, such as ISO/IEC/IEEE 29119, IEEE 829 and BS 7925, complements ITIL recommendations and processes, but is at the same time in agreement with the common language and practices in testing. It currently meant to support the GÉANT SA4 software service validation and testing process, but may grow into a a tool for strengthening of the common ground between software developers and testers and the computer networking community, and be used in consolidation of evaluations of various products, solutions and solutions developed and applied within the GÉANT project.
Streamlining of the testing process for software product has been the subject of significant standardisation effort in the last two decades. As software becomes an intrinsic part of almost every system, an effort has been has to align its validation with needs, practices and experiences from many industries and different fields of engineering. However, this field is still rapidly evolving and there is currently a strong opposition to its its formalisaton. There is no No guarantee that any specification can ensure fully repeatable testing practice that is applicable in all possible contexts. Often, the inherent fluidity of the environment and goals and complexity of tested system systems is likely to preclude any exhaustive formalization.codification.
Therefore, the here presented material should be see as a customisable toolbox. It is intended to prevent the reinventing of the wheel, and to digest the existing expertise, tools, practices, and processes. An over-administrative approach must be avoided, as there is no single size that fits Avoid over-administrative approach; no one-size-fits- all. Schematic and formal application and implementation of the standards, procedures or procedures templates “by the book”, this document included, endangers to may petrify, bureaucratise, and over-proscribe validation and testing process and impede innovation. What matters is the quality. HoweverActual testing, even with automated execution, is designed and supervised by humans. On the other hand, running of production-level services may does require formalised processes, traceability or even auditability of validation, and particularly unification of check control procedures.Generic Testing Process Diagram
General Testing and Validation Governance
XXX This document provides GN and, more specifically, GN4.1 SA4 with a generalised technical specification for testing and validation. It can be seen as a Although this document can be seen as a a contribution to the general specification of GN overall GÉANT approach to validation and testing, on the base of its the overall testing policy (it is beyond its scope to discuss the general validation and testing policy and strategy for implementation and management of services - any sources/refs?), and as an (tentative?) technical expression of the shared testing strategy.The overall policy and strategy, as well as this technical document, must . Such policy and strategy may be further discussed and agreed refined by the project management and stakeholders and published.Should
Both the strategy and supporting guidelines should be periodically reviewed and updated on the base the experience from the actual
...
evaluation, organizational and environmental changes and evolution of
...
general policies and goals.
...
Evaluation Planning
...
As a technical approach to validation and testing, this material supports the practical organisation and documentation of individual evaluations. They consist of two groups of processes: test management and test enactment.
Here proposed descriptions and outlines of documentation artifacts and related comments are informative and guiding nature. This material does not provide any guidelines for organisational or strategic level of testing related documentation. However, the top level policy and strategy may mandating the overall approach to validation and testing on the base of documents such as this one.
Test Management
Test management primariky deals with the evaluation level documentation.
The actual testing is initially specified through the development of the test plan. In order to produce the test plan, its creators need to familiarise with the context in order to define the scope, organise its development of the test plan, identify and estimate risks, establish approach towards
...
them, design the strategy for the particular testing, determine staffing profile and schedule, and after the plan is drafted, establish consensus or get approval for it, and distribute it to all involved.
The assessment relies on test status reports and test completion reports to monitor the progress, introduce corrective measures and appraise testing results. The monitoring of test progress is performed on the base of test status and related measures/metrics/reports. The stakeholders receive reports on test status, and issue control/correction directives are issued and corrective/adaptive actions are made for/on the test design, environment, execution, and perhaps even evaluation level test plan.
The test enactment may also provide some suggestions or inputs for updates of the general approach and strategy on testing and validation. It is on those responsible for test management to further articulate and propagate such initiatives.
Traceability is the key characteristic of both Evaluation and Test Enactment related evaluation planning and test enactment processes and artifacts. Repeatability for is highly desirable in tests related to core requirements and functionality. There is no need to repeat and functionalities. For less importan or involving features and requirements, there is no maintain repeatability of tests that were already done unless it is possible to fully automate the test execution, or if it is required for compliance or periodic audits.
Test Enactment
The enactment consists of test design and preparation and execution of all planned tests. its processes produce and update test level documentation.
Test design includes specification of requirements with conditions for their fulfillment. The conditions may also be contextual and include repeatability or presentability. It is followed by specification of individual tests cases and procedure for their execution. The subsequent steps are test environment set-up, test execution and related reporting. The feedback from environment set-up and execution may lead to an update of the original design. For example, the constraints in capabilities of the test environment may lead to the update of test cases. Alternatively, actual preparation of test cases and environment may require additional elaboration and refinement of test cases. On the other hand, significance /or priority of some requirements may lead to modification of the test environment in order to enable execution of corresponding test cases. Both will lead to modification of the test procedure.
In parallel, monitoring of test progress is performed on the base of test status and related measures/metrics/reports, stakeholders receive reports on test status, and issue control/correction directives are issued and corrective/adaptive actions are made for/on the test design, environment, execution, and perhaps even evaluation level test plan.
The test enactment may also provide suggestions or inputs for updates of the general Testing and Validation Process and Strategy.
Test Documentation
to enable execution of corresponding test cases. Both may lead to modification of the test procedure.
Figure: Test Enactment Processes Diagram
Test Documentation
Below described documentation provides an hierarchy of artifacts that is aligned with common industry practices and is generally compatible with the IEEE 829 standard On the common industry practices and with the strong influence from IEEE 829, and its sequel/update ISO/IEC/IEEE 29119-3, which even provides even detailed. templates for both traditional (sequential and iterative) and agile approaches.
Metadata
All documents should start the following common elements (metadata):
...
At least the reference to the project plan should be provided in its subordinate documents. There is no need to reference other documents that represent the common background that is already listed in the project plan. In lower level documents, only the references that are crucial for its understanding need to be provided, and in order to point to the relevant external documentation that is not already referenced in higher level documents.
Organisational/Strategic Level
...
.
Evaluation Level Documentation
These are the elements of documentation related to testing of as a specific service, product or solution that are intended for communication between those responsible for test management processes (planning, monitoring and control, and completion assessment) and those who actually design, implement, execute and document the planned tests.On the base of of status and completion reports, control and correction directives are issued which lead to corrective or adaptive actions and changes in test level documentation or even the test plan.
...