Qualification Pre-Integration Testing (PIT)
Pre-Integration Testing (PIT) was a requirement of all users of the MHHS arrangements, including software providers to participants.
PIT was performed by all users of the new settlement arrangements to validate every system or service that is included in the MHHS End-to-end (E2E) Design for compliance with its functional and technical requirements. This included back-office systems if affected. The system/service may have several components, and PIT referred to the testing conducted when those components were all internally integrated.
PIT was the responsibility of each Programme participant, using their own systems, test environments, test data, test artefacts (Scenarios and Test Cases), test processes, test tools, test management tool and defect management process.
The majority of PIT activity was an entry criterion for Qualification Testing. However, there were some elements that could only be tested in PIT, for example, performance and volume testing.
PIT Guidance
The latest PIT Guidance document set out what was expected of Programme participants regarding their own PIT testing whether the Programme participant chose to go through the System Integration Testing (SIT) or non-SIT route.
The Programme held a PIT Guidance session on 4 April 2024, to walk participants through the updates and the approach to PIT. You can view the slides, recording and Q&A below:
PIT Guidance Walkthrough slides
PIT Guidance Walkthrough recording
If a Programme participant Placed Reliance on Functional Testing carried out by a third-party software / IT provider, there may have needed to be additional testing requirements to evidence that the software was successfully deployed within the Programme participant’s environment, including any internal integration with the third-party software / IT solution.
Subject to the level of Placing Reliance agreed, there may be some scenarios in which the PIT deliverables expected for participants may have varied. For example, if it was agreed that a participant may place full reliance on Functional Testing, including PIT, then it may be that submitted PIT deliverables only covered Migration, Non-Functional and Operational aspects. This would be agreed on a case-by-case basis.
Please view Section 7 in the PIT Guidance document for diagram examples of PIT scope for different Programme roles.
The Programme uplifted the PIT Guidance document to version 2.8 which included changes to ensure that the guidance reflected updated delivery timelines following Ofgem's approval of CR055: Amendments to M10 and Corresponding Milestones, as well as the role of the Data Integration Platform (DIP Manager) in assuring Non-Functional Testing requirements.
You can view the latest version below:
What did the updated PIT Guidance include?
- An overview of Non-Functional Test (NFT) delivery expected within PIT including:
- A test carried out at volume against Programme Non-Functional Requirements (NFRs).
- Full suite of performance testing carried out.
- Component Resilience and Data Loss Testing at volume, where Resilience Components were defined as layers within the stack e.g., adaptor, web and application server and Design Build (DB).
- Migration at expected volumes alongside daily use.
- Infrastructure and Results to include the Data Integration Platform (DIP) Adaptor and scaled, production sized during PIT. There is no DIP Simulator available.
- Evidence of Observability in place to assure results.
- The document was aligned to the latest Programme position regarding Interim Release (IR) versions to be used during Component Integration Testing (CIT) - IR2 and SIT Functional Test execution - IR5.
- Increased clarity on availability of supporting DIP Simulators for Programme participants PIT activities.
- Amendments to references of DBT2 as DBT2 testing evidence is no longer a requirement for MHHS Qualification based on the following position stated by the Code Delivery Bodies.
- The scope of Qualification under the Balancing & Settlement Code (BSC) and Retail Energy Code (REC) is defined by the relevant Code Delivery Bodies’ vires that were limited to the requirements within the respective Code.
- The Code Delivery Bodies didn't identify any code requirements that were outside of the scope of the MHHS design that Programme participants would be required to cover as part of their PIT evidence for Qualification.
The PIT Guidance additionally included further details about the scope and test coverage of PIT, test evidence, test assurance scope and Programme participant’s adopting the Placing Reliance policy for PIT.
What was the status of the PIT deliverables templates and when are they due?
All Non-SIT Suppliers & Agents were expected to submit the following PIT deliverables:
- Final PIT Approach & Plan, covering Functional, Migration, Non-Functional and Operational - Please note: participants were advised to use the PIT Approach & Plan template available below to prepare Final Approach & Plan submissions and to ensure that all necessary topics were covered.
- PIT Requirements to Test Traceability Matrix (RTTM), including list of PIT Test Scenarios
- Qualification Testing Interim Readiness Report, including PIT progress
- PIT Completion Report, including Final PIT RTTM and all PIT evidence uploaded
The timelines for PIT deliverables and PIT completion across Functional, Migration, Non-Functional and Operational) were detailed in the Qualification Approach & Plan, available on the QA&P page of this website, and the PIT Guidance document (available above).
The exception to this was testing relating specifically to Volumetrics and Performance, as it was recognised that participants may wish to complete this testing on as final a code base as possible. In this scenario, and by agreement with the Code Bodies, participants may have completed and evidenced their Volumetric / Performance testing alongside their Qualification Testing execution. This needed to be completed before Qualification Testing exit.
Requirements to Test Traceability Matrix (RTTM)
You can view the RTTMs published by the Code Bodies and the Programme for Non-SIT participants on the Qualification RTTM page of this website.
What is the role of the Data Integration Platform (DIP) Manager?
The DIP rules were implemented on 1 October 2024 and the DIP Supplement of the Balancing & Settlement Code (BSC) established BSCCo as the DIP Manager.
The DIP Manager is responsible for assuring a Programme participant can meet its requirements set out in the DIP Rules and approving applications to become DIP Users. As each Programme participant must be approved as a DIP User before they are MHHS Qualified, there are aspects of Qualification activity that will be delivered by the DIP Manager, such as Non-Functional Test assurance.
Responsibility for testing activities that provided evidence to support MHHS Qualification and DIP Manager assurance were split between Code Bodies, the DIP Manager and the MHHS Programme.
LDSO Qualification Testing was managed by the MHHS LDSO Qualification Test (QT) team, which was part of the MHHS Programme, with the Code Bodies and DIP Manager undertaking an assurance role to confirm the scope of testing and the output satisfies the wider Qualification and DIP Manager assurance needs. Code Bodies managed Qualification Testing for the remaining Programme participants.
This didn't impact the work that Programme participants had done so far to prepare for Qualification, but was something that Programme participants needed to know going forward.
To find out more about the role and responsibilities of the DIP Manager, please see the Qualification and End-to-end (E2E) Sandbox Working Group (QWG) papers from the meeting on 12 November 2024, which you can find on the QWG page of the Collaboration Base.
PIT Evidence Obfuscation
The Code Bodies published guidance for participants on PIT evidence obfuscation, and you can find more information on this on the Qualification PIT Evidence Obfuscation page of this website.
