Texas Market Test Plan/Retail Commercial Operations

Version 1.3
January 11, 2005

Prepared by 

Texas Test Plan Team
Retail Market Subcommittee Workgroup

Table of Contents
Table of Contents	2
Document History	3
Texas Market Test Plan	4
Purpose and Scope	4
Exclusions	4
Testing Assumptions	4
Testing Overview	5
Testing Website	5
Testing Signoff Worksheet (TSW)	6
Testing Guidelines	7
In-Flight/Out-of-Flight Testing	10
System Changes	13
Testing Details	16
Data Loading	19
Certification	20
Testing Requirements of ERCOT and Market Participants	21
General Marketplace Requirements	21
CR Requirements	21
TDSP Requirements	22
ERCOT Requirements	23
PUCT Requirements	24
Market Flight Administrator Requirements	25
Details of Testing Phases	26
Technical Connectivity and Verification	26
End-to-End Testing	27
Point-to-Point Testing	27
Appendices	40
Appendix A - Testing Signoff Worksheet	40
Appendix B – Technical Connectivity Worksheet	40
Appendix C - Resources	40
Appendix D - Marketplace Issue Resolution Form	41
Appendix E - Flight Test Bed Load Form	42
Appendix F – Testing Requirements Matrix	42
   Appendix G - Glossary of Terms and Acronyms Used in this Document not defined in Section 2 of the ERCOT Protocols……………………………………………..……….32
   Appendix H - Future Flights Schedule………………………………………………………..……33

Document History 
Date/Version	Summary of Changes	
01-08-05 v1.3	•	Update to reflect market changes associated with the Market Solution to Stacking; Reformatted; Defined Testing Guidelines; Updated Appendices	
01-08-04 v1.2	· Re-formatted TMTP document; Defined Testing Guidelines section; added appendix; updated TX SET business process scenarios	
11-03-03 v1.11	· Updated Technical Connectivity and Verification section to include NAESB EDM data transport method.  Added references to NAESB EDM data transport throughout document.	
01-29-03 v1.10	· Updated to reflect ERCOT role as testing facilitator, Issue Resolution Process, transaction additions and testing requirements applicable to various market participants; deleted the term ITPTA	
09-19-02 v1.09	· Updated the Re-testing guidelines timelines to be consistent across sections.	
04-30-02 v1.09	· Revised Re-testing guidelines to provide for ERCOT notification to the TTPT Chair 	
04-26-02 v1.09	· Revised purpose and scope section and re-testing section to clarify that the TMTP was for the Retail Market	
04-11-02 v1.09	· Updated Re-testing guidelines to add clarification	
03-28-02 v1.09	· Incorporated edit following TTPT review on 3-15-02	
03-12-02 v1.09	· Revised re-testing section to reflect the use of automated testing	
02-10-02 v1.08	· Incorporate information on automated testing · Provided addition explanation for the use of Testing Sign-off Worksheet and Technical Connectivity Worksheet · Provided additional details on testing the Replacement FTP process · Included details on transactions not support by the ERCOT portal · Added section on Point-to-Point testing · Updated Texas Retail Testing website address to etod.ercot.com	
12-13-01 v1.07 Final	· Incorporated re-testing guidelines and change in ERCOT connectivity process	
10-29-01 v1.07 Draft	· Update to reflect changes required for testing during 2002	
08-29-01 v1.06 	· Update to reflect changes required for Flight 1001	
06-17-01 v1.05 Draft	· Update to reflect changes required for Flight 3801	
05-03-01 v1.04 Final	· Moved from DRAFT to FINAL · Changed all ‘Certification’ references to ‘Qualification’	
04-19-01 v1.04 Draft	· Added language to clarify testing for ERCOT Texas Market Link · Added section on Test Plan Change Control · Added language for provisional qualification guidelines · Deleted TTPT membership information and published on website	
03-14-01 v1.03	· Final Draft · Refreshed script table · Clarified success criteria · General syntax/grammar/consistency cleanup	
02-21-01 v1.03d	· Second draft	
01-23-01 v1.0d	· First draft	
Texas Market Test Plan 
Purpose and Scope
The purpose of this document is to define the market plan for testing retail commercial operations systems and business processes to support the Texas Electric Choice Market.  This document covers all retail testing requirements and procedures between ERCOT and the Market Participants (MPs) and Point-to-Point retail testing between MPs.  In an effort to diminish the potential risks that could be introduced into the Texas Retail Electric Market from new unproven systems or MPs or from the effects of new TX SET/ANSI X12 EDI Version releases, the Texas Market Test Plan provides the mechanism for ensuring that the central retail systems operated by ERCOT are functioning properly, and that the retail systems operated by MPs interface properly with both ERCOT’s systems and other MPs’ systems. The Texas Market Test Plan addresses the following: 
•	Testing Guidelines
•	Testing Details
•	Testing Requirements for Market Participants and ERCOT
•	Testing Phases
•	Success Criteria
•	Overview of Testing Scenarios for Certification in the Texas Market

The Texas Test Plan Team (TTPT) is responsible for maintaining and updating the information in this document as defined in Section 23.3 of the ERCOT Protocols. All references to testing in the document are directed to the Retail Market.  
Testing Assumptions
·		MPs who wish to participate in the market using NAESB EDM or the ERCOT Texas Market Link for conducting retail operations in the Texas Marketplace will refer to this document for guidelines on these processes. 
· 	MPs may elect to not participate in testing optional processes as identified in this document but will inform their trading partners and the Market Flight Administrator in advance.  However, a MP does not have the option to refuse to test the basic processes necessary to ensure that the central retail systems operated by ERCOT are functioning properly, and that the retail systems operated by the MPs interface properly with both ERCOT’s systems and other MPs systems.
·		Automated internal processes are required when testing.  Any areas that require manual interaction or data manipulation shall be documented in advance in the Testing Signoff Worksheet and communicated to testing partners at the beginning of the testing cycle. 
•	All entities participating in a Test Flight, with the goal of gaining a certification, will use dedicated test environments that are representative of their production environments. 
•	   MPs planning to use the ERCOT Texas Market Link will so indicate in their Testing Signoff Worksheet.
·	  The Market Flight Administrator is the final authority on all levels of Business Process Certification among trading partners, including the verification that a party has successfully passed testing and is eligible to go into production. 
·	The Market Flight Administrator will moderate testing and report on test status including progress and issues to ERCOT, RMS, TTPT, other appropriate committees, and/or the PUCT.
·	Functional Acknowledgements provide a critical audit trail.  All parties will send Functional Acknowledgements (FA/997) for all EDI transactions (except for the receipt of a 997, which would create an endless cycle) during testing.  Parties using the ERCOT Texas Market Link will receive web-based acknowledgements.  Parties shall monitor acknowledgements sent and received, but are not a checklist item for flight success.

Testing Overview
Testing Website
The Market Flight Administrator maintains a Texas Retail Testing website that details the current status of the testing process.  The URL address for this website can be found in Appendix C.
This website includes:
·		The Texas Marketplace Test Plan (TMTP)
·	Test Scripts
•	Approved Test Flight Timelines
·	Daily agenda and minutes of each conference call
·	TTPT meeting schedule
·	Testing contact lists
·	Frequently Asked Questions (FAQs) on the Testing Process
·	Testing Status - Each organization will be able to obtain a status of the testing process, including its own status.  Information will be secured by organization.
·	Market Links
·	File Cabinet for significant testing materials
·	Testing Sign-off Worksheet (TSW)
·	Technical Connectivity Worksheet (TCW)
Testing Signoff Worksheet (TSW)
Each MP completes a Testing Signoff Worksheet (TSW) on-line. This worksheet includes basic contact information required for effective testing. It also identifies processes that will be tested including optional functions that the MP will use in their business plan and which they plan to test. 
The TSW link can be found in Appendix C.
Contacts
Parties shall provide daily and emergency contact information for the test lead, and the test lead alternate. Issue Resolution procedures require that an executive level contact also be provided.
At least one Business Contact shall be an employee of the Market Participant, not a vendor or service provider.
Exceptions to the Test Plan
Parties cannot arbitrarily require other parties to test certain features, scenarios or scripts, nor can they arbitrarily refuse to test certain features, scenarios or scripts.  This Test Plan details full-testing requirements for MPs.  There are legitimate scenarios where a party will not support a feature or scenario that is identified in a test script.  In these cases, a party can claim an ‘exception to the Test Plan’.  These exceptions shall be documented in the TSW, and shall be approved by the Market Flight Administrator.  The Market Flight Administrator will review exceptions on a case-by-case basis to determine the impact on the Marketplace.  Parties that claim “approved” exceptions will not be required to test those features. Once approved, this information will be shared with trading partners.
Manually-Assisted Processes
Each party shall identify the different processes that directly support data exchanges that require manual intervention.  Manual intervention increases the risk of errors or process failures and could serve to conceal systemic problems that might introduce transaction errors or hazards into the Market.  X12-formatted files shall never be altered manually except in the case of a simulated error for a test script. This information will be documented in advance on the Testing Signoff Worksheet and shared with trading partners.
Testing Responsibilities
The ‘Testing Responsibilities’ section details the responsibilities each party has in the testing process.  This Test Plan is focused on testing the most significant features of the marketplace.  Many tests that were considered ‘internal’ tests were removed from the Test Plan to optimize it.  However, some of these tests were deemed important and as a result appear in the ‘Testing Responsibilities’ section of the TSW.  Also, each party has certain obligations prior to, during and after testing which are outlined in the ‘Testing Responsibilities’ section.

Technical Connectivity Worksheet (TCW)
Each MP completes a Testing Connectivity Worksheet (TCW) on-line. This worksheet includes the specific testing communications information required for the test flight.  Connectivity schedules are arranged by the dates stated on the official flight schedule.
The TCW link can be found in Appendix C. 

Testing Guidelines

Market Testing can be categorized as two types, In-Flight and Out-of-Flight testing.  In-Flight Testing consists of market approved scheduled flight tests.  There are a defined number of test flights adopted by the TTPT and approved each year by the Retail Market Subcommittee, as directed by the PUCT.  Out-of-Flight testing is considered only for those changes deemed an ‘emergency’ or a “Specified Ad-Hoc Testing” for existing Market Participants in a specific service territory. 

As mentioned above, “emergency” changes or those deemed a “Specified Ad Hoc Testing” for existing Market Participants in a specific service territory are the only changes that will be considered for Out-of-Flight testing. Out of Flight testing requires advance notice to ERCOT and the Market Participant testing contacts listed on the testing website.  Upon confirmation of the “emergency” change or “Specified Ad Hoc Testing” by the Market Flight Administrator, based on the scenarios described further in this section, a mutually agreeable out of flight testing schedule will be developed between parties.   If an MP is unsure of the lead-time required, it is best practice to contact the Market Flight Administrator for guidance or clarification.
This section provides baseline requirements to assist a Market Participant in determining whether their change qualifies for out of flight testing, or needs to be tested in an approved test flight. These guidelines are intended to minimize risk to the Marketplace.  Market Participants (MPs) shall follow well-defined internal change management processes that document results and demonstrate due diligence when making changes.
Testing Requirements Matrix 
A tool has been developed to assist in determining the testing requirements for any changes made to systems or contracts.  This matrix is a guide; all testing requirements shall be verified with the Flight Administrator. (See Appendix F) 
In-Flight Testing
There are a defined number of test flights adopted by the TTPT and approved each year by the Retail Market Subcommittee, as directed by the PUCT.  Test flights approved by the TTPT, RMS and Technical Advisory Committee (TAC) and are posted on the Texas Retail Testing Website.
http://etod.ercot.com
Most Flights will take approximately ten to twelve weeks including Connectivity. 

New TX SET/ANSI X12 EDI Version Releases
All market participants, including ERCOT, shall complete required certification Flight Testing as defined by the TTPT when a new TX SET/ANSI X12 EDI Version release is approved by the Market.  On occasion, a Version release will not consist of any system or transaction changes to the Market or its participants.  In that instance the TTPT with the approval of TX SET and RMS may determine that an additional test flight for a particular release is not necessary.  In that event, all parties would not be required to test that specific release.

Contingency EDI Providers
A Market Participant who is certified in the Texas Marketplace with the current TX SET version that chooses to test with a Contingency Market Interface Service Provider shall do so in a scheduled test flight.  Testing for Contingency EDI Provider cannot be done during a TX SET version release.  A CR does not have to acquire an alternate DUNS number for testing with a Contingency EDI Provider.  One CR cannot test with two different EDI Providers in same flight.

Certified Market Participant Changes to a Non-Established Service Provider
A Market Participant who is certified in the Texas Marketplace with the current TX SET version may choose to move from their Market Interface Service Provider to another Market Interface Service Provider.  If the new Market Interface Service Provider  has not successfully completed certification testing for another Market Participant in the service territory in question this “Non-Established Service Provider” is required to execute tests during a scheduled market test flight.  Market Interface Service Provider is a term used to refer to an MP’s internal organization or an outsourced company that provides both connectivity and translation services for an MP. 
An MP that chooses to use a Market Interface Service Provider that has not successfully completed certification testing is required to contact the Market Flight Administrator to determine what Connectivity and Translator tests they are responsible for executing during the next scheduled market test flight.
An MP may not switch to a Non-Established Market Interface Service Provider as an “Emergency” without the express permission of the Market Flight Administrator.  A switch to a Non-Established Market Interface Service Provider by an MP is not considered a “Specified Ad Hoc Testing” and does require flight testing.

Non-Established Market Interface Service Provider Checklist
1.	Provide new Testing Signoff Worksheets (TSW) and Technical Connectivity Worksheets (TCW) to all trading partners (as required by test flight).
2.	Complete all scripts specified under the “New” testing track

New Trading Partnership
All new trading partnerships shall go through the testing process as prescribed in the TMTP during a scheduled market test flight.  The only exceptions are clearly set forth in the Out-of-Flight section.

Market Participants who Fail to Maintain Certification
•	A certified MP may choose to not actively participate in the Marketplace.  Regardless of this decision, they are required to maintain certification according to the current Marketplace baseline, including TX SET version and associated emergency change controls.  A party that fails to maintain the current baseline certification loses its certification.  An MP that has certified in the current version is not required to re-test to enter the market.
•	There are scenarios where the PUCT will revoke a CR/REP’s certification.  In this case, retail market certification is also revoked.  The CR/REP will be treated as a new MP, and shall complete all certification testing during a scheduled market test flight in order to re-enter the marketplace.
Marketplace Functional Changes
Marketplace functional changes can include the following:
·	New tracks, such as 
o	Continuous Service Agreement (CSA)
o	Affiliated Retail Electric Provider (AREP)
o	Competitive Metering (COMET)
o	Service Order Option 1
o	Disconnect for Non-Pay
·	New TX SET versions, including normal and emergency change controls
·	New transactions 
·	Outage proof of concept
·	New Connectivity Options (NAESB EDM)

Banking Changes
Trading partners shall be notified when changes occur with the banking institutions they use. The changes may be caused by any number of reasons including bank mergers or upgrades to newer releases of ANSI standards. These changes may result in new routing codes, account numbers, format changes to the remittance advice or other changes that would affect one party’s ability to deliver and/or reconcile invoices and payments. When such changes occur, it the responsibility of the party whose bank made the change to initiate testing with their trading partners. 
Testing shall use one or more of the following methods to verify that payments and remittances between trading partners remain timely and accurate. 
•	Penny test
•	Invoice/Remittance 

Out-of-Flight Testing
The Texas Market Test Plan assumes that testing will occur during a regularly scheduled test flight. However on those occasions when the only requirements for testing between Market Participants are connectivity with ERCOT, connectivity with the TDSPs and/or connectivity with their Bank, the REP has the option to request testing from the TDSPs and ERCOT at anytime during a regularly scheduled flight.  The definition of a regularly scheduled flight for the purposes of this section is the time frame between the connectivity kick-off call and the offend date for Contingency testing for the Flight.  This request shall be made via e-mail to the primary, secondary, and business testing contacts of the TDSP and ERCOT no later than two weeks in advance of their requested testing timeline.  Upon receipt of request, the TDSPs and ERCOT, within two weeks, will be required to propose a test schedule with the REP.

Timing Guidelines 

If a Market Participant encounters a situation that can be categorized as an “Emergency Change” or a need for “Specified Ad Hoc Testing”, their request for testing shall be made via e-mail to the primary, secondary, and business testing contacts of the TDSP and ERCOT at least two weeks in advance of their requested testing timeline.  Upon receipt of request, the TDSPs and ERCOT have two weeks to propose a test schedule with the REP.  While such testing can take place during or between scheduled Test Flights as defined by the TTPT, there will exist a “black out period” during which such testing will not be conducted.  The black out period is required for ERCOT and the TDSPs to setup up test beds (See Appendix E) and establish connectivity with new trading partners for a regularly scheduled Test Flight.  The black out period would exist from the time ERCOT provides the list of new trading partners for a test flight to the TDSPs and ends on the date the TDSPs provide the test bed loads (See Appendix E) to ERCOT, currently about four weeks.  The Future Flights Schedule link can be found in Appendix H.

Emergency Changes
There are a number of scenarios that may dictate emergency action to resolve production problems. Emergency Out-of-Flight testing guidelines address situations like:
·	System failures, disaster recovery, and/or business resumption plan execution
·	Failure of internal or subcontracted entities - There are a number of situations that may require a party to quickly replace an entity because of production failures.  
•	The party requesting the Out-of-Flight testing has the final discretion on what constitutes a failure providing they choose to use a Market Interface Service Provider that has successfully completed certification testing with another Market Participant in the specified service territory.  
•	If they choose to use a Market Interface Service Provider that has not successfully completed certification testing for another Market Participant in the service territory in question, the Flight Administrator will have the final discretion on what can be tested Out-of-Flight.
·	Current bank used by MP goes out of business
If a Market Participant encounters a situation that can be categorized by one of the above-mentioned scenarios, their request for testing shall be made as outlined in the Timing Guidelines.

Emergency Change Testing Checklist
a.	Provide new Testing Signoff Worksheets (TSW) and Technical Connectivity Worksheets (TCW) to all trading partners (as required by test flight).
b.	Complete the Connectivity Test Scripts as defined by the TTPT
c.	Complete the Penny Test Script as defined by the TTPT
d.	Complete the Basic Enrollment Script(s) as defined by the TTPT, at the discretion of the Market Flight Administrator.

Emergency changes considered for out of flight testing does not include market wide emergency changes that may occur due to a PUCT Ruling.

Changes constituting a Specified Ad Hoc Testing 
There are a number of scenarios when an existing Market Participant may determine that there exists a need for “Specified Ad Hoc Testing: to institute a particular change to their systems or processes and they can do so in a manner that does not impose undue risk to the Market.  Out-of-Flight testing guidelines address the following situations that are considered applicable for “Specified Ad Hoc Testing”:
1.	 Current Market Participant adds a New Additional DUNS by Certified REP
Market Participant who is certified in the Texas Marketplace with the current TX SET version determines that they need to establish a new Additional DUNS by Certified REP (DUNS + 4) under that MP’s existing umbrella.  In this instance the certified Market Participant in a specific service territory is simply adding a new trade name and DUNS number that will be utilizing the same Load Serving Entity (“LSE”), back-end systems, services, EDI Provider, and banking relationships.   
If a Market Participant encounters this situation, their request for testing shall be made as outlined in the timing guidelines.  The REP will be required to complete the following before being certified to enter the Market under the new ADDITIONAL DUNS BY CERTIFIED REP name and DUNS number.  The REP will be required to complete the following before being certified to enter the Market under the new ADDITIONAL DUNS BY CERTIFIED REP name and DUNS number:

New Additional DUNS By Certified REP (DUNS + 4) Checklist
a.	 Provide new Testing Signoff Worksheets (TSW) and Technical Connectivity Worksheets (TCW) to all trading partners.
b.	Complete the Connectivity Test Scripts as defined by the TTPT.
c.	Complete the Penny Test Script as defined by the TTPT.

2.	Current Market Participant Changes to an “Established” Service Provider
A Market Participant who is certified in the Texas Marketplace with the current TX SET version determines that they need to change their Market Interface Service Provider to another Market Interface Service Provider that is currently serving another Market Participant in a specified service territory or to an “Established Service Provider”.  Market Interface Service Provider is a term used to refer to an MP’s internal organization or an outsourced company that provides both connectivity and translation services for an MP.   An “Established Service Provider” is defined as an organization or company that provides both connectivity and translation services to another Market Participant in the same service territory and that has successfully tested in the Marketplace provided they tested using the current TX SET version.  This includes changes to internal organizations, external subcontractors, and/or external companies and service providers. 

If they choose to use a Market Interface Service Provider that has not successfully completed certification testing for another Market Participant in the service territory in question or a “Non-Established Service Provider” they will be required to complete In-Flight-Testing.

If a Market Participant wishes to make this change, their request for testing shall be made as outlined in the timing guidelines.  The REP will be required to complete the following before being certified to enter the Market with the new “Existing Service Provider”:

Established Market Interface Service Provider Checklist
a.	Provide new Testing Signoff Worksheets (TSW) and Technical Connectivity Worksheets (TCW) to all trading partners (as required by test flight).
b.	Complete the Connectivity Test Scripts as defined by the TTPT.
c.	Complete the Penny Test Script as defined by the TTPT.
d.	Complete the Basic Enrollment Script(s) as defined by the TTPT.

System Changes

During the normal course of Marketplace operations companies will need to make changes to their systems, including connectivity systems, translation systems, and other back-end systems including billing, metering, customer information, etc.  Once a party has qualified and is in production in the Marketplace, changes to systems can have a significant impact on trading partners and the Marketplace. 
System Change Assumptions: 
·	System changes made by one MP can have an impact on trading partners (TPs) of that MP.
·	An MP shall do sufficient internal testing, including regression testing, to minimize the impact of changes on its TPs.
·	An MP shall communicate to its TPs clearly and early regarding changes to systems.  This includes advance notice of the change, copies of migration plans, etc…
·	An MP shall identify a ‘back-out’ strategy where appropriate in case problems as a result of changes cannot be resolved quickly.
·	‘Translator systems’ include any hardware, software, and system configuration used to create the ANSI X12-compliant files sent to TPs.  It does not include mapping.
·	‘Connectivity systems’ include any hardware, software and system configuration used to deliver files to and from a TP.  It includes the NAESB--based electronic delivery mechanisms (EDM). 
·	‘Service Provider’ is a vague term that can refer to many different types of entities used by MPs in the Marketplace.  These include connectivity, translation, testing, billing, metering, etc.
·	Market Interface Service Provider is a term used to refer to an MP’s internal organization or an outsourced company that provides both connectivity and translation services for an MP.
•	While many changes to systems will be intentional and planned, there are emergency scenarios, such as a system failure, where advanced planning and notice are not possible.

System Change Categories
Connectivity System Changes and/or Updates
Connectivity is defined as the systems used to send and/or receive files to/from your trading partners.  These include NAESB EDM for CR/TDSP/ERCOT communications as well as changes in security keys, IP addresses, DNS names, and DUNS numbers.

Connectivity Change Checklist
1.	Communicate the planned changes of communication systems  to the Market Participant testing contact listed in the Technical Connectivity Worksheet (TCW) on the website, in accordance with sign-up for an approved flight test
2.	Send new Technical Connectivity Worksheets and new Trading Partner Agreements to TPs where necessary, including changes to DUNS, IP address, name change, or any other change to the information on the Technical Connectivity Worksheet (TCW).
3.	Complete the change during the approved In-Flight test or in accordance with the Out-of-Flight Testing requirements set forth above and schedule migration date with each trading partner.  If connectivity is the only change for that trading partner the migration date is not dependent upon the end of the flight test.

Major changes to communication protocols may require more rigorous testing.  Specific requirements for testing these changes will be defined within the specifications for the new communication protocol.   

Translator System Changes and/or Updates
Translators or systems used to perform the data transformation that create EDI ANSI X12 files may be changed or require upgrades.  These changes pose significant risks to individual participants and the marketplace as a whole.  All participants in the market shall therefore address these changes with a clear understanding that;
·	ERCOT and market participants need to take responsibility for any changes they make in their data transformation system(s) and test with trading partners when they perceive a risk.
·	ERCOT and market participants shall do internal testing to help minimize the risk to the market.
·	Best practices include regression testing of translator changes using historical data.

Translator Change Checklist

1.	Provide new Testing Signoff Worksheets (TSW) and Technical Connectivity Worksheets (TCW) to all trading partners (as required by test flight).
2.	Complete the Connectivity Test Scripts as defined by the TTPT.
3.	Complete the Penny Test Script as defined by the TTPT.
4.	Complete the Basic Enrollment Scripts as defined by the TTPT.

Back-end System Changes and/or Updates 
A company’s “back-end systems” is defined as any part of an MP system that exists behind the connectivity protocol (NAESB, FTP, HTTP, etc.) and the Market Interface Service Provider.  Once the data passes through the communication interface and the Market Interface Service Provider, the data enters the back-end system.  Because each MP's back-end system architecture is different, back-end systems could include, but are not limited to the business process management system, billing system, or the data management system (database).

ERCOT and/or MPs are required to take responsibility for any changes they make in their back-end system(s) and, if a potential risk is perceived, shall test with trading partners to minimize that risk.  MPs shall follow change management best practices, including extensive internal testing, regression testing on historical data, etc.

ERCOT and MPs are not required to test when changes are made to their back-end system(s).  As part of the company’s internal testing procedures, they may request to test with all, some, or none of their trading partners; however, it is considered good business practice for MPs and ERCOT to communicate any changes, replacements, and/or upgrades to their trading partners.

Back-end System Change Checklist
1.	Where appropriate, notify TPs and the Market Flight Administrator of the planned changes to back-end systems. Notification needs to be provided to the Market Participant testing contact listed in the Technical Connectivity Worksheet (TCW) on the website.
2.	Where appropriate, communicate testing plans to TPs including cut-over, migration and back-out plans.
3.	Where appropriate, coordinate and agree on plan schedules/milestones with TPs.
4.	Complete the change and confirm successful migration with each trading partner.

Marketplace Production Failures
Nearly all production failures and/or rejections increase the costs of the Marketplace.  Parties are required to work through any production failures directly with their TPs.  
Parties that habitually cause these failures may be required to conduct additional testing to demonstrate their certification.  Parties that are the victims of production failures can use the defined Issue Resolution Process after several direct attempts to resolve the problems have failed.

Testing Details
Synchronized Testing 
Synchronized testing requires that each party execute the same tasks across all TDSPs.  
For example:
On Day One of End-To-End testing, all CRs will send in all of the transactions identified in Day One to begin scripts they will test.  
If CR #One is participating in five TDSP territories, the CR will send a number of enrollment transactions; i.e., one or more transactions for each of the test scripts that involve enrollment for each of the five TDSPs.
There is significant risk in this type of testing if parties are not prepared and cannot keep pace with the test schedule.  The test schedule is defined and maintained by the TTPT and is published on the ERCOT Texas Retail Testing website.  
MP Testing Flights 
Testing “Flights” have been established to organize testing scenarios into pre-defined groups.  Specific details of these flights are covered in the section “Details of Testing Phases” herein below.  Flight testing will occur according to schedules announced by ERCOT, TTPT and the Market Flight Administrator.   
Scenarios
The TX SET workgroup has defined a number of business process scenarios.  The Test Plan is focused on exercising these scenarios; especially business processes that are frequently executed and/or may cause major problems if not performed correctly in the Marketplace.  See link to TX SET Swimlane Diagrams in Appendix C.
Scripts
The Test Scripts defined in this document detail each step of the testing process for each Scenario.  There may be several test scripts for a defined business process scenario.  For example, a number of processes may be tested for both positive and negative (reject) results.  The Test Scripts are each given a unique script name.  The test scripts can be found on the Texas Retail Testing website link noted in Appendix C.
Each Test Script is assigned a unique ESI ID number (electronic service identifier) by each of the TDSPs.  The TDSP will provide the completed Flight Test Bed Load template to ERCOT and all CRs with whom it participates in the flight testing, on the day scheduled in the Flight calendar. This information will be sent on the standard Test Bed Template, maintained by ERCOT and TTPT (See Appendix E). 
Test Days 
For End-to-End testing, each Test Script involves an exchange (request and response) of data between trading partners. Each step in the process is generally referred to as a ‘test day’.  Each test day equates to a day of simulated time and in most instances a day of actual time. However, the Market Flight Administrator may alter this timing based on progress of the Market Participants.  To accommodate these changes, it may be necessary for an MP to bypass logic that enforces rules for exchanges that require waiting longer than the scheduled number of days.  Transition from one testing day to the next will progress naturally unless the test day is a critical date for the script; in which case, the parties shall successfully complete the critical tasks prior to progressing to the next testing day. MPs are required to keep up with the market testing pace. Should a situation arise where a market participant falls behind, that Market Participant will be required to take action to catch up.  MPs that are consistently unable to keep up may be asked to leave the test.  In addition, the Market Flight Administrator may require that an MP start a script over during the flight. 
Simulated System Dates 
The Test Scripts identify two types of dates for each test day:  the actual calendar date and the simulated system date.  Only business days will be used for the simulated system dates.
All tests will use simulated dates that are at least eight (8) weeks earlier than the actual date of the testing.  This provides time for the TDSPs to condition their Test Beds (See Appendix E). 
There may be times when system clocks are held on a particular date to allow Market Participants to catch up or make fixes to their system, or times when system clocks are advanced multiple days within a short timeframe.  The Market Flight Administrator will provide direction on these actions during the daily conference calls to ensure efforts are made to keep Market Participants’ system clocks synchronized.  The Master Flight Calendar is referenced in Appendix C. 
Meter Read and Switching Cycles
The Test Scripts will test both on-cycle and off-cycle initial reads and switches.  Whether the transaction will be on-cycle or off-cycle will be explicitly identified for each script.  Sufficient accounts will be established for the required profiles and number of testing CRs.
EDI versus ERCOT Texas Market Link
CRs can certify using either TX SET EDI with ERCOT and TDSPs, or by using ERCOT’s Texas Market Link to exchange transactions with ERCOT and EDI to exchange transactions with TDSPs.  CRs shall complete EDI-based testing of Point-to-Point transactions with their trading partners.  For testing purposes, MPs are required to identify how they will test and complete testing using that approach. If a party chooses to test using EDI transactions, the party will automatically qualify for use of the ERCOT Texas Market Link.  All transactions shall be tested prior to use in production. If a party chooses to test using the ERCOT Texas Market Link, they will not automatically qualify for use of EDI transactions with ERCOT.  Parties that complete certification using the ERCOT Texas Market Link that subsequently decide to use EDI with ERCOT will need to complete the certification testing using EDI.
Transactions that are not supported by the ERCOT Texas Market Link are the Point-to-Point transactions sent directly between trading partners:
	814_PC and PD, Customer Information Changes
	810_02, TDSP Invoice
	820_02, CR Remittance
	650_01, _02, _04, _05, Service Orders/Requests, and Disconnects for Non Pay 

Testing Status Checklist 
ERCOT maintains a password-secured section on the Texas Retail Testing website to track the progress of each entity through the testing process.  Through the Checklist link on this site,  the MPs have the capability to view overall progress of the Flight Test on the Complete/Total Tasks by  MP link.  They are also expected to update their individual status with the most current information on their testing progress by selecting View Checklist by Partner or View Checklist by Custom Criteria.  
Risk Mitigation
Testing includes several steps to mitigate the risk that parties cannot maintain the pace:
·		Connectivity Testing – MPs begin End-to-End testing after connectivity protocols have been tested, assuring that connectivity is operational
•	Additional mitigation subjects are covered in the Test Days section above:
o	Weekend work as required to catch up.
o	Adjusted test schedule as indicated by Market Flight Administrator.
Issue Resolution Process
Parties shall work through problems and issues first with their trading partners.  If issues cannot be resolved in these forums, then parties are required to submit a Marketplace Issue form, found in Appendix D, to the Market Flight Administrator.  This form is used by the Market Flight Administrator to frame the issue for further clarification and mediation.
The Market Flight Administrator will hold the initial call on the issue and will report resolution to RMS or other appropriate committees.  When necessary, other parties will be engaged by the Market Flight Administrator to resolve the issue including TXSET transaction experts, TTPT members, and others.  Details regarding the parties involved in the issue will remain confidential.  
If resolution is not achieved, the issue will be escalated through appropriate ERCOT committees and to the ERCOT Board if required.  The PUCT will have the final authority on the issue.  The process is intended to resolve issues at the lowest possible level and in a fair and equitable manner for all MPs.

Data Loading

TDSPs will develop the information necessary to establish test beds (See Appendix E) of customer information to be used by CRs and ERCOT during the testing process.  The number of required ESI IDs will depend on two factors; the number of scripts to be tested and the number of CRs testing with the TDSP’s area.  Test Bed (See Appendix E) information will be available for the CRs and ERCOT approximately three weeks prior to beginning the test. 
Extra ESI IDs will be provided for any contingency testing that may be required.
Loading ESI IDs into ERCOT Systems
TDSPs shall have their ESI ID Test Bed (See Appendix E) established at ERCOT approximately two to ten days prior to testing.   The Test Bed will be loaded into ERCOT systems using the 814_20s.  Prior to beginning a new flight of testing, Test Beds will be refreshed to ensure that all ESI IDs have been properly conditioned for the next flight.
Providing ESI IDs to CRs
Each TDSP will provide unique test ESI IDs to the CRs they will be testing with approximately two to ten days prior to the first day of testing (see Appendix E).  These ESI IDs will be sent to the CR and ERCOT via email.
The ESI ID and service addresses need to be altered so the customer/premise is not identifiable.  Each ESI ID requires a unique service address / ZIP combination.
Certification
ERCOT and MPs shall establish their readiness to participate in the marketplace.  This readiness certification process consists of two steps: Pre-Flight Activities and Business Process Certification.  
Pre-Flight Activities
Pre-Flight Activities are described in the "Prior to Test" list in the Testing Requirements section.  These steps shall be successfully demonstrated to ERCOT and the Market Flight Administrator prior to the Market Participant being allowed to test.  
997 Functional Acknowledgement (FA) Transactions
Parties are required to use Functional Acknowledgement transactions to notify their trading partners that:
a)	  Transactions have been received.
b)  They were either correct (positive) or incorrect (negative) according to X12 guidelines.
Business Process Scenarios
Business scenarios have been defined by the TX SET Working Group. The scenarios below cover various business processes.  Swimlane Diagrams associated with these business processes can be found at the link in Appendix C. 
Business Process Certification
Once a Market Participant has successfully completed the Pre-Flight Activities, they are ready to begin business process testing.  Business Process Certification requires an MP to demonstrate that systems work according to the business processes defined by the PUCT Rulemakings and ERCOT Protocols via test scripts.  Any party that has completed Business Process Certification is considered “certified” to process valid market transactions once formally notified by ERCOT and the Market Flight Administrator.
Business Process testing will involve validation of End-to-End and Point-to-Point processes.   These tests will enable MPs to establish the foundation required for successful trading partnerships in production.

Market Participants may chose to certify for all processes used within the Market, or they may receive partial certification by opting out of testing those processes that are considered optional.  It is a requirement that all participants in the retail market successfully complete those tests that validate their capabilities to switch, move-in, move-out, process meter reads, send and receive invoices, make payments and provide remittance advices.  These processes are considered mandatory and will be covered by a collection of scripts defined by the Texas Test Plan Team.
The only processes that are considered optional or conditional are those listed below: 
§	Continuous Service Agreement (CSA)
§	Service Order Option 1
§	Outage Option 1
§	Affiliated Retail Electric Provider (AREP)
•	Disconnect for Non-Pay (DNP)

These processes shall be validated through testing, and certification confirmation shall be received from ERCOT and the Market Flight Administrator before an MP may use them in production.  

Testing Requirements of ERCOT and Market Participants
General Marketplace Requirements
Each MP in the Texas Market has specific requirements that shall be met before it will be allowed to begin production processing.  The ERCOT Protocols and PUCT rules specify many of these requirements in detail.  Market Participants shall thoroughly understand these requirements.
Trading partner agreements will not be required for a party to begin testing but may be required prior to moving into production. This will be determined by individual TDSPs.
Requirements specific to testing and validating Market Participant’s systems and processes are listed below. The following requirements shall be met before a Market Participant receives certification that its systems are ready to go into production with its trading partners.
Additional certification requirements that fall outside the scope of this document may be specified by the PUCT and/or ERCOT.
CR Requirements
Prior to Testing
·	Apply for PUCT REP Certification (not applicable to MOU/ECs).
•	Obtain DUNS number for testing entity.
·	Adhere to Protocol 15 – Customer Registration 
·	Adhere to Protocol 16 – Registration and Qualification of Market Participants.
·	Adhere to Protocol 23 - Texas Test Plan Team – Market Testing.
·	Implement a dedicated test system that closely resembles production. 
·	Identify whether testing with EDI or with the ERCOT Texas Market Link.
·	Complete the Testing Signoff Worksheet on-line; include specific details on required features that are not supported, test exceptions, manual processes.
•	Complete the Technical Connectivity Worksheet on-line.
·	Identify bill scenarios (MOU/ECs only).
·	Receive, review, and load test ESI IDs and associated zip codes from TDSP.
·	Review Testing FAQs (see Appendix C). 
·	   Review the TX SET EDI implementation guides. 

During Testing
·	Establish technical connectivity with ERCOT and TDSP trading partner.
·	Participate in scheduled testing conference calls with the Market Flight Administrator, ERCOT and MPs.  
·	Adhere to the established test schedule.
·	Notify trading partners via E-mail when transactions are sent and received.
·	Update status on the testing checklist. 
After Testing
·	Receive Certification letter from ERCOT.
•	Continue to work with the PUCT, TDSPs, and ERCOT Retail Client Services to complete any additional requirements prior to going into production.

TDSP Requirements
TDSP Requirements
Before Testing
·	Adhere to Protocol 15 – Customer Registration. 
·	Adhere to Protocol 16 – Registration and Qualification of Market Participants.
·	Adhere to Protocol 23 - Texas Test Plan Team – Market Testing.
·	Implement a dedicated test system that closely resembles production.
·	Complete and submit the Testing Signoff Worksheet on-line; include specific details on required features that are not supported, test exceptions, manual processes.
•	Complete the Technical Connectivity Worksheet on-line.•	
·	Establish Test Bed of ESI IDs and zip codes; include enough ESI IDs to cover all required scripts for each of the CRs (See Appendix E).
·	Provide ERCOT and CRs with all required Test Bed data. 
·	Review Testing FAQs prior to testing (see Appendix C).
·	Review the TX SET EDI implementation guides. 

During Testing
·	Establish technical connectivity with ERCOT and CR trading partners.
·	Participate in scheduled testing conference calls with the Market Flight Administrator, ERCOT and MPs. 
·	Adhere to the established test schedule.
·	Notify trading partners via E-mail when you send and receive test transactions.
·	Update status on the testing checklist.
After Testing
·	Receive Certification letter from ERCOT.
•	Continue to work with the PUCT, MPs, and ERCOT Retail Client Services to complete any additional requirements prior to going into production.

ERCOT Requirements
Before Testing
·	Adhere to Protocol 15 – Customer Registration. 
·	Adhere to Protocol 16 – Registration and Qualification of Market Participants.
·	Adhere to Protocol 23 - Texas Test Plan Team – Market Testing.
·	Establish a dedicated test system that closely resembles production.
·	Update the on-line Technical Connectivity Worksheet. 
·	Review Testing FAQs prior to testing (see Appendix C).
·	Review the TX SET EDI implementation guides. 
·	Establish technical connectivity with all testing Market Participants.
During Testing
·	Participate in scheduled testing conference calls with the Market Flight Administrator and MPs.  
·	Adhere to the established test schedule.
·	Notify the MPs via e-mail when you send and receive transactions.
·	Update status on the testing checklist.
After Testing
·	Develop Flight Test Lessons Learned.
Distribute certification letters.
•	Assist MPs with production migration.

PUCT Requirements
Before Testing
·	Confirm REP Certification application has been submitted (excludes MOU/ECs).
·	Inform associated MPs of new MPs entry into marketplace.
·	Identify AREPs for specific TDSP areas.
·	Participate in the development of the Flight calendar to ensure key market dates are met.
·	Provide relevant information to new entrants to the market.
During Testing
·	  Monitor progress of the testing process.
After Testing
·	Review certification notification from ERCOT and the Market Flight Administrator on each Market Participant.
·	Award appropriate certification to the MP.

Market Flight Administrator Requirements
The Market Flight Administrator will act as a neutral facilitator throughout the testing effort.  Primary duties for the Market Flight Administrator will be to:
·	Maintain a testing contact list on the Texas Retail Testing Website.
·	Verify testing eligibility of MPs with ERCOT.
·	Ensure TSW and TCW are completed on-line by all testing MPs.
·	Develop a consolidated list of testing FAQs and post on the Texas Retail Testing Website.
·	Attend TTPT meetings.
·	Review and provide input to TTPT agenda prior to meetings.
·	Assist in facilitation of TTPT meetings.
·	   Assist TTPT in developing a standard Test Plan for Point-to-Point and End-to-End business processes.
·	Assist TTPT in developing Test Scripts.
·	Facilitate End-to-End testing between ERCOT and MPs and Point-to-Point business processes between trading partners.
·	Facilitate daily testing conference calls with MPs and ERCOT and document results.
·	Act as an issue resolution agent for technical and process issues between all MPs including ERCOT.
·	Confirm that MPs have completed certification testing.
·	Verify adherence to TX SET standards by all MPs and ERCOT.
·	Maintain MP testing status on a password-protected section of the Texas Retail Testing website.

Details of Testing Phases
As indicated in the Testing Overview/Certification Section, certification testing consists of two steps: 1) pre-flight activities; 2) and business process certification, which is accomplished through End-to-End and Point-to-Point testing
Technical Connectivity and Verification
Each MP is required to establish technical connectivity with their trading partners using the NAESB EDM communication protocol. All parties shall review the NAESB EDM document (see Appendix B).  
Establishing technical connectivity is a time-consuming process.  New entrants to market testing shall begin the effort as soon as possible.  Technical connectivity shall be completed before any End-to-End Business Process testing can begin. 
Technical connectivity begins with prospective trading partners completing on-line Technical Connectivity Worksheets.  TDSPs will schedule a date to begin connectivity testing with each new trading partner. 
Technical connectivity shall be established for each unique DUNS number that is used by a Market Participant.  
NAESB EDM Testing
Transactions exchanged point-to-point and to/from ERCOT will be sent via NAESB EDM.  The Texas Data Transport Workgroup prepared a plan for the use of NAESB EDM.  A link to this document is provided in Appendix C.  Participants shall review this information.  The primary goals of this test are to establish connectivity and to exercise the exchange failure process.
Scripts for establishing NAESB EDM connectivity can be found on the Texas Retail Testing website. Testing parties will use a standard X12 formatted file in this test.  TDSPs CRs, and ERCOT are required to establish and use PGP or GPG keys to send encrypted EDI transactions to their respective trading partners.  This information is exchanged between trading partners using the Technical Connectivity Worksheets.
TX SET Verification
Transactions that are not compliant with X12 standards will be rejected using an FA/997 transaction in test and production.  Transactions that are not compliant with TX SET standards will be rejected using the defined transaction based on TX SET standards and ERCOT Protocols.
TX SET standards require that parties receiving a transaction send an FA/997 upon receipt of each transaction.  If the transaction is compliant with X12 standards, a positive FA/997 is sent.  If the transaction is not compliant with X12 standards, a negative FA/997 is sent rejecting the transaction. 
FA/997’s from ERCOT will be placed in each MPs mailbox. 
End-to-End Testing
End-to-End testing will be conducted between ERCOT, TDSPs, and their respective CRs. End-to-End testing includes processes that involve all three parties as well as those point-to-point processes that affect only the TDSP and CR.  This testing is designed to validate the End-to-End and Point-to-Point business scenarios outlined in the test scripts. This includes CRs using either EDI or ERCOT Texas Market Link.  End-to-End testing will utilize MP and ERCOT back-end systems to process through entire business scenarios.  End-to-End test scripts can be found on the Texas Retail Testing website (for the link, see Appendix C).  
Each participant will be responsible for fulfilling its role within each test script by either sending or receiving the specified transactions.  Participants are dependent on each other to correctly send and receive transactions in a timely manner as detailed in the testing schedule to realize successful completion.
Point-to-Point Testing
Point-to-Point testing is performed between TDSPs and their CR trading partners. It covers the processes in the marketplace that do not involve ERCOT.  Those processes include customer information updates, invoice and remittance, and service orders.  Point-to-Point testing is included within the End-to-End scripts covered above and can be found on the Texas Retail Testing website.  For the link, see Appendix C.

Appendices
Appendix A - Testing Signoff Worksheet
The Testing Signoff Worksheet can be found online at: 
<http://etod.ercot.com/Signoff.asp?Method=TOC>
Appendix B – Technical Connectivity Worksheet
The Technical Connectivity Worksheet can be found online at: 
< http://etod.ercot.com/TechCon.asp?Method=TOC>
Appendix C - Resources

Market Participants:
http://www.ercot.com/participants/mparticipants/mparticipants.htm

The Texas Testing Retail website can be found online at:
http://etod.ercot.com/

The TSW link can be found online at:
http://etod.ercot.com/Signoff.asp?Method=TOC

The TCW link can be found online at:
http://etod.ercot.com/Signoff.asp?Method=TOC

The TX SET Transaction Names and Swimlane Diagrams can be found online at:
http://www.ercot.com/Tac/RetailISOAdHocCommittee/Customer/SET/keydocs/SwimlanesVersion2.0.htm

TX SET Implementation Guidelines can be found at:
http://www.ercot.com/Tac/RetailISOAdHocCommittee/Customer/SET/keydocs/Keydocs.htm

Protocols can be found at:
http://www.ercot.com/AboutERCOT/PublicDisclosure/ProtocolRev.htm

The NAESB EDM Plan can be found online at: 
<http://www.ercot.com/Participants/Committees/TDTWG_comm.htm>

ERCOT Registration information can be found at: 
<http://www.ercot.com/mkt_part_apps/market_participant_application_f.htm>

The Master Flight Calendar can be found online at::
<http://etod.ercot.com/MasterCalendar.xls>

The FAQ spreadsheet provides questions and answers relating to Retail Testing and it can be found online at::
http://etod.ercot.com/Issue.asp?Method=TOC

Appendix D - Marketplace Issue Resolution Form
Issue Title:	
Date Identified:		
Date Submitted:	Submitted by:	
Parties Affected by Issue:	

Position 1	
Parties supporting Position 1: 	
Position 1 Summary of Supporting Logic (include any legislation, standards, etc.)   	
Position 1 Recommendations:   	
Position 2	
Parties supporting Position 2: 	
Position 2 Summary of Supporting Logic (include any legislation, standards, etc.)   	
Position 2 Recommendations:   	
Market Flight Administrator Position	
Comments:   	
Recommendation:   	
Status:	

Appendix E - Flight Test Bed Load form 
The Flight Test Bed Load form can be found online at: 
http://etod.ercot.com/FileCabinet.asp
Appendix F - Testing Requirements Matrix 
The Testing Requirements Matrix can be found online at: 
<http://etod.ercot.com/FileCabinet.asp>

Appendix G – Glossary of Terms and Acronyms Used in this Document not defined in Section 2 of the ERCOT Protocols
•	Additional DUNS by Certified REP – determined by a Market Participant who is certified in the Texas Marketplace with the current TX SET version; involves adding a new trade name and DUNS Number for a Market Participant in a specific service territory.

•	ANSI X12 - The American National Standards Institute X12 standard relates to shared ways of defining formats and procedures for exchanging documents.

•	Contingency EDI Provider – used for testing purposes by a Market Participant who is certified in the Texas Marketplace with the current TX SET version; cannot be used for testing during a TX SET Version Release.

•	EDI Provider - used for testing purposes by a Market Participant who is certified in the Texas Marketplace with the current TX SET version.

•	Established Service Provider - an organization or company that provides both connectivity and translation services to another Market Participant in the same service territory and that has successfully tested in the Marketplace provided they tested using the current TX SET version.

•	Market Interface Service Provider - refers to a Market Participant’s internal organization or an outsourced company that provides both connectivity and translation services for an MP.

•	NAESB EDM – North American Energy Standards Board Electronic Delivery Mechanism

•	Non-Established Market Interface Service Provider - refers to a Market Participant’s internal organization or an outsourced company that provides both connectivity and translation services for an MP that has not successfully completed certification testing for another Market Participant in the service territory in question.

•	Non-Established Service Provider – a Market Interface Service Provider that has not successfully completed certification testing for another Market Participant in the service territory in question.

•	Specified Ad Hoc Testing – refers to “emergency” testing to institute a particular change to a Market Participant’s systems or processes; these changes cannot impose undue risk to the Market.

•	TP – Trading Partner
Appendix H – Future Flights Schedule 
The schedule for Future Test Flights can be found online at: 
<http://etod.ercot.com>