3.5   Software Sensitivity Tests

This section details the eligibility requirements for an applicant simulation program for use as compliance software under the 2013 Title 24 ACM. A series of quantitative tests called the Software Sensitivity Tests shall be performed to measure the change in energy consumption when changing specified input parameters. Applicant software results will be compared against predetermined Reference results to demonstrate that the applicant software is acceptable for use in code compliance. There are a total of 96 tests. All the test cases described here shall be performed and results summarized in the forms contained in Appendix 3B.

3.5.1.1 Overview

The applicant software shall perform a suite of Software Sensitivity Tests to demonstrate that their performance is acceptable for code compliance.  The applicant software test results shall be compared against a base case called the Reference test case. The Reference Test case is the corresponding match of a particular test case simulated already on EnergyPlus (v.8.1) engine. The Reference test case results, as determined by the Commission, are tabulated in Appendix 3B.

Test cases specific for Simplified geometry are meant for software with 2D inputs for building geometry. Software with 2D geometry approach shall seek certification by submitting the Simplified geometry test cases. In addition, they are also required to produce results for HVAC tests which will be compared against the HAVC reference test results which are performed on detailed geometry.

The test cases will assess the applicant software’s sensitivity to various inputs ranging from envelope thermal conductance to HVAC system performance. Each case tests the impact of the input component on building end use energy and annual TDV. The following six building components will be tested through a series of tests:

•  Opaque envelope

•  Glazing

•  Lighting

•  Daylighting

•  Receptacle loads

•  HVAC System Parameters

3.5.1.2 Prototype Models                                                         

The Software Sensitivity Tests are performed on four Prototype Models which are a subset of the DOE Prototype Building Models developed by PNNL for analysis of ASHRAE Standard 90.1. The Prototype Models are EnergyPlus model input files of the DOE prototype building models, modified to comply with the requirements of Title 24-2013. The Prototype Models will be the reference baseline runs for the test cases. The applicant software shall replicate the four building models using the same inputs as the Prototype Models. The models so replicated will be the applicant baseline models for the test cases.

A summary of the Prototype Models is provided in Appendix 3A Detailed EnergyPlus input files of the Reference baseline models are available from the California Energy Commission’s Building Energy Efficiency Software Consortium webpage http://bees.archenergy.com/

Prototype Models used for Software Sensitivity test cases are:

•  Small Office (02000CZ-OffSml):

The Small Office model is a single story rectangular shaped building of 5500 square feet area. It has punched windows and a hipped roof with Attic space. There are five zones, each served by packaged single zone air conditioner units. For more details refer Appendix 3A This ptoptotype is used for the Simple Geometry test cases only.

•  Medium Office Building (0300CZ-OffMed):

The Medium Office Building model is a three storied rectangular building with an overall area of 53600 square feet. It has a window to wall ratio of 33% with fenestration distributed evenly across all four facades. The zones are served by DX cooling and Gas furnace heating with hot water reheat. For more details refer Appendix 3A. This prototype is used for both Detailed geometry and Simple Geometry test cases.

•  Large Office Building (0400CZ-OffLrg):

The large office building has twelve floors and a basement floor with glass windows with a window to wall ratio of 40% on the above-grade walls. The total area of the building is 498,600 square feet.  The HVAC system type used VAV system. For more details refer Appendix 3A.

•  Stand-Alone Retail  (0500CZ-RetlMed):

The stand-alone retail is a single story rectangular building measuring 178 ft by 139 ft. The total area is 24695 square feet. Windows are located only on the street facing façade and occupy 25.4% of that facade. The building is divided into five thermal zones which are served by Packaged Single Zone systems as described in Appendix 3A. This prototype is used for both Detailed geometry and Simple Geometry test cases.

•  Strip Mall Building Strip Mall-PSZ System (1000CZ-RetlStrp):

The strip mall building area of 22,500 square feet. It has ten zones each with rooftop units. The building has windows in the street facing façade and has an overall window to wall ratio of 10.5%. For more details refer to Appendix 3A.

3.5.1.3 Climate Zones

The Software Sensitivity Test cases use building models for five of the 16 California climate zones. Most tests are performed with two or three climate zones to capture the sensitivity of the input characteristics to extremes in weather conditions. The test cases are performed on climate zones 6, 15 and 16, which represent mild, hot and cold climates respectively. Daylighting tests are performed on climate zone 7, which has a high annual daylighting potential and climate zone 1, representative of the climate with least annual daylighting potential.

Table 2-Climate Zones Tested

3.5.1.4 Labeling Test Runs

Each test case in the Software Sensitivity test is labeled uniquely to make it easier to keep track of the runs and to facilitate analysis. The following scheme is used:

XXYYZZ-Prototype-RunDescription

Where,  XX denotes the Prototype Number

YY denotes Test Run Number

ZZ denote Climate zone and

Example:

05 01 15-RetlMed-EnvelopeRoofInsulation                Run Description

                                                                                                       Prototype

                                                                                                       Climate Zone

                                                                                                       Test Run Number

       Prototype Number

Example for Simple Geometry Test case

0211015-OffSml-SG-EnvRoofInsulation                                 Run Description

                                                                                                                Prototype

                                                                                                                Climate Zone

                                                                                                                Test Run Number

                                                                                                                Prototype Number

3.5.1.5 Test Criteria

Applicant software vendors shall perform a series of computer runs. Each of these runs shall be a systematic variation of the applicant base case model as described in 3.5.1.7. The applicant test case results will be compared to the reference results to verify that applicant software meets the requirements of the ACM. Simulation results for each test case will be compiled in forms provided in Appendix 3B. Compiled results will include annual site energy consumption for each end-use, overall site energy consumption, total unmet load hours and annual TDV and % variation of annual TDV and total end use site energy. 

The annual TDV % variation shall be calculated using the formula:

TDV_%= (TDV_b – TDV_n)/TDV_b

Where, TDV_% is the TDV % variation,

TDV_n is the annual TDV for test case number n and

TDV_b is the annual TDV for the base case run.

In order to be accepted, the applicant software should fulfill the passing criteria as determined by the Commission:

When vendor software integrates the CBECC-Com Application Programming Interface (API)

 The change in energy for each test case must be in the same direction as the Reference Method test case result, and must be equal to the Reference Method test case percentage change in TDV energy, plus or minus 0.5% of baseline TDV energy. 

When vendor software does not integrate the CBECC-Com API

Applicant software should fulfill the passing criteria for both the baseline TDV energy comparison and the TDV energy variation tests.

Baseline TDV energy comparison tests:  For each test case, baseline TDV energy results from Applicant Model must be within ±15% of the baseline TDV energy of Reference Method results.

TDV energy comparison tests: For each test case, the change in energy must be in the same direction as the Reference Method test case result, and must be equal to the Reference Method test case percentage change in TDV energy, plus or minus 1% of baseline TDV energy.     

Any vendor software meeting these requirements and certified as compliance software under this criteria (not integrating CBECC-Com API) is valid for demonstrating compliance through December 31, 2014. After this date vendor software must integrate CBECC-Com Application Programming Interface (API) and resubmit for certification to the Energy Commission.

3.5.1.6 Reporting Test Results

For each test case, the TDV energy use of the modeled building is reported (kBtu/ft²), along with the TDV energy use attributed to the major fuel types (electricity, gas), site energy use, and energy end use intensity for the regulated end uses (cooling, heating, lighting, etc.).  The following energy totals are reported:

1.   Annual TDV EUI (kBtu/ft²)

2.   Annual Site EUI – Electricity (kWh/ft²) 

3.   Annual SiteEUI – Natural Gas (therm/ft²)

4.   Annual Total End Use Site Energy EUI – kBtu/ft²

Site Energy End Uses

5.   Site Energy: Heating (kBtu/ft²)

6.   Site Energy: Cooling (kBtu/ft²)

7.   Site Energy: Interior Lighting (kBtu/ft²)

8.   Site Energy: Interior Equipment (kBtu/ft²)

9.   Site Energy: Fans (kBtu/ft²) (Airside Fans, does not include tower fans)

10.  Site Energy: Pumps (kBtu/ft²)

11.  Site Energy: Towers (kBtu/ft²)Water heating (kBtu/ft²)

12.  TDV % Variation – this field is used for the compliance test

13.  Total End Use Site Energy % - percentage change in site energy use

14.  Pass/Fail – test fails if it does not meet passing criteria

15.  Unmet load hours – these are defined as the zone with the most unmet load hours

a.  Reference Model Occupied Unmet Load Hours

b.  Applicant Model Occupied Unmet Load Hours

c.  Reference Model Number of Zones with excess unmet load hours (>150)

d.  Applicant Model Number of Zones with excess unmet load hours (>150)

Figure 3 – Results Spreadsheet Excerpt from Appendix 3B

The results spreadsheet provides the results of the Reference Method for each test, and provides a column (in orange) for the vendor to report the results from their candidate compliance software.

The variation from baseline section of the spreadsheet shows the percent change in TDV energy use (kBtu/ft²) from the base case for testing.  The percentage must be within the passing criteria for the candidate software to pass this test.

Also reported is the number of unmet load hours during occupied hours for the building. An unmet load hour for a specific zone in Title 24 compliance is defined as any hour when the zone has an unmet cooling or heating load.  This is typically reported by the software for each zone in the building. For the test case results, two unmet load hour metrics must be reported: the number of unmet load hours for the zone with the greatest number of UMLH, and the number of zones that fail the ACM Reference Manual criteria for acceptable unmet load hours (any zones with greater than 150 hours fail the criteria).

The spreadsheet where the results are documented indicates whether the candidate software passes or fails a test. The result in column AL of the spreadsheet indicates whether or not the candidate software passes the test.

3.5.1.7 Software Sensitivity Test Cases

Test cases assess the energy impact of one or more of the building or system input characteristics on the baseline model. Each test suite consists of a series of unique test cases aimed to test the impact of a specific characteristic on building energy performance. Simulations are grouped according to test criteria and sub-grouped based on the reference model type to allow for direct comparison of results.  For each test case, the applicant software will modify the applicant baseline model with specific inputs as described in the Test Case description section.

The test cases are simulated on multiple California weather files to evaluate the sensitivity of the building or system input to extremes in climate. Results of the test case runs and the TDV percent variation over the baseline run shall be compiled and compared against the reference results.

Detailed descriptions of the Standard Design models are provided in Appendix 3A. CBECC input files for all baseline and test case models are available from the California Energy Commission’s, Building Energy Efficiency Software Consortium webpage http://bees.archenergy.com.  Details on each Test Description can be found in Appendix 3A under the Test Criteria tab.

Table 3 – Baseline Runs for Test Cases Test Case Summary

3.5.1.8 Results Documentation

The applicant shall perform simulations for all tests specified above. Detailed description of each test case is provided in Appendix 3A and report results in the forms provided in Appendix 3B. Some of the prototype models have some variants of the baseline model. These include 1) Stand Alone Duct Loss baseline- a variant of the Stand-Alone Retail model 2) StripMall-PTAC model- a variant of StripMall- PSZ model and 3) StripMall- Fan Coil model- a variant of StripMall PSZ model. For details please refer Appendix 3A.

A test case is presented here as an example: one for building envelope, one for lighting and daylighting, and one for HVAC. The development of the other required test cases follows the same process.

Example Test Case: 19CZ16MediumOfficeGlazingWindowUSHGC

For this test case the U-factor and SHGC of all vertical fenestration is decreased by 20%. The prototype used for this test case is a medium office building.

Before the test cases are run, the first step is to generate the prototype models for the four reference buildings, which are required for all of the tests.  The four prototype models are defined in the PrototypeModel spreadsheet of Appendix 3A. (Note: while many of the prototype model inputs are based on Title 24 prescriptive requirements, the prototype models do not exactly conform to minimum Title 24 requirements, but rather, are intended to test the sensitivity of the candidate software simulation results to common variations in building inputs

Step 1. Generate prototype models. The first step is to generate the prototype building for the medium office building.  The detailed specification of the medium office building is 'listed in Appendix 3A.  A portion of the inputs are shown in the Figure below.  The prototypes are defined for the reference models on the Prototype Model tab of Appendix 3A. 

Figure 4 – Prototype Model Definition from Appendix 3A

The prototype model definition in the spreadsheet contains links to other input definitions:

Row 19, 26,45: Links to layer-by-layer exterior construction assembly definitions in the ConstructionAssembly tab

Row 52: links to layer-by-layer interior construction assembly definitions in the ConstructionAssembly tab

Row 85, 105, 107, 109, 112: links to fan and equipment hourly schedules in the Schedules tab

Row 112, 116, 119: links to space use interior load schedules in the Schedules tab

Step 2. Define base case and variation for test run.

The base case is defined as the starting point for each test. In many tests, the base case will be one of the prototype models. However, in some cases, a variation of the prototype may serve as the base case for the test.

For this test, the base case is found by looking at the Test Criteria spreadsheet:

Figure 5 – Base Case Definition from Appendix 3A

For this test, the Baseline field in row 3 of the TestCriteria tab shows that the baseline is CZ06MediumOffice, the medium office prototype in climate zone 6.

This same TestCriteria tab shows the input(s) to be verified, which are highlighted in purple. For this test, the solar heat gain coefficient of all vertical fenestration is reduced by 20%, from 0.25 to 0.20.

Figure 6 – Input Parameter Variation for Medium Office from Appendix 3A

Step 3. Run the base case model and generate test results.

Once the base case model is developed, the simulation is run and the results are recorded onto the spreadsheet of test cases, Appendix 3B.

The candidate software shall report electricity use by end use, gas use by end use, TDV energy and unmet load hours. For purposes of compliance, unmet load hours are defined at the zone level, and the zone with the greatest number of unmet load hours must pass the criteria specified in the sizing procedure.

For the Reference Tests, the HVAC system’s capacities and flow rates can be found in Appendix 3A under the Sizing Values tab.:

Step 4. Run the test case model (with the reduced solar heat gain coefficient) and report the results.

The model is re-run and the energy results and outputs are reported. The percentage change in energy use is reported.

Step 5. Report the change in regulated TDV energy use from the base case as a percentage change.

The reported percentage change in energy use from the candidate compliance software must fall within the passing criteria for the Reference Method.