Recruit Sufficient ,Sample to .Aclirwe ?1us1F* Goals Sample an re&n to the number of participants being evaluated in a sdd^% and it is a cudc ai factor in detarmung how pcieeise a sWs fiad'mgs can be while remaining shtistit ally siguv6eaut Iiaane%mr, sample size is also a key cost driver Utilities xinist weigh the beor.fits of increased precision against the COSIS of MMRMMM# and i1111plelleubbOR to detamiae the appropriate balance . Although sample size cakulabous, we mpLe to the type of statistical testing being per£iume+d, the €alCUlalim will Uely include the Mowing a n a • a- =apIE sia:lepsramis the marober ofpaticqwts wfian each groUp' being ftsird A a E-mqnsent the amount of climige that most be Observed and the tolerable erW (+ivalue) seslotrMy; these values may be adjusted in accordance with the program's ynk a aadP - represent the probability of ea+aor there is some flexibility in sssigoiq dose values ytiklik certain generally wcqAed boundaries a- standard devistiok in The planmiog shqes, Ibis is usually an esfimate and is not within the eapeiimentefs control to 3naaipula6e (Olt & Laagnealer 2010) AltLoirgh it is desirable to have a low A at E value and low a and P values, lowering these x4boes increases n. Tbudcffe, The program team must eucfuIl^r assess the goals of The evakwhosi as well as the feasibility and cost of increased sample to dekn3me the appropriate balanceOver-Recruit in Axceunt for Dropouts Owe the sample size goals are estikrshe1 it is important tint program, maWers remember that not everyone recruited in the beginniog of the program will and up providing useful data at The paog[am's conclusion Gbs6ouners, may enuofl but not follow through with mstlllabam, name before the evaluation period is comcbuie& or iotso3xe other vanables that cannot be controlled for For these reasms., it is important that programs recruit more parbckxmh than required for the final aaalysis to ensure that safficimt data is available at The stody's conclusion Appropriate rates of ov cr-raauitmeat vary, but faetars to consider include_ Demands an the castomner- Iiow eW is it to perticipabr^ Leigh of the evahiatiun period - How zwor}; dapsfmo^s^years of daft do are needed? Participant chimicleristics - Who is being evaluated? Homeowners? Renters? Commercial property oavaers? Each pogulatim will have unique cluractrristics impec1iag the Iilwhlnod+of Mm4lroughP6a for an Assessment Peaioil Sufficient for Evaluation of Long-Term Impacts Allowing tam to assess lomg-tra impacts is especially critical for behai,i;aor-bosed savings when persistence cannot be assumed Savings observed within the fast ym of parhcqwban my be inflated based an short-term engagement at they may mark the beginning of 1omg-las6ag energy savings habats_ What Iamg-Eerrd means for any given Program will depend an the zhlity's goals, but it is iaI t^t to remember that behavior-lased UvmP Cannot be assumed to edend for a period lmpw than the period during which they weffe evaluated. CW12/IGEEE Summer Shady ao EneFgy E1icierwy in BuldiMgs 228T 100 Combine Various Data Soorces to Improve Understanding of Impacts I:nsMs conamWoon dab is, of emu, The most essential component m enerU savings imqacts- Hoavever, addiaand dab can maRY expami the opportunities for analyxis_ WeaAha data can be used to normalize eangy' consumption- fsrilifaf3ng caaapariam between groups- Wenther nonuslindm however, may not be necesauy if fim is a randomly asa*aed cautra1 groupDeft from o&w utility progiaans can be used to klenhfy energy savings not athiluemble to the progaan's fteatmeni(s)Pa^icapant feedback can provide msighi into outside fs^o^s that may impact enein cousunuphon, as well an a qmlit&vr assessment of the ausbomw expuience, ?lEP Texas SM.^LR.T View Pi-ujW; A Case Study To illustrate what these best practices look like when appiied- a hqot4eM overview of the AEP Texas SWRT View Project research methodology is provided belcrw. This research plan illustrates the implementation of The best practices deacn-bed above as well as provikWig an oppatuuzt^ to discuss alternative appmoarhes and oppohunties for unprovment of 4Le plant Inaernal Device POK In preparation for launch of the fsdl. sdrdy_ a pilot study was camdocted with Pia Team employees to. Identify= devices that are the best fit for the study Assess the level of support that will be reqdmed for lauach of the project Idmafy c+oaoID^m *xU&al issaes, so that ^ pmventative measures can be aipleummied project sappm# slaff is prepared to Made common s • Participants m the internal pilot were re=Jed through an email to local AEP Tem officrL Volunteers received a fee device as well as instructions for pffovisioning dw device (Le,, establishing a coaner.tian between the device and The me6er)- PMcipaat support was nianaged through a dedicated email account, and all issues were logged m a simple backing syslrmSevrenl weeks a&r Pronsioning their &-me= pilot Porticipats. completed a short survey about Ibdr expaieuce installing and interacting with the deviceAs a result of the pilot find- , the project vom made several key adjustments to the research pLm, which will facilitate a saaoodta rollout to paticrpants- The changes mduded: Device Selechm: The pilot revealed that one of the eligible devices had too mony bamen to iastallatiom and too little visidality to occupants once msbdle& Far these reasons, Or project vom eleded. not to include it in the fuIl shulg- Q20i2 ACEEE Sunwner Shm* on EneFW Elicieecy in 6uldW 2312 101 Pre-delivery veiifiCation: AEP Texas idesifified two pm-ddivey verification skah*s that significaldy reduced the pacerdpc of participants who encountered tecloncal issues duinig the device, setup pw%L Staggered Uelivray Schedule: Based an support needs encoambered daring the pulot, the project teem detesommd. plissaug the delivery schedule over a period of 4-6 weeks would keep the support needs of participants at a manageable level Tbus internal pilot may be considered a type of soft launch. However, 8re rerroitouat Phase also begam with a so81nmcL (see be6ow). Samplmg PI= and Screening hftmdiodebp. Because 1I1Iy programs require an opt-in recrudhmat melhodology, tbe pncWt team dc6eammed dot the population for tLe¢ study would be likely particqmmb in an AEP Tens IED program Knowing that AEP Texas caukd not rJann savings for an 1I1L1 tmiless a program parbcpW had chosen. to receive one from the ati[ity, the hum knew that there was no value in atteoipt,mg to apply the sesuhs mom geaecaUyPa:iheqmmb will be recanted through an -1 survey to c4zfim that they nod The pwject's screening czibeii Because AEP does not lave emad addresses. targeted postcards with a fink to the aunvy will be mailed to randomly selected siagk-faorily residemlial homeowners with advanced me6em To encourage a high response nCe, survey invitations will promote availability of a saeepshloes incentive (e.g-, respond to be entered to win one of tea $100 VISA gift ^) Qudtfymg survey perbeipants will be grim an opportunity to indicate &w willingness to participate in the AEP Texas SMART View Project', i3eezuae knaqvlydpe of the suivey's purpose could influence the responses provided by respondents, 6e, oRniftaidy to p" new 1erbnology will not be mmhamed until eligibility has been confirmed ln addition to allowing the project to pre-screen pabcTonts, recruiting participants through a survey facilitates the recruit-amd-dmy methodology for establishing a control group. Because qualified participants will indicaDe fim willingness to participate rather than actually applying to be a parbeipesd, the project beam will be able to confirm that members of the caotool group aresimlar ID members of the treatment group in every respect other than the treaho®t. Itw-rokment Soft Launch Because this study relied an a recruitment mNhod onu before used by the oftlity it was essential to confim the response rates and qualifying rates estimated in. the research PILL The cost and reach of fie postcard recruitment effort made the soft launch especially critical; with a plm to send over 100,000 postcai& it was very unpiaQaant that the team knew what ID expect and make no mistakes- The postcard outreach was soft launched with 5°a of the total anticipated mvitaaane to use= actual response rase and qualifying rate, as well as test =uIple icer^t messagesTo ensm the highest possible response rme, the team developed two unique reannunmt postcard templates to test dining the so8lnmch Both templates included essential dernmis such as the survey ud the sweepstakes offer, and the emhq conducting the research. One template was designed to use be least possilile copy wide The otler was fo®atled as a leflu and included adddmmat elcoments often listed as best prachm for survey reawtmesit, including a privacy 4=2 ACEEE Sunrner Slody an IEmtHy Etisieacy in 8uitlinps 2-269 102 disclaimer and an explanation of the xseaTch pm.pose. Each tesoplaie included a uniqum idmti6uer in The survey on to back resu[4sInd^mgly- tue team found that the more formal invitation was less effCtive in ma>sting Lomeammis; the response rAe was one half of The response rate observed wfih The more concise mvi,iaton- The learn also evaluated whether qualifying rain vaned based on the postcard zzceived, but no sinifuea^t difference was obsaved As a resuit, the team sdopted the higher response aate template as the basis for the final template AMamigh the qualifying rate of survey respondents was qmtdeae to the eshusled value of 35%., the response rule for the postcards was signu6canfly loner flan projected MAN. as opposed to 1.5%). As a resal>', Or project team made several modifications to The posicard to, Memsr response IRL- including n n n n addition ofan expected time&amue, for response to mobvate action adddwn of the uitiWs logo to mcmw perceived credibility ^ sin of the survey url to draw atlealion. and improve reodability use of colored cudsbo& (ahiie was used in the initial soft lausch) to draw attention The early discovery of the low response rate also allowed the tow. to prqme an alternative research design in case the sample size required for the anginal design could not be aclurued. To e9nme that suffnent sample could be obbuned. tlamo this zeenaluseut nuedadolog,y, a second soft launch was conducted before the postcard recaudmeat was folly launched (resolts are pmdiag)Treml®ent Groups Became the project tusm was iaarirsted in detamnoing if the impect of HMs might be augmented by The diahNbntioa of mqqdemeoW edacafiosat materials, the project uses a matrix shucds+e to Wiese the tealaumt and control groups. FFacch fteameot is being tested in isolation as well as in combination with other tteammLs, so 69 when the project twmeludrs, the ham wilt be able to assess the impact af esch individually. Ta61c 2. .AU Texas SAURT Project Treatment cawocatis -- -,-I Device Type ltesot Not PHes®t ^evioeli Al A2 Deaioe B BI B2 Nb ^ics 01 W Pwbcqmb who meet the srneming criteria will be randamly assigned to one of lhe sus. poaps. The number initially recrmted into each group will be 20'% `. higher than tire targeted sample size to, account for participants who do not successfully set up &m 11M or most be removed fain the assessment for odw reasons. c Ple9e main fLti this is a poftmmy de9iBa md is MWt m^o®e Is sdswlababq nfdq*lEpwtiripmtr.. C012 1UCEEE Sunmer SWdy an EneFW Efi¢iermy in Buildops on the ^Ns of an YecaEftmeoa effim 2-M 103 Sample size Like many ptopems (espeeiallg Fogrants with opt-in reenutmmO, the AEP Texas SMART View Project had limitations that r+equimed the tum to make campaomsiaes regarding necessary sample sim, Mi^ sample size needs fur the hypadiew test were catwlsted based an The followint Sninuta for a two-sided lest oanwiag means of iadepmdicoR s®ptes_ ra=^a (Zg^`2 + ZP)W A2 (Oft & Ianpecker 2010) The a and P values were set at 0-05 and 0-1 respeetivety-The o was estimaled based on past energy c.aosumption data flour the repon-2 The estimation of tl was based on the savinp that would be required to Justify the cost +oAf 1he devices to the utility and the range of sarinp that could be reasamd* amliopa6e,d based on previous resmwh- Due to flit relatively high Situ raao, the catt11taled. A was qY&E ldWBecause the CitC'Otied niumimin sample size was low, a second calculation was used to deinoustrahe the value of increasing the ssimple Sls^+2 beyond the inumminn required for the hypodiesis test This calculation was based an tlle confidence minvat for comparing means of ^ samples- E 1 R 1 -j- - R CUtk ^ Laogµecker 21i113} The some a and o values were used m this calcoluhaa T.hm K was nonipubded upwards to demonstrate how increased samPle size allows far a reduced aamficlmoe admvaL We*bW the results of this calculation and the interest m poEC1Si0R (aSkdwAkd by a re&iced canfidencE mkua^at^ apinst the compliaftms of recumbag participants and &e need to control costs, the project fem dehsinined that a sanple size of 100 participants per cell (pAus 20% to account far dnogouts) was required and that, when appropaiabe, cells would be oonikuned during analyns, to provide increased precision for comparisons. between larger participant graPSAssessinen.t Period Due to a hard deadline at the end of 2013. the AEP Texas SMART View Project will indudr an assessment period of 14 to 16 monfLs. This Van of tine, will allow the project team to assess impacts beyond tLc first year It will also indUde two suamris, which is a critical pmud for madpnti demand m Texas- However. if savings are abmenu+ed and c4mdiuur into the final mo>fis of the assessuient penud, The team will net have sufficient data to delmmiar the fill peimlmae of 8tie impacts- (If this ocacmm the goject team may recommend a fatlow-up evaluation in saes aqumt yews,) H¢u®e tie so* win We ' zo*sis, aaas bmd on yew-1o-pw chop a dmMxeq ^mopnea, ndw d= total elecMatg moMmpom CD'12 AGEEE Sumaner SWdy an EmffW Efieieneyr in Buik%nps 2-291 104 Past Tr eatnent Surr+eiA$rt the completion of the ssoessmeot periA the paojact teinn will c?on&x.^t a survey of participants tnducling the comlrd group) to assess_ ' • • • PWhCVWA perception of IHM' zmPact mcbmlinU • Hehavim cLang;es • Eqmzornd upgrades and borne effidenc)r miprmemenis Participant satidwhon With the &wiec Changes m awmenesshaidershm&W 1t-mls evaluated during initial su+e3^ Other facm that may have impacted energy asagr (i+enovatims, change m number of occuqwmiz, etc-) This survey data will be combined with pxracqmtLon data for aW MhRT View participants who also participated in other AEP Teses energy efficiency pQoqpmms- Analyzing this data in camAwnhon with the consmupbon data will allow the pQapet beant to develop a deqw understmuling of the proW€t's impacts, incbxbw. Clross-pwgcm► benefits - Are pea* with a display mffe likely to participate in other uffd3`Vonsoffed Progmm`3 Quaktahve participant experience EeeAa& -17ro the participants parocive a bowWAdditiaoal factors that impact energy savings - Are certain sepments of the population (yovWcad_ Lqghllm mcamr,highJiow canmumWhw etc_) mo®e likely to show savings that others? C"onch.sioa There are always caukands that make it difficult for programs to achieve optimal design, and each program tmm mat auei& The tndeoffs,. These best practices pum& an outline of zoqatandt considerations for the program dmp proem_ • • • • • • • • Begin with a soft launch Define the population and align the saoqpie to auaacinoim external validity Include a€antr©1 emap thDvommiumrt internal validity Design treatment groups In isolate impacts of research variables Recruit suffideut sample to achieve anatg^sis goals ChTaae+enut to account ifor dropouts Plan for an amassment period sufficient for evaluation uf kmg-trsm impacts Combine various data sauces to improve understanding of unpacis While liaise rer^mdatims are especially important fior research and development psojxta, they can improw the appostimities for asaessmmt of any program the will require verification of energy savings. jIf swdin=Wct is aLanaed, itwill de ueomuy to di9c+omode ervmpm nrodd doutsltaomtio& @ID612 AiCEEE Summer Study on EnerW Elieiency in HulMMgs 2-292 105 References El^uItIatatinea, K_, K A- Damfly, & B. A- Laibaer. 2010_ `=Advanced Mewing Initiatives and ResidmW F®edbwJk Ptoga®s: AMeta-Redrvr fimr Household Elechicity-Saving Oppatunoties.' Washinghw D.C..: American C:auxa1 for an EmeWr-Efficiadt FconaW_ EPR.i. 2009_ Readmlim[ ebcirncity ecsefoadback A research syvmdew and ecaanoxwfiamwwork. Palo Alto Cali£_ Electrwal Pvaw ^. ewach IosWWe. Ott, R L & M Lampeckeor_ 2010_ ,In IAbnvdWctian to Shwishcal Mddsa& and Dab Analysis, Sixth E*fim Belnwt Calif.: BrooksXo1e. Q2012IlCEEE Sumner SWdy on EntW Eiicieecy in 9uldiW 2-293 106