you're reading...
Energy and Green Intiatives, Healthy Homes Initiative, Weatherization

NASCSP Response to Michigan “Greenstone” Report Around Weatherization & Energy Efficiency Investments

The study released this week titled “Do Energy Efficiency Investments Deliver? Evidence from the Weatherization Assistance Program” by Meredith Fowlie, Michael Greenstone, and Catherine Wolfram on energy savings from residential energy programs is seriously flawed and does not present a balanced assessment of the benefits of investing in energy efficiency in either its final or working draft form. This study was supported by a grant from The Alfred P. Sloan Foundation and run through E2e. E2e is a joint initiative of the Energy Institute at the University of California at Berkeley’s Haas School of Business and the Center for Energy and Environmental Policy Research at the Massachusetts Institute of Technology.

Residential energy efficiency is responsible for 22% of the energy consumed in the United States, accounting for more than $230 billion annually in home energy bills. Many assessments have confirmed the energy savings potential that exists in residential homes. The focus of the recent study was the Weatherization Assistance Program, which has been responsible for weatherizing more than 7.4 million homes during the nearly forty year history of the program. Weatherization has helped ease the disproportionate burden of high energy costs that falls on low income Americans, especially senior citizens, those who are disabled, families with children, and low income households. The program also is responsible for creating new work pathways as well as contributing to community revitalization and reduced carbon emissions.

Other more broad-based and comprehensive evaluations of the weatherization program negate the findings in the recent study and prove what those who have received or provided weatherization services know: this program saves energy and reduces the burden of utility costs for the most vulnerable populations in our society. (Examples of evaluations include but are not limited to those linked here from ORNL, here from Berkeley Lab, here from Policy Matters Ohio, here from independent contractors, and here from Harvard)

Weatherization is the largest residential energy efficiency program in the country. Conclusions by studies, including but not limited to those referenced here, are important and relevant to shaping policy; therefore, it is absolutely imperative that such studies are held to rigorous standards and sample a sufficient number of homes to be conclusive, are peer-reviewed for accuracy and efficacy, draw sound and logical conclusions without relying on assumptions or unfair estimations, and hold up under scrutiny. The weatherization study conducted in Michigan by Fowlie, Greenstone, and Wolfram does not meet these standards. The sample included only two weatherization providers and as few as 349 homes were sampled for specific data segments. Additionally, though the abstract cites more than 30,000 households in the sample, the researchers were only able to match audit data for 1,638 homes — an insignificant and statistically irrelevant number compared to the more than 1 million homes weatherized during the American Recovery and Reinvestment Act period.

Additionally, the study draws illogical conclusions and makes unfair assumptions about the populations of weatherized customers. The researchers conducted what they refer to in the study as an “encouragement campaign,” in which potential customers are inundated with information to encourage voluntary participation in the program. However, what the researchers fail to articulate is that advertising does not drive people to the program. Instead, crisis largely drives participation. For example, the research states that ” given that households had detailed, specific information about the program, it seems reasonable to surmise that some combination of high perceived costs of applying for the program, low expectation of an application leading to weatherization, high unmeasured process costs, and low expected benefits of participating in the program are impediments to WAP participation.” This is a broad and unfounded assumption. Any first year weatherization staff person realizes that people do not apply for weatherization because they are convinced of the importance of energy efficiency measures for their home — they are driven to request services because they are in crisis. They cannot afford their utility bills. In Michigan, where the study was conducted, customers may reach out for help because their heating system has failed and the weather grows cold, they might not be able to afford to pay their energy bill, or are seeking help to remain safe and healthy in their homes.

People reach out for weatherization assistance due to practical or crisis concerns, not because lots of flashy materials told them their homes needed energy efficiency measures. Low-income weatherization customers live day-to-day and their main concern is surviving rather than thriving. Fortunately, weatherization provides for both. The researchers admit that “…the aggressive encouragement efforts were disappointing.” Indeed, they were, not because of any lacking on the part of the customers, but because the researchers themselves failed to understand the values of the population served, their circumstances, and their motivations.

We would be as remiss if we also failed to note that some homes have no heating or cooling system (example) or have a dysfunctional system prior to weatherization (example). Correction of this health and safety concern by installing a high efficient appliance will mean that utility bills and usage actually increase because the family has a working system post-weatherization, and had no system prior to the weatherization work. It is unclear whether the researchers considered this key factor when conducting the study. There are no references to elimination of these homes from the sample or correction for this variable. Failure to take into account this important factor would unfavorably skew the data conclusions drawn by the research.

Of critical note also is the fact that the research uncovered — but elected not to emphasize — a number of positive outcomes of weatherization, including the following:

  • WAP energy efficiency investments reduce monthly energy consumption by 10-20 percent on average.
  • Weatherized households have historically consumed significantly less natural gas than non-weatherized homes during both winter and summer months.
  • WAP participation reduced energy consumption by roughly 8-10 percent in the relatively small sample of homes from two agencies in one state.
  • The study estimated energy savings of approximately $155 per year for weatherized homes. This represents a savings for vulnerable participants who may then apply those savings to other household needs. Although the savings number is only an estimate and is based on assumptions that may or may not be valid, any savings is positive for the populations mainly serviced by weatherization (priority for service is given to the elderly, disabled, families with children in the home and those with high energy usage of burden).
  • Average natural gas bills during winter months are $85.95 for non-weatherized households. For weatherized households, the average bill drops to $57.96.

These positive outcomes were mostly buried in an overall negative and unbalanced perspective provided by the researchers. Outputs by studies such as this receive attention from policymakers and should therefore adhere to rigorous standards of quantitative analysis and present facts in an unbiased light. It is clear that this report misses the mark. Policy debate regarding the costs of energy efficiency should be driven by the best available analyses, and the goal of research should be to narrow the range of uncertainty and produce better cost estimates useful in the ongoing discussion of energy efficiency policy.

The study focuses almost exclusively on the impact of weatherization from an energy-saving perspective. This is to be expected. Weatherization is, after all, an energy efficiency program. However, energy savings is not the only benefit conferred by weatherization. There are tremendous benefits to households that are not reflected in increased indoor air temperatures — to compare the benefits of weatherization exclusively using this one metric is akin to judging a painting based on one brush stroke. There are comfort, safety, and health benefits to participants; structures are improved; communities are enhanced as housing stock increase in value; the health and safety of the residents in the home is improved, and customers served often are able to live independently and remain in their homes thanks to the work of the program. These unmeasured benefits are not recognized by the research produced by Fowlie, Greenstone, and Wolfram, yet they are powerful testaments to the value of weatherization.






13 thoughts on “NASCSP Response to Michigan “Greenstone” Report Around Weatherization & Energy Efficiency Investments

  1. Does NASCSP know who funded the study?

    Posted by Marcy Oerly | June 25, 2015, 2:24 pm
  2. Does NASCSP know how much funding went towards the study? $475,000 alone was used to hire a firm that “encouraged” clients to sign up for the Weatherization Program.

    Posted by Marcy Oerly | June 25, 2015, 3:12 pm
  3. Please also note Minnesota energy costs are low, for example 40-60% cheaper that in the Northeast where Wx does a large portion of it’s work. That alone miserably skews the results, and efforts should have been made to normalize the savings proportionate to how much is spent according to location, local demand (HDD), and local value of fuel.

    Posted by William Gregg | June 25, 2015, 3:25 pm
  4. Working a lot in the weatherization field before moving to another state, I know how important this program is. I see how important energy savings benefits are, but I urge everyone to go to a local community action weatherization open house and meet these people, hear their stories. This program is keeping a roof over people’s heads; most of which are extremely grateful. This program benefits in many way above energy efficiency. People just need to take the time to look past the negative headlines and see how positive this is.

    Posted by Chase | June 25, 2015, 4:04 pm
  5. Folks –

    Following are a series of short talking points we have developed to put the study into perspective. I’ve pasted them below, and attached a version on our letterhead. I hope this is helpful if you get or have questions about the white paper.

    The E2e Project Study of Five Michigan Weatherization Agencies During ARRA

    The E2e Project, a joint effort of the University of Chicago, MIT, and UC Berkeley, released a study on the operation of the ARRA Weatherization Assistance Program (WAP) in an area of Michigan served by five community agencies. In a working paper, the authors describe a study focused on three issues: the effectiveness of an outreach program using canvassers; to confirm the fact that computer-based audits and engineering analysis consistently overstate actual energy savings by a substantial margin; and, the cost-effectiveness of using WAP as a greenhouse mitigation strategy. The study was recently the focus of stories in or on the Associated Press, The New York Times, The Wall Street Journal, Vox, Bloomberg, The Washington Post, Phys.org, Fox, ABC, and CNN.

    One of the authors was quoted as saying, “I would not feel comfortable generalizing from our study in Michigan.” Unfortunately, the press has done just that.

    1. The authors claim that the study is ‘a first of its kind’ and ‘the gold standard’. However, it fails to follow the protocols used by the weatherization and home performance industry to measure effectiveness. Sometimes the first time isn’t golden.

    2. The opening of the paper is devoted to a recruitment experiment using paid canvassers to convince low income families to enroll in the weatherization program. They were not very good at it, probably because people did not trust them. Our agencies have long time deep relationships in the community. People trust us. Nationwide the network weatherized over 800,000 homes, serving 25% more families than the program goal. We found plenty of clients. Now that the ARRA money is gone, 1–2 year waiting lists are again common.

    3. The study did not include over 30,000 weatherized homes as some have reported. The savings study appears to be based on some of the 1,600 homes actually weatherized during the early days of ARRA. The huge infusion of funding affected quality initially, and clearly did so in this part of Michigan. Other states had more training infrastructure; Ohio operated four training centers. Every state got better as the program matured.

    4. The study says that over 34% of the units had furnace replacements. Furnace replacements, after insulation and air sealing have been completed, are almost never cost-effective. Furnaces are replaced as a health and safety measure because they are not working – ‘no heats’ we call them — and paid for with other funds. The study authors somehow missed this and classed one of the most expensive measures as an efficiency investment, leading to the assertion the program has a poor return on investment.

    5. Savings exceed the cost of air sealing, attic insulation, sidewall insulations, hot water, and electric baseload savings measures such as refrigerators and lighting. Those are the services that evaluation professionals measure.

    6. Agencies don’t do windows unless they are broken. Payback is over 35 years. It is a health and safety measures.

    7. Engineering models, including NEAT and the model developed by Berkeley for DOE’s new Home Energy Score program, notoriously overstate savings and have other flaws. Everyone in the industry knows it. Audits are a tool to identify high end uses, not outcomes, and do not use a 6% discount rate to determine net present value.

    8. In Ohio, weather-normalized bill analysis shows natural gas savings for low income families in the 28-34% range in independent studies funded by utilities looking at combo units – units combining federal and utility funded measures. The numbers have been consistent since 1995. Electric savings to the families are in the 8-12% range. Several program designs used in Ohio have been designated as exemplary programs by ACEEE. Programs in other states are also effective, though savings will vary by climate.

    Anyone who had ever struggled with poverty knows how extremely expensive it is to be poor. James Baldwin (1924 – 1987), Fifth Avenue, Uptown.

    Dave Rinebolt
    Executive Director & Counsel
    Ohio Partners for Affordable Energy
    PO Box 1793
    231 W Lima St
    Findlay, OH 45839-1793
    Office: (419)425-8860
    Cell: (614)975-8692

    Posted by Dave Rinebolt | June 25, 2015, 5:39 pm
  6. Ensuring energy efficiency is critical not only to low-income households, but to our global community. Huge volume of research supports NASCSP, the U.S. DOE Weatherization Assistance Program, and others in demonstrating the results of quality residential energy efficiency measures. Investing in energy efficiency is investing in our future. Thank you NASCSP for all you do to keep us advancing green!

    Posted by Kelly Kupcak | June 25, 2015, 6:19 pm
  7. Studies such as this should encourage more efficiencies and better results, but this paper seems to judge whether a program works on a ‘one size fits all’ concept. Some thoughts:

    • The difficulty in performing outreach to lower-income clients is common to several community action agencies. Clients are hard to reach, working, distrustful of authority, and have difficult lives (dealing with myriad social agencies for food aid, illness, child care, etc.). However, the fact that over $1000 was spent per customer in signing them up should highlight how this is a separate (and serious) problem, rather than just lumping it in as another strike against weatherization.
    • The study (performed from June 2008 – May 2014) found that 94% of projected savings were based on using natural gas as a fuel. However, June 2008 was the highest price nationally for natural gas ($12.41/thousand cf). The following year natural gas prices dropped 70%, and continued a further 10% over the next 3 years. Nowhere does the report address this.
    • Not all measures are equal. The bedrock of weatherization is infiltration and insulation, both of which are incredibly energy efficient. Efficiency drops off with ‘big ticket’ items, however, like furnaces, heat pumps, boilers and windows. Weatherizing a property with a non-functioning heat source is essentially worthless, so we must include this factor. In addition, such items are often addressed as Health and Safety issues (along with vapor barriers, venting, water heaters, furnace repairs, etc.), which have no efficiency requirement and can radically skew project costs. Again, the report does not appear to address this.
    • Much of the study focused on weatherization performed during ARRA, and the study recognizes that Michigan received over $200 million but was delayed implementation until early 2011 (like many states). Funds needed to be spent by March 2012, which meant a glut of projects and spending. As a result, agencies were forced to consider more expensive measures that normally might not have been considered, which would also skew overall efficiency. This was still important work, but at the time ARRA was concerned more with providing jobs and turning around a recession bordering on depression. Focusing strictly on energy effectiveness during this time period is somewhat like rating the cleanliness of a house during a fire.
    • The study mentions that NEAT evaluates items based on a projected lifespan. This is simply a function of the program, requiring that all items are assigned a ‘lifespan.’ An item like attic insulation (again, a bedrock item of weatherization), with a 20-year lifespan, will actually last the lifetime of the building. So energy savings may continue well past 20 years.
    • The issue of tying weatherization to carbon emissions is curious. The study’s convoluted math seems to suggest that reduction of carbon emissions is negligible or even detrimental. Energy savings will reduce emissions. Period. But the unspoken message is that we should wait on future technologies involving hydrogen fuel cells and fusion reactors, where there are 0% emissions. So in the meantime, why bother?

    I think this study is important in pointing out that weatherization is a complex issue, rather than trying to compare apples to apples. Saying that weatherization is ineffective is simply a no-win argument, and groups like Passivhaus, Leadership in Environmental and Energy Design (LEED), and others will vigorously defend that. There’s no doubt that supporters of weatherization may exaggerate the benefits (as supporters of any program will do), but trying to address several competing factors by assigning costs and then gathering them together neatly and orderly, is herding cats. And at its worst, concluding the study by saying ‘WAP does not appear to pass a conventional cost-benefit test’ is detrimental to a program that’s been vital to communities for decades. This study seems to be wrapping a bow on an erroneous conclusion, and I would urge the authors to take another look at their work.

    Posted by Patrick Stuart | June 26, 2015, 11:10 am
  8. The Department of Energy has sponsored an independent evaluation of the effects of the Weatherization program, conducted by program evaluation experts under contract with the National Renewable Energy Laboratory. Results of the national evaluation are due to be released later this summer. The evaluation looked at all aspects of the program, and included studies of various types of housing (single-family, multi-family, mobile homes), in all regions and climates, examined the costs and benefits of installing energy efficiency measures in homes heating by gas, oil and other fuels and the impact of weatherization measures on health, safety and building conditions. The evaluation also examined the standards and practices followed by installers to measure the impact of certification and standard work specification.

    This is a rigorous, controlled study using accepted evaluation practices under strict supervision by the Federal government. The evaluation is expected to find that in most states that follow accepted protocols, the program is highly effective at reducing energy use in buildings occupied by low-income households, and is a sound investment of public funds. Savings tend to be higher in multifamily buildings and those heated by oil, but overall, reductions of energy consumption of 20% or more can be expected.

    The Fowlie Greenstrom study was weak and doesn’t deserve the attention it’s getting. One wonders why it was released before peer review – perhaps it’s because the authors know that they will be called out by their academic peers for their shoddy work and the methodological flaws that characterize the research. No mention was made in their paper of how the agencies that performed the work were selected, or of what types of buildings were worked on (wood frame vs. brick, old or new multifamily or single family) or even whether the occupants were owners or tenants. Anyone working in the field knows these are all important factors. Finally, there was no mention of whether the Michigan weatherization agencies or their workers meet the accepted standards for working in the program.

    The national Weatherization network has led the way for the home performance industry to adopt standards and a certification system that ensures that energy efficiency work, when performed by firms certified by the Building Performance Institute and inspected by a Quality Certified Inspector, is installed to the highest standards, using techniques that have been proven to be effective. Weatherization and home performance contractors that hold BPI and QCI certifications know how to get results, and energy consumers that choose firms that follow these standards to provide efficiency work can expect to see significant energy savings for years to come.

    Posted by Tom Carey | June 28, 2015, 2:19 pm


  1. Pingback: VAEEC Blog: Controversial Research Paper Attacks Energy Efficiency, and in the Process Generates Widespread Support for It - VAEEC | Virginia Energy Efficiency Council - August 4, 2015

  2. Pingback: DOE Blog Responds to E2e Study | The State of Poverty - August 31, 2015

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Enter your email address to follow this blog and receive notifications of new posts by email.

Follow NASCSP on Twitter

Like NASCSP on Facebook

Like us on facebook

Follow NASCSP on Pinterest

NASCSP Pinterest
%d bloggers like this: