Since 2013, ERCOT has historically overpredicted the summer peak by an average of 1.15 GW in their yearly Long-Term Hourly Peak Demand and Energy Forecast report. As an alternative, we present a simple linear benchmark, which has a 31% smaller mean absolute error and 86% lower year-over-year variation. In this report, we will offer a study of ERCOT’s methodology and make the case that it does not outperform a simple benchmark.
Over the years, ERCOT has developed a framework for its long-term load forecasts (LTF), where linear regression and autoregressive models per weather zone are driven by 15 years of weather data, economic and demographic variables, provided by Schneider and Moody’s. These models are trained, validated, selected and tested to achieve the best possible accuracy.
As ERCOT decides to report only the current forecast and that of the previous year, and doesn’t publish the accuracy of their historical forecasts, it limits their accountability and makes it difficult for the average reader to properly evaluate the forecasts. It appears that no benchmark model is used to verify the validity of the methods and data sources and how adequate they are for the load forecast. The only validation process mentioned in the report is the use of random data points between 2015 and 2020, but tests which cover a ten year horizon are absent.
All models are wrong, some are useful, as George Box said. The complexity of reality does not always mean that complex models prevail over simpler ones. A good practice in forecasting is to challenge any complex method by a simpler benchmark model to make sure the more complex method is actually better. In addition, one should not forget to look backward and examine the results of past forecasts.