The Double-Edged Sword: Using LLMs in Electricity Market Forecasting

ChatGPT launched in November 2022 with Google, Microsoft, and Meta rushing to join the AI race. Now, tech giants, utilities, and other companies are estimated to spend ~$1tn on GenAI in the coming years. But is GenAI a useful tool for every industry?

The electricity market is a complex, ever-changing ecosystem that depends on accurate forecasting for everything from price setting to grid stability. With the advent of large language models (LLMs) like GPT-4, LLAMA, and Gemini, the possibilities for augmenting forecasting capabilities have expanded. However, while these models possess significant potential, they also present risks that make them unreliable and potentially challenging for short and long-term forecasting. In this blog I explore the dual nature of LLMs in the context of the electricity market.

The Opportunities of LLMs in Electricity Market Forecasting

1. Natural Language Processing: LLMs excel at processing vast amounts of text and extracting useful information. For electricity market participants, this capability can be harnessed to analyze news articles, regulatory updates, weather reports, and social media sentiment, providing valuable context that might affect electricity supply and demand.
2. Pattern Recognition: The self-learning algorithms behind LLMs can detect patterns that might go unnoticed by traditional statistical methods. By training these models on historical electricity market data, one could potentially uncover trends and correlations that improve forecasting accuracy.
3. Speed and Scalability: LLMs can process large datasets quickly, providing real-time insights that can be crucial for short-term decision-making. They can be used to automate forecasting processes, freeing up human experts to focus on more complex tasks.

The Challenges

Despite their advantages, LLMs also have significant drawbacks that make them unreliable for rigorous forecasting tasks in the electricity market, and unusable for daily operations or hedging for market participants.

1. Lack of Domain-Specific Understanding: LLMs are generalist models trained on diverse datasets, and they lack the deep domain-specific understanding required for accurate electricity market forecasting. Unlike specialized models, LLMs might miss critical industry nuances, leading to faulty predictions.
2. Susceptibility to Data Bias: LLMs learn from the data they are trained on, which can include biases and inaccuracies. In addition, some supporting data sets may be missing, and data samples can be skewed from normal outcome due to known events not specified in the training data. In the context of the electricity market, such biases could lead to consistent forecasting errors, potentially causing financial losses and grid instability.
3. Overfitting Risks: Given their enormous capacity, LLMs are prone to overfitting, particularly when trained on limited or unbalanced datasets. Overfitting means the model performs well on historical data but poorly on new, unseen data—an especially critical flaw in an industry dependent on accurate forecasting.  
4. Interpretability: The "black box" nature of LLMs makes it difficult to understand how they arrive at their predictions. This lack of transparency can be problematic for market participants who need to validate and trust their forecasting models. Without interpretability, it is challenging to diagnose or correct errors. This is especially critical considering the biases and known errors in GenAI forecasts.
5. Vulnerability to Nuances in Data Changes and to Adversarial Attacks: LLMs are susceptible to small, often unnoticeable changes in input data. Thus, such changes, which can stem from benign differences in data captured at different times, resettlements of published data, or adversarial attacks, can lead to significantly different outputs. In the high-stakes environment of electricity markets, such vulnerabilities could go unnoticed and manipulate forecasts, leading to disastrous outcomes.

LLMs for Short-Term and Long-Term Load Demand Forecasting

Short-Term Load Demand Forecasting

Appeal: The speed of LLMs can provide real-time insights, allowing for rapid adjustments in load management and grid operations. They can also integrate real-time and unstructured data inputs such as immediate weather changes or social events that could impact electricity usage.

Drawbacks: Short-term forecasting with LLMs can be highly sensitive to noise and short-lived anomalies. The lack of accurate, domain-specific understanding means LLMs might misinterpret these fluctuations, leading to erratic predictions.

Long-Term Load Demand Forecasting

Appeal: For long-term forecasting, LLMs can incorporate a wider range of historical and contextual data to identify broad trends and shifts, potentially offering insights into future electricity consumption patterns.

Drawbacks: Over longer time spans, the risks associated with data biases, model drift, and overfitting become pronounced. With the lack of true weather forecasts, which are a central driver in load demand, and coupled with a lack of domain-specific expertise, LLMs may produce forecasts that are unreliable, leading to strategic missteps in resource planning and policy setting.

The Hybrid Model

To mitigate the risks associated with LLMs while reaping their benefits, consider using a hybrid model that combines LLMs with traditional forecasting models: use sophisticated timeseries models for the primary forecast while utilizing LLMs for supplementary insights to tweak the forecasts for events that cannot be captured in the more traditional models. Like in any other application of machine learning model, when using LLMs that way, remember to ensure the training data is comprehensive, up-to-date, and domain-specific to minimize biases and reduce overfitting risks.

Conclusion

While LLMs offer exciting possibilities for demand forecasting in the electricity market through their impressive data processing and pattern recognition capabilities, they come with significant risks that make them unreliable for both short-term and long-term forecasting tasks. By adopting a hybrid and cautious approach, leveraging LLMs for supplementary insights rather than primary forecasting, market participants can benefit from their strengths while mitigating their inherent limitations.

For example, Amperon does not use LLMs in our energy forecasts, but we are working to add LLMs for internal tools and later for a Conversational AI Bot. This bot will learn from existing documentation and provide answers to questions such as, “What do you use for weather assumptions when producing long-term forecasts?” or “Are the same models used for all suppliers?” The bot will learn as others interact with it and be retrained whenever new documentation is added. This will enable it to answer an increasing array of questions over time and provide more transparency into how our models work.  

Harnessing the power of LLMs in electricity load demand forecasting is indeed a double-edged sword; wield it with caution to gain valuable insights without falling into the pitfalls of inaccuracy and unreliability.