Energy trading consumes and in turn produces massive amounts of data, with every transaction initiating or referencing dozens or even hundreds of supporting data points. 

Those data points may include contract/counterparty related data, locational data, time series data, price data (both deal data and prices from interfaced feeds), multiple buckets of volumetric data, transmission/transportation routing data (including additional contract data), and accounting data (GL, subaccount, tag, etc.).

Additionally, as each transaction is committed and recorded, processes that aggregate or analyze those transactions – such as position reports/screens or risk metrics – produce new and additional data.

The primary systems for capturing and analyzing all this data are the ETRM systems used by energy trading companies to buy and sell natural gas, power, crude and crude products, liquid natural gas (LNG), coal and emission certificates.

In order to manage this flood of data – whether internally generated and fed in from external systems – ETRM solutions are typically designed to be efficient at data storage through the denormalization of timeseries data, batching processes in overnight runs, or archiving data that, while valuable for macro analysis purposes spanning multiple years, is largely irrelevant to and within the current accounting windows (i.e. data older that x number of months or years).

Though these data management approaches may be moderately effective at addressing some problems, trading companies with high volumes of transactions will often find their systems slowing as more deals (and subsequent data) are committed to the system.

Additionally, because of data denormalization and complex, often obscure data structures, virtually all energy trading companies say that their ETRM make it very difficult to extract reports and retrieve data or information that is not provided either by vendor supplied reports or via exports from system screens – and hence usually requires vendor support or highly experienced internal IT skills. 

These problems are compounded if a company is using more than a single ETRM solution, a situation not uncommon in multicommodity trading shops that have opted for a ‘best of breed’ approach by deploying the solution most capable of addressing each of the various commodities in which they trade.

These might include one system for managing a complex portfolio of wholesale natural gas (often including C&I sales), and one or more additional systems for managing power generation, trading or renewables across multiple regional markets. In these instances, not only must the data be managed within the endpoint systems, it must also be aggregated daily at various levels for reporting (including PNL), analysis or accounting.

This lack of aggregated data visibility access can hinder ad hoc analysis of rapidly emerging or immediate market opportunities, such as a short-term capacity release, potentially leading to wrong sized bids and/or opportunities missed entirely.

Addressing the data problem has proven to be a daunting task for most organizations, and has commonly been accomplished via complex data extracts (often requiring vendor support to write and maintain) and/or consolidations via one or more complex spreadsheets requiring manual manipulation and reconciliation. This approach can and often will lead to errors, delayed analysis and incomplete market insights. Truly solving the data and analytics issue in energy trading requires a more strategic approach, one that is supportable, accessible, reliable and secure.

Data Stores and Data Analytics

We believe the best approach to solving the data problem is to deploy a data store (or data lake) that collects, normalizes, transforms, and stores the critical data and information that resides in and across the numerous systems necessary to succeed in today’s energy markets.

Using the latest available cloud technologies, such as Azure DataHub and Data Factory, this approach can more readily and reliably integrate data from such systems and provide the transformations necessary to make the data readily readable and accessible for reporting and analysis by approved users. 

Though simple in concept, creating an effective data store/data lake requires proper planning and a willingness to invest in the tools and support to not only create it, but to ensure its long-term success and utility.

When moving forward with a new strategic data strategy, there are several keys to securing success:

• Have a clearly defined data strategy. Which transactional or operational systems (both internal and external) will be incorporated and by what method (vendor supported API, direct access to the system’s database, etc.), the tenor (how often) the data to be collected (end of day, hourly, each new update, etc.), and the required granularity of data (do you need to capture sub-hourly data?).
  
• Governance must be established. Who ‘owns’ what data and who has access to that data? How will any design changes be managed as new systems are brought onboard as your business grows?
  
• Data quality and normalization is vital. What system or data is the original (golden) record? How is the data stored in the system and will it require normalization and transformation to a common format?
  
• Identify the right tools. Clearly identify the best and most manageable reporting, analytics, and visualization tools to be deployed in conjunction with the solution. This could include simple user-friendly reporting tools and more sophisticated analytics/development packages such as Python.
  
• Ensure you have an appropriately skilled team. Employ the properly skilled and experienced resources to not only design and deploy the solution, but also ensure you have a solid support team, either inhouse or through an existing relationship with a consulting firm.
  
• Create a culture of data consumers and decision makers. Establish a rich repository of key data and information that is readily accessible, timely and accurate, enabling improved decision making and a timelier view of enterprise performance.

The Benefits of Strategic Data Approach 

Companies that have successfully employed a strategic data approach using data lakes and the latest data management, reporting and analytics tools have reaped significant benefits.
Perhaps the most important of which is improved efficiencies across all aspects of the business by accelerating information flow and ensuring the right data and information is readily available to decision markers.

This is enabled by having an infrastructure that can consume, transform, and store data from multiple end point systems (such as ETRM solutions) and external systems such as price feed, scheduling systems, or market interfaces; and, present that data back, normalized and properly formatted for more flexible and rapid data analysis, reporting and enterprise consolidations.

Additionally, with a virtually unlimited capacity to store historical data, data archiving in ETRM systems can be more ‘aggressive’, lessening data processing loads and improving responsiveness of the system. Furthermore, by moving historical transaction information to the data store, trading analysts, originators and traders can readily access customer data to better understand usage patterns or buying trends without impacting ETRM system performance.

Benchmarking and performance analysis can also be improved by having a single source of data, allowing a more granular analysis and review of performance over time or by comparing trading books, desks, etc. over longer horizons.

Beyond the obvious benefits of improved reporting and analysis, a comprehensive data store can also enable the development of custom applications such as dashboards to improve performance monitoring or custom-built solutions to identify arbitrage opportunities that span multiple markets in trading or logistics.

Though the benefits of a strategic data solution are numerous, prior to the advent of advanced cloud-based data management tools energy trading companies often struggled to build, maintain, and support these types of solutions in-house. However, with these new cloud tools and a growing pool of experienced resources to leverage them, these solutions have become more economical, simpler to support, and more valuable in the energy trading space.