Open Energy Data at DOE

June 28, 2019

The Geothermal Data Repository (GDR), the Map of US Wind Farms, and the Regulatory and Permitting Information Desktop Toolkit (RAPID). What do they have in common? They are all about renewable energy and they are all free applications built on the Open Energy Information (OpenEI) platform, sponsored by the Department of Energy (DOE) and implemented by the National Renewable Energy Laboratory (NREL).

The Opportunity

OpenEI and the associated Open Energy Data Initiative (OEDI) support DOE’s fulfillment of open government tenets: transparency, public participation, and collaboration. They aim to improve and automate access to high-value energy datasets across the DOE’s programs, offices, and national laboratories. OpenEI and OEDI strive to provide open data and make data actionable and discoverable by government staff, researchers, industry, and the public, thereby enabling more new datasets to be created, accelerating analysis, empowering game changing innovations, and growing businesses.

Screenshots of Geothermal Data Repository (GDR), Map of US Wind Farms, and Regulatory and Permitting Information Desktop Toolkit (RAPID)

OpenEI – First Generation of Open Data

Since the beginning of the 21st century, energy-related information and data have grown exponentially around the world, from Research & Development (R&D) data to market trends to policy developments. However, these valuable resources were dispersed among numerous individuals and organizations, available in widely disparate formats, and highly variable in quality and usefulness.

OpenEI was developed in 2009 by NREL to link the world’s energy-related information and data with the energy community (e.g., policy makers, developers, researchers in the national laboratories, and industry). The platform is geared toward facilitating access to data and empowers the energy community to use and contribute to the collection of information and data.

A conceptual diagram of OpenEI showing Information Community, and Data supported with a Wiki, Linked Open Data, and Open Government Standards

Users on OpenEI can view, edit, add, and download data for free. OpenEI has more than 1,700 datasets, both acquired and generated, across all major sectors, such as geothermal, wind, water, solar, biomass, hydrogen, buildings and smart grid. It also enables the development of derived data like the Utility Rate Database (URDB), contributed data, and data catalogs. More can be found at openei.org/wiki/Data.

OpenEI was created using open-source technology, like Semantic Media Wiki, as the base platform and employs a custom data access feature that provides access to the public and connects data with external platforms (like Data.gov). NREL calls this custom and reusable code “XDR” (aka. Customized Data Repository), which has similar functionality to CKAN, the powerful open source data portal platform. Thanks to this approach, many developers have been able to build applications on OpenEI (e.g, GDR, RAPID and the Map of U.S. Wind Farms, as mentioned above).

OpenEI became an internationally trusted source of energy data, especially for data related to renewable energy and energy efficiency. Over the period of May 2018 to April 2019, OpenEI has 2.46 million pageviews, referred in over 3,000 Facebook articles, over 21,500 downloads, and 831, 600 users from every single country on Earth.

“OpenEI is at the vanguard of the Federal Government’s efforts to increase transparency, portability, and usability of information produced at the National Laboratories. The OpenEI team, at the National Renewable Energy Laboratory, is the best in class with exceptional knowledge, skills, and experience. They are my go-to team when I need to develop federated or distributable information solutions for customers across the Nation.”
– Rob Bectel, Senior Program Analyst, the Office of Technology Transitions (OTT)

OEDI – Cutting Edge Open Data Initiative

Entering the era of big data and the Internet of Things (IoT), the energy community faces a new set of challenges: many researchers have difficulty accessing and utilizing big data and complex data because 1) data can be difficult to find and use, 2) data aren’t publicly available, 3) data are rarely in standard sizes or formats, and 4) large data sets can be costly to store and manage.

In July 2018, DOE started the OEDI project to help remove these barriers and improve accessibility for analysts and researchers. The platform, built on the foundation of OpenEI, will enable improved open data access and help energy scientists and analysts explore, mash-up, and analyze data. The new framework will speed innovation, facilitate rapid computation, catalog data assets, and allow the public to contribute data.

There are three key components in the OEDI framework:

Cloud Partner Relationships: Based on a successful federal partnership at NOAA with a variety of cloud hosting vendors, OEDI will build similar partner agreements with all major cloud hosting vendors including Amazon Web Services (AWS), Google, IBM, Microsoft Azure, and Open Commons consortium. These partnerships will enable big data to be openly shared with the energy community.
The Data Lake: OEDI will build a robust data lake in the cloud to enable faster, easier, more advanced analysis and computation to accelerate novel data mash-ups, analysis, and innovation. It will support next-generation research and analysis by allowing researchers to analyze big data without making expensive hardware investments (i.e. supercomputers).
The Data Catalog: In order to easily find available data, OEDI is creating a data catalog that merges data from multiple cloud hosting locations. The catalog will include information about the data and data owner, licensing information, provenance, links to all relevant data locations (if the data is housed on multiple cloud hosting vendor sites), and other metadata. The catalog will be designed to get users to the data as quickly as possible.

Concenptual visualization of the OEDI Data Lake)

OEDI already has two large datasets — examples of high-impact, high- demand resource data:

The WIND Toolkit data include meteorological conditions and turbine power for more than 126,000 sites in the continental United States for the years 2007–2013.
The Solar Radiation Database (NSRDB) is a serially complete collection of hourly and half-hourly values of the three most common measurements of solar radiation—global horizontal, direct normal, and diffuse horizontal irradiance—and meteorological data.

Visualization of NSRDB and Wind Toolkit in the Cloud)

Future datasets will be dependent on laboratory researchers and other stakeholders. Criteria for OEDI datasets include usefulness to a broad audience and researchers who will support the inclusion of their data. Initial plans will focus on additional resource and technology data (such as solar, wind, geothermal, water, bioenergy, and transportation), utility rates, technology performance data, materials and market data, information from cities and states, and data related to grid systems.

Future Directions

The data and information coming out of DOE national laboratories are valuable assets. OEDI gives laboratory staff a platform through which they can easily share datasets with the public. Access to big data analytics and computation in the cloud will enable innovation and data mash-ups like never before. Using key analysis tools only found in the cloud, researchers can analyze data faster and industry can leverage large-scale federal datasets in new ways. “Through this program, the Office of Energy Efficiency and Renewable Energy (EERE) will make more widely available the largest and most highly trafficked datasets it maintains,” said Solar Energy Technologies Office program manager Garrett Nilsen. “This will make it easier to find and compare energy data across the country and allow America’s talented academic, national lab, and industrial researchers to develop new innovative products, make scientific discoveries, and create more value from federal research dollars.”

For the coming year, the NREL/OEDI team is planning to leverage and expand the data catalog on OpenEI and utilize it for OEDI – providing access to big data. They are working to secure new big data sets that are currently not accessible to the public. Furthermore, they are partnering with the Linux Foundation’s LF Energy to build an open source community around this project.

Postscript

To learn more about OpenEI and OEDI, you may contact Debbie Brodt-Giles, Director at DOE/NREL, at Debbie.brodt.giles@nrel.gov.