Jobs and People

Hydro Review: Development of a Simulator for Hydro Operator Training

Tags
Hydro Review: Development of a Simulator for Hydro Operator Training

Development of an online model to assist with hydro operator training should help Portland General Electric ensure reliable operation of its West Side Hydroelectric Project.

By Christopher Goodell, Nathan Katz and Balaji Kannan

Portland General Electric’s (PGE) West Side Hydroelectric (WSH) Project in Oregon consists of six dams on the Clackamas River that impound water for powerhouses with a total capacity of 192 MW. Ranging in age from the 1.7-MW Timothy Lake powerhouse, which began operating in December 2018, to the 44-MW Faraday powerhouse built and commissioned in 1907, the plants contain a patchwork of generations of equipment all connected and controlled at the remote-control center via supervisory control and data acquisition (SCADA), with individual powerhouses and unit control achieved through Allen-Bradley-based programmable logic controllers (PLC).

The Clackamas River plays a vital role for hydro generation, agriculture, recreation and drinking water. It also is an essential ecological habitat, with species including the only significant run of late-winter Coho salmon in the Columbia Basin and other species listed as endangered under the Endangered Species Act. PGE operates the WSH project as a run-of-river facility per its Federal Energy Regulatory Commission (FERC) operating license that includes a slate of rules and requirements mandatory for compliance, including minimum flow rates, tolerances restricting the delta between inflow vs outflow within certain reaches, and seasonal mandatory spill gate operations to promote water dissolved oxygen content.

PGE’s hydro control operators (HCO) ensure river flow license requirements are kept within regulatory compliance, regardless of external/internal disturbances that could kick multiple units offline, restricting flow or sudden changes to the river. Figure 1 highlights the complexities of controlling flow through the reach and illustrates the dynamic nature of the river. Clackamas River flows vary drastically season-to-season, tapering to less than 1,000 cubic feet per second (cfs) as the rains depart and snow pack depletes during the summer. Flows increase to 5,000 cfs to 10,000 cfs during the rainy season, with swings in excess of 20,000 cfs during atmospheric river events.

Figure 1. Clackamas River flows measured by the U.S. Geological Survey from summer 2017 through spring 2018.

The tools available for the HCOs to regulate the river consist primarily of ramping generation and spillway actuation. This manipulation to ensure a constant inflow to outflow is often described as a seesaw or balancing act of shuffling water between reservoirs. Although assisted by PLCs for controlling ramp rates and operations constraints, tables for correlating one dam’s throughput vs river inflow to determine drafting/ponding within individual sections, and the operational “rules of thumb,” the success of the WSH project relies on the decades of experience and knowledge of the HCOs. The HCOs’ capability of managing the WSH project stems from their in-depth understanding of the watersheds and dams and the nuances of the river system, along with the experience of many “trial-by-fire” abnormal river and maintenance events.

Like many other utility companies, PGE faces the challenge related to an aging workforce, the irreplaceable years of experience that will leave with them, and the need to transfer this knowledge to a successive generation of employees. In addition, labor trends point to the fact that the newer generation of workers are less prone to turn any job into a career-long endeavor, thus making it less likely to maintain an ongoing workforce of seasoned veterans. Further, an increasingly stringent regulatory environment and scrutiny of compliance violations by FERC presents an elevated risk of simply allowing new HCOs to learn through the “trial-by-fire” type scenarios that served as valuable training for operators of years past.

The WSH project has an in-flight project for migrating its obsolete control system to a new controls platform. This will include implementation of new control strategies and a complete revamp of the human-machine interface (HMI). This migration presents its own challenges in change management, the need to train even the most experienced operators using the new system, and the need to validate control code to minimize, if not eliminate, bugs surfacing during the commissioning process. Glitches can lead to unintended operational situations, putting PGE at risk for a compliance deviation or violation.

Thus, PGE saw that it was prudent to identify an effective way to facilitate new operator training, promote change management among veteran operators, and provide a test bed for the new control system without the risks of operating a live system. In partnership with real-time simulation experts GSE System Inc., PGE implemented real-time power plant simulators for its thermal fleet. Because GSE’s focus has been on the nuclear and thermal sectors, engineering consultant Kleinschmidt Associates was tasked to provide river modeling expertise. Kleinschmidt was already developing a dam breach model of the Clackamas River system as part of a separate PGE project.

As a major investment in a first-of-its-kind real-time simulation of a hydro generation project involving multiple dams, PGE decided to test concept viability through construction of a pilot project. The scope of this pilot was to include simulation of the WSH North Fork Dam and a reach from its reservoir to the tailrace of River Mill Dam. The purpose of this project is to provide an integrated high-fidelity operator training simulator (OTS) that meets the immediate and long-term training needs for PGE’s WSH generation system.

Completed in 1958, North Fork Dam is a variable-radius thin arch concrete dam. Two Francis turbines facilitate a combined output of 48 MVA. In addition, three spillway tainter gates and a sluice gate assist in regulating flow past the dam. North Fork Dam was chosen for this pilot project because of its simplicity for modeling as two identical generation units, and it is at a critical licensing point for regulation of the Clackamas River. With operational control being simulated in real time at North Fork Dam, the flow through the downstream dams was programmatically adjusted proportionally in the river model to evaluate the accuracy and time response of the GSE simulation software, working in conjunction with the HEC-RAS river model.

HEC-RAS model development

The virtual system is constructed as a HEC-RAS Version 5.0.7 combined one-dimensional/two-dimensional (1-D/2-D) hydraulic model. HEC-RAS is a software application developed by the U.S. Army Corps of Engineers Hydrologic Engineering Center that can simulate 1-D, 2-D, and combined 1-D/2-D unsteady and fully dynamic flow in rivers.1 HEC-RAS includes geometric elements to simulate dams, levees, weirs, gates, bridges, culverts and other hydraulic structures. The HEC-RAS model was developed to be the virtual system previously discussed and included the Faraday Diversion Dam and Forebay.

To serve as an operator training tool, the HEC-RAS model must: run quickly, be numerically stable, and produce accurate and reliable results. To keep the model running quickly, it is primarily constructed using 1-D elements because 2-D areas are much more computationally burdensome. One 2-D area is used in the model to capture spreading flows that spill into the left overbank adjacent to River Mill Dam during high flow events. The rest of the model is simulated using cross sections. Figure 2 provides the model layout at North Fork Reservoir, while Figure 3 presents a close-up view of North Fork Dam. The cross sections, shown as yellow transects, are used to represent the geometry of the system.

Figure 2. This is a schematic of North Fork Reservoir created using the HEC-RAS model.

Figure 3. This is a schematic of North Fork Dam created using the HEC-RAS model.

The HEC-RAS model used flows from the upstream end of the model, along with gate settings at and flows through the dam determined by another model set up by GSE, called the GSE dam model. This model uses forebay and tailwater stage records as inputs in determining flow rates, power output and various other metrics at North Fork Dam.

Automating HEC-RAS using the HECRASController

The HECRASController is used to facilitate communication between HEC-RAS and the GSE dam model. The HECRASController is part of an application programming interface (API) that is part of the installation of the HEC-RAS software. It is a library of HEC-RAS-specific procedures that can be called from programming code during run time to automate and control HEC-RAS, set input values, and retrieve output.2 The HECRASController is an integral part of the OTS software developed by GSE.

In addition, data exchange between HEC-RAS and the GSE dam model is handled through direct manipulation of the HEC-RAS input files. These are stored as ASCII text files and can be read and written to using command statements in any programming language.2 The HEC-RAS geometry, flow, plan and project files can all be accessed this way. Specifically, for the OTS, the unsteady flow file is rewritten at every exchange with updated upstream inflow data, as well as total flow through the dam.

Operations simulator

OTS will provide the following benefits to the plant operations and instrument and control (I&C) staff on the Clackamas River:

Normal start-up and shutdown training on the WSH project units.

Development of best practices in the form of operational procedures for abnormal river flow or equipment situations.

A platform for training operators to address regulatory issues and the zero tolerance established by management for future deviations from the WSH operating license regarding river flow.

A virtual platform of the control system to allow I&C engineers to use the simulator to test the logic and control changes on the simulator before implementing the changes in the plant. This will help remove any bugs and shorten the time of plant outage, should that occur.

Early discussions of this project revolved around building the river model using the GSE modeling tools to interface with the controls systems. However, plans were under way for a separate project to develop a Clackamas River hydraulic model using HEC-RAS for performing dam breach studies. It was evident that only minor modifications would have to be made to the dam breach model to serve as the river model for the OTS. Hence, it was decided to use the HEC-RAS model and integrate it to the simulator.

Figure 4 shows the GSE dam model interface. The blue icons represent volumes of liquid and vapor or, in this instance, water and air. The volumes are connected to each other or to boundaries by green links that represent flow paths. These flow paths — which may include gates, valves or pumps — define the elevation and flow capacity of the connections between the volumes.

Figure 4. This figure shows the GSE dam Model interface.

The volume representing the North Fork Dam Forebay has its water level driven by the HEC-RAS model, which in turn accepts flow feedback from the GSE dam model for penstock, spill gate and tailrace flows. In addition, the instructor can set values for the diversion tunnel gate position and the River Mill and Faraday powerhouse loads.

The water turbine, represented by the turbine icon, provides the hydraulic power output to the generator model.

Model implementation

The GSE Simulation Executive (SimExec) controls the simulator and serves as the interface between the HEC-RAS and GSE dam models, as well as the Operator Station and the Instructor Station. SimExec controls the simulator-specific functions like Snapshot, Reset, Freeze and Run.

In the current system, the North Fork Dam controls, operator HMI, and plant electrical panel controls are emulated within the GSE Java Application Development Environment (JADE) tool set. This allows for all systems to be run on the same machine.

All models are run at 20 cycles per second. HEC-RAS is limited to a 1 minute or greater output interval. During this period, the GSE dam model accumulates the volume of water for that minute and passes this value as a discharge to HEC-RAS as an input for its next iteration. Then HEC-RAS is run for 1 minute before passing output to the GSE dam model. This requires HEC-RAS to effectively “pause” while it transmits output every minute, so a restart file was used. At the end of each 1-minute HEC-RAS simulation, forebay elevation is sent to the GSE dam model and a restart file is written for all flows and stages throughout the model domain. After changing inflowing discharge and gate positions (from GSE model output), the HEC-RAS model resumes from its previous state by reading in the previous run’s restart file.

SimExec can perform the following functions before, during or after a training session:

Freeze: Values are not recalculated in the models and the distributed control system (DCS) stops updating values.

Snapshot/Initial Condition (IC): A file or set of files containing all simulation and control system variables stored at a plant condition. The state of the simulator can be changed from one plant condition to another by freezing the simulator, reading the IC file to re-initialize the simulation and control system variables (referred to as resetting the simulator), and then placing the simulator into Run.

Reset: All simulation variables are re-initialized to values stored in an initial condition or backtrack file. Where a DCS is present, internal control system variables such as controller outputs and integrator values are re-initialized to the values in an initial condition or backtrack file.

Run: The simulation models are allowed to calculate new values based on dynamic conditions being simulated or command inputs from the operator; the opposite of Freeze.

The HEC-RAS model is interfaced to SimExec using the HECRASController API. This interface allows SimExec to send Snapshot, Reset, Freeze and Run commands. These are critical commands required to train operators on normal and abnormal scenarios.

The instructor station was developed using JADE object-oriented programming software and is the interface used by the training instructor to run the simulator. The instructor can start the training session from a certain initial condition and allow the operator to run the plant. During normal operation, the operator calls maintenance staff to perform certain actions. In the simulator, these actions are defined as Remote Functions (RF) and the trainee calls the instructor to perform these operations. The instructor can also introduce abnormalities in the plant known as malfunctions, which the trainee must recognize and take corrective actions. Some of the RFs available to the instructor are:

Diversion Tunnel Gate, Lockout Relay Reset, Main Cooling Water Valve, Raise Fish Net, Stream Flow, Rivermill Dam MW, and Thrust Bearing Cooling Water Valve.

Some of the malfunctions available to the instructor are:

Broken Wicket Gate, Generator 2 Thrust Bearing Temperature High, Governor Unit 1 Oil Leak, Load Rejection Trip, Loss of Excitation Trip, and Unit 2 Trash Rack Diff Trip.

Table 1 shows global failures that will apply for all components controlled from the PLC and in the scope of simulation. This provides the instructor with a larger range of abnormalities that can be introduced during the training session.

Table 1. Global Component Failures

The operator station was developed using JADE object-oriented programming software and is used by the trainee to operate a virtual plant. It is composed of replicas of the North Fork Dam HMIs that are used in the project. During a training session, these replica interfaces talk directly to the SimExec, vs in real life they would directly control the systems of the project. Figure 5 shows one of the replica interfaces used to simulate the North Fork Dam HMIs.

Figure 5. Operator’s overview screen interface

Plant electrical panels were also emulated using JADE and included in the operator station. The operator can operate these switches either on a touch screen monitor or using a mouse.

Summary

Operator training has not yet begun using this system, but it has demonstrated the ability to link HEC-RAS with the GSE dam model and can be controlled by a trainee and instructor. The system is anticipated to be installed on site by the end of October 2019.

The HECRASController API served as a vital link between HEC-RAS and the SimExec and made it possible to use the HEC-RAS Dam Breach model as the virtual river in the OTS. With this connection, the trainee can train on the OTS and be unaware of the difference between the virtual system and real system. The HEC-RAS model can continue to be updated with newer and/or better terrain and geometric data without affecting the OTS software.

However, there are some issues with linking to a virtual model that were discovered during this process. Very low water depths in the HEC-RAS model will cause numerical instabilities that can lead to a run-time crash. Any cross section that goes dry will immediately crash HEC-RAS. Normally water depths that low would not happen. However, to prevent the HEC-RAS model from crashing, water levels are monitored by the SimExec and the operator is warned if they become too low.

The OTS will serve as a valuable tool for training operators at PGE’s WSH project. Once this training software is pilot tested for North Fork Dam, the OTS software and the virtual HEC-RAS river model will be extended to include the entire WSH project.

Notes

1“HEC-RAS River Analysis System User’s Manual, Version 5.0,” Report Number CPD-68, U.S. Army Corps of Engineers, Institute for Water Resources, Hydrologic Engineering Center, Davis, Calif., 2016.

2Goodell, Christopher, Breaking the HEC-RAS Code: A User’s Guide to Automating HEC-RAS, h2ls Publishing, Portland, Ore., 2014.

Christopher Goodell, P.E., D.WRE, is principal consultant with Kleinschmidt Associates. Nathan Katz, P.E., is staff engineer with Portland General Electric. Balaji Kannan is senior program manager with GSE Systems.