Operational Excellence Program -- Avoiding Events by Information Sharing

A new program, called Operational Excellence or OpEx, is available from the National Hydropower Association to help owners and operators of hydroelectric facilities avoid accidents and events by learning from others’ experiences.

By David Zayas and Jim Miller

David Zayas is senior manager of regulatory affairs & technical services with the National Hydropower Association. Jim Miller is a consultant with Signal Hydropower Consultants LLC.

All industries experience accidents and events. The question becomes: Are there tools available to help limit and minimize the impact when something goes wrong? At the National Hydropower Association (NHA), we think our Operational Excellence (OpEx) program can do just that, assist the industry in reducing and avoiding events.

Consider this real-life example: Runoff for the year was 136% above normal at a hydro facility. The plant had been spilling water for 28 days and the reservoir was 1.3 feet above the flood guide level. Unit 1 had been operating continuously for just over 90 days, with all units experiencing similar run times. During the weekly inspection, personnel noted Unit 1’s brushes needed work, but brush wear did not warrant immediate action. Three days later, Unit 1’s brush rigging components failed, leading to an emergency shutdown.

The results of this forced outage were more than two weeks of lost service, equipment repair costs of $142,000 and lost revenue of $162,000, for a total event cost of $304,000. The owner performed a causal analysis and completed a corrective action plan to avoid an event of this nature happening again. The lessons learned will prevent a similar incident at this facility. But what about at another facility?

Weeks after NHA’s OpEx program received this event report (ER), a second hydro plant owner described a similar brush failure that resulted in a three-week forced outage, repair costs of $395,000, and lost revenue of more than $200,000. If the earlier event had been shared across the industry, perhaps the second owner could have avoided a $600,000 event.

NHA’s OpEx program was designed to assist the hydropower industry with sharing the types of operating experiences described above. Safety is a priority for the hydro industry and it is the cornerstone of OpEx. However, the average age of a hydro facility in North America is reported to be 52 years and operational reliability is an increasing concern. Sharing operating experiences, solutions, best practices and lessons learned through NHA’s OpEx program is one way the industry can respond to these increasing concerns to ensure safe, reliable and efficient operations.

What is NHA’s OpEx program?

OpEx is a voluntary event reporting system that receives, distributes, archives and catalogs hydropower operating experiences and any resulting best practices and lessons learned. OpEx is an NHA program and member benefit and lives within NHA’s Hydraulic Power Committee (HPC), an organization with a more than 40-year history of exchanging technical hydropower information.

OpEx is built on trust and integrity. Ensuring that participant information remains confidential and secure has been a priority for OpEx since its inception in 2011, and the website was designed with confidentiality in mind. Before an ER is released for other participants to view, it goes through multiple rounds of quality control and information scrubbing.

OpEx’s mission is to instill a culture of operational excellence at existing and future hydro facilities that encourages the aggressive critical self-review and analysis of safety, operations, maintenance and environmental events, and the sharing thereof. Its scope and focus is on four functional areas: operations; maintenance; safety (employee, public and dam); and environmental performance.

How do I participate in OpEx and what are the benefits?

Participation is easy. NHA members simply visit hydroexcellence.org and register. Once participants are cleared, they can access the database and begin searching and submitting ERs. For non-members, contact NHA for information.

The benefits are numerous. The value of sharing information and operating experiences is well-documented. Perhaps this is even more so for the hydropower industry, as our assets are aging, including our workforce. OpEx’s benefits are best described in three program areas: event reporting, best practices and lessons learned, and workforce development.

Event reporting

OpEx has received dozens of ERs from utilities, including the federal sector, and equipment manufacturers. The ERs cover the four functional areas listed above. Below is a sample of ER titles:

Incomplete Shutdown — Unit Brake Ring Damaged;
Near Miss — Unauthorized Use of Gas Welder Resulting in High CO Levels;
Drill Press Injury;
Turbine Shrink Ring Failure; and
Uncontrolled Loss of Load on Gantry Crane.

When submitting an ER, the participant fills out a form that includes sections such as the event description, cause analysis, corrective action plan, best practices and lessons learned. There is also the option to attach media, such as photos and video, and tag the ER for search function purposes. ERs can be saved as drafts (all revisions are saved) and participants from the same company can collaborate on the ER before submission.

The submitted ER goes through a review and approval process. NHA’s OpEx administrator reviews, categorizes and redacts any sensitive information before posting for other participants to view. Posted ERs can then be searched based on criteria and keywords and downloaded for sharing.

We encourage participants not to wait for an event to occur at their facilities. Rather, review your records and identify historical events you can submit to OpEx. Historical events are just as important and beneficial as an event that happened recently. An added benefit of submitting historical ERs is that it helps familiarize a participant with the system, and it increases OpEx’s value by building the database.

Finally, not all ERs are created equal, but they are all important. A seemingly small event to one company could be a significant event for another.

Best practices and lessons learned

Often, ERs describe best practices, lessons learned and corrective actions to remediate the event. To capture this invaluable information, we created a best practices library. From the ERs received to date, we have documented 124 corrective actions, 62 lessons learned and 53 recommendations.

To give OpEx’s best practice’s library a boost, we worked with the U.S. Department of Energy’s Oak Ridge National Laboratory to transfer information developed as part of the Hydropower Advancement Project. The best practices library is separated into categories: civil, mechanical, electrical, instrumentation and controls, and environmental. In 2015, we will expand the library, guided and developed based on the trends we are identifying through analysis of the ERs submitted.

Workforce development

Workforce development seems to be a standing topic at hydro industry conferences. Numerous organizations and initiatives focus on workforce development in the energy industry, whether providing resources to help train a new workforce or assisting in the placement of new talent. Some examples include the Center for Energy Workforce Development, First Lady Michelle Obama’s Joining Forces initiative and Hispanics in Energy.

These photos show a brush rigging failure that occurred on a hydro unit, and the event has been recorded in the Operational Excellence program.

Although great resources, none of these initiatives focus exclusively on hydropower. OpEx will fill this void and through ER development and analysis, webinars, and workshops, be a premier resource for educating and training the next generation of hydropower professionals.

Current OpEx participation

More than a quarter of NHA’s membership participates in OpEx, with more members joining regularly. Here are a few examples of how member companies have incorporated and institutionalized OpEx participation into their operations:

Included participation and ER submission in employees’ performance goals.
Including OpEx updates and presentations at weekly and monthly plant meetings and sharing recent ERs, corrective actions and best practices.
Adopted the ER template to match their own internal reporting process, streamlining the ER submission process.

Conclusion

OpEx will be highlighted at NHA’s annual conference, held April 27-29 in Washington, D.C., where you can learn more at Session 8C — Industry Trends and Analysis: What Are We Learning? We will also present OpEx and share results at all of NHA’s regional meetings, HydroVision International, and other industry events throughout the year.

In a new era of growth in the industry, NHA’s OpEx program is our contribution to ensure hydropower assets operate at the highest standards possible. Join your colleagues today, register at hydroexcellence.org. Or contact [email protected].

Example OpEx Event Report – 13.8 kV Bushing Failures

Below is an example of an event report (ER) in the OpEx system.

Description of Event and Investigation

On Dec. 9 at 2300 hours, a transformer differential alarm was received and the Unit 3 generator tripped offline. Immediate investigation revealed failure of one of the C-phase standoff bushings in the 13.8-kV leads tunnel between the generator breaker and step-up transformer.

Further investigation revealed rainwater intrusion through cracks in the concrete deck of the transformer yard. Calcium deposits were formed, copper grounding bars were oxidized, and other bushings and sections of the bus bar were wet.

Water intrusion and wet bushing and bus

The crack was sealed, the failed bushing replaced, and all buswork and bushings cleaned and dried until test results were acceptable. The unit was returned to service Dec. 16 at 1955 hours.

On Dec. 21 at 0516 hours, the same unit tripped offline on a generator neutral overvoltage relay operation. The neutral overvoltage cleared when the generator breaker opened, indicating the condition was once again between the generator breaker and step-up transformer. Investigation revealed insulation failure on the bottom end of B-phase through-bushing leading through the top of the leads tunnel into the transformer yard. Once again, water intrusion appeared to have been the cause.

“Back” side of the failed through- bushing after it was removed

A spare bushing was on hand and was installed. The unit was returned to service on Dec. 26 at 1315 hours.

On Jan. 22 at 1730 hours, the unit again tripped offline on a generator neutral overvoltage. Initial inspection this time revealed failure of the C-phase through-bushing. Once again, it was evident that water was leaking in and running down the through-bushings on all phases.

After further inspection, it was discovered that the water intrusion was not coming from the gasket area of the through-bushings but instead was entering between the base of the concrete pedestal that the through-bushings are mounted on.

Water intrusion at base of through-bushing mounting pedestal

Two spare through-bushings were on order but were several months away from being fabricated. With the unit “dead in the water,” a support engineer came up with the idea of building a temporary through-bushing using high-voltage cable and stress cones. The unit was returned to service on Jan. 25 at 1730 hours. The interface between the concrete pedestal and transformer yard slab was sealed and the temporary setup worked flawlessly until the replacement bushings arrived.

Top-end of temporary through-bushing with sealant around base of pedestal

Corrective action plan

All concrete cracks in areas where leakage could cause problems were sealed under a contract.
All buswork and bushings were visually and electrically tested, cleaned and replaced as necessary.
Transite bus duct panels were removed and replaced with non-asbestos panels that can be easily removed to allow for future bushing/bus inspection.

Lessons learned

Perform periodic inspections (if possible) of generator stand-off and through-bushings during routine generator step-up transformer outages. The inspection should include supporting concrete and metal structures.
Perform periodic megger testing of generator bus during generator step-up transformer outages.
Have spare bushings in on-site inventory due to long lead time purchase.

To view additional ERs like this, and others, visit hydroexcellence.org.

More HR Current Issue Articles
More HR Archives Issue Articles