On Thanksgiving Day 2012, a fire began at the 84-MW Thermalito plant three floors below ground level and spread upward into a control room on the next floor. An investigation revealed the probable ignition point, and the author discusses lessons learned.
By David Roose
David Roose is the chief of Utility Operations for the California Department of Water Resources and responsible for the operation of the California State Water Project.
Every hydroelectric power plant owner is loath to experiencing an uncontrolled fire burning at its scheme, especially within generation facilities. But, on Thanksgiving Day in 2012, a devastating fire burned out of control for nearly 48 hours at the 84-MW Ronald B. Robie (Thermalito) Pumping-Generating Plant, operated by the California Department of Water Resources (DWR).
Thermalito is part of the California State Water Project (see Figure 1 on page 36). The role of this project is to deliver water to the residents of California.
Indications that the unthinkable was truly happening at the unmanned powerhouse 4 miles west of the city of Oroville in Butte County, Calif., came around 6:50 a.m. when supervisory control and data acquisition alarms tripped, forcing Units 1, 2, and 3 out of service.
Thermalito, completed in 1969, has five floors with the top two above ground. There are four units in its underground powerhouse complex; a 36-MW Kaplan type generator and three 27-MW Francis type pump-generators.
The fire began at Elevation 136, three floors below ground level, and spread upward into the control room on the next floor. Although there were no injuries to plant personnel, 2014 revenue loss from hydroelectric generation is estimated at US$6.5 million to US$10.5 million.
There was no damage to the bottom two floors of the structure.
At 7:20 a.m., an operator apprentice dispatched to the scene reported to the Oroville Field Division Area Control Center that heavy smoke was coming from Elevation 136, the second floor. The control center called the Butte County Emergency Communications Center and they dispatched the California Department of Forestry and Fire Protection (Cal Fire) and local fire resources, which arrived on scene at 7:42 a.m. Personnel battled the blaze until “significant smoke and fire conditions affecting firefighter safety” forced the responders out of the building at 1 p.m.
Prior to evacuating the plant, Cal Fire personnel installed an unmanned cellar nozzle that continued fighting the fire, ultimately bringing it under control late Saturday morning, Oct. 24, 2012.
The following is an edited version of the report submitted to the California DWR.
Data
Although not absolutely conclusive but with high probability, fire forensic experts that included a lead fire forensics expert, an electrical engineer and mechanical engineer who both specialize in fire forensics identified the point of ignition as a fault in circuit No. 19 in Control Cabinet A (CCA) at Elevation 136. The fire spread through a “cable access point,” leading from Elevation 136, one level higher to Elevation 149.
During its investigation, the group of forensic experts discovered that circuit No. 19, which originates at elevation 136 from the 125-v DC Power Supply Control Cabinet A (DCA) at Elevation 149 — which supplies power to the Unit 4 relay board DC distribution panel located in CCA, also at Elevation 149, in the control room — had both conductor-to-ground and positive-to-negative short circuits in the cable tray at back to Elevation 136. In other words, a cable leads from Elevation 136 to Elevation 149. The cable travels a length at Elevation 149 and returns to Elevation 136. The fault likely occurred in the length of cable at Elevation 149.
Heavy smoke billows from within the control room of the 84-MW Ronald B. Robie (Thermalito) Pumping-Generating Plant on Thanksgiving Day in 2012. |
Both wire conductors became welded to the steel cable tray. Circuit No. 19 was routed between DCA and CCA at Elevation 149 through a cable access point in the floor into the plant cable tray system at Elevation 136 and back up through another cable access point into CCA.
When examining fires for the point of ignition, fire forensic investigators look for burn patterns. A common pattern is the “V” or conical pattern, formed as the fire burns up and out. By tracing the “V” pattern, investigators found the lowest point of the burn and from that were able to more clearly determine the origin of the fire.
Engineers set up an experiment from which data indicated the Thermalito powerhouse cable tray fire went out shortly after the breaker for the system tripped, removing the source of energy for the induced series arc. |
A “V” pattern is evident on the front of CCA, which contained various relays for Thermalito plant equipment at Elevation 149. The pronounced “V” pattern places the burn point at the bottom (floor level) of CCA between the relays controlling Thermalito Unit 2 and Unit 3. This “V” pattern is directly above the cable access point from Elevation 136 into Elevation 149.
Point of ignition
While the plant burned, fire fighters pointed a thermal imaging camera at a cable access point in the floor between Elevation 149 and 136. The camera measured 900 degrees Fahrenheit on the cable access point on Elevation 136. The collective gathered data indicated: a “V” burn pattern at Elevation 149 burned wooden batten boards between Elevations 136 and 149; thermal imaging of extreme heat of 900 F at the cable access point at Elevation 136; and concrete spalling at Elevation 136 confirmed the point of ignition occurred at Elevation 136.
Contributing factors
Although the state Fire Marshal found no fire code violations in the suppression or detection systems during the most recent inspection (2008), forensic expert Mark Schafer, of Electro-Mechanical Recertifiers, did identify possible contributing factors to the fire. Some of the contributing factors, but not all, included: aged cables, mixed voltages and over-stacked cables in the cable trays, a lack of fire stops between elevations, an inoperable dry chemical fire extinguisher cart, and combustible materials such as plant schematics and additional historical items printed on large paper sheets stored within the plant.
DC short-circuit arcing fault — Circuit 19 analysis
After determining the possible point of ignition, DWR retained relay expert Derrick Stewart, of Pacific Power Engineers Inc. (PPE) to further analyze certain components of the electrical system.
To the greatest degree possible, PPE designed and built a DC system to replicate the faulted circuit in order to determine if delayed fault clearing could have contributed to the severity of the fire. This lab experiment used some new circuit components, such as DC breakers, because the actual circuit No. 19 components were destroyed in the fire. But, the experiment was able to use cable taken from other parts of the Thermalito plant that was the same type used in circuit No. 19.
The relay engineers determined that because of inherent design vulnerability, a short- circuit arcing fault in DCA circuit No. 19 would not be reliably cleared (an electrical engineering term that indicates electrical mitigation processes) in fewer than 8 seconds for the identified fault location. This means circuit No. 19 met design criteria but the circuit has an inherent vulnerability.
The relay engineers performed lab-controlled experiments to assess the impacts of this design vulnerability, and they concluded that the DC short-circuit arcing fault, despite a delayed fault clearing, likely could not produce flames or fire without other unknown conditions being present.
The 84-MW Ronald B. Robie (Thermalito) Pumping-Generation Plant, completed in 1969, is part of the California State Water Project, established to deliver water to the residents of California. |
The relay engineer’s experiments demonstrated the difficulty in starting a cable-tray fire from a circuit faulted in a manner similar to DCA circuit No. 19. For this specific type of fault, lab-controlled tests produced small amounts of molten copper slag and vaporized copper, but no secondary ignition of cable-tray components occurred. The relay engineers think that for a short circuit in DCA circuit No. 19 to have started the fire, one or both of the following conditions would likely have had to be present at the time of the fire, conditions that were not possible to replicate exactly in laboratory testing:
- An unknown source of fuel was available that could be and was ignited; and
- The actual circuit No. 19 fault produced far more thermal energy than was generated during testing.
Inspection and testing of the tripped circuit breaker for DCA circuit No. 19 also revealed significant degradation of the breaker’s internal moving electrical connections, likely the result of exposure to the airborne contaminants present during and after the fire.
Series arcing faults
Although not identified as the point of ignition or the type of fault of this fire, series arcing faults are far more common than short-circuit arcing faults. In fact, they are the dominant cause of DC arcing faults in the U.S. Navy.1 This type of fault can develop when a faulty electrical connection causes a high resistance point. This high resistance generates heat, causing connections to melt and leads to an inline (series) arc. Inline arcs add resistance to the circuit, which in turn lowers current, making the arc impossible to detect using traditional over-current devices, such as the breakers at Thermalito.
To test the effects of a DC series arcing fault, the relay engineers and DWR applied a resistive load across the battery that drew 2.5 amps. The circuit was routed through the cable trays from circuit No. 19’s laboratory tests. Slowly separating the conductors resulted immediately in an arc, creating a fire with little effort. Notably, the cable tray fire went out shortly after the breaker tripped, removing the source of energy for the induced series arc.
An attempt to ignite all the different types of cables in the trays using the series arcing fault produced similar results — none of the cables sustained the fire after the breaker tripped and the arc extinguished.
Conclusion
The results of the fire forensic and relay engineer investigations lead DWR to think the Thermalito fire’s point of ignition occurred at Elevation 136. However, lab testing on circuit No. 19 was not able to confirm the fire expert forensics. This inability to confirm circuit No. 19 as the root cause can be attributed to the extent and severity of fire damage (cables and breakers destroyed) that limited the fire forensic investigation and the relay engineer’s inability to replicate absolutely the actual event in a laboratory setting. Additionally, although no forensic evidence was found to support a DC series arcing fault as starting this fire, literature and laboratory experiments suggest the DC series arcing fault as a possible root cause of plant fires.
Such a possibility gives rise to the need to identify proactive steps to address and mitigate these types of potential faults in all State Water Project (SWP) plants. Because so much evidence was incinerated by the Thermalito fire and because of the complexities and limitations of a root cause fire — forensic and laboratory — testing analysis, the root cause of the fire cannot be conclusively determined. However, DWR is committed to evaluating and applying the lessons learned and taking the next steps outlined to secure the safety and integrity of all SWP facilities.
Lessons learned
After a careful review of all of the findings, DWR assembled a Fire Systems Modernization Team — an internal group of experts consisting of operations, maintenance, safety and management staff — to evaluate the following suggested improvements and corrective measures to protect against such incidents and losses at hydro plants in the future:
- Ensure that fire stops are made from materials approved by the National Electric Code (NEC) and are properly installed;
- Install remotely monitored fire and smoke detectors;
- Regularly test all fire suppression systems to ensure operability;
- Test and replace aged cables where possible;
- Use best practices to better manage cable tray capacities and voltages of cables within the same tray;
- Remove unnecessary combustible material from the premises;
- Upgrade the alarm notification system to provide more useful information;
- Develop and implement a maintenance program for low-voltage AC and DC distribution systems;
- Include hot-spot testing of low-voltage AC and DC distribution systems; and
- Review DC distribution system design and maintenance practices at all hydro facilities.
Next steps: SWP fire systems modernization
The first project to be implemented as a result of the evaluation, the SWP Fire Systems Modernization Plan, consists of six goals:
- Review and rank the suggested corrective measures and lessons learned, as identified in the fire reports;
- Develop documented action items for immediate implementation such as removing combustible materials in all SWP facilities (completed during 2013);
- Contract HDR Inc. to address tasks that align with plant modernization;
- Contract consultants who specialize in hydroelectric power plant fire systems, risk reduction methods and the California Fire Code; and
- Develop a modernization plan for enhancing/updating fire systems statewide.
The second project, the SWP Fire Systems Design and Construction, will involve undertaking the design and installation of SWP fire systems needed for modernization.
DWR estimates the Thermalito plant will be restored sometime in late 2018 with estimated cleanup costs amounting to US$90 million and estimated restoration costs of US$135 million.
Note
1Land, H.B., C.L. Edding, and J.M. Klimek, “Evolution of Arc Fault Protection Technology at APL,” Johns Hopkins APL Technical Digest, Volume 25, No. 2, 2004, page 141.
Reference
Roose, David, and Derek Stewart, “Fire at Ronald B. Robie (Thermalito) Pumping-Generating Plant,” Proceedings of HydroVision International 2014, PennWell Corp., Tulsa, Okla., 2014.
Learn more at HydroVision International
Several sessions at HydroVision International, July 14-17 in Portland, Ore., cover fire prevention and safety at hydro plants. Below are a few:
- Session 7E: Power Plant Safety: It Can Save Your Life
- Session 5J technical paper: Protecting Hydroelectric Generator Enclosures with Hybrid Fire Suppression Systems
- Session 2I technical paper: Rim Fire Emergency Response Provides Valuable Lessons Learned to Minimize Risk from Future Disasters
- Wednesday Poster Gallery: State Water Project Fire Modernization
- Friday Poster Gallery: Arc Flash Safety and NFPA 70E Compliance
More HR Archives Issue Articles