Presented By:
Byron Shovlaini, P.E.
San Deigo Gas & Electric
TechCon 2019
Abstract
San Diego Gas & Electric Company (SDG&E) was an early adopter of Condition Based Monitoring (CBM) in 2008 before wide scale development of such programs. Safety and reliability are core values at SDG&E, which drive the use of technology to monitor and maintain equipment. Without a roadmap, SDG&E Substation Maintenance and Operations department worked with other departments, field crews, contractors, and equipment providers to develop their own program. This presentation gives insight into how the program came to be, how it was done, challenges, results, ongoing development, and thoughts on what could have been done better. By sharing this information hopefully other utilities can learn something about developing their program and by giving feedback on how they do their program SDG&E can benefit as well.
Introduction
SDG&E’s expansion in the mid-2000’s of its Supervisory Control and Data Acquisition (SCADA) communication infrastructure for remote monitoring and control of load and equipment to improve intelligence and operability gave idea to leverage other technology as part of the Operational Excellence 20/20 initiative.
As substation infrastructure started aging and needing replacement, tightening resource constraints demanded a better way to maintain equipment, prioritize repairs and replacement rather than the traditional time-based. The idea of using new technology to improve safety, reliability, be more efficient, and save money for maintenance and replacement strategy was embraced by SDG&E’s upper management under the Operational Excellence 20/20 initiative.
In 2006, we started researching using CBM concurrently as we upgraded our operational monitoring and control system. We had a big bubble of transformers with an average age of 55 years old that needed replacement with lead times of up to 2 years. Our business case leveraged prolonging the life of an asset and delaying replacement while preventing failures, resulting in cost savings.
Our time-based maintenance intervals used manufacturer’s recommendations in conjunction with our experience with equipment robustness, loading, operations, and environmental conditions. We had been routinely conducting annual manual oil samples and performing periodic dielectric testing but wanted a better handle on incipient conditions.
In 2007, looking to expand system-wide, we started with a vision statement and asked stakeholders for a wish list of capabilities and a minimum of what they could live with. The project team included engineering, planning, telecommunications, asset management, system operators, information technology, information security, construction and maintenance, and system protection. Most importantly, all the different departments managers were on board to support the project collaboratively. We had weekly meetings with all parties to lay out our vision, track progress, and fix problems. Having a project manager to coordinate the work and be the face of the project allowed subject matter experts autonomy to do their tasks.
Our initial research started with interviewing other utilities for best practices and advice, which only a few had even piloted monitor systems and none had deployed system-wide which we hoped to do. We chose the initial equipment based on the level of vendor support and their willingness to work with us. We sought equipment that had minimal operating and maintenance expenses. We did not find turn-key solutions viable and so went to work assembling and testing our own system in a mock-up lab situation integrating all the equipment, wiring, and back office technology to help work out as many bugs beforehand and test different equipment. Now, new transformers are purchased with pre installed monitors.
In 2008, we chose to pilot the new equipment at two substations to develop material, design, construction procedures, implement back office data collection, notifications, and data analysis. We discovered more efficient ways to do things like routing the signal back to the control house by way of a magnetic coupler versus a more complex and expensive capacitive coupling. The output of the monitors was wired into the control house using power line carrier technology, avoiding the need to install new cable or dig trenches, and into the annunciator panel to send alerts.
The biggest issue encountered in the field was that an outage was typically required to install equipment and so load profiles influenced timing. Calibrating alerts was a big issue to get over false positives and crying wolf. Today we have established dissolved gas baselines and sampling rates and modify them for individual equipment with known issues.
We did not have enough regular staff to do a system-wide deployment, so we partnered with a design firm to do location specific field designs, performing pre-job walks with the contractor which helped refine the designs to be more complete. We sought committed crews from the installation contractors for the duration of the contract which we helped train based on our pilot project. We long negotiated price and leveraged the fact that the project included over 300 transformers in more than 100 distribution and transmission substations over a projected 7-year period. As the project progressed, the remaining substation installations were continually re-prioritized as needed, based on equipment maintenance history.
We tried three different versions of dissolved gas analysis (DGA) monitors. We looked at solely hydrogen, 3-gas monitor, and 8-gas monitor, with and without load-tap changer (LTC) monitoring. Due to the added expense with LTC monitoring, we chose to monitor only certain LTC’s based on deteriorated LTC manual oil lab samples. We also desired to monitor bushings, which were responsible for a few transformer failures. Typically, we only tested bushings and transformer insulation on an 8-year cycle. Further measurements we desired were loading, oil temperature, moisture, cooling, LTC operations, and later, battery status. This would also allow us to know and forecast the effects of dynamic and emergency transformer loading. We continue to perform manual oil samples which give additional information such as color, acid, interfacial tension, and furan count as a good check against the monitored DGA.
The CBM system was developed with its own secure enterprise gateway and communication path to separate it from the SCADA system and relay information as we were aware of future Critical Infrastructure Protection (CIP) data security requirements.
At the time we started, there was no commercially available software package to bring all the data back and so we worked with a vendor to develop such a system but found it difficult to do. We wanted the data to go into our OSI-PI data historian and existing maintenance management system. We found much erroneous information causing mismatches, which took more manual data cleanup of serial numbers, model types, locations, and other data, than we expected.
The second phase of the project RFP in 2010 sought visualization software that had the ability to select multiple data points, graph them, create reports on demand, represent with Duval triangles, export, and analyze criticality, thermal loss of life, health, and risk. Originally, we used Excel spreadsheets and PI graphing of the incoming data. We wanted to have email alert notifications for abnormal operation, so we could respond real-time but also a dashboard that would track and prioritize trends and anomalies that would be available to stakeholders. We have worked with a couple of software vendors in the last few years to develop better analytics with more features that we feel give us the best use of our data.
Because of implementing CBM, we have taken advantage of the efficiencies and cost savings such as extended time for manual oil samples, major overhauls of gas circuit breakers, and major overhauls of LTCs. We eliminated doing time-based transformer insulation testing and instead test for baseline and as needed. We do not have to take equipment out of service to test it and we can exercise it remotely further reducing maintenance costs. We continue to upgrade our monitor system based on regulatory compliance, for example, California’s SF6 gas release standard and NERC PRC-005 battery monitoring which caused us to install monitors on SF6 gas circuit breakers and on substation battery banks.
Making the data valuable demands diligent review of alerts and trends. Real-time PI email alerts from monitors allow maintenance supervisors to respond quickly to developing situations, sometimes faster than notifications from our system operators. It is also necessary for someone knowledgeable to interpret the data to see if it is truly valid. There are many excellent seminars and training on interpretation of monitored values. We consult our vendors often for their opinion and interpretation as well and check against prior maintenance history, manual oil samples, and insulation tests. We believe we have prevented imminent failure of transformers in a few cases by taking them out of service preemptively based on monitor data that showed rapidly increasing bushing power factor or LTC coking and resultant gassing.
We continue to mature our analysis and use of the thousands of data points from our monitors to evaluate asset health. We work with our analysis software vendor to refine data alerts, reduce unnecessary notifications, add desired data on the dashboard to supplement existing data and correlate it better, and tweak data weighting in formulation of a health and risk ranking.
Hopefully, our experience may benefit other companies in the development and refinement of their monitoring and condition-based maintenance programs. Some of the important lessons learned are to involve all stakeholders in the organization, ask for a wish list and define what is possible, get buy-in and commitment from all managers, ask other utilities for their experience, find equipment that works with legacy systems as much as possible, find vendors that are willing to work with you in development, deployment, and support; have committed personnel for installing monitor equipment, programming, repairing, interpreting data, and tracking trends; proceed incrementally away from time-based maintenance towards condition-based maintenance; attend conferences to learn more, and tout your successes.
References:
- T&D World, September 2010. “Equipment Condition Dictates Maintenance”. Neal Bartek P.E., San Diego Gas & Electric Co.
- Doble 80th Boston Conference, 2012 presentation. “Transformer Operations and Loading A User’s Perspective”. William E. Yturralde, P.E. and Kristina Lukin, San Diego Gas & Electric Co.
- T&D World, January 2013. “SDG&E Takes Proactive Approach to Maintenance” Janet Lonneker and Brent Baker, San Diego Gas & Electric Co.
- TechCon Asia-Pacific 2013 presentation. “Deployment of Monitors System Wide for Condition Based Maintenance of Substation Equipment”. Terry Snow, P.E. Dynamic Ratings Co.
Acknowledgments:
- Brent Baker, Construction Supervisor; San Diego Gas & Electric Co.
- Terry Snow, Principal Engineer; Dynamic Ratings, retired from San Diego Gas & Electric Co.
- Sherwin Yari, Engineer I; San Diego Gas & Electric Co.
