Seven Strategies for Bullet-proof Substation Networks
The joint imperatives of 9/11 and the smart grid have created a massive amount of power utilities development and retrofit activity.
The joint imperatives of 9/11 and the smart grid have created a massive amount of power utilities development and retrofit activity. The need to protect the security of power installations and the data that is passing in increasing quantities within and among substations and central offices is high. The smart grid requires two-way communications with users to encourage smart use of power. Opening communication while protecting user privacy and transmission security adds another layer of complexity.
Figure 1 displays the smart grid opportunity-and the challenges. A simple operation suddenly became more complex with two-way communication and multiple stakeholders replacing a simple one-way transaction from an omnipotent and, from the user's standpoint, arbitrary source. Control functions increased in complexity, but non-operational data management also dramatically grew; because it was transported far beyond the boundaries of a single facility, issues including timing and security had to be addressed at a more comprehensive level.
To address these issues, knowledgeable substation managers use the following strategies as the basis for a comprehensive Internet Protocol (IP) implementation program that consists of high-reliability, high-performance and secure solutions for substation management.
1. Use scalable bandwidth to handle the steadily increasing demand for data.
Figure 1 shows that the increased component sophistication and increased pressure for automation mean additional bandwidth planning is critical. Fiber backbones are a basis of most large-scale data management strategies because of fiber's properties in providing high bandwidth over long distances, noise immunity and inherent security features. Fiber is also flexible enough to support the installation of new nodes as demand on the network increases. With increased acceptance, coupled with the rise in copper costs, fiber is seen as a cost-effective and secure alternative to dedicated T1 or dial up lines, and it is well matched with IP infrastructure solutions.
2. Specify industrial-grade switches and routers to support expanding demands for more equipment attachments.
Reliability is the number one attribute for substation managers. Industrial-strength networking equipment is required with extended temperature ranges, along with a strong electromagnetic compatibility (EMC) design to protect against electrical magnetic interference (EMI), convection-cooling, shock and vibration resistance, fiber configurability to support security and high-bandwidth demands, dc power and ac to support installation in areas requiring specialized power sources, and redundancy options to ensure high availability.
The smart grid infrastructure requires increasing numbers of nodes to serve the proliferation of substation devices that need to be included in the control and monitoring networks. Figure 2 shows the increasing number of intelligent IP-enabled devices available for connection-from sensors and monitors to security devices such as video cameras, card readers and intelligent access control devices including fingerprint or iris scanners.
To support data and control systems demand generated from increased substation complexity, designers must be able to choose Ethernet switches and routers equipped with varying ports. At the core of the network it is inefficient and expensive to pile multiple low-port-count switches together, wasting two ports per device for connectivity. In addition, this practice adds unnecessary potential failure points. Where larger port-count devices were once only deployed in climate-controlled central offices, today one sees installations of 24-port and 36-port switches at the nerve center of the substation.
3. Integrate wireless communications for simple, cost-effective data links to remote sites.
Smart grid networks often need wireless connectivity to support the larger grid and demands within specific facilities. Distributed alternative power generation resources, as well as the need for two-way communications at users' meters, often require wireless connectivity support. Wireless is not a monolithic concept, and the broad variety of wireless connectivity options are beyond the scope of this article. It is important, nonetheless, in planning a network to ensure wireless connectivity is an option-at least at the router level.
4. Integrate serial equipment into the IP network-it's not going away any time soon.
Power utilities serial equipment is here for the long run because infrastructure investments are expected to have life-spans of up to 30 years. While some utilities might have greenfield projects where they are deploying fully IP-based networks, most will be using legacy serial components, and, possibly, newly purchased serial devices for years to come.
IP technology advances are making it possible to more fully utilize and integrate serial data and include it in IP security protocols. For this reason, ease in connecting serial devices into the IP architecture is a high priority. Terminal servers and routers that support Ethernet and serial devices reduce complexity and provide greater security options. See Figure 3.
A typical substation will have intelligent electronic devices (IEDs) and other equipment outfitted with a wide range of standard Ethernet and serial connectors. Modular technologies that support the mixing and matching of blocks of ports on individual switches and routers provide cost-effective and easy-to-deploy alternatives to fixed-port boxes.
5. Upgrade to equipment with precision timing features to enable synchronized data management and control actions.
Continued integration has made precision timing more important. Most of us are aware of the communication challenges that result from coordinating different time zones, especially since some states don't follow daylight savings practices. With mission critical applications, it is necessary to have tight synchronization of all devices. The time stamps on data from various cameras and intrusion detection devices, for example, must be synchronized to a universal clock to support forensic security work to ensure the input sequencing from multiple security devices is exact. When there are operational events, it is equally necessary to ensure comparisons of data-even from serial devices in the network-are based on a single time standard.
IRIG-B, developed by Inter-Range Instrumentation Group, the standards body of the Range Commanders Council, is a one-time code standard that makes it possible to synchronize geographically separated instruments throughout a power delivery system.
6. Integrate a plan for cyber security and physical security to keep control networks safe.
In addition to the challenges of ensuring consistent system-wide timing synchronization, flexible access to information in a distributed environment creates security issues that must be addressed to ensure the integrity of the operation. Stringent North American Electric Reliability Corp. (NERC) mandates require security network components such as:
• CIP-002: Critical cyber asset (CCA) identification-requires identification of switches, routers and data concentrators with access to the outside world;
• CIP-005: Electronic security perimeters-requires switches and routers with access to the outside world to be protected by access control applications, such as firewalls;
• CIP-006: Physical security of CCAs-typically requires an integrated cyber and physical security strategy to protect the communication cabinet and the supervisory control and data acquisition (SCADA) cabinet, and the entire plant;
• CIP-007: System security management-includes test procedures, ports and services, patch management, prevention of intrusion by malicious software account management, and security status monitoring via syslogs; and
• CIP-009: Recovery plans for CCAs-includes change control and basic recovery kits or protocols.
The momentum toward shared data networks offers the potential for higher operational efficiency and distributed decision-making, so cyber security issues are here to stay. In addition, as the Stuxnet computer worm proved in 2010, even unconnected systems can fall victim to attack caused by employees intentionally or unintentionally downloading malware at unsecure systems on memory sticks. Developing a strong cyber and physical security strategy is critical in today's world. Figure 4 shows a network that is open to attack. Figure 5 shows the same type system with a stringent physical and cyber security layer inserted.
Cyber security starts with physical security. If outsiders cannot gain access to the premises, it is harder for them to access sensitive data. On the data side, it is necessary to protect cyber assets with firewalls at the cyber perimeters of critical cyber assets, just as the physical perimeter is protected. Other prudent strategies include disallowing unauthorized devices to be plugged into ports on switches and routers, fiber cabling (previously described) and encryption for data transmission between secure facilities. Virtual private networks and virtual local area networks (LANS) provide extra security layers for transmissions over multi-purpose transport networks.
Other prudent practices include authorization: knowing the right password, a strategy that is enhanced when employees or system administrators change passwords regularly; making sure passwords are long enough and complex enough to be difficult to crack; and authentication that goes one step further by ensuring the person or device requesting access is who they say they are.
Finally, security is only as good as the policies and practices in place. Employees, without meaning to create a security breach, can be lax with passwords, security codes and other primary measures unless they are educated and reminded of the importance of security.
7. Bring corporate information technology into data management as a partner.
As the security discussion made clear, substations are increasingly hard-pressed to seamlessly integrate data flow with corporate information technology (IT). While conflicting priorities and needs have traditionally made the two groups friendly adversaries at best and outright enemies at worst, there are growing reports about how the two groups have collaborated to bring about the best results. Simply put, the two groups have different goals and objectives in many cases-precision timing issues and maintenance schedules in the substation can conflict with corporate information flows. Multi-discipline workgroups, however, are identifying and solving these types of problems-and providing more information and greater efficiencies across entire systems.
Creating an efficient network is a work in progress. Progress is measured in increments and phrases-from quality circles and continuous process improvement (CPI) to the planned phasing of NERC-CIP requirements to the practical demands of resource planning. In the latter case, it is rarely feasible to implement the entire overhaul of substations that have hundreds of thousands or millions of dollars invested in equipment that has not reached the end of its life cycle. The combination of NERC and smart grid initiatives requires a major review of assumptions and objectives in collecting, managing and analyzing data. The seven lessons discussed, consequently, become increasingly critical to success.