SANS, Interstates, Veracity, Dynics and INL detail software-defined networking and other strategies

BY JIM MONTAGUE

IT can seem like a disadvantage that cyber-threats, -probes, -intrusions and -attacks are always changing. However, this changeability can also be a benefit because cybersecurity solutions can also evolve at an equally rapid clip—if developers and users are proactive in creating and applying them.

“Industrial cybersecurity is meant to allow processes and machines to run as intended, and anything that interrupts them malicious or not—falls under the definition of a cybersecurity event,” says Jeffrey Shearer, coauthor and cybersecurity instructor at SANS Institute (www.sans.edu) and chief automation officer at Morris Thermal Solutions (morris-coolideas.com/process-cooling-solutions). “Software programs found in embedded systems like the PLC or HMI and those found on a computer system, which are connected or related to the machine or process, can be used to create a cyber-incident by simply misusing the poorly written code or misusing the network in a poorly architected system against the machines or processes.”

Shearer adds that users must ask themselves, “Did I program the machine or process to do the right thing?” and “Did I program the thing right?” This is an important difference because programmers often don’t fully understand what a machine or plant-floor process is supposed to be doing.

“They often don’t spend time with the mechanical or process engineering teams to understand use cases and especially abuse cases,” explains Shearer. “As programmers, we need to evolve to the point of asking not only how to make something work, but we should also ask how do we spot abuse cases and guard against them with the follow-up question, ‘What should we do when we’re in an abuse scenario?’”

Easier eyes-on automation

Because humans can’t be vigilant enough against the 24/7 deluge of cyber-probes, intrusions and attacks, some cybersecurity solutions are becoming as automated as the processes and networks they protect, including some that are transitioning from virtual local area networks (VLAN) to software-defined networking (SDN).

For instance, the prefabrication shop at system integrator Interstates (www.interstates.com) in Sioux Center, Iowa, has manufacturing requirements that used to make its regular enterprise network vulnerable and made it difficult for staffers to do their jobs. Some of the shop’s machines needed building information model (BIM)-generated data files to drive manufacturing processes, while third-party vendors needed remote access to provide software support and troubleshooting. This meant giving the machines and vendors network access, even though Interstates’ operators-technicians and IT department didn’t always know when other parties were in the system or what they were doing. Vendors could gain remote access, potentially reach other resources on the enterprise network, and even view the activities of different vendors.

“Historically, we’ve used VLANs for segmentation, and deployed firewalls above the controls network. However, this only got us so far in handling what was going on within our environment because we couldn’t control communications within the VLANs,” says Dave Smit, senior systems analyst at Interstates, which is a certified member of the Control System Integrators Association (CSIA, www. controlsys.org). “We could add an access control list (ACL), but they’re often messy and hard to manage. Over the last three or four years, we’ve seen lots of customers adding cybersecurity inspection software such as Nozomi, Armis or Claroty to monitor network traffic. However, these products typically only provide visibility, not the ability to control east-west traffic.”

To further improve its cyber-awareness and protections, Interstates has also worked with Veracity Industrial Networks (veracity.io) for several years, and participated in developing its OT Network Controller software, which intuitively manages an SDN-capable switch from Dynics Inc. (www.dynics.com),and uses zero-trust and deny-by-default methods to improve network visibility and security. The controller also manages switches, microsegments network traffic, creates device-based firewalls on endpoints like PLCs and HMIs, supports OT Ethernet-based protocols, and presents data in formats preferred by plant-floor personnel. It’s been running at Interstates’ prefab shop (Figure 1) for a year and a half and is presently being added to its facility in Omaha, Neb.

Jeff Smith, CTO at Dynics, explains that SDN-capable switches employ an agent that supports OpenFlow, an industry standard SDN management protocol. The agent replaces the switch’s control plane, which is generally unique from vendor to vendor and often switch-family to switch-family. The control plane is where the switch is configured, either via a command line interface (CLI) or a web-based interface.

“If a switch supports OpenFlow, the control plane becomes nothing more than an agent that interfaces with the SDN controller. The agent, at the direction of the controller, determines how the switch will behave and what it will do,” says Smith. “Because SDN is deny-by-default, it prevents communication until it’s allowed. By regulating these flows between devices, it controls communications point by point, and micro-segments the network, far exceeding the granularity and security offered by traditional concepts such as VLANs. This network is controlled by an SDN controller that uses Dykstra algorithms to calculate the fastest path between two endpoints. This allows automatic redundancy and recalculation of flows, lays in flows automatically, and provides nearly real-time network status at the controller, so users don’t need to monitor the protocol and network operations.”

PROTECTS INL’S CYBER PROACTIVELY -INFORMED ENGINEERING

Just like rustproofing, it’s better to think about and adopt cybersecurity ahead of time than treating it as an afterthought and trying to add it later.

For instance, besides diligently evaluating what type and how much cybersecurity is needed, Idaho National Labs advocates using Cyber-Informed Engineering (inl.gov/cie) to design cybersecurity into equipment, process applications and networks early in their lifecycles. Similar to process safety for physical risks, CIE uses design skills, physics and think-like-an adversary practices to engineer out security risks, and emphasizes the partnership needed for designers and engineers to work with cybersecurity professionals to determine possible and worst-case consequences possible from cyber-attacks and related failures.

“CIE started as a philosophy inspired by Mike Assante at INL with a series of research projects in conjunction with the U.S. Dept. of Energy (DoE) to test systems and find and solve vulnerabilities through engineering approaches as opposed to just applying patches,” says Virginia “Ginger” Wright, the Energy Cyber Portfolio Manager at INL’s Cybercore Integration Center (inl.gov/cybercore) . “It shifts the focus from seeking a completely vulnerability-free system to understanding that any digital system can fail or be subverted, and that cyber hygiene can’t mitigate all threats. This allows practitioners to concentrate on engineering out as many weaknesses as possible, early in the systems engineering lifecycle.

Wright reports that maturation of CIE is guided by the DoE’s National CIE Strategy, released in mid-2022. The five pillars of the strategy drive INL’s research and development of the body of knowledge around how CIE can be best applied in different organizations, and determining where mitigations can be applied. The five pillars are:

• Awareness to promulgate a shared and universal understand of CIE;
• Education to embed CIE into formal education, training and credentialing;
• Development to build the body of knowledge by which CIE is applied to specific implementations;
• Current infrastructure that applies CIE principles to existing, systematically critical infrastructure; and
• Future infrastructure that conducts R&D and develops an industrial base to build CIE into new infrastructure systems and emerging technologies.

In addition, INL is taking its CIE strategy to engineering schools to incorporate fundamental cybersecurity education that most don’t receive in their engineering curriculums. In fact, INL has already been working with Auburn and the University of Texas, San Antonio, to incorporate CIE in their programs “INL’s patented Consequence-driven Cyber-Informed Engineering (CCE) methodology is the first operationalization of CIE principles to go into widespread use,” explained Sam Chanoski, a technical relationship manager with INL’s Cybercore Integration Center. INL conducts training and engagements for CCE customers, and licenses CCE to selected practitioners. For example, water engineering firm West Yost (westyost.com) recently became the first organization to license the CCE methodology, which will allow it to help U.S. water utilities protect their operations from cyber-threats.

Likewise, starting this year, INL will launch a “community of practice” to educate users about CIE, and develop cybersecurity mitigations for their applications and facilities. Its first product will likely be an implementation guide to walk engineers through the process of developing a cybersecurity program for applying CIE principles to their work, including guidance on who needs to participate and what data is needed, identifying security considerations early in the engineering lifecycle, mitigating risks, deciding how to handle risks that can’t be mitigated, and tracking and trending progress for continuous improvement.

Smit adds, “The advantage of SDN and Veracity’s OT Network Controller is that we don’t need to do another VLAN segmentation, which is usually complex and requires a lot of overhead and management. Plus, SDN lets us go back to a flatter network that needs less maintenance ,but has more cybersecurity benefits than a regular VLAN. Now, we don’t just control north-south network traffic, but also east-west traffic thanks to per-device microsegmenting. With SDN and OT Network Controller, one device can talk to another, but only via a pre-authorized defined protocol, which creates a network-based firewall for every device. We can also allow devices to communicate on a very small subset of ports. For example, we can restrict Server Message Block (SMB) client-server protocol for file sharing between only approved devices. With SDN switches and networking, as soon as anything is plugged in, we know where it is and when it happened.” Past informs the future

Similarly, even though industrial network connections keep multiplying, the vulnerabilities that go with them can still be understood and addressed as failure modes, safety issues and process interruptions. Tim Conway, industrial control system (ICS) curriculum director at SANS Institute, adds that thinking about cybersecurity as an important component in process safety programs is important, especially after some of the lessons learned from the well-known Trisis malware attack against a Triconex safety controller in 2017 at a Middle East petrochemical facility.

“Similar to traditional physical areas of concern like equipment failure, degradation, storm-related impacts or animal induced problems, cybersecurity issues are also failure modes that can planned for and potentially designed around,” says Conway. “Physical faults, misconfigurations, and human error can often be the culprit behind process and system failures, but that doesn’t mean there was malicious intent behind them. Adversaries and defenders alike can learn from these failures”

Conway concedes that most process applications and plants have grown more complex and interconnected, so they can’t be manually operated as easily as 30-40 years ago. However, to maintain defense-in-depth despite their increasing connections, they must recognize that their cybersecurity tools need to access their operations environments, and that probes and intrusions will likely follow.

“Many organizations are adding two factor authentication systems and endpoint antivirus software like Symantec, McAfee, or asset monitoring solutions like Solarwinds, but all these tools must have a pathway and two-way data access to devices that defenders care about or they wouldn’t need cybersecurity,” explains Conway. “However, these hub-and-spoke solutions can also provide the same trusted path route for potential attackers.”

To close these avenues, some users with mission-critical processes in nuclear and military settings physically isolate their equipment and networks with data diode hardware and single-strand, fiberoptic connections that can only transmit data outward. Conway reports this method is still more secure than publish subscribe communications protocols like MQTT that utilize bidirectional digitalized pathways that can still be manipulated.

SDN separates, controls by software

Smith states that SDN isn’t an application that runs on the network—it is the network, so it can control east-to-west connections across switches unlike purely monitoring solutions or most other OT cybersecurity solutions. All this happens at Layer 2, so SDN can work with any industrial protocol such as EtherNet/IP and Profinet. “Users don’t need to use VLANs or enable spanning tree protocols such as RSTP, MSTP, or PVST+, and it’s 100% secure out of the box,” says Smith. “SDN has been used in the enterprise and cloud for years, but Veracity Industrial Networks is the first to apply it to the OT space. SDN is standard IEEE Ethernet; the only difference is how the network is managed. Any application that runs on an IEEE ethernet network today, will run on an SDN managed network”

Once the prefab shop at Interstates installed OT Network Controller, it achieved several objectives:

• Isolating separate skids, including a Scotchman cold saw, Haco press brake, ShopSabre CNC plasma table, and a ShopSabre CNC router;
• Isolating the facility’s manufacturing network from Interstates’ corporate network;
• Controlling access to and from particular aspects of the network;
• Controlling third-party access to the shop’s equipment; and
• Meeting the cybersecurity requirements of the company’s insurance carrier, which requires separating production and business networks “Veracity OT Network Controller lets us isolate and limit communication between our skids, so vendors can only access their own skid,” says Smit. “Because we installed and configured this solution, we can support it long-term with a nuanced understanding of the initial configuration and SDN learning modes. The goal, however, is for SDN to be simple and easy to manage once set.”

In addition, Smit reports that using Varacity’s solution at the shop reduces the risk of cutting over a single station to the new network. “This is important because at least one of the skids we worked with was a high-volume machine that we didn’t want to take down for an extended period,” he explains. “When the time is right, the new switch can be wired up and ready to go with simple steps. Shop workers can put the switch in learn mode at lunchtime, and no longer worry about having the VLANs configured on all the ports or making sure they plug the right devices into the exact right port.”

Finally, end users at Interstates and its partners haven’t noticed any technical issues after switching to SDN. “In fact, it improved and simplified their work,” says Smit. “Previously, they’d take a USB key, download the files they needed, and then physically run it out to the skid, which was risky and inefficient. Veracity OT Network Controller lets them securely access the file they need on the network.”

Closer connections, more IT interest

Likewise, Shearer reports that SANS is seeing more IT professionals showing interest in getting educated about cybersecurity paths into ICSs and assuming more responsibility for securing those paths.

“Because the pandemic caused more connections due to more remote work, more users who aren’t as familiar with their plant-floor are realizing they need to support it,” says Shearer. “Another reason for this interest is that cloud computing services like Amazon Web Services (AWS) and their counterparts use a lot of data processing and these data centers are dependent on ICS systems. For example, these data centers rely on cooling and power systems being always available but we find these systems are programmed and supported by ICS teams without being part of the IT cybersecurity program. When an ICS cybersecurity incident occurs and it affects IT systems, then the response to involve the IT cyber team. We need to identify these ICS and IT reliance relationships and include them into a broader cybersecurity plan.

We’re also seeing an explosion of wireless and pervasive sensors that are a dime a dozen, and they’re creating even more connections and producing more data, too. Again, this means more remote applications, more logs and managed services at the center, and more work for the cloud and the ICSs.” While not all data is sent to the cloud, nor should it be, we do need to accept that this architecture is real, it exists, companies are doing it so we, the ICS cybersecurity teams need to figure out how to support these requirements in a secure manner.

Fortunately, cybersecurity tools for ICS have also been improving, and aren’t just relying on traditional IT based detection signatures or using active directory for basic authentication, but with available security offerings asset owners can now be tailored to the individual needs of the applications and users implementing them. These include authentication packages in Rockwell Automation’s FactoryTalk software and similar solutions from ABB and Siemens, as well as third-party software solutions from Dragos, Nozomi, Tripwire and Industrial Defender, according to Shearer.

In conjunction with these software tools, users are often told to follow cybersecurity standards such as IEC 62443 or guidelines such as the NIST Cybersecurity Framework. Unfortunately, Shearer reports these directives are mainly being adopted by large organizations, and encompass so much content that many smaller users avoid understanding how to use it effectively and simply don’t apply them. SANS suggests that users instead follow its five ICS Cybersecurity Critical Control (www.sans.org/white-papers/fiveics-cybersecurity-critical-controls).

“All of this starts with asset owners doing an inventory and cybersecurity risk assessment (RA), and identifying what’s critical,” says Shearer. “While manipulation can some from direct physical connection, most attacks that we see come from probes and intrusions via networks. This is why the most common defense and most solid response comes from our five principles: ICS incident response, defensible architecture, ICS network visibility and monitoring, secure remote access, and risk-based vulnerability management. This is how to build a threat informed defense.”

“The advantage of software-defined networking (SDN) and Veracity’s OT Network Controller is that we don’t need to do another VLAN segmentation. With SDN switches and networking, as soon as anything is plugged in, we know where it is and when it happened.”

This article was previously published by CONTROL Magazine in the June/July 2023 issue.