Whitepaper
IT/OT Convergence: How to Achieve Success and Avoid Pitfalls
Collaboration between Feyen Zylstra and Belden
Just a few years back, an Information Technology (IT) network and plant floor Operational Technology (OT) network were wholly separate, with the personnel who ran each having very little to do with one another. With the advent of the industrial Ethernet replacing fieldbus protocols on the plant floor, the two networks now share a common “language,” creating valuable opportunities to combine resources and collaborate on goals for overall organizational success. This newly termed “Network Convergence” also sets the stage for challenging interactions that can occur between IT and OT network personnel. With different training, backgrounds, and cultures, merging the two groups can cause a rift that is detrimental to the network’s reliability and efficiency. Fortunately, the extent to which these collaborations become adversarial or collaborative is in large part up left up to the organization and people involved.
In addition, there is also a great deal of misunderstanding about what convergence is and what it entails. These common misperceptions, combined with challenging personnel, create a steep and expensive learning curve. A successful convergence plan combines both IT and OT best practices in order to produce a functional and strategy-driven methodology to approaching your network. Fortunately, as more organizations work to converge their networks, there is a robust and growing body of experiences and best practices available to learn from. In this whitepaper, we’ll explore ways that you can reduce your learning curve while jumpstarting gained benefits of a converged IT/OT network.
The benefits of moving to Ethernet based communications in the OT environment are significant and practical. To many manufacturers, data collection, process visualization, and reporting are primary drivers. With the speed and immediacy of Ethernet communications, operators can collect real time production data that can be applied to create improvements in their operations that translate into improved production margins and quality. For those manufacturers who are not interested in implementing the power of Ethernet-driven tools like data capture, it has become widespread enough that key competitors in their industry are probably already doing it, gaining competitive advantage. In short, the trend is now spreading exponentially.
The value of production data is evident. For example, one of our customers who manufacturers high quality water heaters was looking to significantly increase production to meet their order backlog. They started collecting data on the amount and type of steel that went into each unit in order to identify ways to improve efficiency and speed. Another customer, a firm that treats industrial bolts with functional additives, used data to better understand why bolts were damaged and defective throughout the production process. By applying the data, they were able to significantly increase their yield and productivity along with minimize waste.
Many companies like these are finding that having production data for each individual component is extremely valuable and provides the ability to remedy future issues. Furthermore, many companies are finding that collecting and storing data, even if they don’t have the right questions to ask upfront, is still valuable. Stored data can be analyzed retroactively. Manufacturers might want to investigate something later and having months’ worth of production data to crunch is very valuable in the pursuit of such knowledge.
In the pre-Ethernet days, if data was collected it would be hand captured on a clipboard. If ever used in the future, the operator would be confronted with illegible, transposed digits littered with human error throughout. Moreover, studies have shown that if a production event takes less time to solve than to document, it rarely gets documented. This is commonly referred to as the death of a thousand shutdowns. If a 10 second interruption to production is happening every 10-30 minutes, the aggregate downtime becomes considerable when viewed holistically from a macro level. Using Ethernet to capture and analyze information makes it usable and useful information as opposed to just pen scribbles. Nearly every industrial manufacturer can benefit from data and with the technology becoming more readily available, moving forward is more accessible than ever.
One Network and Selective Sharing
With Ethernet running on both the office (IT) side of the business and the plant floor (OT) side, the days of managing and maintaining two isolated networks makes little sense. Real time business intelligence requires real time plant floor data. Just-in-time scheduling requires updated business schedule needs. Recipe, routing, and quality data are routinely kept in ERP based systems, but require interactions on the plant floor. Thus, we are seeing the emergence of a single segmented network. The idea of one, single network may be a frightening proposition for many. It many conjure up images of strangers dabbling in parts of the network where they have no business, like Sue from Accounting messing with PLC data or Bob from Operations accessing personnel records, not to mention outside cyber security risks.
This is an unfortunate, but common, misconception. Use the Internet as a model. The Internet is one single network, with you, me, and billions of other people using it. Still, I can’t print to your home printer and you can’t access the files off my desktop. And yet, either of these actions can be readily performed through the network if we both agree to it.
Like this scenario, a properly converged IT/OT network is not one flat, open network, but one network fortified and protected so that only appropriate connections along the network are allowed. Selective sharing makes connectivity possible and controllable so that information and resources are accessed only as they should be. Specific data might flow one way, back and forth, or not at all. Selective sharing is a key to an effective converged network.
How to Design a Converged Network
Sometimes the term “converged network” causes confusion. A converged network should not be formed from two existing networks simply connected together wholesale. Instead it should be planned and designed as a “new” network to include both office and plant floor elements, appropriately structured with strategically predetermined interface points.
The first step is identifying what is on each network, where devices are located, and what each device is talking to (or needs to talk to). If you’ve never audited and inventoried your network, you’re likely in for some surprises. OT networks are notorious for growing organically throughout the years, without concern for the holistic nature of the network. This provides you with an opportunity to start with a clean, streamlined, efficient slate.
There may be OT devices that have been piecemealed into the IT network that are mislabeled or uncatalogued. Don’t find out the hard way. An international pharmaceutical company we’ve worked for once had a virus issue. They hired an outside IT consultant to come in and take a look. Throughout his work, he identified a slew of IP addresses that were unaccounted for, so he shut them down. Meanwhile, across the ocean, one of their manufacturing facilities was panicking because their key components were suddenly offline. Since the corporate office never communicated the issue, the manufacturing facility fixed things locally at a great expense. They didn’t identify the true source until months later. Evidently, they had one flat network and people just kept plugging things in as needed without inventorying or selective sharing.
The concept of a handful of OT devices being joined with an IT network is not unusual and is usually only discovered when there is an IT-side incident or shut down. In addition, OT people often find “hidden” communication nodes that are continually connecting to the outside world, not only generating unnecessary expense but opening the channel up to serious cybersecurity vulnerabilities.
Once everything is inventoried, you can identify the purpose of each device, decide what should be talking to what, and create the optimum data flows in each case. These decisions require technical and strategic discussions. The job of keeping everything flowing appropriately—and shared selectively—is the work of firewalls and operations in the area known informally as the DMZ.
Structuring IT and OT and The Space Between
As noted, a network audit will help separate OT machine functions from the IT world and vice versa. This not only ensures that things are appropriately tied to the right network, but also that the proper security protections, resources, and connections are applied.
The Purdue Architecture Model is a solid, simplified, at-a-glance illustration of a basic network architecture. In Levels 4 and 5sit the main IT domain functions. These functions include Accounting, Finance, Human Resources, Enterprise Resource Planning, Marketing Databases, and other data outside of operations. Anything to do with email, telecommunications, and the internet reside here. On the plant floor side of the business in Levels 0 through 3sits everything to do with production. This includes controlling robots and VFDs, monitoring conveyer speed, operating the mixer or extruder, etc.
Gray areas do exist and it’s important to remember that the network a device is tied to depends on what it does, not where it is located. For example, there might be an email terminal or a VoIP phone on the plant floor that should be connected to the IT network, not the OT network. A COO might have a terminal accessing real time production data in his office, although located in the office space it should be tied to the OT network. Purposes should NOT be mixed—the PC on the plant floor with the browser and email function should NOT also be used to gather production data. PCs are not expensive and trying to mix their capacity opens up serious vulnerabilities.
The space between the IT and OT domains is often referred to as the demilitarized zone or DMZ. Physically, this area is a collection of servers and PCs, with information flowing “up” from OT and “down” from IT, directionally protected by firewalls. Data is processed here, and then directed back to the pre-determined location. The information flowing in and out is carefully controlled and selectively shared. The DMZ exists to provide a data transmission layer between the IT and the OT networks, as seen between levels 3 and 4. What resides in level 3 or 4 is dependent on the needs and data flows of the end user. The outside world (WAN) and the IT network should never directly touch the OT network. Conversely the OT network should never directly connect to the IT network or the internet. This method of segmentation provides a more robust and time tested cyber security model. The OT and IT networks while converged should be treated as logically and physically separate with convergence points being monitored and controlled by firewalls or similar technology. If a virus strikes the IT network causing it to shut down, the compartmentalized OT network can continue to run, avoiding costly downtime. The DMZ also helps ensure that production equipment is void of IT priorities, such as scans or upgrades, allowing maximum availability.
The Cultural Differences of IT and OT
With the creation of the Ethernet network, it is inevitable that the IT Department needs to be involved. After all, IT has been working with Ethernet in the office for decades and has extensive experience in technology, specifications, available products, and vendors. With the newer OT Ethernet network converging with the existing IT network, it is inevitable that IT and OT professionals will need to do something that they have never done before—interact.
Many organizations find this to be a serious challenge and difficult to navigate. Due to their different backgrounds, trainings, knowledge, experience, and priorities there seems to be an inherent us vs. them mentality within the groups. It’s important to understand the differences between groups as it serves as the first step towards bridging the gap.
One of the most important things to understand is the difference in priorities between IT and OT personnel. Unfortunately, their priorities are exactly the opposite. To OT personnel, constant equipment uptime is the number one priority, overshadowed only by safety. Confidentiality of data has only recently become a priority. To IT personnel, working daily in a clean, static, climate-controlled, low voltage environment, there is little need to think about safety. If a printer or PC fails, the user goes out for a lunch break and IT brings a new one up from the basement, so lack of availability represents little significant loss. For IT, data confidentiality is a main priority, such as if financial records should fall into the wrong hands.
This difference in priorities shows itself in many ways. For IT, the go-to fix is to reboot, resulting in a period of unavailability that is deadly for production. Performing software upgrades and patches are routine and poise low risk. In OT, patches and upgrades mean taking something offline, and so are reserved to be completed during a scheduled downtime. IT loves to scan the network, a proactive measure meant to ensure that there are no viruses or threats on the network. This scan does not affect PCs and printers, but delicate PLCs on the plant floor can crash from being pinged—another example of how availability must overshadow other concerns on the plant floor.
It is also important to understand the relative urgencies of the functions. IT, for the most part, is a 9-5 operation. OT is a 24/7 operation. When there is an issue in OT, the clock could be ticking at thousands of lost production dollars per minute, so everyone jumps in immediately to remedy the situation. IT usually has no such urgency. You have an issue; you submit a help desk ticket and hopefully it will be fixed to promptly.
Another important difference between the groups is the difference in reference points. Although IT and OT people are both “T” people, their technology training and experience leads them in different directions. As suggested, IT people are cybersecurity savvy and know about the technology and products that keep networks protected. While OT people can code and deploy a PLC, a valuable skill that’s fairly foreign to IT people. With their differing backgrounds, even the “same” language can have a completely different meaning. Once, we were talking with a mixed IT/OT group about “SIP.” It soon became apparent from the dialog that we weren’t all talking about the same thing. One group was referring to CIP as in “Common Industrial Protocol” and the other SIP as in “Session Initiation Protocol.” Miscommunications like this could lead to major problems—like if someone was told verbally to ensure that a new device is compatible with “SIP.”
Read more about the differences between IT and OT personnel.
Overcoming These Differences
So, how can these groups work together? The first step is understanding and acknowledging the differences. This might mean reading the passage above to your team and discussing. Another ideas is having each person explain what they do in a typical day.
Both IT and OT people are smart and knowledgeable, which sometimes means they carry an ego. They may talk to impress each other and if they don’t understand something, they may not feel comfortable asking for clarification. In a converged environment this is an issue. Create an environment where it’s clear that there are no dumb questions—only questions worthy of a sincere answer. Big egos, rattling off jargon, and using unexplained acronyms should be discouraged, and people should be encouraged to elaborate as if they are speaking to non-technical colleagues. Everyone should be prepared to generously teach their area and respectfully listen and learn about the other area, with each side explaining technologies, best practices, and describing the reasons behind their daily workflow needs as appropriate.
Create a Virtual IT/OT Organization
Even in an organization where IT and OT people work well together, the inevitable question of ownership will come up. Does IT or OT have the final word on equipment and operations? Who gets to spec network-wide Ethernet equipment?
The answer to these questions often has to do with company culture. Some may appoint IT since they are the traditional Ethernet experts, others may appoint OT because production and operations drive bottom line. Whichever way, this may lead to a winner/loser mentality. A better way to view responsibility is by creating a dotted line organization, frontloading universal buy-in from both IT and OT, at all levels of the organization. IT/OT convergence is not only about converging the networks, but just as important, it is about converging the specializations. Both sides of the firewall are experts in their areas and a successful convergence plan leverages this aspect. IT and OT are stronger together and a company that leverages both skillsets into a single force will be served far better than one that does not.
In most organizations this starts with support from the c-suite. Having the head of IT, generally the Chief Financial Officer, as well as the head of OT, generally the Chief Operating Officer, partner together and express their joint support for all IT/OT convergence activities.
In the same vein, the day-to-day functions of the network can be approached through a hybrid model. A joint task force that meets regularly and includes key members from OT, IT, and related disciplines is important to accomplish this. The National Institute of Standards and Technology (NIST) recommends that individuals in the discussion should include at a minimum:
- A member of the IT staff
- A control engineer
- A control system operator
- A network and system security expert
- A member of the management staff
- A member of the physical security department
- A process engineer
- An operations manager
- A quality manager
When it comes to appointing a leader of the taskforce, we suggest having the group search for and select a professional who understands firsthand both IT and OT functions, priorities, and language. This individual, often called an Automation and Data Exchange (ADX) Engineer, must be cross-trained in both OT and IT practices. This may be a networking engineer who has spent time working on the plant floor learning about automation operations, or this could be an automation engineer who has completed networking classes and trainings. Led by the ADX Engineer, the task force should have governance responsibilities for all things related to the network. One of their early duties should be creating and posting standard operation procedures for the converged network. The group also owns all convergence establishment and maintenance activities.
A common false start is to exclude operations, process, and quality in the strategy discussions. These groups are the network’s main customers and their requirements must be met. The networks exist to support operations, operations exist to make product, product is made to generate revenue. Revenue is the backbone to corporate survival. Any successful strategy must always remember that the network services operations and not vice versa.
In the case that you are a multi-location organization, we recommend starting with a “pilot project” at a smaller location and apply key learnings to additional locations. After assessing the extent of the convergence challenge at each location, you can also decide, case by case, whether internal resources possess the expertise needed to tackle each project.
Encouraging Cooperation
In the drive for successful IT/OT convergence, it is likely that issues will arise. We’ve seen individuals refuse to cooperate, be secretive in their dealings with outside partners, or refuse to bend for the better of the group. So, how do you deal with an uncooperative team? To start get involvement by the c-suite executives helps set the precedence for cooperation.
Another effective strategy is involving a third party to lead the charge, draw upon experience, flatten the learning curve, and avoid the many costly pitfalls. This third party should be equipped to deliver both technical and psychological assistance and understand and speak the language of IT and OT. By bringing in someone with a proven track record of understanding IT and OT immediately, the organization is set up for IT/OT convergence success.
Conclusion
The convergence of IT and OT networks on a single Ethernet network is inevitable for companies that wish to maximize the benefits of connectivity, while also optimizing efficiency. While this convergence may come with challenges and growing pains, it’s benefits and the competitive advantage is undeniable. Should you wish to speak to an expert on how to best approach IT/OT convergence within your organization, we would love to help.
Founded in Grand Rapids in 1980, Feyen Zylstra is a knowledge-based electrical services and industrial technology firm focused on solving complex problems associated with the design, installation and maintenance of electrical, low voltage and automation systems. Over the last 41 years, Feyen Zylstra has grown to employ more than 550 people at six locations throughout the states of Michigan, Tennessee, South Carolina and North Carolina, and consistently works to be the first-choice electrical contracting and plant floor technology resource for the most demanding industrial and commercial applications in the marketplace.
By combining business know-how with the technologically advanced products of their leading brands, Belden provides signal transmission solutions that help make their customers operate faster, better, longer, safer, and more economically. Their combined product portfolio includes data cables, connectors, I/O modules and network equipment. Belden is not just a supplier, but a strategic business partner delivering solutions that help their customers perform better. Belden Inc., a global leader in high-quality, end-to-end signal transmission solutions, delivers a comprehensive product portfolio designed to meet the mission-critical network infrastructure needs of industrial and enterprise markets. With innovative solutions targeted at reliable and secure transmission of rapidly growing amounts of data, audio and video needed for today’s applications, Belden is at the center of the global transformation to a connected world.
