Operational Technology (OT) networks are the backbone of essential systems—lighting, HVAC, elevators, and entry systems—making any unplanned downtime a critical event. Unlike IT networks, it’s unlikely a campus, hospital, or corporate data center has access to redundant building systems. The risks are manifold, not only to productivity but also to the health and safety of your occupants.
When designing, deploying, and maintaining a large OT network, many facilities teams will bring on an external systems integrator (or several) to help manage and maintain the system. Can these two teams work together, not only during regular operations but also when the chips are down in a crisis?
Turns out, there are a lot of opportunities for a collaborative crisis response, from paper plans to ongoing training and drilling. When internal teams and external vendors work together effectively, they can streamline crisis management, reduce downtime, and mitigate costs.
Let’s look at some of the ways these two teams can benefit from acting together in an OT network crisis: Preparing a collaborative crisis response before an incident occurs, reacting rapidly and with a plan during an event, and working together post-crisis to measure outcomes and prevent future emergencies.
Before: Build a Foundation
1. Create an Emergency Response Playbook
Developing standardized emergency response playbooks with input from both internal teams and external vendors is essential for efficiency. These playbooks outline each party’s role in specific downtime scenarios specifically relating to your environment, helping everyone transition seamlessly from planning to action. By assigning clear roles and responsibilities ahead of time, teams can act quickly and avoid confusion during an emergency.
Likely, your systems integrator has already gone through this exercise with other clients. If not, the US National Institute of Standards and Technology (NIST, PDF link) and the Canadian Department of Public Safety provide free best practice guides for creating incident response plans and protecting OT networks.
2. Proactive Emergency Drills
Routine emergency drills allow both teams to practice their roles, test communication protocols, and assess network resilience. Drills often expose gaps in procedures or technology that need improvement. Ongoing crisis-focused training sessions also help both teams collaborate under pressure, manage system interdependencies, and address common points of failure. This shared experience is invaluable when real crises arise.
There are a variety of resources online to help you create your training scenarios, as well as 3rd party organizations that focus on training and hosting “Tabletop crisis management” workshops.
3. Cross-Training Exchange
Cross-training between internal facilities staff and external integrators deepens each team’s understanding of network operations. By shadowing each other, both teams learn each other’s processes and challenges, creating a shared knowledge base that’s useful in emergencies. When a crisis hits, this familiarity leads to faster, more effective problem-solving.
We recently focused on the benefits of collaborative training for OT teams and ways you can get started in this article.
4. Coordinated Communication Plans
Structured communication plans are essential for emergencies. This doesn’t need to be complex. A centralized contact list, notification protocols, and a shared communication tool accessible to all stakeholders keep everyone updated in real-time. This coordinated approach prevents misunderstandings and ensures all parties remain aligned on the situation.
5. Implementing OT Cybersecurity Best Practices
Securing an OT network is crucial for preventing emergencies. Best practices like network segmentation, access control, continuous monitoring, and regular patch management protect systems from potential vulnerabilities. For more on OT cybersecurity, see this guide to OT cybersecurity best practices from SCADAfence.
During: Rapid and Effective Response
1. 24/7 Rapid Response Teams
Dedicated rapid-response teams of internal staff and external integrators are invaluable in emergencies. These teams are (optionally!) on-call around the clock, ready to mobilize with pre-established escalation processes and remote access to network systems. Having teams readily available ensures quick action on complex issues without unnecessary delays.
2. Shared Access to Real-Time Monitoring Tools
Tools like OptigoVN with Site Scopes provide internal teams and external vendors with real-time network data, enabling immediate troubleshooting—even when teams aren’t on-site. With shared remote access, both parties can diagnose issues at the same time, accelerating problem resolution without the delays of on-site visits.
3. Vendor-Specific Expertise During Critical Failures
In high-stakes network issues, tapping into an external vendor’s specialized knowledge can speed up troubleshooting significantly. While internal teams know the network well, external vendors offer deep expertise on their products, allowing for faster and more targeted responses.
4. Resource Sharing in Disaster Recovery
During extended downtime, external partners can quickly supply critical equipment, tools, or replacement parts that may not be available through regular supply channels. This access to resources is invaluable for time-sensitive recovery efforts.
This is just the tip of the iceberg. Here are a few more great suggestions for disaster recovery strategies to keep your network resilient from Upstack
After an Emergency: Learning and Adapting
1. Root Cause Analysis and After-Action Reviews
After an incident, collaborative post-incident reviews with internal and external stakeholders allow for critical insights. By examining the root cause and analyzing each team’s actions, teams can identify improvement areas and refine response strategies for future emergencies. Tools like OptigoVN provide the historical data needed for effective root cause analysis. Here are six reasons for continuous monitoring of OT networks for more insights.
2. Data Logging for Incident Tracking and Reporting
Both internal and external teams should maintain synchronized logs of incidents and repairs. Detailed records are essential for post-emergency reporting, regulatory compliance, and identifying recurring issues that may signal underlying vulnerabilities. This tracking helps refine the response and strengthen the network over time.
Remember, not all of these solutions are for you. Some may not be adequate for your needs, and some might outstrip your budget. The critical action we recommend is that you start the conversation with your systems integrator or your client. A collaborative approach to OT network emergency management benefits everyone involved, creating a resilient, well-coordinated response framework.
OptigoVN can play a critical role in this collaboration. With its suite of diagnostic tools, continuous monitoring capabilities, and shareable access, OptigoVN empowers both internal teams and external partners to work more effectively during a crisis. Real-time insights, accessible to all stakeholders, allow for swift identification and resolution of issues, minimizing downtime and ensuring a more cohesive response. With the right tools and teamwork, OT networks are not only prepared to handle emergencies but positioned to emerge stronger from them.
How many issues will you solve today?
Collaborative Crisis Response FAQ
What is collaborative crisis response in OT networks?
Collaborative crisis response involves teamwork between internal facilities teams and external systems integrators or vendors during emergencies, like network outages. The goal is to diagnose and resolve issues quickly, minimizing downtime.
Why is collaboration important in OT network emergencies?
Emergencies often require diverse expertise. Facilities teams understand the building’s operations, while external partners bring specialized technical knowledge. Working together ensures a more efficient resolution to complex problems.
What are the benefits of having external vendors during a crisis?
External vendors bring in specialized knowledge and tools. They can access diagnostics remotely or on-site, helping pinpoint and resolve issues without delay.
How does Optigo Networks facilitate crisis collaboration?
Optigo Networks offers tools like OptigoVN to give both internal and external teams visibility into the network’s health. This shared access allows for faster troubleshooting and better-informed decisions.
How can proactive collaboration prevent emergencies?
Regular communication and planning between teams can help identify potential issues before they escalate. Utilizing continuous monitoring tools ensures network health is maintained, reducing the risk of critical failures.
