How a global tech giant traced 118,000 hours of missed runtime to a single hidden bottleneck.
At a Glance
- The Client: World Headquarters for a top-tier global tech company, located in Washington state.
- The Scale: 80+ buildings, 50,000+ occupants, and a massive OT network managing 16,000 BACnet devices.
- The Complexity: A mixed-vendor environment (Alerton, Siemens, and more) integrated into a legacy Iconics front-end that had evolved over a decade.
The Challenge: Groundhog Day
In May 2024, the CBRE digital engineering team faced a crisis of confidence. Tenants at the client’s headquarters were complaining of uncomfortable environments—freezing in the morning, overheating in the afternoon. The schedules were technically “set” in the Building Management System (BMS), but the equipment simply wasn’t running.
The troubleshooting process became a painful, months-long cycle. The client’s facilities teams were buried in help-desk escalations. They would call the BMS vendors, who would investigate, claim to fix a programming error, or reset a server. The ticket would close, but the next day, the schedules would fail again.
“We couldn’t permanently fix it,” admitted April Yi, Director of Digital Engineering at CBRE. “Traditional troubleshooting failed after months of trying. The BMS vendor would report the issue fixed, but the next day it would all start over again.”
The team investigated every logical culprit in the automation layer. They assumed it was broken hardware, or a failing device. The answer was something easy to miss without visibility into the network.
The Investigation: The Invisible Bottleneck
The team shifted focus to the OT network layer, deploying Optigo Networks’ diagnostic suite to capture traffic across 20 different automation servers, including Alerton, Siemens, and Iconics instances.
The packet capture data revealed a shocking reality: the automation logic was fine. The network was choking it. The Culprit? A single BACnet router acting as a bridge between the East and West sides of the campus.The Root Cause
It wasn’t a hardware failure; it was “configuration drift.” Over ten years of expansion and integration, the Iconics server had been misconfigured to route a massive amount of unnecessary cross-campus traffic through this one choke point.
The router was simply too busy to pass the command packets. These “Busy Router” messages were invisible to the BMS front-end, so the system looked operational even while it was physically failing to execute start/stop commands.
The Solution: Re-Architecting The Traffic
Once the invisible bottleneck was visualized, the fix was architectural rather than mechanical.
- Traffic Analysis: OptigoVN pinpointed the specific controller bridging the campus and identified the misconfigured points on the Iconics server that were flooding the network.
- Server Split: The team split the 10 Iconics servers—dedicating five to the East campus and five to the West.
- Elimination: This simple move stopped the heavy traffic from forcing its way through the “middle” BACnet router.
The Result: Quantifying The "Missing" Hours
The impact of the fix was immediate. Response times stabilized, and the “random” failures vanished. The network Health Score climbed immediately from a critical 32% to a stable 72%: a strong rating for a brownfield site of this complexity.
Crucially, the team used the diagnostic data analyzed by Optigo to conduct a retrospective analysis, proving that the issue hadn’t started in May 2024, but as far back as June 2023. The cost of invisibility was massive:
- 118,000 Hours: The total time systems did not run as expected.
- 38% Failure Rate: The percentage of equipment that failed to turn on for more than half of their scheduled days.
By solving the network bottleneck, the client didn’t just stop the comfort complaints; they restored the integrity of their energy data and their trust in the system.
"We were also able to balance a problem between energy efficiency management and improving our client’s comfort. It made the case for why it’s worth investing in network visibility."
