Network horror stories

halloween network horror story
Tales of terror from building automation systems

There are goblins and ghouls that go bump in the night; and then there are BAS misconfigurations that give you a fright! For Halloween, we’ve collected some of your spookiest network horror stories. Read on for stories of:

  • Black Saturday
  • The Fallout Shelter Security Panel
  • Hotter Than Hell
  • Phantom Devices
  • Princeton’s Poltergeist
  • A Haunted Hospital
  • Spooky Surveillance
  • An MS/TP Monstrosity
  • PLUS a bonus tale of horror from Ryan Hughson, our Director of Business Development!

Black Saturday

By Hendra Nugraha, National Engineering Manager, Delta Building Automation

We were just working away in the basement of one of the large federal department HQs in Canberra. The building is so large that we have to split the BAS network into three IP segments, separated by managed switches. An area controller manages each network segment, and the CAT-6 cabling to these three area controllers cannot be mixed into one segment. Because we used the Delta Controls’ derived network addressing at the time, mixing them would cause the rest of the subnet controllers to “jump over.” In each network segment we had critical subnet controllers which performed load-shedding of general power (lighting, power, etc.) to the building.

My rookie engineer (let’s call him Bob) at the time was just tidying up the CAT-6 cabling at the back of the switch by plugging/unplugging the cables, and I was explaining away to him the importance of NOT mixing the area controllers together into the same VLAN. Bob then called out, “Oh you mean I shouldn’t do this?” He pointed out where, to my horror, the three CAT-6 cables from those three area controllers were plugged into the SAME VLAN.

I quickly looked into the front-end to confirm if everything was ok. 

Strangely, everything was ok. Nothing had happened to the controllers.

Maybe I was wrong here with my theory about not mixing the area controllers together. How could I be so silly? I started doubting myself…

WRONG. Mayhem ensued.

All lights and power in the building dropped OFF in the next split second. You could hear a series of loud bangs in the background as all the load-shedding contactors latched. All BAS controllers were on UPS, so power-wise all controllers were still online, but you could see on the front-end that ALL the hundreds of subnet controllers had “jumped” from under one area controller to the next and then to the next. While doing this, all load-shed controllers performed a reset and dropped the lighting and general power to the building, via the load-shedding contactors.

I heard the security guard run down the basement corridor yelling “Are you guys ok down there? What the #$%^ happened?”

Before he could reach us, I quickly pulled out the CAT-6 cables, put them back in the right spots, and checked the network again to see if everything settled down.

Eventually, after a tense few minutes, I could see all area and subnet controllers sitting under the right “network tree,” and all was back to normal again. All lights and power were restored to normal after a few minutes. Thank God it was a Saturday so no one was around and we didn’t have to write a report to the client (the federal department) about the incident — other than some confused looks from the security guard at the time.

We have since “software addressed” all controllers to ensure this incident never happens again.

Fallout Shelter Security Panel

By Nicklaus Reeves, Service Tech, Parallel Technologies

I was young and generally fearless. I worked electronic security systems for the State of Nebraska, and I would occasionally be on call for after hours service. 

One dark and very stormy night, I got a trouble alarm from the Emergency Management Center. The Center resided within a fallout shelter underneath this nondescript, brick, government building. To access the Center you went down a flight of stairs, opened a door, and walked down a hallway to come face to face with an access keypad. Once you had access, you walked through another hallway, except this hallway sloped down and curved. One of the strange mechanics of this hallway was that your own footsteps would echo back at you, as if somebody was walking toward you. Even though I was aware of this, it still raised the hairs on the back of my neck. 

I reached the vault, disarmed the system, and accessed the panel. The panel was showing no trouble codes, even though we had been receiving tamper trouble alarms. I started testing the tamper circuits, when suddenly the lights went out! 

For a brief moment I was in complete darkness inside a vault with no emergency lighting. I heard the generator roar to life and the lights outside the vault came back on. But in that moment of darkness, I swore I heard footsteps. 

I was alone. I called out but there was no answer. I could hear my blood rushing as my heart began to pound. “Let’s finish this and get out of here,” I thought to myself. 

I finished testing the circuits and found no problems, but the entire time I felt as if someone was watching me. I reloaded the panel and verified everything was working. Just then, the power returned and the lights blinked momentarily. During that blink, I heard the footsteps again. I locked the vault, set the alarm, and ran out of there as fast as I could possibly run. 

Just as I reached my van, I got another trouble code from the alarm. Code 022, “system disarmed under duress," and then a handful of other trouble notifications followed. This automatically calls the police, even if you have the system set for test. I called the police, but they’re required to respond to this alarm. I was fine with that as there was no way I was going back down there alone. 

We got downstairs and the system was still armed, with no trouble codes. While I had been waiting, I called the emergency manager and informed her that they may need to man the emergency center overnight until we could replace the panel. Two emergency personnel arrived while the policemen were checking out the center. One of the on-call personnel told me that they will never work alone down there at night, as they kept hearing footsteps, and sometimes whispers. We found out the next day that there was water leaking down onto the demarc, causing the phone line voltage to spike and scramble the security panel. After replacing the panel and repairing the phone line, we never had a problem with it again. New company policy after that night still became, “Never go to the Emergency Center alone at night.”

Hotter Than Hell 

By Steve R., Facility Manager 

It was the middle of the night, at the height of summer in August. I was sound asleep, probably snoring peacefully to the roar of my AC at home. 

The office building I managed was not so peaceful that night. In fact, it was more like hell. At 2 a.m. the blower motor there suddenly stopped running, with no warning. The condensing unit was working just fine, but there was just no blower motor. My coworker told me I probably needed to replace the blower motor and the capacitor and it should work just fine. 

I did. It didn’t. 

Meanwhile, the building was getting hotter and hotter all day, with no signs of relief. They say “Man plans, and God laughs.” Well, the devil was probably laughing at me as I struggled to fix whatever was making this building hotter than hell. 

Finally, the next day, I tried jumping the blower motor. It sputtered to life, and I realized that the control board needed to be replaced. The part of the control board that managed the blower motor had just burnt out completely. As soon as I replaced it, everything went back to normal and started working again. But I never did forget that hot summer day, desperately trying to get the AC working again. 

Phantom Devices

“I was checking my network’s health after some new devices were added, and saw the ‘Unresponsive Device’ check had failed. A router was sending out a Who-Is, but wasn’t getting any response. I thought maybe it wasn’t connected to power, or had gotten unplugged from the network. But when I went to check on the device, there was nothing there.

“Turns out it was a misconfiguration that I was able to quickly fix lol.”

Princeton’s Poltergeist

It was the middle of the night. Princeton University’s controls engineer Gary Brancato was jerked out of a deep sleep by the shrill ringing of a telephone. “It’s the stadium lights. They’re on again,” a weary voice said on the other end of the line. Gary stumbled out of bed and called his techs to drive up to the university and reset the panels. Just like he had last week. And the week before that. And the week before that…

What sort of poltergeist was behind these sporadic wake-up calls? How did Gary finally put a stop to these hauntings? Read our case study to find out!

Haunted Hospital

“One night, I was on site at this creepy old hospital, for some controllers that were going offline. Now, I don't believe in spooks, but there was something super eerie about that place. I kept looking over my shoulder expecting to see some phantom floating behind me.

“Eventually we figured out that the problem was because of a duplicate network, and we managed to fix it pretty quickly. Good thing too, I swear I had goosebumps the entire time I was there.”

Spooky Surveillance

One Seattle stadium was having some serious security issues. Their surveillance video flickered and glitched. Sometimes it cut out completely. Whenever fans flooded the field, traffic on the converged network increased and the surveillance video became completely fuzzy. It was almost like someone — or something — was haunting the stadium…

Or, was there something more technical behind the ghostly, glitching surveillance? Find out more in our case study!

MS/TP Monstrosity

“Here’s a nightmare for you. Our heating was being a huge pain a while back. It was acting up, and we were getting tons of complaints coming in. There were controllers in the ceiling, and that meant waiting until after hours, when everyone had left the office for the day. We had to spend something like 20 or 30 minutes per box, just trying to figure out where the problem was.

It was dark and dusty up there. The air was stale. Particles of fiberboard kept falling on my head. At one point a wire dropped down out of nowhere. I definitely heard a scurry of feet up in the ceiling with me — you can’t tell me that wasn’t a rat. All told, I spent almost four hours that night just digging around troubleshooting MS/TP before I could finally uncover the source of the problem.”

Ryan Hughson’s network horror story

You know that feeling you get in your gut, when you’re terrified but can’t put a name to it? It’s a cold sense of dread that you feel to your core. No matter how old you are, or how hard you try, you can’t shake the idea that there’s a boogeyman hiding in the dark, around the corner.

I was on site at an old school. It was just a regular building, but it creeped the bejeezus out of me. Maybe it was the creaky set of wooden stairs leading down to the boiler room. I could barely see where I was going, it was so dimly lit. There was just a single, flickering light at the very bottom. The stench of mouldy wood filled the air, wet and a little bit rotten. Each squeak of the stairs under my boots sent a shiver straight through me.

The basement door was metal, covered in peeling grey paint. It almost looked like someone had scraped the paint off, dragging sharp fingernails all the way down. The door swung open slowly on rusty hinges, squealing in protest.

I walked in, across the compacted dirt floor. It was cold, dank, and dark like a grave. One incandescent bulb shone a small circle of light in the middle of the room. I could just make out the boiler in the corner; could hear the sound of leaking pipes and a slow drip-drip-drip on the dirt floor. Other than that, the air was thick with silence. Dead silence.

And then suddenly…

I heard a howling screech as the compressor rattled to life. I turned right around and got the hell out of there. No job is worth having a heart attack over, and the controllers probably didn’t need to be serviced that badly anyways.


This article has been updated since original publish date (October 2017), with new tales of terror for October 2018. Happy Halloween from the team at Optigo! 

Recent Blog Posts

November 27, 2018, Vancouver, BC – Vancouver-based Optigo Networks, the network connectivity, monitoring, and analytics company, has released a whitepaper on the future of BACnet and the Building Internet of Things (BIoT).

Gary Brancato had a problem. As a controls engineer at Princeton University, he has a small team and a big campus. It’s just him and his fellow controls engineer Adam, juggling 1,000 Ethernet devices from more than a dozen vendors.

The BAS industry is at a turning point, with IP connectivity shifting the way we see our brick and mortar buildings. 

There are goblins and ghouls that go bump in the night; and then there are BAS misconfigurations that give you a fright! For Halloween, we’ve collected some of your spookiest network horror stories. Read on for stories of:

Have you voted in the annual ControlTrends Awards yet? The ceremony will take place at AHR Expo 2019, in Atlanta, Georgia January 14–16.

Recent Projects

Penn State University Optigo Networks Visual BACnet

PENN STATE UNIVERSITY

When Tom Walker looked at Penn State University’s Navy Yard network, he saw huge issues. The system was busy and loud, to the point where the overrun network was bringing down the entire building. Because this was happening on the MS/TP network, pinpointing the problem would mean boots on the ground to segment and test the chain, piece by piece.

Penn State University Optigo Networks Visual BACnet

PENN STATE UNIVERSITY

When Tom Walker first started working at Penn State University four years ago, there were a lot of network issues. Buildings were dropping offline. Broadcast traffic was pushing 90,000 packets per hour. Walker was on the phone almost every single night because devices were down or had to be reset.

 

Torre Manacar Mexico City Optigo Connect

TORRE MANACAR

When MR Soluciones began work on Torre Manacar, they knew they needed a flexible and scalable network infrastructure to support a wide array of integrated systems. Optigo Networks was a natural fit for the massive project, designing a robust network at a competitive cost.

short

SHORT PUMP TOWN CENTER

Short Pump Town Center, an upscale retail center, underwent a complete renovation in 2014. The flexibility of Optigo Networks’ solution meant the retail center’s unknown final design was not a barrier to placing IP surveillance equipment in the field.

BOULEVARD MALL

BOULEVARD MALL

Optigo Networks connected New York-based Boulevard Mall’s security surveillance devices in December 2015, using a Passive Daisy Chain topology.

Visual BACnet tech support team

TECH SUPPORT TEAM

One tech support team at a manufacturer purchased an account with Visual BACnet in April 2017, for technical problems around the world.

Aster Conservatory Green Optigo Connect

ASTER CONSERVATORY GREEN

The Aster Conservatory Green is a community comprising 352 residences across 24 low-rise buildings. The buildings use advanced surveillance and access control technology, including 40 HD video cameras and 60 FOB-access-tele-entry points for access control.

25 NATIONAL CIRCUIT

25 NATIONAL CIRCUIT

When Delta Building Automation (Australia) won the BMS Upgrade at 25 National Circuit for the Australian Trucking Association, they partnered with Optigo Networks to create a secure and robust Building Services Network (BSN). Optigo Connect more than delivered on this project with a scalable solution that restored the building network to perfection.

Optigo Connect Seattle Stadium

SPORTS AND ENTERTAINMENT CENTER

Optigo Connect offered a simple, cost-efficient solution for a premier Seattle-based stadium. Optigo Networks’ design improved the surveillance system to crystal clear perfection, made it dependable, and allowed the security system to scale with the addition of more than 40 16MP cameras.

TELECOM DATA CENTER

TELECOM DATA CENTER

Optigo Networks and OTI offered a secure and scalable solution for four data centers’ HVAC and Access Control systems throughout the United States. Optigo Connect’s performance in the first data center was so impressive, the client asked that Optigo replicate the network design for three other data centers.