Some goblins and ghouls go bump in the night, and then there are BAS misconfigurations that frighten you! This Halloween, we’ve collected some of your spookiest network horror stories.
For the last few Halloweens, we’ve collected some of your spookiest network horror stories. Read on for stories of:
- Great Ball of Fire
- Spectre? Or a Figment of Imagination?
- A Steep Drop
- A Little Light Spooking
- Black Saturday
- The Fallout Shelter Security Panel
- Hotter Than Hell
- Phantom Devices
- Princeton’s Poltergeist
- A Haunted Hospital
- Spooky Surveillance
- An MS/TP Monstrosity
- PLUS a bonus tale of horror from Ryan Hughson, our Director of Business Development!
Great Ball of Fire
By Hendra Nugraha, National Engineering Manager, Delta Building Automation
Back in 2004, we’d just completed a fit-out for an office building in Sydney, with each floor having VAV controllers connected via BACnet MS/TP. In some VAV controllers, we mounted expansion cards for various purposes, such as multiple temperature sensors, tenant exhaust fans, and the like.
In one particular VAV on level 4, we had one of the expansion cards to monitor and control various I/Os in a tenant comms/electrical distribution board (DB), installed inside the panel itself. The panel had separate high-voltage (240VAC) and low-voltage (24VAC) sections.
We had our electrician wire up this panel based on wiring connections provided by the tenant. At this stage of the story, we hadn’t commissioned the panel as the power wasn’t yet available.
I received a call one afternoon that level 4 was “stuffy” and it felt like there was “no airflow.” There were also reports of a “burning smell” on the floor, and the lighting was “quite dark.” Back in the day, remote access was a luxury, so the only way to check was to attend the site.
True enough, once I arrived on site a few hours later, most of the lights were off and the floor had no airflow whatsoever. There was also a strong burning smell, which I put down to electric elements reheating when they first turned ON.
Suspiciously enough, I could not see any flashing lights on our controllers, and the network was OFFLINE on level 4.
The smell became stronger as I approached the tenant’s electrical DB, so I thought I’d just check the VAV that had the expansion card connected.
It all became apparent once I removed the ceiling tile to inspect the VAV controller:
The entire VAV controller was covered in big black circular char as if a great ball of fire had struck the controller. All plastic terminal, PCB, and MS/TP cabling were fused and charred completely black. This then shorted the rest of the 24VAC power to all other VAV controllers and tripped various circuit breakers. Back at the electrical panel, this also tripped the safeties and killed power for the entire floor. As for the expansion card, it was burnt to a crisp inside the electrical DB.
It even left a giant black mark on the VAV box itself. It’s probably like that to this day.
It appeared that our electrician had mixed up high- and low-voltage wiring by accident. It wouldn’t have been a problem if someone hadn’t turned the power on ahead of schedule. 240VAC power then traveled to the VAV controller via RS-485 cabling and “struck” it with great vengeance. The VAV controller appeared to have put up some resistance, albeit for a very short amount of time.
We quickly replaced the VAV controller with the new one and re-established the network and power, as the tenants had become quite annoyed.
As a tongue-in-cheek exercise, we sent the “faulty” VAV controller back to our manufacturer for RMA with the comment “Controller fails to communicate, please check for the issue.”
Spectre? Or a Figment of Imagination?
By Bill Winger
I crawled into the back of a mech room in a 100-year-old building, and I swear I saw someone walk by. (Active imagination?) The operator swears he saw someone in a locked room, and management freaked out because no one should be in there. They opened the door — but no one was there…
Of course, the price sheets and inferior delivery from some contractors also scare the hell out of me!
A Steep Drop
By Gary Schrader
I got to work in a cool old middle school here in KC. The architecture was gorgeous: marble flooring, wainscot, oak stair rails, doors, and trim. Masonry was impeccable. The building was sold by the school district and it was abandoned for a few years before developers bought it.
Anyway, I needed to figure out how to get to the dampers in the “Foul-Air” Cupola. (Actually called that on the original blueprints!) I climbed up a rickety ladder to a wooden slat attic above the stage, then crawled over the hand-bolted truss separating the stage from the auditorium to get to the cupola. There was no catwalk; I was standing on trusses and girders.
Once at the bottom of the cupola, I saw one of the six wooden-framed pneumatic dampers from the 30s had given way and was being held on by a thread… or, in this case, an 80-year-old actuator and its associated galvanized pneumatic line. If the wind had blown just right, it would have fallen right on me and taken me down the 60–70 feet to the bottom of the auditorium floor. At the time, I was slightly oblivious to how far up I was or how I could have slipped and fallen. The view was pretty awesome in that old attic, though. Boy, they don’t build them like that anymore.
A Little Light Spooking
By Adam Grant
We have a building out here at Texas A&M campus and up in the penthouse is all this super old pneumatic equipment and as you walk towards the back of the penthouse where the BAS panels are there is a single light fixture that flickers and sways all by itself.
Black Saturday
By Hendra Nugraha, National Engineering Manager, Delta Building Automation
We were just working away in the basement of one of the large federal department HQs in Canberra. The building is so large that we have to split the BAS network into three IP segments, separated by managed switches. An area controller manages each network segment, and the CAT-6 cabling to these three area controllers cannot be mixed into one segment. Because we used the Delta Controls’ derived network addressing at the time, mixing them would cause the rest of the subnet controllers to “jump over.” In each network segment, we had critical subnet controllers that performed load-shedding of general power (lighting, power, etc.) to the building.
My rookie engineer (let’s call him Bob) at the time was just tidying up the CAT-6 cabling at the back of the switch by plugging/unplugging the cables, and I was explaining away to him the importance of NOT mixing the area controllers into the same VLAN. Bob then called out, “Oh you mean I shouldn’t do this?” He pointed out where, to my horror, the three CAT-6 cables from those three area controllers were plugged into the SAME VLAN.
I quickly looked into the front end to confirm if everything was ok.
Strangely, everything was ok. Nothing had happened to the controllers.
Maybe I was wrong here with my theory about not mixing the area controllers. How could I be so silly? I started doubting myself…
WRONG. Mayhem ensued.
All lights and power in the building dropped OFF in the next split second. You could hear a series of loud bangs in the background as all the load-shedding contactors latched. All BAS controllers were on UPS, so power-wise all controllers were still online, but you could see on the front end that ALL the hundreds of subnet controllers had “jumped” from under one area controller to the next and then to the next. While doing this, all load-shed controllers performed a reset and dropped the lighting and general power to the building, via the load-shedding contactors.
I heard the security guard run down the basement corridor yelling “Are you guys ok down there? What the #$%^ happened?”
Before he could reach us, I quickly pulled out the CAT-6 cables, put them back in the right spots, and checked the network again to see if everything settled down.
Eventually, after a tense few minutes, I could see all area and subnet controllers sitting under the right “network tree,” and all was back to normal again. All lights and power were restored to normal after a few minutes. Thank God it was a Saturday so no one was around and we didn’t have to write a report to the client (the federal department) about the incident — other than some confused looks from the security guard at the time.
We have since “software addressed” all controllers to ensure this incident never happens again.
Fallout Shelter Security Panel
By Nicklaus Reeves, Service Tech, Parallel Technologies
I was young and generally fearless. I worked electronic security systems for the State of Nebraska, and I would occasionally be on call for after-hours service.
One dark and very stormy night, I got a trouble alarm from the Emergency Management Center. The Center resided within a fallout shelter underneath this nondescript, brick, government building. To access the Center you went down a flight of stairs, opened a door, and walk down a hallway to come face to face with an access keypad. Once you had access, you walked through another hallway, except this hallway sloped down and curved. One of the strange mechanics of this hallway was that your footsteps would echo back at you as if somebody were walking toward you. Even though I was aware of this, it still raised the hairs on the back of my neck.
I reached the vault, disarmed the system, and accessed the panel. The panel was showing no trouble codes, even though we had been receiving tamper trouble alarms. I started testing the tamper circuits when suddenly the lights went out!
For a brief moment, I was in complete darkness inside a vault with no emergency lighting. I heard the generator roar to life and the lights outside the vault came back on. But in that moment of darkness, I swore I heard footsteps.
I was alone. I called out but there was no answer. I could hear my blood rushing as my heart began to pound. “Let’s finish this and get out of here,” I thought to myself.
I finished testing the circuits and found no problems, but the entire time I felt as if someone was watching me. I reloaded the panel and verified everything was working. Just then, the power returned and the lights blinked momentarily. During that blink, I heard the footsteps again. I locked the vault, set the alarm, and ran out of there as fast as I could.
Just as I reached my van, I got another trouble code from the alarm. Code 022, “system disarmed under duress,” and then a handful of other trouble notifications followed. This automatically calls the police, even if you have the system set for the test. I called the police, but they’re required to respond to this alarm. I was fine with that as there was no way I was going back down there alone.
We got downstairs and the system was still armed, with no trouble codes. While I was waiting, I called the emergency manager and informed her that they may need to man the emergency center overnight until we could replace the panel. Two emergency personnel arrived while the policemen were checking out the center. One of the on-call personnel told me that they would never work alone down there at night, as they kept hearing footsteps, and sometimes whispers. We found out the next day that there was water leaking down onto the demarc, causing the phone line voltage to spike and scramble the security panel. After replacing the panel and repairing the phone line, we never had a problem with it again. The new company policy after that night still became, “Never go to the Emergency Center alone at night.”
Hotter Than Hell
By Steve R., Facility Manager
It was the middle of the night, at the height of summer in August. I was sound asleep, probably snoring peacefully to the roar of my AC at home.
The office building I managed was not so peaceful that night. It was more like hell. At 2 a.m. the blower motor there suddenly stopped running, with no warning. The condensing unit was working just fine, but there was just no blower motor. My coworker told me I probably needed to replace the blower motor and the capacitor and it should work just fine.
I did. It didn’t.
Meanwhile, the building was getting hotter and hotter all day, with no signs of relief. They say “Man plans, and God laughs.” Well, the devil was probably laughing at me as I struggled to fix whatever was making this building hotter than hell.
Finally, the next day, I tried jumping the blower motor. It sputtered to life, and I realized that the control board needed to be replaced. The part of the control board that managed the blower motor had just burnt out completely. As soon as I replaced it, everything went back to normal and started working again. But I never did forget that hot summer day, desperately trying to get the AC working again.
Phantom Devices
I was checking my network’s health after some new devices were added and saw the ‘Unresponsive Device’ check had failed. A router was sending out a Who-Is, but wasn’t getting any response. I thought maybe it wasn’t connected to power or had gotten unplugged from the network. But when I went to check on the device, there was nothing there.
Turns out it was a misconfiguration that I was able to quickly fix lol.
Princeton’s Poltergeist
It was the middle of the night. Princeton University’s controls engineer Gary Brancato was jerked out of a deep sleep by the shrill ringing of a telephone. “It’s the stadium lights. They’re on again,” a weary voice said on the other end of the line. Gary stumbled out of bed and called his techs to drive up to the university and reset the panels. Just like he had last week. And the week before that. And the week before that…
What sort of poltergeist was behind these sporadic wake-up calls? How did Gary finally put a stop to these hauntings?
Haunted Hospital
One night, I was on site at this creepy old hospital, for some controllers that were going offline. Now, I don’t believe in spooks, but there was something super eerie about that place. I kept looking over my shoulder expecting to see some phantom floating behind me.
Eventually, we figured out that the problem was because of a duplicate network, and we managed to fix it pretty quickly. Good thing too, I swear I had goosebumps the entire time I was there.
Spooky Surveillance
One Seattle stadium was having some serious security issues. Their surveillance video flickered and glitched. Sometimes it cuts out completely. Whenever fans flooded the field, traffic on the converged network increased and the surveillance video became completely fuzzy. It was almost like someone — or something — was haunting the stadium…
MS/TP Monstrosity
Here’s a nightmare for you. Our heating was a huge pain a while back. It was acting up, and we were getting tons of complaints coming in. There were controllers in the ceiling, and that meant waiting until after hours when everyone had left the office for the day. We had to spend something like 20 or 30 minutes per box, just trying to figure out where the problem was.
It was dark and dusty up there. The air was stale. Particles of the fiberboard kept falling on my head. At one point a wire dropped down out of nowhere. I heard a scurry of feet up in the ceiling with me — you can’t tell me that wasn’t a rat. All told, I spent almost four hours that night just digging around and troubleshooting MS/TP before I could finally uncover the source of the problem.
Ryan Hughson’s network horror story
You know that feeling you get in your gut, when you’re terrified but can’t put a name to it? It’s a cold sense of dread that you feel to your core. No matter how old you are, or how hard you try, you can’t shake the idea that there’s a boogeyman hiding in the dark, around the corner.
I was on-site at an old school. It was just a regular building, but it creeped the bejeezus out of me. Maybe it was the creaky set of wooden stairs leading down to the boiler room. I could barely see where I was going, it was so dimly lit. There was just a single, flickering light at the very bottom. The stench of moldy wood filled the air, wet and a little bit rotten. Each squeak of the stairs under my boots sent a shiver straight through me.
The basement door was metal, covered in peeling grey paint. It almost looked like someone had scraped the paint off, dragging sharp fingernails down. The door swung open slowly on rusty hinges, squealing in protest.
I walked in, across the compacted dirt floor. It was cold, dank, and dark like a grave. One incandescent bulb shone a small circle of light in the middle of the room. I could just make out the boiler in the corner; could hear the sound of leaking pipes and a slow drip-drip-drip on the dirt floor. Other than that, the air was thick with silence. Dead silence.
And then suddenly…
I heard a howling screech as the compressor rattled to life. I turned right around and got the hell out of there. No job is worth having a heart attack over, and the controllers probably didn’t need to be serviced that badly anyway.
This article has been updated since its original publish date (October 2017), with new tales of terror for October 2018 and 2019. Happy Halloween from the team at Optigo!
