Gary Brancato had a problem. As a controls engineer at Princeton University, he has a small team and a big campus. It’s just him and his fellow controls engineer Adam, juggling 1,000 Ethernet devices from more than a dozen vendors. When there are network issues, there’s lots of finger-pointing between the vendors, passing the buck and saying it’s someone else’s fault.
Gary had been improving his Network Health for a while. But there was only so much that he and his colleagues could accomplish with new additions and updates keeping the network in constant flux, and vendors pointing fingers when something needed to be fixed.
He couldn’t stretch his team any thinner. He didn’t have the resources to fix the variety of network issues. There were no overlooking network problems when they popped up on campus. And he couldn’t get anything fixed if his vendors were busy pointing fingers. So, he came up with a brilliant idea that would maximize his resources without maxing out his budget: Gary decided to throw an on-site “SharkFest.”
What’s this “SharkFest”? How does it help him stretch his resources and fix network problems without exhausting his budget? We sat down with Gary to get the scoop and find out how other colleges and universities can bring a “SharkFest” to their campuses.
“SharkFest,” as Gary refers to it, isn’t the Wireshark Conference, and it’s not an offshoot of the Discovery Channel’s Shark Week.
It’s a few days out of the year that Gary pulls in his two biggest automation vendors, Siemens and Automated Logic, and gets them to work through network problems. The top techs from each vendor come in, sit next to each other in the control room, and run Visual BACnet. They find problems on the network and just fix them as they go.
“It’s all in an attempt to stop finger-pointing,” says Gary. “Everyone knows that when a vendor has a chance to point the finger at somebody they go ‘Well that’s not my fault, because their controller is sending all the Who-Is’ and stuff.’ And then you let them get away with it because you don’t know.”
For more on finger-pointing, check out our interview with Gary on reducing finger-pointing and pinpointing problems on your BACnet network.
With the two vendors troubleshooting together in the same room and Visual BACnet to keep them accountable, they couldn’t point fingers at each other. It wouldn’t get them anywhere. Instead, they rolled up their sleeves and worked together. It’s a genius move from Gary that gets these separate groups to collaborate, without extra work for him and his colleague.
Gary came up with the idea of using Visual BACnet. When he was first trying to improve his Network Health Score from 16%, he brought in a couple of his vendors to work together and fix all the network issues.
“We would run captures and send it up and see what the issues were, and they would dive into their servers and go down to the controller levels, make changes to their software — whatever they had to do.
“In the Siemens case, Visual BACnet found that the Siemens controllers were constantly sending out Who-Is’ every two minutes or five minutes. It was a short period. And my tech from Siemens said ‘Oh, I can change that! We don’t need to do it that frequently.’ I think he changed it to every 10 minutes or something, from every two minutes.”
Since bringing his vendors into the control room way back in 2016, Gary’s continued to have SharkFests once or twice a year. He pays to bring these techs in for two to four days — it’s not part of their service contract — and he supervises while they make changes to bring up the Network Health.
“After the very first SharkFest, the Network Health was up to almost 70% after just those four days. It was awesome,” says Gary. He continues to host these SharkFests to keep the Network Health from falling again.
“The more hardware you put out there, things dwindle, and you don’t know what’s happening. My Visual BACnet Reports come to me every morning, and as that Network Health starts dropping off, that’s when I’ll just say to my boss, I need another four days, two for each vendor, and we’re gonna have to do it again. It’s money well spent because Visual BACnet tells me that things are going downhill, and I can catch it before the proverbial shit hits the fan.
“I don’t have the network resources like Penn State does. We don’t have network analysts here — it’s just myself and Adam. We’re monitoring this whole thing. And it’s the perfect thing to do if you don’t have the people that Penn State does.”
Gary encourages other colleges to try their own SharkFest, though he warns that the more vendors you have the more challenging it will be. “If you have two or three, I think it could be done relatively easily. I’m not sure how you could get seven vendors in there at once to do it. But if you have two or three, it’s doable. I would highly recommend it.”
Thinking about trying your own SharkFest? Gary has some advice for you.
“Make sure you put aside four days in the beginning, and make sure you put aside the same amount of time. In the very beginning, I was sitting in that room with those guys the whole time so things didn’t get pointed at and just left. It’s like managing a boxing match. You know, just so things don’t get out of hand. I mean, I’m super lucky. I have two good vendors and they care about working here. So I’m a little spoiled that way. But you have to make sure that somebody’s there with them. You don’t leave them, you don’t walk out, and you just keep plugging away.”
