This week is all about the meat and potatoes of network management: troubleshooting! In this episode, we’re tackling your submitted questions failing devices, ADPU timings, port mirroring and more.
Have more BACnet questions? Keep them coming! We love diving deep into these technical topics that help make OT networks more reliable and efficient. Send us a message on LinkedIn, Reddit, or Bluesky, or email us at marketing@optigo.net .
Here’s what Ping covers in Episode 4:
❓ What are some obvious signs of failing devices on the network?
Ping breaks down the warning signs of failing devices—from slow response times and dropped packets to unexpected reboots. These behaviors are often signs that a device is overloaded or encountering environmental issues like overheating or power loss.
❓ My BACnet devices can’t see each other. Where should I start?
Ping shares a solid troubleshooting approach that starts at the physical and network layers. He highlights how broadcast traffic, subnet boundaries, and IP misconfigurations can cause devices to seem “invisible” to one another.
❓ What is port mirroring, and how can I use it to help diagnose issues on my OT network?
If you’re not using port mirroring, you’re missing out on one of the best diagnostic tools for your OT network. Ping explains how to set it up and what kind of traffic insights it can reveal when you’re tracking down BACnet issues.
❓ Is port mirroring done with limited switches? Do most people have these switches already?
Not all switches support port mirroring—but many managed switches do. Ping talks through what to look for in your existing hardware and how to check if you already have this capability available.
❓ Does APDU timeout have anything to do with slow answers to broadcasts?
Great question! Ping dives into how APDU timeouts are connected to BACnet communication timing and how they can indicate deeper issues with broadcast storms or response delays.
Transcript
So signs of a device that’s failing. You can think that an IoT device will have memory, it will have a CPU, it has I/Os to transmit and receive information. When any of these three or all these three are full, they tend to slow down. So your response time goes down. They tend to drop processing, drop packets. You send a request, it doesn’t come back. Sometimes. And then third, but not last, it might lock up. That blue screen of death we see in Windows. The device locks up. Now, a well-designed device will lock up and unlock itself. It may reboot itself. That’s still a bad thing. Devices that reboot without human intervention, a human command to reboot, that is a sign that the device is overloaded. The device is failing. I should also mention that, of course, I’m not talking about hardware issues. Right? The power supply going out. The temperature of the device going too high and therefore it loses operation. But those are obviously potential issues. Focusing from a network standpoint, from a software stack standpoint, those are the most common signs.
If devices can’t see each other, that probably means the broadcast traffic is not being sent or it’s not being received. Start from the base level. Is it a network problem? Is it a layer 3 problem? Meaning broadcast can go across IP subnets. So is there a router in between? Did you enable the BACnet broadcast forwarding? Are you using BBMDs to help this broadcast go across IP segments? If it’s not a layer 3 problem, then it’s a network level problem. Meaning is the broadcast being sent? Did the BACnet stack in the device misconfigure something so it’s not even sending the broadcast? Did it put in the wrong IP address? Or is there a firewall blocking it? Because UDP traffic can be blocked. So it could be at the network layer, or it could be a BACnet protocol issue. Meaning the message is being sent, the device received it, but it didn’t like the message so it didn’t respond.
Port mirroring is a feature on a switch that allows you to copy the data going through one port and send it out another port. Why this is important is because in today’s managed switches, every device has its own port. So if you plug in Wireshark into another port, you see nothing. You won’t see any data unless you’re doing port mirroring. Port mirroring allows you to mirror one port, say port number 5, to port number 24, and on port 24 you run Wireshark. Then you can capture all the data going in and out of port 5. That is critical. Otherwise you’re capturing an empty pipe.
Not every switch supports port mirroring, but all managed switches support port mirroring. If it’s a managed switch, typically there’s a way to set port mirroring up via its web UI. However, unmanaged switches do not support port mirroring. So when you’re buying network switches, make sure it supports port mirroring.
APDU timeout means that a device sent out a message and it didn’t get a response. And the timeout kicked in. That is usually indicative of one of two things. Either the other device never received it and didn’t reply, or it did receive it and chose not to reply. APDU timeout is common in busy networks or misconfigured networks. Now you ask, does it relate to slow broadcasts? It can. Broadcasts send a lot of data. If you’re broadcasting too much, devices get overloaded and they can’t respond to all the messages. So yes, too much broadcast traffic can lead to APDU timeouts.
