BACnet over IP, or BACnet/IP, has gradually become the new standard in OT networks for years. Compared to much older MS/TP networks, BACnet/IP is faster, has more bandwidth, scales more easily, and is generally less of a headache to troubleshoot. It also brings OT networks into the modern age, addressing the rise of IoT and the ongoing convergence between IT and OT systems. But like any protocol, it brings with it a new set of devices, programming, and ultimately, opportunities for common BACnet/IP issues to arise.
Like our companion guide for some of the most common MS/TP Issues, we’ve put together this helpful list of some of the most common BACnet/IP issues we see, and how to fix them using management and diagnostic software, like Optigo Visual Networks. These issues include unreachable devices, duplicate addressing, and excessive request traffic.
1. Unreachable Devices
Strictly speaking, an unreachable device does not answer a ‘Who-Is’ request with an ‘I-am’ response. In reality, it’s a “ghost in the machine”. That’s because an unresponsive device can appear as both fully unreachable – completely offline to the network – or partially unreachable. Partially unreachable devices pop up from time to time, answering some requests but not others, and are generally intermittent in their communication.
There are a few different reasons a device can be unreachable, and can typically be resolved through a process of elimination:
- Is it a “phantom” device? We recommend you start by checking your front-end BMS to see if old programming or a misconfiguration is leading to a request for a device that is simply no longer on your OT network.
- Is it a physical issue? If you know the device is still present, it might be time to get up close and personal. A physical inspection may reveal disconnected, loose, or damaged cabling or connections that are causing sporadic communication.
- Is it a routing issue? If the device is present, and powered up, the last step would be to path trace for any potential communication issues. Check any switches, MS/TP to IP routers, etc., to see if there are any failures.
Tip: Duplicate devices trying to communicate with your hardware can create a flood of traffic to the point where it simply cannot respond fast enough, which can mimic the symptoms of a partially unresponsive device.
If you’re troubleshooting common BACnet/IP issues with OptigoVN, you’ll see several diagnostic results that will clue you into possible issues, including Fully and Partially Unreachable Device alerts, as well as Busy Router Backpressure Message, Error Messages of Programming Type, and several duplication error alerts.
OptigoVN’s Site Scope add-on feature then allows you to drill down to specific devices, saving you hours manually inspecting devices and configurations through physical inspection or packet sniffing.
Already an OptigoVN user? Adding a Site Scope to your account and assigning it to an OT Network is just a few clicks away.
2. Duplication
Duplication, another set of common BACnet/IP issues, occurs when two or more devices are unintentionally assigned the same designation – appearing to the rest of the network as one device. Duplication can cause a range of issues, including communication failures that can lead to unresponsive behavior in devices.
There are typically three issues we see around duplication of hardware that you should prioritize: duplicate BACnet broadcast management devices (BBMDs), duplicate network numbers, and duplicate device instance numbers.
Duplicate BBMDs
BBMDs manage all the broadcast messages generated by BACnet/IP devices to keep the network from getting overwhelmed. We created a great primer on what BBMDs are and how they work on an OT network. In this scenario, you’ve got multiple devices acting as BBMDs — all rebroadcasting the same message to the whole network. A duplicated BBMD means double the traffic – and the destination BBMD now has more traffic to route. This kind of duplication of unnecessary traffic can quickly take down an OT network.
Duplicate BBMDs are normally caused by an installation issue, typically on a mixed vendor site. In a common example, a vendor on a new site sets up multiple networks and connects them with their BBMDs. A later site upgrade or change occurs, and a new vendor adds devices to the expanding network, using their BBMDs to connect the subnets. This can continue, resulting in two, three, or more BBMDs on each subnet, doubling or tripling (or more) your network traffic load.
To resolve the problem, start by:
- Isolating networks if you don’t need integration.
- If not, assign one vendor’s BBMDs to the task.
OptigoVN’s bank of diagnostic tests includes analysis for duplicate BBMDs. With the Site Scope add-on, you can identify the addresses of all devices acting as duplicate BBMDs.
Duplicate Network Numbers
Similar to duplicate BBMDs, this common BACnet/IP issue relates to routing within your network. We dove into the details in our article, Dealing with Duplicates in BACnet with OptigoVN:
A situation where two or more BACnet routers are pointing traffic at the same network segment. Think of them like an elevator door – connecting the outside world (the outside network) to its hallway of rooms (the devices on its segment). And just like floors in an elevator bank, every BACnet router needs to have a unique network number so data exchange between devices can happen seamlessly. When two or more BACnet routers have duplicate network numbers, it can lead to communication conflicts and disrupt normal operation.
Fixing duplicate network numbers with OptigoVN is fast and easy. The Duplicate Network Number diagnostic report will alert you to any duplicate network number issues, and with your Site Scope insights, pinpoint the unique device addresses of the two devices. From there, you can reassign unique network numbers, and re-test to ensure the issue is resolved.
Duplicate Device Instance Numbers
Also referred to as “Device IDs” or “BACnet Instance Numbers”, a device instance number is a unique identifier for every device on your OT network. This is different from a device address in that addresses can be repeated if on separate networks/subnets, but BACnet instance numbers have to be globally unique. No two devices anywhere on any part of your BACnet system can have a repeated instance number.
Referring back to our Dealing with Duplicates article:
A common issue that leads to duplicate BACnet instance numbers is human error. Because just like device addresses, BACnet instance numbers need to be manually assigned. New hardware will typically arrive with a default BACnet instance number, and it’s all too easy to forget to change it. Soon, you’re starting to see network issues like the duplicate address issue: communication errors, devices appearing and disappearing, and inconsistent data reporting.
Resolving duplicate BACnet instance numbers is essentially the same process as resolving duplicate addresses. OptigoVN’s diagnostic reports will alert you to any duplicate BACnet instance number issues, and a Site Scope pinpoints the duplicate devices’ unique MAC and IP addresses. From there, you can reassign unique BACnet instance numbers and re-test to see if the issue is resolved.
3. Excessive Traffic
Smart building automation systems generally work by collecting information from devices scattered throughout a building – think thermostats, lights, humidity sensors – relaying that information to a program in the building management system (BMS) that decides what to do with that information, then sending out commands to devices to act. Three of the most common types of this data used to run this system are:
- Data polling requests, or “Read” requests
- Programming and configuration updates, or “Write” requests
- Change of Value (CoV) messages: a message from a device to the system reporting a change in a measurement exceeding a reporting threshold.
So in any BACnet system, there’s always going to be a fair amount of traffic related to this functionality. And that is completely normal. In a perfectly functioning system, measurements and adjustments occur all the time to ensure the best results. But sometimes these requests can spiral out of control, generating way too much traffic for the system to handle. As a Gen Z might put it, “you’re doin’ too much.”
Let’s look at three common BACnet/IP issues responsible for generating unnecessary traffic: excessive read/write rates, and excessive CoV messages. Then let’s tamp them down to a manageable volume.
Excessive Read Rate
As we laid out in our deep dive article, What is Excessive Read Rate, and How Do You Solve It? “If [a] device is being constantly polled for updates, it’s generating data on the network, and consuming CPU cycles. Now assume you have hundreds of similar sensors all generating a constant stream of data across a network with limited bandwidth. The traffic adds up fast.”
Excessive read rates are almost always a programming mistake – creating blanket polling policies for all object types across all devices. In a large system, this has the potential to trigger a flood of unnecessary data traffic. Excessive traffic from reads can lead to an increase in response times, decreased resources, and instabilities across your system.
Fortunately, there are a few easy ways to resolve the issue:
- Review your configurations. Inspect config settings on devices for data polling rates to ensure they align with your requirements.
- Reduce Poll Rates. Don’t use factory defaults. Set polling frequencies based on how important the data is, how often you need the data to be updated, and your system’s operational needs.
- Optimize Control Programming. Similar to config settings, review your control programs and scripts to avoid unnecessary or redundant data polling.
- Convert Reads into Change-of-Value. Instead of polling the value, use a CoV “push” notification when the value changes, reducing traffic and asking for updates that may have nothing to report.
OptigoVN has an out-of-the-box diagnostic check for Excessive Read Rates. Warnings are displayed in the expanded list under Rate and Frequency of Object Reads, Writes, and COVs>Excessive Read Rate, flagging requests that are being sent to the same device less than five minutes apart. From here you can use a Site Scope to dig into your diagnostic test results to identify the culprit devices that are triggering a diagnostic warning.
Excessive Write Rate
The flip side to read rate, excessive write rate occurs when devices are sending settings and program updates to one particular device at an unusually high pace. Also centered around programming mistakes, excessive write rates can lead to high levels of network traffic, impacting network performance, and contributing to increased wear and reduced device lifespan.
The recommended actions are similarly related:
- Review configurations and set write frequencies based on the frequency of control changes, data logging requirements, and system performance goals.
- Optimize programs and scripts to avoid unnecessary or redundant messaging.
- Tune control loops to prevent rapid and unnecessary adjustments that lead to excessive write operations.
- Minimize unnecessary write operations for logging.
OptigoVN also has a diagnostic warning users of Excessive Write Rates that exceed our threshold under Rate and Frequency of Object Reads, Writes, and COVs>Excessive Read Rate. From here you can use a Site Scope to dig into your diagnostic test results to identify the culprit devices that are triggering a diagnostic warning.
Excessive Change of Value (COV) Rate
The Change of Value rate flows in the opposite direction. CoV messages originate in the reporting devices, but the issues are essentially the same: a device is sending off a high rate of messages reporting a change in a flagged data point.
For example, a thermostat that detects a temperature change is likely programmed to report any change past a certain threshold back to the BMS. However, misconfigured programming might have that thermostat reporting changes of 0.1 degree, which is unlikely to have any real-world impact, but might generate exponentially more data traffic than is necessary. Now multiply that by 200 thermostats across your building configured the same way, and you can see how that can overwhelm a BACnet system’s capacity.
Once again, the fix is almost always in the programming:
- Review the configuration of your device to set the reporting thresholds to a value that matches its importance and your overall control needs.
- Consider hardware upgrades if you cannot tune reporting to the degree you need.
OptigoVN includes a separate diagnostic warning for potentially excessive CoV messages on your BACnet network that exceed our threshold. Add-on Site Scopes can identify the addresses of all devices with excessive COV rates. From there, it’s easy to inspect and tune your devices to get the messaging rates back within your system needs.
Remember, some high rates of reporting from devices might be what you need for your system, like in a sensitive lab area or data center. In these cases, it’s valuable to consider how that traffic is flowing through your system. Are there device bottlenecks? Are there areas where load balancing may help? Are there any rate-limiting techniques you can implement?
For common BACnet/IP issues like excessive read, write, and CoV optimization, regular diagnostic checks should be performed to keep polling and writing frequency under control! We highly recommend using our plug-and-play, Optigo Networks Hardware Capture Tool for all BACnet MS/TP packet capture activities, or installing our free Optigo Networks Software Capture Tool (available as a Windows or Linux application) for BACnet IP or BACnet Ethernet captures.
This list is a great place to start your troubleshooting processes for common BACnet/IP issues, but it’s by no means the end of the list. That’s why OptigoVN’s 25 different diagnostic tools provide you with best-in-class OT network visibility. And by adding Site Scopes, you’ll get faster and deeper insights into your OT networks than you’ve ever had before. Ready to learn more? Sign up for free today.
