Recent Outages Reinforce the Need for 911 Center Diversity

How many botched light-pole installations does it take to cause 911 service interruptions across multiple states? This sounds like the beginning of a joke, 

but it was no laughing matter last week when a third-party company installing a light pole accidentally sliced a fiber-optic cable owned by Lumen Global. The result was 911 service disruptions that reportedly lasted about three hours, in the states of Nebraska, South Dakota, and Nevada, including the cities of Las Vegas, Rapid City, and Sioux Falls.

Such outages happen for a wide variety of reasons — and they’re going to keep happening. The inability to receive emergency calls is the greatest nightmare for a 911 center, even worse than if the center’s computer-aided dispatch (CAD) system failed. If that were to occur, the center could resort to using run cards (assuming they still exist) to dispatch emergency responses. But there’s nothing they can do if emergency calls don’t reach the center.

In the public-safety sector, especially the 911 community, the terms “resiliency” and “redundancy” are heard often. They often are used interchangeably, but they are not synonyms. Resiliency describes how quickly the current situation can be restored to the ideal state. Redundancy means having multiple pathways/devices for achieving resiliency. That brings us to “diversity,” which means having multiple pathways/devices that are separated geographically. Diversity can be accomplished in several ways, as follows:

Carrier diversity — This means having two or more Internet Protocol (IP) network providers. It is highly unlikely that both providers would suffer an outage simultaneously, so each provider’s system could be engineered to failover to the other provider’s system if such an event occurs. However, this level of carrier diversity is not the norm. In some less-populated areas, i.e., rural, there might only be one provider. But that doesn’t necessarily mean that carrier diversity is not an option.

For example, we have one client, a regional 911 system that ironically serves a large, highly populated area, that has deployed Next Generation 911 (NG911) service. This client uses a single IP network provider, but the provider uses two IP selective routers — which are at opposite ends of the state and 300 miles apart — to deliver calls to the organization’s emergency services IP network (ESInet). This occurs at three separate locations, with dual fiber feeds at all of them, effectively achieving diversity. In addition to this level of diversity, a center ideally would have a completely different system that can handle basic 911 calls if all else fails.

Physical diversity — This means having emergency calls enter the center at two separate connections. These connections should be at least 25 feet apart.

System/media diversity — This means having two or more microwave links or fiber-optic cables in place. Regarding the latter, a ring-configuration is ideal — if the cable is cut, as it was in the Lumen Global situation, traffic can be rerouted seamlessly in the opposite direction.

Backup centers — Many 911 centers maintain a backup center stocked with essential systems and equipment that can be turned up quickly if the primary center is rendered inoperable, inaccessible, or uninhabitable. A common mistake made concerns locating the backup center too close to the primary center. There is some logic to this — the closer the two centers are, the faster personnel can get to the backup center in a bugout situation. However, a natural disaster — think tornado, hurricane, wildfire, flood — that causes the primary center to bug out likely will have the same effect on the backup center if there’s not enough distance between them. Consequently, the two centers should be separated by several miles.

Intergovernmental agreements — Perhaps the easiest way to achieve diversity is to develop IGAs with neighboring 911 centers, including those in other counties and even states, that would allow seamless forwarding of emergency calls in real time. However, care must be taken when forwarding the calls of a large center to a small center to avoid the temporary call-volume surge from overwhelming the latter. Another similar approach concerns IGAs that enable one 911 center’s telecommunicators to set up temporarily in another center, especially if both are NG911 compliant. This is accomplished far more easily today than even a few years ago. First, the NENA i3 standards for NG911 systems have interoperability baked into them. Second, because of the COVID-19 standards are working remotely. They can bring their laptops, plug them into the hosting center’s connectivity, and then log onto the home center’s call-handling and CAD systems — it’s as if they never left.

NENA suggests that 911 centers implement as much resiliency and redundancy as financially and technically viable. It’s great advice and we would welcome the opportunity to help you do that — please reach out.

Phil Rizzo is an MCP senior technology specialist. Email him at PhilRizzo@MissionCriticalPartners.com.