On the evening of March 8th, PSAPs in multiple states were notified that wireless AT&T customers were unable to access 9-1-1 with a voice call, or were having difficulty reaching a 9-1-1 calltaker.
It appears that the outage was extensive, perhaps nationwide, and lasted for several hours before being resolved. PSAPs in Tennessee, Texas, California, Florida, Colorado, and Pennsylvania are just a few states that publicly confirmed 9-1-1 service degradation. The full impact is still unknown at this point.
As the vendors involved work to investigate the root cause of the system failure, many within the public safety sector are beginning to reflect on the situation and how the industry can be better prepared for when—not if—a similar problem occurs in the future.
Meanwhile, at the federal level, the FCC has announced an investigation that will likely to take some time to complete.
A seemingly uncoordinated effort
Many of our 9-1-1 clients reported that notifications to state 9-1-1 leaders and PSAPs were sporadic and narrowly targeted. Agencies shared bits of information via email with their colleagues as the situation unfolded. Network operation centers from service providers such as West Safety Services and Comtech forwarded notifications to some of their PSAP customers; however, many were never notified by a service provider during the incident or after the situation was resolved. The notification process and subsequent communication appeared to be an uncoordinated effort that leaves significant room for improvement.
Although requirements codified in Federal regulations (§47 CFR 4.9 - Outage reporting requirements - threshold) mandate notification to the FCC within 120 minutes for system outages, contacting affected 9-1-1 authorities is required “as soon as possible” which unfortunately is not well defined.
It's critical that the 9-1-1 community mitigate the effects and impacts associated with unplanned outages. Agencies need to prepare themselves to respond effectively to emergency situations like this outage.
time for a policy and procedure refresh
In light of the outage, Mission Critical Partners urges each 9-1-1 authority to review their local/regional policies and procedures to determine if revisions (or testing) are necessary. The National Emergency Number Association (NENA) describes contingency planning in its “PSAP Disaster & Contingency Plans Recommendations.”
A few steps we suggest include:
1. alternative methods of contact
Every PSAP should have 10-digit emergency phone numbers or other alternate methods of contact designated and ready for use. Examples include a
- 10-digit local or
- toll-free phone number that routes directly to your PSAP.
2. easily accessible contact information for local media outlets
Current contact information to local news media outlets should be retained by PSAP supervisors and members of the management team. In the event of an emergency that impacts public access to the 9-1-1 system, the news outlets should be contacted as soon as possible with details on the impacted geographical area (borough, township, city, county) and the designated alternate methods of contact. Leveraging existing relationships, often through a county’s public information officer (PIO) can be extremely valuable.
3. a credible presence on social media
Social media platforms such as Facebook and Twitter should be leveraged by the PSAP’s designated account managers. The event information should be aligned with the information provided to the news media outlets.
To expand your social media visibility, it’s recommended that PSAPs maintain frequent and regular presence on their account, which increases the public’s perception of the agency as a legitimate source of news and information. During the outage, some jurisdictions received calls from outside their service areas because social media posts were not specific to geographic coverage areas for alternate emergency numbers.
4. Local emergency alert systems
Local emergency alert systems that send text messages, emails, and/or automated calls to subscribers should be considered. The type or breadth of outage, as well as time of day, may impact how and if a community-wide alert is transmitted.
5. A wireless emergency alert plan
Wireless emergency alerts (WEAs) can be sent by state and local public safety officials through the Integrated Public Alert and Warning System (IPAWS). PSAPs should work with their county emergency management officials to seek approval and training for designated PSAP personnel to access IPAWS for the WEA function. Time will be required for account registration and configuration, and should not be attempted at the height of an emergent situation.
As additional information becomes available on the March 8th AT&T wireless outage, we will share it with our clients. We look forward to working with industry leaders, our partners, and NENA to improve information sharing that will help during a service-affecting crisis.
Identifying, mitigating, and eliminating single points of failure are critically important to improve the reliability and resiliency of our nation’s 9-1-1 system. This “sunny day” outage is just another in a series of failures in the legacy infrastructure and as we learn more about the root cause, we may find even more reason for our need to push forward with Next Generation 9-1-1.