According to the old saying, time flies when one is having fun. The adage also applies when a new business unit is being built.
A little more than five years ago—has it really been that long?—MCP landed its first network monitoring contract, which marked the birth of our Lifecycle Management Services division. The initial concept for MCP to get involved in this business was the brainchild of Dave Boyce, with a helpful push from Brian Bark. It all started when the Pennsylvania Region 13 Task Force decided that once their Emergency Services Internet Protocol Network (ESInet) was complete, the region needed to monitor the ESInet. Because MCP subject matter experts had designed and deployed the network, Region 13 asked us for help in securing a monitoring service. The search didn’t go well. There weren’t many options, and those that were qualified to do the work were seeking fees that were markedly cost prohibitive, or they wanted to monitor only a specific network segment.
That’s when the proverbial lightbulb went on. “Maybe there’s a way that we can do this ourselves,” Dave said.
To create the service offering, MCP decided to partner with Athena Networks. At the time, Athena—which MCP acquired in January 2019—was focused on computer-aided dispatch system monitoring, but they believed they could extend that expertise to the ESInet technology. We took a leap of faith. By the time I arrived at MCP, at the end of July 2015, the service offering was well into its development, and I supported Dave in the development of the statements of work and sample contracts. Dave’s and Brian’s vision finally came to fruition when Region 13 offered a network monitoring purchase order, and the contract was signed on October 15, 2015.
Everything went reasonably well for the next year and a half. Eventually, though, a shortcoming of the service offering became apparent—no dedicated network operations center (NOC) existed. MCP’s Dennis Matzen, who was working for Athena at the time, had customized the Nagios open-source, off-the-shelf software with filtering logic that did a good job of sifting through all of the alarms that were generated to determine the ones that actually were problems. But the solution was better suited for smaller networks, not regional networks, like the one operated by Region 13, or statewide networks.
The Need for a Network Operations Center
That realization led MCP to seek a partner that could provide outsourced NOC services 24 x 7 x 365. We initially thought about building our own NOC, but we didn’t have a large enough portfolio of clients at the time to do so cost effectively. Lacking such a portfolio, we turned to our NOC partner, which has done a fabulous job.
Today we are supporting 47 clients with monitoring services, including the Southern Alleghenies 911 Cooperative and the Lubbock (Texas) Emergency Communication District, in addition to Region 13.
The next step in our evolution was to obtain our own network monitoring platform, which we did by leveraging the ConnectWise information technology management and managed services platform software suite. The result was the NetPulse suite of services—Essential, Advanced and Custom.
24 x 7 x 365 Visibility
Sometimes, the terms “network monitoring” and “network assessment” are used interchangeably. While they might seem to be synonyms, a subtle but important difference exists. When we assess a network, we provide our clients with a “snapshot in time,” essentially what their network environment looks like at the moment of the assessment. When we monitor a network, we are continuously assessing that network 24 x 7 x 365. Either way, the goal is to provide clients with unprecedented visibility into their networks so that performance-affecting problems can be prevented before they become real-time operational problems.
Here’s an example. A client thought they had covered itself from a redundancy perspective by designing bidirectional paths into their public safety communications network, which was being provisioned by a telecommunications carrier. Unfortunately, this carrier neglected to tell the client when one of the redundant paths failed, and compounded that mistake by having the other path fail—which resulted in 28 emergency communications centers served by the network having zero ability to communicate with each other during the outage. Though the network provider was monitoring the network, the client did not have a view into their network, which would have been the case if they had their own network monitoring solution in place. If the client had been able to see into their own network, this problem would not have become nearly as critical.
Here are a few examples of how we have helped other clients prevent such incidents through this effort:
- A telecommunications carrier was monitoring another client’s public safety communications network, but refused to provide the client with any visibility into that network. This is not unusual. Often carriers will say “trust us—we’ll let you know when there’s a problem.” So, the client asked us to assess their network environment and then provide them with a monitoring view into their network. The client, by using our monitoring platform, discovered that there was traffic on the client’s backup circuit. The client confronted the carrier, which admitted the transgression and removed the traffic. The presence of the carrier’s traffic on the backup circuit could have affected public safety communications during a major emergency incident when traffic needed to be diverted to its backup circuit.
- Another client discovered, after an MPC network assessment, that there were backdoors built into their network for maintenance and support, and that it had been paying maintenance on network devices that had been decommissioned for three years. They didn’t have an up-to- date network map to compare with the invoices. They also didn’t realize the risks of having backdoors built into their network, not necessarily maliciously, until we were able to show them what was tied to their current network design.
- Another client in a similar circumstance discovered through one of our NetInform assessments that all of their redundant equipment was located in the same equipment rack as their primary equipment. If the building had been struck by lightning and a power surge found its way to the equipment, all of the client’s primary and backup would have been fried. This has since been corrected.
It’s very easy to see from these examples that visibility into the network is imperative, and not just from a network performance perspective. Cybersecurity is becoming a bigger problem for public safety and government entities. Ongoing visibility into the network helps to identify vulnerabilities that can be addressed before a disaster occurs.
The way to achieve that visibility is through network assessments and monitoring. MCP has numerous platforms and a great many subject matter experts to help you view into your network—please reach out.
David S. Jones is president of MCP’s Lifecycle Management Services division. He can be emailed at DavidSJones@MissionCriticalPartners.com.
