Incident Summary
On Tuesday, May 27, 2025, at approximately 11:14 AM ET, CallHarbor began receiving reports of one-way audio on some calls originating from or terminating through our Detroit Point of Presence (PoP). Our team was able to reproduce the issue through internal testing. Following reproduction of the issue the CallHarbor team initiated a failover to our other Points of Presence to mitigate any impact. While this did resolve the issue some clients may have had a brief period of one way audio on calls between approximately 11:14AM ET to 11:30AM ET provided that their calls traversed our Detroit, MI PoP.
We sincerely apologize for any inconvenience or disruption this incident may have caused to your communications. We understand the critical nature of reliable audio for your business operations.
Timeline of Events
Initial reports and resolution put in place:
11:14-11:18 AM ET: Initial reports of one-way audio through Detroit, MI PoP received by CallHarbor Support.
11:19 AM ET: Network engineers engaged to begin verification of issue.
11:20 AM ET: Internal testing confirmed the one-way audio issue.
11:25 AM ET: Decision made to initiate failover of services from Detroit to our Atlanta and Phoenix PoPs.
11:28 AM ET: Failover initiated. Devices began successfully re-registering at the Atlanta and Phoenix PoPs.
Investigation and Restoration of Detroit, MI PoP:
11:32 AM ET: Status page updated outlining steps taken and that issue has been identified.
11:32 AM ET - 8:30 PM ET: Investigation into the cause of the one-way audio during failover. All CallHarbor-managed equipment and services in Detroit were confirmed to be online and stable, indicating an external factor.
9:00 PM ET: Failback to the Detroit PoP commenced.
9:06 PM ET: Services in Detroit were fully restored with two-way audio confirmed, and no further issues were observed.
Root Cause
Our investigation determined that the one-way audio incident was not caused by any CallHarbor internal systems or equipment. All CallHarbor infrastructure in our Detroit PoP remained stable and operational throughout the incident. The issue has been attributed to a connectivity problem with an upstream carrier providing service to our Detroit PoP. This upstream issue prevented the proper flow of audio in one direction for some calls.
Resolution
The immediate resolution involved a successful failover of services to our geographically diverse Atlanta and Phoenix PoPs. This proactive measure ensured minimal service disruption for our customers, allowing calls to be routed through unaffected infrastructure. Once the upstream carrier issue appeared to be resolved or mitigated, a controlled failback to the Detroit PoP was executed without incident.
Future Prevention and Resiliency
While we do not anticipate a recurrence of this specific upstream carrier issue, this incident underscores the critical importance of our multi-point of presence architecture. This design, combined with our relationships with multiple upstream carriers, provides significant resilience against external outages. Our ability to quickly failover to alternative PoPs demonstrates our commitment to maintaining high availability and reliable service, even when faced with issues outside of our direct control. We will continue to monitor our carrier connections closely and explore further diversification where appropriate.
