CallHarbor - Notice history

We're happy to report that the data center has restored connectivity at our Atlanta, GA point of presence, and all services out of this facility are now fully operational.

Incident Summary

On Tuesday, May 27, 2025, at approximately 11:14 AM ET, CallHarbor began receiving reports of one-way audio on some calls originating from or terminating through our Detroit Point of Presence (PoP). Our team was able to reproduce the issue through internal testing. Following reproduction of the issue the CallHarbor team initiated a failover to our other Points of Presence to mitigate any impact. While this did resolve the issue some clients may have had a brief period of one way audio on calls between approximately 11:14AM ET to 11:30AM ET provided that their calls traversed our Detroit, MI PoP.

We sincerely apologize for any inconvenience or disruption this incident may have caused to your communications. We understand the critical nature of reliable audio for your business operations.

Timeline of Events

Initial reports and resolution put in place:

• 11:14-11:18 AM ET: Initial reports of one-way audio through Detroit, MI PoP received by CallHarbor Support.

• 11:19 AM ET: Network engineers engaged to begin verification of issue.

• 11:20 AM ET: Internal testing confirmed the one-way audio issue.

• 11:25 AM ET: Decision made to initiate failover of services from Detroit to our Atlanta and Phoenix PoPs.

• 11:28 AM ET: Failover initiated. Devices began successfully re-registering at the Atlanta and Phoenix PoPs.

Investigation and Restoration of Detroit, MI PoP:

• 11:32 AM ET: Status page updated outlining steps taken and that issue has been identified. 

• 11:32 AM ET - 8:30 PM ET: Investigation into the cause of the one-way audio during failover. All CallHarbor-managed equipment and services in Detroit were confirmed to be online and stable, indicating an external factor.

• 9:00 PM ET: Failback to the Detroit PoP commenced.

• 9:06 PM ET: Services in Detroit were fully restored with two-way audio confirmed, and no further issues were observed.

Root Cause

Our investigation determined that the one-way audio incident was not caused by any CallHarbor internal systems or equipment. All CallHarbor infrastructure in our Detroit PoP remained stable and operational throughout the incident. The issue has been attributed to a connectivity problem with an upstream carrier providing service to our Detroit PoP. This upstream issue prevented the proper flow of audio in one direction for some calls.

Resolution

The immediate resolution involved a successful failover of services to our geographically diverse Atlanta and Phoenix PoPs. This proactive measure ensured minimal service disruption for our customers, allowing calls to be routed through unaffected infrastructure. Once the upstream carrier issue appeared to be resolved or mitigated, a controlled failback to the Detroit PoP was executed without incident.

Future Prevention and Resiliency

While we do not anticipate a recurrence of this specific upstream carrier issue, this incident underscores the critical importance of our multi-point of presence architecture. This design, combined with our relationships with multiple upstream carriers, provides significant resilience against external outages. Our ability to quickly failover to alternative PoPs demonstrates our commitment to maintaining high availability and reliable service, even when faced with issues outside of our direct control. We will continue to monitor our carrier connections closely and explore further diversification where appropriate.

Tonight, we'll be failing back phones to our Detroit, MI point of presence. We expect this change to happen at 9 PM ET and be completed by 9:30 PM ET. There's a possibility of a brief period of one-way audio as we finish testing. We will provide updates as we complete the change.

This as well as previous changes from this incident only impact phones that are set to default to our Detroit, MI point of presence, our Atlanta, GA and Phoenix, AZ points of presence remain unaffected.

We are continuing to see calls completing as normal through our Atlanta, GA and Phoenix, AZ data centers.

This incident is now being marked as resolved as we are continuing to see outbound calls processing at normal rates.

Update: The upstream carriers have implemented a fix and we are monitoring the situation. We will consider this incident resolved after monitoring for some time.