This story is a few tears old, but I'll try to remember all the fun parts.

Back then I was working with a company that among other stuff also outsourced telephone services to customers. So they would get their phones from us, all the infrastructure, we did all the technical stuff with the ISP, got everyone their extension and call groups etc.

We had only a hand full of customer who used this service from us, but or company itself of course also relied on it.

Most parts of the infrastructure were customer specific except one. The main entrance/exit server (+backup) into/out of our datacenter. But for our cause, they were so oversized, that no amount of traffic would even be closely able to bring them down. (Or were they)

On usual days we would handle maybe 50-100 external calls simultaneously. Cause remember, those servers were to the outside. All other traffic would not touch them. The servers were (according to the specs) able to do 4000 simultaneous calls.

To the day of the incident. It began around 8 in the morning. We would get a few incidents reporting calls not being established, which we brushed off at first, cause it was more probable that the other site was at fault.

Later one of our customers also opened up incidents reporting this in mass. At this point, we were getting a little worried and looked into the logs. What we found was not fun. Much to our dismay, we saw that we had around 7000 simultaneous calls trying to bomb our system. Most of which were trying to reach one specific customers call center.

After a while we found out that this customer had a countrywide mandatory survey they didn't tell us about. For this survey an external call center was hired to handle all the calls.

We hopped into a call with them and found out a few things: They were expecting about 15-20k calls a day, and their contract said something about "up to 2k" and when questioned, how that would work, they told us about a specific rule in their contract with their ISP. This rule meant that all calls above the 2k limit would get a "number is busy" kinda answer and had to wait or hang up.

We called the ISP. They just told us (and the customer in the same call): "Yeah, we sell that feature, but that doesn't really work and mostly isn't even used..."

So the ISP broke their contract but were to big to fail and the customer didn't tell us enough, but was angry our stuff didn't work.

End of the story was, that we rerouted all the calls directly to the call center and then the call numbers dropped back to a few hundred.

Edit: Survey was mandatory.