Gandi Bar

Home > Gandi > Incident that occurred on Monday, January 22

Incident that occurred on Monday, January 22

A (highly unusual) chain of events occurred that led to a kind of 'traffic jam' in the processing of requests. Please note, I am not referring to a DDOS.
Nonetheless, our machines did not respond as we expected, and more specifically, our database showed us its limits...

To those customers that may feel our service is not up to par, I would like to reassure them that our new network architecture is comprised of 3 nameservers located at three different locations, and connected by a fiber optic loop.
We anticipated a gradual increase in load on the new nameservers, though it was abrupt (multiplied by a factor of 60), and this caused the outage: my fault.

To those that wished we could have been more open and transparent with regards to this situation, I must say that I am rather surprised at that, but since this is not the general opinion, I am somewhat reassured. :)
All last night, and all throughout the night, at the critical time of the incident, we were more concentrated on fixing the problem than making excused. This is why everything returned to normal this morning, and I would like to apologize to all of you in the name of the entire staff at Gandi!

We have made several changes in order to consolidate our platform, notably changing our database to BerkeleyDB. This will have an effect on the time needed to update your DNS changes, unfortunately; Changes may now take up to 5 minutes to take effect. This is the price to pay for much more stability, and this what seems to be the most reasonable thing for us to do. The good news is that the response time has been multiplied by 100.

Thank you for your understanding in that our Customer Care Department received a large number of requests yesterday (around 3% of our customers were affected by the outage), and they will be working twice as hard to return to an acceptable response time to your requests.
Today is another day :)