Disk problems (being fixed)
By Ryan on Sunday 18 May 2008, 15:32 - Gandi - Permalink
Things are progressing nicely, and the incident caused by the loss of filer 13 which impacted customers is now in the process of being fixed.
The servers have been re-created one at a time since 10:00 AM (GMT) this morning and this long process will be completed by tomorrow for the last affected customer. We therefore ask for just a bit more patience...
It is to be noted that new disk and server creations have been suspended until Monday in order to minimize the waiting time needed for customers who are still blocked.
As mentioned at the beginning of the incident, customers on filer 13 who were impacted will receive a full refund.
Also, we will keep you informed as to the RAID architecture changes that we are in the process of developing, and that will prevent this type of scenario from ever reoccurring.
Additionally, we are going to discuss all this with the manufacturers.
Console access will be granted to everyone tomorrow, which will allow those that were not on the Filer of Death though nonetheless affected by the side effects to regain access to their machines.
Once more, we apologize for the inconvenience this may have caused.
The servers have been re-created one at a time since 10:00 AM (GMT) this morning and this long process will be completed by tomorrow for the last affected customer. We therefore ask for just a bit more patience...
It is to be noted that new disk and server creations have been suspended until Monday in order to minimize the waiting time needed for customers who are still blocked.
As mentioned at the beginning of the incident, customers on filer 13 who were impacted will receive a full refund.
Also, we will keep you informed as to the RAID architecture changes that we are in the process of developing, and that will prevent this type of scenario from ever reoccurring.
Additionally, we are going to discuss all this with the manufacturers.
Console access will be granted to everyone tomorrow, which will allow those that were not on the Filer of Death though nonetheless affected by the side effects to regain access to their machines.
Once more, we apologize for the inconvenience this may have caused.
Comments
It's "tomorrow", and my server is still "Status: Blocked". Do you have an updated estimate? Thanks.
Ya! I'm also getting a little impatient here to, my server is down since Thursday
Hello Chris, Desi,
Thanks for your patience. The disk creation process is taking a bit longer than anticipated, and so while we planned on having the situation return to normal this afternoon, we unfortunately require a bit more time to do so in a careful manner.
We will post updates here on a regular basis so that you will be able to follow the developments as they unfold.
I'm not very impressed by the speed and continual problems with Gandi Mail...the systems don't seem robust enough for people who are trying to run businesses that rely on the internet and email...are things going to improve - and get faster...I really need to know. I came to Gandi on the reccomendation of Moonfruit/Sitemaker and I'm regretting it.
I also find it irksome that it defaults to French everytime I use it - is there not a way to set it in my native language once and for all?????
I understand that some of you (Gandi team) are tired but some emails from customer service and some of your posts on these forums make me feel as if we customer were responsible (...guilty!) for what is happening.
w00t tout remarche bien pour moi. Merci Gandi!
I don't feel that Gandi are blaming on customers (at least the forums' posts don't look that way) but it's true that _again_ the mail service is down and if there's something critical nowadays, that's email.
@Pablo
Well, reading today "We therefore ask for just a bit more patience... " while being told on Friday that the last (and unlucky) will have their server working on Monday afternoon (and just after thinking on Thursday last week that a filer crashed all data would be lost but that we could recreate virtual servers on another filer) I just feel as if we are bothering them asking for news. Moreover because I received an empty email from customer service, another with a link to this blog while I was asking a console access as recommended precisely in this blog (one 10 hours after sending, the other one 20 hours after). I am not complaining that the incident happened nor that it is taking a really long time to solve it, I like that they are not hiding information but I dislike a lot the false hopes that they are giving : the first one see advertisement of the system (watching it anyone would believe that the system has a lot of redundancy - not only a RAID6 on 1 filer) the second one that everything would be solved for everybody today afternoon.
Sebastien, the empty mail was due to a problem with our support's email system (we are working with the software provider to see what happened)...I assure you that Sam gave you a nice reply (reply to the mail and we'll re-send the message). For instructions on console access, just add that to your mail
Otherwise, the very existence of this blog (in addition to the groups and unlimited free support) is to give you information...we love hearing from you! We do our best to provide time tables for these fixes, though as you know, sometimes things end up taking longer than expected (especially when fixing something other than treating just the symptom) We'll continue to keep you posted here - but please feel free to write to support for specific questions!
And as a reminder: the hosting service is in beta testing phase, meaning that it is not the final product, and we are using this phase to find all the problems *before* we go 'live' ;)...
Ryan. The hosting service is in beta testing phase, that's correct. What about the email service? I keep getting connection errors and I'm fearing the worst.
Are those problems (hosting+email) related?
Regards,
Pablo
Email (pop, imap, webmail) is completely down since Sunday. Does this filer problem have any impact on emails ? I'd like to know if I have really lost my 3G of data. Thank you for your answer.
@Pablo and Chris: The e-mail and hosting platforms are not related. The problem with the e-mail that you are experiencing is because we are getting hit with real attacks of spam from all over which is causing an unanticipated and unusual load on the mail servers.
So with regards to the mail, the problem's origin it is more human than technical in nature. Unfortunately, it so happens that right now both are occurring at the same time, and since our developers have been working day and night (yes...that is more than just an expression) to fix the hosting, well the mail issue came up at a really bad time...
You see, the mail platform is surprisingly complex, but despite all this, the devs (as we call them) managed to implement improvements to the mail platform this afternoon, thus helping the situation - though we still have some work to do there.
What they are doing is heroic, and the devs really deserve all the credit (and thanks) that can be given to them. The entire team is making real sacrifices to assure that we get everything up and running for you as quickly and as professionally as possible, because we honestly do care!
Do you have any estimate of when the email will be up again ??
Thanks Ryan!
Email seems to be up and running again though if it goes down during the next hours I'll just re-read your post
Keep up the excellent work.
Pablo
@Ryan
thank you for your answer that clarifies the point of the emails. As for the things taking longer than expected, you're right it happens everywhere and it is the case here. It is close to 5am in France and still no virtual server. I guess I'll have to spend the night making my setup for tomorrow on another machine. Actually I would have preferred being told "Filer 13 is out and we don't intend to replace it immediately" (if that was the reason of the delay after filer14 started to make weird things) or "if things are working correctly it sould be fine by Monday afternoon; it may take longer" rather than "by Monday afternoon everything should be fine for the last ones, some of them are already up"
i can't attach my disk to server
it said my disk attached but it doesn't really attach
i cant see my disk being mount in /srv/
How can one found out which "filer" is ones partition/share(s) hosted on?
/J
N6t s4re what 5s g65ng 6n w5th the 2eyb6ard+...4t 4s b5t eveb tt64bg r4ggt / 4 d5b-t 1b5w 4f t5y see a22 tge bynbers abd wr6ng 3etters+++ Any h6w, 5 cann6t 36g 5nt6 0y e0a53 acc64nt+ and 5t 5s 12-42 4s 0st t50e+++