Hi all,
as you’ve probably noticed by now, there was an outage on 14 March, caused by a malfunction in our host’s datacenter. As a result of this, our host had to restore the site from a backup — which was dated 7 March. This unfortunately means that every single page, wiki and user account created after 7 March has been lost and needs to be recreated.
We at ShoutWiki realize how frustrating this is. It was frustrating for us, too, when we noticed the situation and realized that it’s basically out of our control. We’ve been working on getting the site back up since yesterday (15 March), but right now MySQL daemon process is having issues, and as a result the wikis do not work.
Our tech team are working on getting the site back up as soon as possible. As a result of this unfortunate data loss, we will be looking into building a dump system that allows the steward (founder) of a wiki to generate a database dump on-request.
Apologizing for the behalf of everyone,
Jack Phoenix
Head of the Customer Support Team
30 replies on “Site outage on 14 March 2011”
Thanks for letting us know what’s going on. Does this lost data affect bans and image uploads as well? We banned a few users sometime last week for spam.
@Icelilly
Bans don’t matter so much, spammers usually leave when they get banned. If the users in question come back, just re-ban them.
I would like to know if binary logs were enabled for the databases, where are located in the servers, and what happened to them so they are uncrecoverable.
Also, what is the backup plan for the databases… More than 1 day for each backup is unacceptable.
I assume there’s no database replication on ShoutWiki… too bad.
Site outages are unpredictable, but having a good database backup plan ensures that the outage has the minimum posible impact. It seems that on ShoutWiki somebody hasn’t done well it’s job. On whatever business, the responsible of that would have lost his/her job already.
The site is still down.
Database dumps are not an effective way to prevent this, specially if every bureaucrat needs to request them instead of being generated automatically.
Yeah the site is down again. Any idea when it will be back up?
This is unacceptable on two counts: First, that it’s taking so long to get things working agin; and second, that there wasn’t a more recent backup available than March 7th.
I hope your next blog entry consists of an even bigger apology and a Roll Call of Rolled Heads … speaking metaphorically, of course, but this is the kind of SNAFU that in a business environment would result in jobs being lost. Please make sure that the appropriate consequences are meted out.
@Icelilly: I’m afraid so.
@Ciencia Al Poder: You might wanna email our tech team (technical at shoutwiki) about that; I’m not exactly the local MySQL guru.
@FH14: No ETA yet, but the techs are working on it to get the site back up as soon as possible.
@Mythigator: While you do have a valid point, blaming doesn’t help anyone at this point. We need all hands on deck, so to speak.
I take it this means ShoutWiki doesn’t do off-site backups of their own.
Keep in mind that kind of reliance on your hosting provider is the same thing that killed ScribbleWiki… history is crying.
Hey Myth, you know this could easily happen whatever server we are on…
Good to finally get some notice of whats going on. Also a good thing i did very little editing over that week.
I founded new Wiki (rodokmen wiki) after 7 March. Does it means, that this Wiki now doesn’t exist at all or just doesn’t have articles?
Good luck with getting the site up! Did some major changes to my wiki after 7 March, so that’s a real bummer. Oh well, I just hope you can get it back up and running as soon as possible 😉
Ouch, what a thing to happen on White Day! ;_; I hope you get the problem fixed soon, or at least as soon as YOU guys can; I know you’ve got your lives and other such things to deal with as well. Good luck!
OK I’m starting to get annoyed…….
I’m with Escyos on this one. This is starting to get pretty ridiculous. On monday it’s going to be a whole week, come on guys.
It’s been almost a week now. Is the site coming back up or not?
I’m getting annoyed as well. My wiki was deleted and I really want to re-create it. Please fix the website soon. Thanks.
Hopefully it will be brought back soon… 🙁
Im now beyond annoyed and am pissed off. When is the site coming back? Is anyone even working on this? Can we get some sort of timeline or update on this?
I’m right there with FH14 and Ecosys about being fed up with the lack of progress and wondering what in the world is going on. Could we PLEASE have some news, or an update?
Their comments also underline the point I made earlier about your needing to mete out some accountability to the parties responsible for the outage/inordinately long recovery and the data loss. If you fail to do so, you will have an uphill battle getting people to take you seriously as a wiki provider and trust you with their content.
In other words: Roll some heads, regain your cred.
As for your “we need all hands on deck” response, I submit that the long outage and data loss seem to indicate that you DIDN’T have all hands on deck to start with, and may even have some deadwood issues to address. I repeat: Roll some heads, regain your cred. Best of luck.
1 week has passed.
Status of the situation?
A WEEK? STILL DOWN! I need most of the information I write, it’s a big project a do!
Newest status update has been published; we’re still working on it, but this recovery process is something we cannot afford to mess up.
@Mythigator, deferring blame to subordinate workers is how you lose credibility, not gain it. The person in charge appears to be taking full responsibility and that is exactly how it should be.
So that’s why I can’t get to the Hetalia Wiki……
That sucks big time. =-=
@Molly: **Somebody** lower down than Jack screwed the pooch, hence the data loss. If it was a third party that failed to keep the backups current, then SW needs to make a decision about whether to continue doing business with them. That’s accuntability, too.
I do agree, though, props to Jack for being the point man and taking ownership of the situation. That’s certainly not an easy job.
I can get back on the shoutwiki website! Does that mean it’s fixed for good? Or is there another version of the website? Thanks!
Hey, sorry to bug you guys so soon after my last posting on April 29, but is it true that Hetalia World Series has come to an end and the entire Hetalia anime is over?? I thought something was kinda fishy when I saw Seychelles in the anime rather than just in the manga… I was wondering if you guys had that info. Thanks again!
Annnnnd the site seems to be out again… though I figure you guys know that. Here’s hoping it comes back soon!