Categories
General Technical

Update on the System Recovery

Hello,

I realize that for many of you this has been a long and troublesome process, and I can thankfully say that ShoutWiki will soon be back up and fully functional.  Currently, I’m working on recovering all the databases; it’s a slow process, but so far all the databases that have been recovered (there are around 700 in all) are completely intact which means that you did not lose any pages, etc. from this crash.

After all the databases are recovered, we will need to basically reinstall the server, and once that is completed ShoutWiki will back in action.

Thank you all for your patience during this time, and I hope that you continue to choose ShoutWiki!

Regards,

Skizzerz
ShoutWiki Technical Team

UPDATE 1 (31 March 2011, 3:14am UTC): Current status is 125 done, 521 to go. Will continue to update as this progresses.

UPDATE 2 (31 March 2011, 3:52am UTC): Current status is 274 done, 372 to go.

UPDATE 3 (2 April 2011, 6:55pm UTC): There were about 10 wikis that didn’t recover properly, so I’m working on a way to recover them. Once this is done, everything else should go relatively quickly.

UPDATE 4 (4 April 2011, 11:56pm UTC): Clarification on the above: When I said they did not recover properly, this does NOT mean that they can’t be recovered. The issue lies with some stuff wrong with the backend behind the dbs, so certain large wikis with lots of revisions and log entries took too long to dump in one go before MySQL crashed. What I’ll be doing for these 6 wikis (got 4 of them to work normally since the last update) is dumping them table-by-table, so your stuff will not be lost. As for an update on how other stuff is going, we are now waiting for the Head Tech to reinstall the Operating System on the server so that we may begin restoring our backups. Due to the nature of the backup restoring process, wikis will become available as the backups get restored, so check back in a day or three to see if your wiki is up.

UPDATE 5 (9 April 2011): OS is going to be reinstalled sometime later today, and provided all goes well, so will the wiki restorations. Wikis will be restored in alphabetical order and will become available as they get restored (meaning some will be up before others). I’ll post another update when the restorations actually start.

UPDATE 6 (16 April 2011): The tech team has taken backups of databases and important configuration files. The CTO estimates that we might get started on the recovery as soon as this weekend, but it might take a major part of the next week. We’ll keep you posted on this. –Jack Phoenix

UPDATE 7 (19 April 2011): The operating system has now been reinstalled and the backup files are being copied to the server currently. Recovery will begin later today. –Jack Phoenix

UPDATE 8 (19 April 2011): Data recovery proper has begun. During this process the site may display various error messages. –Misza

UPDATE 9 (19 April 2011): The wiki restoration process has begun. Wikis should become available as they get restored. They are going in random order, so we cannot give any ETAs on specific wikis. Please note that just because your wiki is back, it does NOT mean that all the restores are done. As such, if issues arise, please alert a staff member on IRC so that we can look into it. We are also considering rolling out the 1.16 upgrade after the wikis get restored, so you may see some downtime there as well. –Skizzerz

UPDATE 10 (21 April 2011): We have restored the vast majority of the wikis, as people were quick to find out, we ran into a LOT of issues trying to restore Autopedia and Simpsons Fanon. We’re going to continue working on ways to restore these two wikis, but in the meantime, we will be pushing out the 1.16 upgrade to the rest of the wikis. After the upgrade finishes, editing will be re-enabled. –Skizzerz

UPDATE 11 (27 April 2011): We’re working on both issues, the MediaWiki upgrade and restoration of Autopedia and Simpsons Fanon databases, but it turns out that MediaWiki’s default upgrade script is not suitable for our purposes as-is, which means that we have to write our own upgrade script. This is something that we want to do right, without rushing, in order not to mess up anything, which is why it might take a bit longer then expected. Once the techs have finished writing the upgrade script, the actual upgrade process should be relatively quick, after which the database will be unlocked and you can resume editing. We’ll keep you posted on this. –Jack Phoenix

UPDATE 12 (1 May 2011): The update process has begun. In the meantime, a “ShoutWiki is currently undergoing maintenance” page will show until updates are complete. Once complete, editing will be re-enabled. ETA is about two hours from now, provided no issues are found. –Skizzerz

UPDATE 13 (1 May 2011): Updates are complete and editing has been re-enabled. If you run into issues, please let someone know on IRC and open a bug in our bugzilla: http://bugzilla.shoutwiki.com — At some point in time we’ll be re-updating to the newest 1.16 version, but this will not cause any downtime and should be for the most part invisible to you guys.

By Skizzerz

ShoutWiki staff member

387 replies on “Update on the System Recovery”

I am really concerned about Autopedia and Simpsoms Fanon. Do you guys know if your able to recover the wikis?

What’s the progress on the MediaWiki upgrade? Also, good luck with recovering Autopedia and Simpsons Fanon.

Six days without and edits. Please make sure we won’t have to wait another month to gain back access to editing.

(YAWN) Falling asleep from boredom while waiting for editing capability to return! Can we PLEASE have an update on the situation? (Even if there’s nothing new to report, saying anything is better than saying nothing.)

Can you tell us how the update is going? I’m not as inpatient as most of the others, but I just want to know how many days/weeks it will take. I’m asking very nicely, y’know.

What are you guys doing at this moment, updating MediaWiki to 1.16, or attempting to recover Autopedia and others?

What;s going on with MW 1.16 upgrade. And how are you progressing with Autopedia and Simpsons Fanon?

New update: UPDATE 11 (27 April 2011): We’re working on both issues, the MediaWiki upgrade and restoration of Autopedia and Simpsons Fanon databases, but it turns out that MediaWiki’s default upgrade script is not suitable for our purposes as-is, which means that we have to write our own upgrade script. This is something that we want to do right, without rushing, in order not to mess up anything, which is why it might take a bit longer then expected. Once the techs have finished writing the upgrade script, the actual upgrade process should be relatively quick, after which the database will be unlocked and you can resume editing. We’ll keep you posted on this. –Jack Phoenix

Is there a way to get a database dump now? Or is that something I have to wait for them to implement? If we can do it now, would somebody please give me some idea how? Thanks.

@Silver Knight

The more careful they are, the less likely that this will never happen again, even if it takes more than three months. As they say: “It tastes better if you wait for your ice cream.”

@Silesianus

I understand you are inpatient, but we should not give up now. We’ve already lost one wiki, y’know.

@Escyos

So what? Just to get Autopedia back will be good enough for me. “You can’t run with the hare and hunt with the hounds.”

@Escyos

You lost more than 1,000 pages when you tried to install an extension without any possibility of backup. You just have one wiki, imagine when you host more than 700 wikis. For the good news, you’re learning from scratch and from your mistakes.

Many people will be leaving ShoutWiki. I will do the opposite once editing is back online. I will be moving a wiki from Wikkii to ShoutWiki once I can. ShoutWiki will not make the same mistake twice and if they do, they’ll be able to sort it out quicker any way.

@Silesianus

I dont think there will be a ”next time”. I think that this is a unique situation that wont happen again. If it does, I will move.

@Escyos

Please try to be realistic. Do you really think that staff were sitting on their asses for a month before attempting to recover it? If you understand the nature of the crash, you’ll understand why it’s been taking his long. I’m a close friend with some of the staff, so I know what they’re doing behind the scenes while you’re all moaning. Frankly, I don’t like your mentality. I figure that you’re not very knowledgable in the technical area, so I’ll just say that theres a big difference between recovering a wiki and recovering a whole will farm. Please look this up before making comments before thinking…

Leave a Reply

Your email address will not be published. Required fields are marked *