Hello,
I realize that for many of you this has been a long and troublesome process, and I can thankfully say that ShoutWiki will soon be back up and fully functional. Currently, I’m working on recovering all the databases; it’s a slow process, but so far all the databases that have been recovered (there are around 700 in all) are completely intact which means that you did not lose any pages, etc. from this crash.
After all the databases are recovered, we will need to basically reinstall the server, and once that is completed ShoutWiki will back in action.
Thank you all for your patience during this time, and I hope that you continue to choose ShoutWiki!
Regards,
Skizzerz
ShoutWiki Technical Team
UPDATE 1 (31 March 2011, 3:14am UTC): Current status is 125 done, 521 to go. Will continue to update as this progresses.
UPDATE 2 (31 March 2011, 3:52am UTC): Current status is 274 done, 372 to go.
UPDATE 3 (2 April 2011, 6:55pm UTC): There were about 10 wikis that didn’t recover properly, so I’m working on a way to recover them. Once this is done, everything else should go relatively quickly.
UPDATE 4 (4 April 2011, 11:56pm UTC): Clarification on the above: When I said they did not recover properly, this does NOT mean that they can’t be recovered. The issue lies with some stuff wrong with the backend behind the dbs, so certain large wikis with lots of revisions and log entries took too long to dump in one go before MySQL crashed. What I’ll be doing for these 6 wikis (got 4 of them to work normally since the last update) is dumping them table-by-table, so your stuff will not be lost. As for an update on how other stuff is going, we are now waiting for the Head Tech to reinstall the Operating System on the server so that we may begin restoring our backups. Due to the nature of the backup restoring process, wikis will become available as the backups get restored, so check back in a day or three to see if your wiki is up.
UPDATE 5 (9 April 2011): OS is going to be reinstalled sometime later today, and provided all goes well, so will the wiki restorations. Wikis will be restored in alphabetical order and will become available as they get restored (meaning some will be up before others). I’ll post another update when the restorations actually start.
UPDATE 6 (16 April 2011): The tech team has taken backups of databases and important configuration files. The CTO estimates that we might get started on the recovery as soon as this weekend, but it might take a major part of the next week. We’ll keep you posted on this. –Jack Phoenix
UPDATE 7 (19 April 2011): The operating system has now been reinstalled and the backup files are being copied to the server currently. Recovery will begin later today. –Jack Phoenix
UPDATE 8 (19 April 2011): Data recovery proper has begun. During this process the site may display various error messages. –Misza
UPDATE 9 (19 April 2011): The wiki restoration process has begun. Wikis should become available as they get restored. They are going in random order, so we cannot give any ETAs on specific wikis. Please note that just because your wiki is back, it does NOT mean that all the restores are done. As such, if issues arise, please alert a staff member on IRC so that we can look into it. We are also considering rolling out the 1.16 upgrade after the wikis get restored, so you may see some downtime there as well. –Skizzerz
UPDATE 10 (21 April 2011): We have restored the vast majority of the wikis, as people were quick to find out, we ran into a LOT of issues trying to restore Autopedia and Simpsons Fanon. We’re going to continue working on ways to restore these two wikis, but in the meantime, we will be pushing out the 1.16 upgrade to the rest of the wikis. After the upgrade finishes, editing will be re-enabled. –Skizzerz
UPDATE 11 (27 April 2011): We’re working on both issues, the MediaWiki upgrade and restoration of Autopedia and Simpsons Fanon databases, but it turns out that MediaWiki’s default upgrade script is not suitable for our purposes as-is, which means that we have to write our own upgrade script. This is something that we want to do right, without rushing, in order not to mess up anything, which is why it might take a bit longer then expected. Once the techs have finished writing the upgrade script, the actual upgrade process should be relatively quick, after which the database will be unlocked and you can resume editing. We’ll keep you posted on this. –Jack Phoenix
UPDATE 12 (1 May 2011): The update process has begun. In the meantime, a “ShoutWiki is currently undergoing maintenance” page will show until updates are complete. Once complete, editing will be re-enabled. ETA is about two hours from now, provided no issues are found. –Skizzerz
UPDATE 13 (1 May 2011): Updates are complete and editing has been re-enabled. If you run into issues, please let someone know on IRC and open a bug in our bugzilla: http://bugzilla.shoutwiki.com — At some point in time we’ll be re-updating to the newest 1.16 version, but this will not cause any downtime and should be for the most part invisible to you guys.
387 replies on “Update on the System Recovery”
That pisses me off!!! Autopedia is the one that is truly suffering the most and ShoutWiki hasn’t recovered yet.
Out of all the wiki hosts I’ve seen, I’ll stay on ShoutWiki. Just make sure that after this is over it won’t happen again.
For all those that are thinking of leaving ShoutWiki:
This will most likely be a one off. ShoutWiki will learn from their mistakes and this will probably not happen again and if it does, it won’t be to this scale.
There are no other hosts that are as good as ShoutWiki (except everyone’s least favourite host, Wikia) and most others don’t come close to ShoutWiki in how good they are.
Whether the Simpsons wikis stay on ShoutWiki or go independent, I will keep coming to ShoutWiki to make other wikis or work on other wikis.
Sure, this was bad. But other hosts are slow and have less features than ShoutWiki (Wikkii doesn’t allow favicons unless you pay for premium, makes ShoutWiki’s Monaco premium seem much better).
So people, bear in mind that even though ShoutWiki cocked up, they are still the best host out there.
Also, do you have a time that editing will be allowed again?
I agree with Solar Dragon.
@Solar Dragon
I hope Autopedia is back up before editing starts. We’ve easily lost over 40 cars.
Get Simpsons Fanon back up! I’ve got about 9 fanon stories ready to put on Simpsons Fanon! Also, get Autopedia back up!
Both Stargate Wiki (FR and US) are back, good job guys, just a problem with svg conversion.
Goddamn it!!! Why are all the sites up except Autopedia and Simpsons Fanon? I’d be really happy to finally see the main page of Autopedia.
The censorship going on in the comments is interesting. Wikia was supposed to be the one trying to stifle anyone suggesting that any Wikia wiki would be moving away. But anno1404’s comment linking to a post suggesting awa was moving somewhere else was deleted.
Then when Escyos tries to say you can have favicons on Wikkii that gets deleted too.
Solar Dragon followed up with a comment about Escyos’ favicons being a js hack not actual favicons, and it was deleted too.
Interesting double standard.
You can also add 1 post I made to the pile that was deleted.
It’s been nearly two days without editing. Come on ShoutWiki! Get Simpsons Fanon and Autopedia up! Off topic: It’s 36.7 Celsius where I’m living!
okay, this is getting ridiculous, I appreciate shoutwiki deleting my two comments first reprimanding then mocking the two editors who were fighting here and deleting many of their posts. It stopped the argument.
But deleting my post where I give a link to a backup of our uneditable site on wikkii and ask two editors here to comment there on whether we should move? That is very wikia like.
@ solar dragon
RE: But other hosts are slow and have less features than ShoutWiki
Shoutwiki for months was so slow it was uneditable. It is so nice to be on a wiki for the first time in months which is not slow.
RE: (Wikkii doesn’t allow favicons unless you pay for premium, makes ShoutWiki’s Monaco premium seem much better).
Have you volunteered to pay for the monaco premium on shoutwiki? Favicons (the small icon at the top of the page next to the address) are minor to me. A stable editable wiki is much more important to me.
OK. Could you please stop blaming ShoutWiki for this? THE PROBLEM WASN’T THEIR FAULT, AND IT’S NOW SOLVED! They just want to be sure that the wikis won’t crash again. CAN’T YOU WAIT TWO OR MORE DAYS?
What’s the problem now? ShoutWiki deleted comments in which you were insulting them? And some in which you were linking to other websites? If you had a wiki host, would you like people linking to other hosts? In this case, Wikkii. JUST WAIT TWO DAYS, PEOPLE! PLEASE!
And you moved awa to Wikkii. Just because ShoutWiki is uneditable for now. You’re not patient, are you?
Any word on when editing will be enabled again? It’d be nice to know if it’ll be a few more hours, few more days, or few more weeks.
OK. I’m getting nervous that there is a problem with the backups of Autopedia and Simpsons Fanon. Why are they not back yet?
Yeah, why can’t you recover Autopedia and Simpsons Fanon? Is there something wrong?
@Silver maybe ask on IRC. Skizzerz recommended to do that.
I’m on ShoutWiki’s IRC channel now; Skizzerz isn’t responding.
Is he on the channel, though?
I just quit and before I did he was.
@Daniel Friesen
>Interesting double standard.
There are no double standards. It is about “no link to competitor wiki”.
I saw anno1404`s comment – it contained the link to another wiki-hosting service. From ShoutWiki`s point of view – he has advertised a rival site. Would you allow anyone to advertise a rival site in your blog?
@TheHomer
I can blame them for one thing only – they should have been posting more often. Maybe even every 72 hours.
In some situations I may understand the approach “Do not post until everything is fine”. But in such critical situation it works against them: Even if you are working for the solution, each day you are not posting anything, they will think you are not doing anything.
@anno1404
I think they did not delete your comments but restored this blog to some older state (my guess is they were setting the system through trial and error method that’s why it took so long).
@ShoutWiki: You’ve promised to implement the on-demand wiki backup and when the dust will settle please, guys, do not forget this!
WHAT IS WRONG WITH Autopedia and Simpsons Fanon?!
Oh, Just saw update.
Not liking the sound of update 10…
They’ll find a way to fix it, I’m pretty sure.
They’d better. Autopedias my second most edited wiki. But I have confidence in Skizzerz and the team to get it done.
Any idea when the MW 1.16 upgrade will be complete? And when and if Autopedia will come back.
Jesus Christ!! Why can’t you recover Autopedia? I thought it was just an normal wiki.
@silver
Probably some tech issues, it can’t be database size, as my wiki with 43,511 pages was restored just fine. Just nervous about this 🙁 Better inform RM.
Simpsons Fanon has only got about 150 pages. But many contributors like the wiki and it was beginning to grow rapidly before this. Hopefully, it can be recovered.
Murphy’s law. “Anything that can go wrong, will go wrong”. Something I don’t want to happen is happening. I hope Autopedia can recover.
I don’t know how many wikis use SVG images, but there seem to be issues arising with them pot-restoration – “Error creating thumbnail: sh: rsvg: not found”.
@Michael Von Preueben
Thats happening to some images on my wiki. The same problem happened after it was imported
@Silver I would imagine they can recover it. Just unfortunate that the 2 wikis theyve had problems restoring aren’t empty (1 page).
I hope i can edit to easterday, this sundy!
What do you mean the two wikis they’ve had problems with have only got one page?
I wish the two wikis that they have problems with only had one page, instead of Thousands.
So, two wikis are blocking other wikis from editing?
@Silesianus
No. They are experiencing problems but the reason editing is still locked is because they are installing MediaWiki 1.16 to ShoutWiki.
Well? Is Autopedia coming back or not? The community has put a TON of work into the site, it’s got 4500 pages, and it would be CRUSHING to the community if it doesn’t come back. I guess the same can be said for Simpsons Fanon.
I just hope that the two wikis return. And I bet that the MediaWiki 1.16 will offer more stuff to ShoutWiki.
Please….I beg you. bring Autopedia back. If you can’t, our whole community will be destroyed.
@Hockeyben, Silver Knight:
I can appreciate your frustration, but please understand that Autopedia and Simpsons Fanon are not still down because of our ill will.
As Skizzerz wrote in update #10, we are working on it. And no amount of shouting, begging or writing in ALL CAPS will make it any faster.
Once we’ve made progress, you can count on an update.
@Silver Knight, Hockeyben:
They’re also focusing on getting all the other wikis up and running first.
I can see that SVG images bug is fixed. It’s a progress.
Thanks for the update, Misza. I was wondering if you had an ETA on when the MediaWiki updates will be done and editing can resume?
Nicely done so far on the restorations. I’ve been checking around Wikisimpsons, and except for the changes lost due to your (hopefully former) host’s apparent ineptitude with backups, everything seems to be in place and looking fine.
Can we have another update please?
@Misza
Thanks for the update. I guess myself (and Silver too) were frustrated because we thought it wasn’t being worked on. Anyway, thx for the update and good luck with your efforts.:)