Categories
General Technical

Update on the System Recovery

Hello,

I realize that for many of you this has been a long and troublesome process, and I can thankfully say that ShoutWiki will soon be back up and fully functional.  Currently, I’m working on recovering all the databases; it’s a slow process, but so far all the databases that have been recovered (there are around 700 in all) are completely intact which means that you did not lose any pages, etc. from this crash.

After all the databases are recovered, we will need to basically reinstall the server, and once that is completed ShoutWiki will back in action.

Thank you all for your patience during this time, and I hope that you continue to choose ShoutWiki!

Regards,

Skizzerz
ShoutWiki Technical Team

UPDATE 1 (31 March 2011, 3:14am UTC): Current status is 125 done, 521 to go. Will continue to update as this progresses.

UPDATE 2 (31 March 2011, 3:52am UTC): Current status is 274 done, 372 to go.

UPDATE 3 (2 April 2011, 6:55pm UTC): There were about 10 wikis that didn’t recover properly, so I’m working on a way to recover them. Once this is done, everything else should go relatively quickly.

UPDATE 4 (4 April 2011, 11:56pm UTC): Clarification on the above: When I said they did not recover properly, this does NOT mean that they can’t be recovered. The issue lies with some stuff wrong with the backend behind the dbs, so certain large wikis with lots of revisions and log entries took too long to dump in one go before MySQL crashed. What I’ll be doing for these 6 wikis (got 4 of them to work normally since the last update) is dumping them table-by-table, so your stuff will not be lost. As for an update on how other stuff is going, we are now waiting for the Head Tech to reinstall the Operating System on the server so that we may begin restoring our backups. Due to the nature of the backup restoring process, wikis will become available as the backups get restored, so check back in a day or three to see if your wiki is up.

UPDATE 5 (9 April 2011): OS is going to be reinstalled sometime later today, and provided all goes well, so will the wiki restorations. Wikis will be restored in alphabetical order and will become available as they get restored (meaning some will be up before others). I’ll post another update when the restorations actually start.

UPDATE 6 (16 April 2011): The tech team has taken backups of databases and important configuration files. The CTO estimates that we might get started on the recovery as soon as this weekend, but it might take a major part of the next week. We’ll keep you posted on this. –Jack Phoenix

UPDATE 7 (19 April 2011): The operating system has now been reinstalled and the backup files are being copied to the server currently. Recovery will begin later today. –Jack Phoenix

UPDATE 8 (19 April 2011): Data recovery proper has begun. During this process the site may display various error messages. –Misza

UPDATE 9 (19 April 2011): The wiki restoration process has begun. Wikis should become available as they get restored. They are going in random order, so we cannot give any ETAs on specific wikis. Please note that just because your wiki is back, it does NOT mean that all the restores are done. As such, if issues arise, please alert a staff member on IRC so that we can look into it. We are also considering rolling out the 1.16 upgrade after the wikis get restored, so you may see some downtime there as well. –Skizzerz

UPDATE 10 (21 April 2011): We have restored the vast majority of the wikis, as people were quick to find out, we ran into a LOT of issues trying to restore Autopedia and Simpsons Fanon. We’re going to continue working on ways to restore these two wikis, but in the meantime, we will be pushing out the 1.16 upgrade to the rest of the wikis. After the upgrade finishes, editing will be re-enabled. –Skizzerz

UPDATE 11 (27 April 2011): We’re working on both issues, the MediaWiki upgrade and restoration of Autopedia and Simpsons Fanon databases, but it turns out that MediaWiki’s default upgrade script is not suitable for our purposes as-is, which means that we have to write our own upgrade script. This is something that we want to do right, without rushing, in order not to mess up anything, which is why it might take a bit longer then expected. Once the techs have finished writing the upgrade script, the actual upgrade process should be relatively quick, after which the database will be unlocked and you can resume editing. We’ll keep you posted on this. –Jack Phoenix

UPDATE 12 (1 May 2011): The update process has begun. In the meantime, a “ShoutWiki is currently undergoing maintenance” page will show until updates are complete. Once complete, editing will be re-enabled. ETA is about two hours from now, provided no issues are found. –Skizzerz

UPDATE 13 (1 May 2011): Updates are complete and editing has been re-enabled. If you run into issues, please let someone know on IRC and open a bug in our bugzilla: http://bugzilla.shoutwiki.com — At some point in time we’ll be re-updating to the newest 1.16 version, but this will not cause any downtime and should be for the most part invisible to you guys.

By Skizzerz

ShoutWiki staff member

387 replies on “Update on the System Recovery”

Great job, ladies and gentlemen of the technical team! Thank you so much for keeping us informed, it made all the difference.

Skizzerz,

That’s great news, and thanks for the updates. Quick question, though: Jack said earlier that any changes (new pages, edits, new wikis, new users, etc.) made after March 7th would be lost, apparently because that was the most recent backup to which the databases could be recovered. So there would be about a week’s worth of data loss. Is that no longer the case?

If you’re able to recover the data to a point in time right before the crash, that would be great! At Wikisimpsons, some of us editors have been able to copy recently changed and newly created articles (between March 7 and the crash) from web browser caches and Google’s cache, but I wouldn’t mind at all not needing them after all. 🙂

Anyway, I’d just like a bit of clarification on the situation with the data. Thank you!

@Mythigator: That is still unfortunately the case, everything made after March 7th is gone. The issue we are dealing with here is that the backup that the datacenter restored was corrupted beyond repair.

Not the news I was hoping for, but thanks for the answer, Skizzerz. Please choose your words a bit more carefully — saying “all the databases that have been recovered are completely intact” conveys the impression that there was no data loss at all.

@Hockeyben: Troo dat! I just think it’s wise to not inflate people’s expectatioons.

Your handle prompts a question … do you edit on or own a hockey-related wiki on Shout?

@Myrhigator: Yes, I own icehockey.shoutwiki, and nhl.shoutwiki, the latter will have to be re-created, because it was made on March 12. You’re welcome to come and contribute there if you want.

Do you have any idea on how long it should take to recover all of the databases, and reinstall the server?

I believe dynamic hosting from big hosts like 1and1.com can dynamically handle almost any bandwidth, memory, server, and backup needs. I see their multi-page ads in PC World magazine frequently. They can instantly allocate server space, RAM, RAID, and bandwidth depending on the load. Here are some direct links to virtual and dynamic cloud server info:
http://order.1and1.com/xml/order/Server;TCpfix141b
http://order.1and1.com/xml/order/CloudDynamicServer;TCpfix141b

@Timeshifter, if you’re looking for cloud hosting there’s also Rackspace Cloud’s Cloud Servers, and ElasticHosts. If you can take a shared-like setup MediaTemple’s (gs) and Rackspace Cloud’s Cloud Sites scale on their own.

Isn’t this a good chance to delete some of the inactive wikis? For example, the mass made in October to December, which haven’t had any edits since.

The only one of mine that is active is Ice Hockey Wiki. My other 3 are in the import queue. @George Some of the wikis with no edits haven’t been imported yet.

@UxieLover1994 All wikis are still down. I think mine might be one of the ten, but everyone thinks that one of the then is theirs despite the fact that there are over 700 wikis on shout!

Is there any way of getting things up and running for the 690-plus wikis that are “safe in the fold” (as it were), while you continue to work on the ten that didn’t restore properly?

I take a wild guess that it’s not the problem with the wikis restoration at all. And the things will look more grim if I continue to guess from there. 🙁

@Mythigator: Can’t get anything back “up-and-running” until the OS is reinstalled. After that, the restoration of backups will be fairly fast (give maybe 5-10 minutes per wiki being restored). However, since the restoration can be automated, that should not be a huge deal compared with the fact that dumping the databases required 2 manually-entered commands per dump that could not be automated.

Skizzerz, you left me with the impression you had ten wikis which you needed to figure out how to recover, since the auomated process didn’t work, and THEN you’d be able to reinstall the OS, and from there things would go fairly quickly. Let me rephrase my question: Are those ten wikis the “stopper” on making further progress? I GET that the OS reinstallation needs to happen before anything will work, but is there something preventing you from doing that reinstallation, getting the 690-plus successfully recovered wikis working again, and then at your relative leisure re-addressing the ten wikis that you’ve been having problems with?

I’ve been working in IT for 15 years, so please don’t talk down to me. Also, strictly as an advisory note, I’m a professional writer and researcher (which means I know better than to settle for half-baked answers), so don’t try to BS me. Comprende?

What I would like is some straight-up answers about how things are going and what is the anticipated timeline for getting things working again.

To be fair, I don’t think you’re deliberately trying to BS me or anyone else. It’s more on the level of “what we have here is a failure to communicate”, and not even all that great a failure.

Anyway, I would like some clear answers. If you need clearer questions, let me know and I’ll have another go at asking. Thank you.

This is absolutely atrocious! Do you want to know what you did to our Autopedia community? We’ve lost over 2 weeks worth of car updates. It’s been so long that recovering them may seem impossible.

@Silver At least the site should be back soon. Yes, we did lose a lot, I lost all those Jeep pages. I think we lost some stuff from Geneva Auto Show too. The merge from Wikicars could be done by now.

I really hope the site is back by Friday. I’m sure my wiki is one of the six — It is very large.

F**K!! This is such bulls**t!! I switch over from Wikia and Wikicars and me and Red marquis get this!! There are way too many missed car blogs and updates that we can ever recover from!! Its all your FAULT!! Shoutwiki…..

I feel your anger Silver Knight, I wanted to destroy the wikia stargate site with better written articles, more pages and more detail yet had almost missed a month worth of work.

I sure as hell want some sort of assurances that this kind of thing won’t happen again. I also want the use of the monaco skin for free for the days that were lost. Only fair considering the amount of time lost.

Okay, I’ve been trying to wait patiently, but my patience is running out. Please provide us with some more updates to show us that you are working hard to get everything back together.

Okay, I’m giving these guys one more week. If ShoutWiki is not back up by next Friday, I’m moving my wiki (again).

Anyone who agrees with me should also move away from ShoutWiki. Anywhere but Wikia should do.

What happened to one to three days for having the wikis back? First it was a few days, then April 1, then April 4. It’s April 7 now. I recognize there’s a lot to do, but you guys keep setting dates for them to be back up, and keep missing them. I will not be moving, as I’ve been very happy with everything here at Shout. But if another debacle like this happens, chances are I will move.

OK now, I have been very patient and been trying to stay neutral on a decision to move away from ShoutWiki being head of Wikisimpsons and feeling very attached to ShoutWiki. However, this is really p***ing me off now and I am leaning towards moving away now just as you guys can’t keep to your dates you set. I like ShoutWiki and you guys were great until the crash but this is ridiculous!

Leave a Reply to Silver Knight Cancel reply

Your email address will not be published. Required fields are marked *

buy male enhancement pills near me do penis enlarging pills wirk biolyfe cbd gummies ed penguin full spectrum gummies donde venden power cbd gummies target gummy candy enhancement pills for male rhino 24k male enhancement pill where can i get cbd gummies for sex does penis enlargement pills really work male enhancement pills happy passenger extensions iv male enhancement pill male enhancement pills on walgreen super cbd gummies ed best over the counter male enhancement pill rhino para que sirve keoni cbd gummies best sexual enhancement pills for males male enhancement pills on priscilla tulsa stores how do you make cbd gummies ultra cbd gummies 300 mg cbd gummies for pain best healthy gummies top penis enlargment pills the black ant male enhancement pill nature made gummies review steriods and penis enlargment pills keoni cbd gummies donde comprar rhino x male enhancement pill where can i find cbd gummies for ed regan cbd gummies shark tank fantasyland male enhancement pills what gummies are good for erectile dysfunction tevida male enhancement pills caliberx male enhancement pills reviews full body cbd gummies cost roman male enhancement pill do cbd gummies help ed naturamax penis enlargement pills animale cbd male enhancement gummies reviews test sample best male sex enhancement pills

celexas male enhancement pills black strong male enhancement pills individual male enhancement pills stimuli rx cbd gummies reviews extreme bio sex male enhancement pills side effects of herbal male enhancement pills best penis enlargement pills 2024 keoni ed gummies advanced me gummies full spectrum cbd gummies for pain blue fusion male enhancement pill libido gummies in the mood where can i get free male enhancement pills regen cbd gummies walmart side effects to male enhancement pills sex enhancement pills for males gnc nitric oxide gummies new age premium hemp gummies gladiator penis enlargment pills full body health cbd gummies penis kingsize male enhancement pills wild horse male enhancement pills fda maelstrom male enhancement pills pelican cbd male enhancement gummies price skinny gummies precio gummies for growth is king size male enhancement pills safe blue chew gummies male enhancement pills san antonio hillstone hemp cbd gummies reviews full body cbd male enhancement gummies how long cbd gummies to work

proper cbd gummies pure organic hemp extract benefits of cbd gummies 600 mg what are the side effects of cbd and thc gummies cbd gummy for pain and sleep gridiron cbd isolate bio gummie are just cbd gummies broad spectrum ulg cbd gummies difference between cbd oil and gummies how many mg of cbd gummies to sleep cbd miracle gummies cbd gummies near chapin sc cbd gummy wholesale cbd gummies tucson az cbd gummies bloating how long do cbd gummies take to hit cbd edibles gummies is cbd gummies good for depression and anxiety biogold cbd gummies walmart 25mg cbd gummies wholesale cbd thc gummies dc cheech and chong cbd thc gummies are cbd gummies bad can cbd gummies cause shortness of breath what does cbd gummies help super cbd gummy bears cbd gummies sleep apnea how quickly do cbd gummies work benefits of cbd gummies 750 mg cbd gummies lemon cbd sleep gummies hemp bombs do cbd gummies test positive for marijuanas katie couric cbd gummies for sale cbd gummies for kids for sale cbd gummies vegan dragon fruit 300mg cbd gummies vs delta 8 monjour cbd gummies review

what is cbd gummies 300 mg used for are cbd gummies marijuana joe rogan cbd gummy cbd dosage calculator gummies what dose of cbd gummies should i take where can buy cbd gummies unabis cbd gummies for tinnitus reviews clinical cbd gummies official website panda cbd gummy bears vitafusion sleepwell cbd gummies erectifil cbd gummies reviews price of pure cbd gummies bellingham cbd gummies sweet cbd gummies reaction to cbd gummy high times cbd gummies winners green apple cbd gummies reviews chill cbd oil gummies cbd life gummies cbd gummies faq gummy cbd drops 1000mg bio lyfe cbd gummie cbd gummies for nerve damage cbd gummies enlargement how much cbd in gummies is absorbed cbd gummies worth it best cbd gummies for neuropathy quit smoking cbd gummies how to cancel natures boost cbd gummies top rated cbd gummies for anxiety

extreme fit keto plus acv gummies royal keto gummies website is royal keto gummies legit how to take gemini keto gummies do ace keto gummies really work ps keto gummy simple keto gummies oprah why am i getting gummy bread vegan keto fitlife keto gummies ntx nutrition keto acv gummies reviews optiplex keto gummies reviews keto life gummies oprah can a diabetic take keto gummies keto bhb gummies ingredients apple cider vinegar gummy singapore pro burn keto gummies customer service number does oprah promote a weight loss gummy reviews for truly keto gummies where can i buy algarve keto gummies quantum keto apple cider vinegar gummies apple cider vinegar gummies dollar tree are keto gummies sold in stores reva xtend keto gummies reviews how do you make gummy bear slime keto cannabis gummy recipe did oprah make keto gummies goli ashwagandha gummies help with weight loss weem apple cider vinegar gummies dolly parton and keto gummies dakota apple cider vinegar gummies

weight crasher keto gummies royal keto gummy review keto bhb gummies official website how much do keto blast gummies cost oprah slimming gummies south africa dolly parton keto cbd gummy how do you use keto gummies hydroxycut gummy diet pills what do you take apple cider vinegar gummies for diet gummy bears comments do keto acv luxe gummies work simpli keto gummies how to take goli gummies for weight loss keto one gummies k3to keto gummies are gummy vitamins ok on keto where can i buy royal keto gummies bioscience keto gummies where to buy where can i buy weight loss gummies ultra slimming gummies is truly keto gummies safe how long does it take for keto gummies to work how to cancel keto gummies order keto pink gummies are apple cider vinegar gummies any good reviews of keto plus acv gummies where can i buy keto gummies in australia does oprah really back weight loss gummies two apples apple cider vinegar gummies price for keto gummies keto aurora acv gummies keto blast gummies as seen on shark tank what stores sell goli acv gummies lifeline keto acv gummies trisha yearwood pure acv gummies vs goli oprah wingrey gummies shark tank gummy bear weight loss mother apple cider vinegar amazon gummies celine dion keto gummies diet gummy kakao