Categories
General Technical

Update on the System Recovery

Hello,

I realize that for many of you this has been a long and troublesome process, and I can thankfully say that ShoutWiki will soon be back up and fully functional.  Currently, I’m working on recovering all the databases; it’s a slow process, but so far all the databases that have been recovered (there are around 700 in all) are completely intact which means that you did not lose any pages, etc. from this crash.

After all the databases are recovered, we will need to basically reinstall the server, and once that is completed ShoutWiki will back in action.

Thank you all for your patience during this time, and I hope that you continue to choose ShoutWiki!

Regards,

Skizzerz
ShoutWiki Technical Team

UPDATE 1 (31 March 2011, 3:14am UTC): Current status is 125 done, 521 to go. Will continue to update as this progresses.

UPDATE 2 (31 March 2011, 3:52am UTC): Current status is 274 done, 372 to go.

UPDATE 3 (2 April 2011, 6:55pm UTC): There were about 10 wikis that didn’t recover properly, so I’m working on a way to recover them. Once this is done, everything else should go relatively quickly.

UPDATE 4 (4 April 2011, 11:56pm UTC): Clarification on the above: When I said they did not recover properly, this does NOT mean that they can’t be recovered. The issue lies with some stuff wrong with the backend behind the dbs, so certain large wikis with lots of revisions and log entries took too long to dump in one go before MySQL crashed. What I’ll be doing for these 6 wikis (got 4 of them to work normally since the last update) is dumping them table-by-table, so your stuff will not be lost. As for an update on how other stuff is going, we are now waiting for the Head Tech to reinstall the Operating System on the server so that we may begin restoring our backups. Due to the nature of the backup restoring process, wikis will become available as the backups get restored, so check back in a day or three to see if your wiki is up.

UPDATE 5 (9 April 2011): OS is going to be reinstalled sometime later today, and provided all goes well, so will the wiki restorations. Wikis will be restored in alphabetical order and will become available as they get restored (meaning some will be up before others). I’ll post another update when the restorations actually start.

UPDATE 6 (16 April 2011): The tech team has taken backups of databases and important configuration files. The CTO estimates that we might get started on the recovery as soon as this weekend, but it might take a major part of the next week. We’ll keep you posted on this. –Jack Phoenix

UPDATE 7 (19 April 2011): The operating system has now been reinstalled and the backup files are being copied to the server currently. Recovery will begin later today. –Jack Phoenix

UPDATE 8 (19 April 2011): Data recovery proper has begun. During this process the site may display various error messages. –Misza

UPDATE 9 (19 April 2011): The wiki restoration process has begun. Wikis should become available as they get restored. They are going in random order, so we cannot give any ETAs on specific wikis. Please note that just because your wiki is back, it does NOT mean that all the restores are done. As such, if issues arise, please alert a staff member on IRC so that we can look into it. We are also considering rolling out the 1.16 upgrade after the wikis get restored, so you may see some downtime there as well. –Skizzerz

UPDATE 10 (21 April 2011): We have restored the vast majority of the wikis, as people were quick to find out, we ran into a LOT of issues trying to restore Autopedia and Simpsons Fanon. We’re going to continue working on ways to restore these two wikis, but in the meantime, we will be pushing out the 1.16 upgrade to the rest of the wikis. After the upgrade finishes, editing will be re-enabled. –Skizzerz

UPDATE 11 (27 April 2011): We’re working on both issues, the MediaWiki upgrade and restoration of Autopedia and Simpsons Fanon databases, but it turns out that MediaWiki’s default upgrade script is not suitable for our purposes as-is, which means that we have to write our own upgrade script. This is something that we want to do right, without rushing, in order not to mess up anything, which is why it might take a bit longer then expected. Once the techs have finished writing the upgrade script, the actual upgrade process should be relatively quick, after which the database will be unlocked and you can resume editing. We’ll keep you posted on this. –Jack Phoenix

UPDATE 12 (1 May 2011): The update process has begun. In the meantime, a “ShoutWiki is currently undergoing maintenance” page will show until updates are complete. Once complete, editing will be re-enabled. ETA is about two hours from now, provided no issues are found. –Skizzerz

UPDATE 13 (1 May 2011): Updates are complete and editing has been re-enabled. If you run into issues, please let someone know on IRC and open a bug in our bugzilla: http://bugzilla.shoutwiki.com — At some point in time we’ll be re-updating to the newest 1.16 version, but this will not cause any downtime and should be for the most part invisible to you guys.

By Skizzerz

ShoutWiki staff member

387 replies on “Update on the System Recovery”

Can we have an update please? Werent wiki restorations supposed to happen on Saturday? Fourth time you’ve missed your dates. This is getting RIDICULOUS.

I agree with that last post by Hockeyben. If you say that something is going to happen today and 3 days later we haven’t even been told why it hasn’t happened yet we kind of feel abandoned.

I hope you guys at Autopedia didn’t lose as much as you think. It might help to search Google for your pages and see if any that might have been lost were cached.

@Escyos We’re at 29 days now, what time zone you’re in doesn’t change how long it’s been down.

@Kirbman I managed to save around 100 of them. The other couple hundred were copied from another car wiki in which Autopedia is merging with.

@kirbman
1. 15th (14th)
2. 16th
3. 17th
4. 18th
5. 19th
6. 20th
7. 21st
8. 22nd
9. 23rd
10. 24th
11. 25th
12. 26th
13. 27th
14. 28th
15. 29th
16. 30th
17. 31st
18. 1st
19. 2nd
20. 3rd
21. 4th
22. 5th
23. 6th
24. 7th
25. 8th
26. 9th
27. 10th
28. 11th
29. 12th
30. 13th

30 days. Pathetic.

@Escyos today is the 12th, in times zones where its the 13th, the servers went down on the 15th rather than the 14th

lol at all these people leaving really. I’d hate to see them wait at a restaurant.

It’s not like they wanted the system to fail on them (That’s already Livejournal’s job), and they’ve informed their members of what they were doing. I’d like to see these whiners do better than the team fixing a huge MYSQL database, because I assume it can’t be easy, IT prick or not. Don’t let the door hit you on the way out!

Jack, Skizzerz, whoever else from ShoutWiki might be reading the blog comments: You guys need to SERIOUSLY get with the program. And I don’t just mean in the technical sense of getting the wikis working; I also mean in the public relations sense.

Each time you say that something will happen by a particular date and it doesn’t, ShoutWiki takes a credibility hit. And each day that goes by with no further updates after a non-happening, you take another credibility hit. You need to get your rears in gear, get onto the blog and say SOMETHING — at this point, even a lame excuse would be better than no communication at all.

If your situation is that you’ve decided to close up shop entirely, that would be unfortunate. But if that’s the case, how about at least making data dumps available so we can take our wikis and go find a host that will provide reliable service and regular communication?

@Mythigator

What’s the point, Mythigator. They’re like Wikia; they’ll just ignore you and do things their way. I’m starting to feel that by time they fix this mess, I’ll be an old lady.

@Everyone: I hope that you realize that our systems did not fail, our host’s did, and as a result, we suffered. We have to fix things that they essentially broke and try not to break anything in the process of doing so — you want your wikis to be fully functional, of course, and that’s what we’re aiming for!

@Mythigator: I personally prefer fixing things as quickly as possible and then announcing “we’ve fixed it!” instead of giving lame excuses to the general public.

And no, we have no plans to “close up shop entirely”, don’t worry about that. Closing the site without giving people a chance to grab a backup of their data would be quite dickish in my opinion.

It is no secret at this point that mistakes were made. We’re hoping to improve the service and naturally prevent disasters of this magnitude. Nevertheless, I think that most of you would appreciate the work we do a bit more if you knew the differences between ShoutWiki and other wiki hosting services (I don’t mean just the differences in features and whatnot, for example).

@Jack Phoenix: Thanks for the update, just seeing a new message to confirm that we haven’t been forgotten is nice. Any idea when the restorations will begin?

@Escyos: Nevermind your inability to understand basic math and time zones is hopeless.

Well, thanks for the update, Jack. I really do hope that ShoutWiki is back up soon. But what’s the progress on re-installing the OS? Any estimations on when the wikis will restore? Thanks again.

@Anno

If AWA moves, can the shoutwiki site stay up? I’m an admin on Awa (however, my rights were lost as they were given on march 12), and I would still like to contribute to the shoutwiki one, as well as the new one.

I would also like a status update on the repairs.

@kirbman – the 14th is the 15th, so the 15th is the 16th, 16th is the 17th…….12th is the 13th, if its a day ahead it stays ahead, even if only by one minute into the next day. And I thought all the idiots stayed behind on wikia….

@everyone else – the site is down and has been for a long time, the site was slow before hand and frequently unaccessable. Over 1 month of down time and no progress is not good. Move to other hosts, Shoutwiki screwed up.

@ hockeyben

Then have two awa’s sigh. That is the same problem again we are all having with wikia.

I probably won’t move to my site, as I don’t know how to install new mediawiki myself. But maybe wikii?

We will see how slow the server is when this returns, then we can all vote on it.

@Escyos As Jack said, Shoutwiki DID NOT screw up, the host did, so stop ranting on this blog and go ranting to them.

And don’t bother prompting everyone to leave Shoutwiki, because they won’t unless they are as unpatient as you.

@Escyos

Hey, give them another chance, Escyos. At least Jack was able to tell us something, y’know.

I’m upset that a month has passed, too, but we got an update from Jack, which is better than nothing, y’know. I understand you hate ShoutWiki now, but some are willing to forgive and forget, y’know.

@UxieLover1994

I understand that it was not their fault, but when something you are paying for (shoutwiki paying for their servers) then you get on it quickly.

They lost some data because they didn’t back it up and thats whats taken so long to get back. I don’t hate them, I am annoyed that they took so long. And one update is not good enough, now if they posted updates everyday then maybe that would be better, but they didn’t.

A one month failure is not good nor is it excusable. They knew that it was taking longer than expected yet only gave up an update once every week or so.

I’m not giving them another chance, I subscribe to a one-shot policy with everything in my life.

@Escyos

Yes, it’s fustrating, but I’m not giving up on them (yet), since they are still under constuction with their community, y’know. (Yeah, I talk like Raijan from Final Fantasy VIII)

I’m planing on moving to Wikkii should this last more than two months. You can move if you want (I’m being friendly), but some will stay, ether because they like it here (WikiSimpsons) of that they cannot afford their own host (Spyro Wiki, which is my homebaise).

@Everyone else

31 days… don’t bring it to 40!

@Escyos Would we listen to Queen, Beatles and other excellent groups, for whom noone gave a crap in their beginnings, if everyone had one-shot policy?

What happened to the wikis coming back LAST Saturday. They’re probably not even going to be back by THIS Saturday. Seriously, if you’re gonna say it’s gonna be up by a certain time, it needs to be, or you lose credibility thats kind of what’s happened here. Now, I’m not moving, but if anything like this were to happen again, i will.

@Shoutwiki Staff: If you guys can change your datacenter provider without adding downtime I think that would be best.

All of you above (especially Escyos) need to shut up and get your facts straight before talking crap. Firstly, this crash wasn’t an error on Shoutwiki’s part. FYI, their systems were running better than fine until their hosting company screwed up. Yes folks, that’s right; their HOST COMPANY, not Shoutwiki, screwed up! So if you want to rant at anyone, go rant them. The crash was not in Shoutwiki’s control.

Furthermore after the crash occurred, the Shoutwiki staff indeed (as you would expect) asked their host for details in problems behind the crash so they could know what to do specifically to recover. News for you folks, their host refused to give out any of that info and simply gave them a “your problem, fix it yourself” scenario (why, idk… because they suck, right?). Since the staff had no idea what has gone wrong, they’ve had to restore EVERYTHING from scratch. That should satisfy your query of why the site has been down for so long (yeah, wiki farms aren’t a simple wiki, there’s many wikis that can take ages to recover completely, do some research). Only one staff member has the appropriate access for the re-installation of the server OS, and when you only have one staffer working on such an issue, naturally the process will take a while. Its not (and can’t be) within anybody else’s control. And don’t just think that a re-installation for a server hosting a wiki farm is as simple as your average windows install; there are complicated issues (as well as crash problems) behind it that need to be overcome. Hacks can be done, but is it really worth risking another failure just for the sake of time? If you’re that impatient, write your articles on a text editor or on another wiki for previewing (without saving). Just because your wiki is down doesn’t mean you can’t keep working in the background.

And no, Shoutwiki isn’t closing shop, as per what Jack said above. Suggestion to you is to just wait at least a bit more, even if its been a long time (and its understandable why many of you are upset). I personally find it unfair that you’re ranting Shoutwiki staff for something that wasn’t their fault. Perhaps this might have given you a better insight as to why its been a long wait and open your eyes a bit wider to the situation.

@Nick I agree with just about all of this. But all three techs are working on this. Skizzerz has recovered the databases, and I believe Misza has access to the OS (Not sure, but Just assuming as he is the head tech). If the host basically said that’s not our problem, that leaves the staff in a difficult situation.

@Hockeyben My point exactly. And yes, Misza is the one with access to the OS. Its pretty much up to him to get SW restarted now, except that the underlying issue is that re-installing the OS isn’t as simple as one might think.

@Nick

I am aware that it is not entirely Shoutwikis problem, but they are still at fault, they lost data and have spent too long recovering the system.

Seeing as they are paying for the datacenter and they were the ones to screw up, im not holding them responsible, however 1 month+ is not forgivable. Simple as that.

@Escyos Dude, SHOUTWIKI DIDN’T SCREW UP! In any shape or form!

Blaming Shoutwiki is like shooting a person and when it dies blaming the paramedics, because they couldn’t resuscitate him.

@Nick

I agree with you, Nick. I know that it’s not their fault, but it was not until Jack gave us his update that I regained patience. It seems only one of us (Escyos) gave up on them.

@Escyos

I’m willing to forgive the Staff once the site is back on, but you ”need” to relize that it’s not their fault this crash has happened. Nick’s right, y’know! GROW UP!!! These things happen, and I’m willing to forgive EVEN if it lasts A WHOLE YEAR!!!

FWIW, in this industry relying on your host to keep the server backed up and not having automated daily backups of the entire server IS the company’s fault. It’s inadequate protection of your customer’s data. If ShoutWiki were actually being paid for professional services their customers would have rightfully demanded a refund and found a new host.

FOR THE LAST TIME I AM NOT BLAMING THEM FOR THE CRASH!

I am blaming them for the lack of progress, notification of progress and inadequate back ups.

LEARN TO F**KING READ!

@Escyos

Do you want this to turn into a fight!? Ether get patience or go find a new host and leave your wiki behind! I AM SICK OF YOUR WHINING!!!

@UxieLover1994

I was merely correcting people who are misquoting me. You would do the same.

Im already looking for a new host, and have found one im justing waiting for Shoutwiki to get it together so I can leave.

Intermittent outages and slow times followed by a crash which facilitates the loss of data. There not getting any better and if history has taught us anything its that they won’t. They screwed up by not backing up data. Simple as that.

@Escyos

Just remember: Not everyone is planning on leaving, plus you cannot make everyone leave. I’ll be getting a databaise dump every Sunday, just in case this place crashes again, and it cannot be fixed.

@UxieLover1994

Take your time!? It may not be a problem for you to wait 60 days, but Autopedia can’t. We’ve lost 32 days of car blogs (New cars, car updates, etc). Autopedia admin Red marquis isn’t even sure if we can recover them all.

@Siver Knight

Who knows; we might get back our wikis on Monday night, but if you start to become as inpatent as Escyos, then just create a new wiki and do the missing pages until ShoutWiki returns; then get your remaining pages. This is a tip for the inpatient.

@Escyos
If you have already decided to leave Shoutwiki, then stop commenting and just leave. Stop commenting and find something better to do than wingeing, and once your wikis are back, just go. However, if everyone had the same “policy” as you and left at the first sign of trouble, where would the world be today? Mistakes are always going to be made, but once they are fixed, they are usually never made again. People learn from mistakes, and put measures in place to pervent them, or at least stop them from causing as much damage as they did in the first place. If we all stopped doing something, or gave up on something when it first had a problem, we’d all still be living in caves. It’s people with rationales like yours who stop society from advancing.

Leave a Reply to Hockeyben Cancel reply

Your email address will not be published. Required fields are marked *

keto max science gummies reviews gummies keto acv can you get keto gummies at walmart where to buy weight watchers keto gummies reviews on rapid results keto gummies pro fast keto and acv gummies where can you buy keto one gummies search for info on hydroxycut gummies diet do the gummies work for weight loss advantages of apple cider vinegar gummies simpli health acv keto gummies reviews which acv gummies were on shark tank f1 keto gummies ree drummond keto for health gummies reviews any side effects from apple cider vinegar gummies best gummies for weight loss reviews diet gummies that work what is the best acv gummy slim firm gummies ree drummond apple cider vinegar gummies crossfire keto gummies reviews does simpli acv keto gummies work premium blast acv gummies it works slimming gummies while breastfeeding goli nutrition apple cider vinegar 60 gummies ingredients in keto plus acv gummies

total effect keto gummies avc keto gummies rapid results keto gummies review are keto gummies good for weight loss sure slim keto acv gummies reviews trisha yearwood weight loss gummies legitimate go keto gummies hydroxycut gummies weight loss plus vitamins apple cider vinegar gummies natures bounty keto gummy from shark tank weight loss gummies supported by shark tank shark tank episode keto gummies apple keto gummies au best keto gummies on the market where to buy ketology keto gummies can apple cider vinegar gummies cure bv apple cider vinegar gummies good for diabetes keto flow gummy bears review do keto flow gummies really work algrave keto gummies how to use optimal keto acv gummies what are in keto gummies elite keto gummies reviews pro burn keto acv gummies reviews

cbd gummies how long for effect gummy cbd amazon joy cbd gummies vidspur cbd gummies younabis cbd gummies full body cbd gummies maximum strength how to cancel fun drops cbd gummies cbd gummy text 25g cbd gummies penguin cbd gummies near me cbd gummy benefits ulixy cbd gummies for diabetes best cbd products in pill or gummie vera pure cbd gummies green 8 cbd gummies can you bring cbd gummies to jamaica wyld cbd pear gummies cbd gummies for pain sugar free best full spectrum cbd gummy does cbd gummies help with high blood pressure is hemp oil the same as cbd oil gummies cbd gummies for lowering blood sugar

buy biolyfe cbd gummies how long do cbd gummies take to kick in cbd gummies salem oregon how long do cbd gummy bears stay in your system cbd joint pain gummies jolly cbd gummies side effects condor cbd gummies for sale hemp bombs cbd gummies buy in store cbd green roads gummies how much cbd gummies should i take canada puppy cbd gummies low dose cbd gummies for anxiety trident cbd gummies near me anatomy one cbd gummies for sale best cbd gummies for penis enlargement best cbd thc gummies online best cbd gummies with thc for pain elderberry cbd thc gummies can you take cbd gummies with fluoxetine what to expect when taking cbd gummies how much cbd in a gummy bear cbd gummies have pot in them koi cbd delta 9 gummies boosted cbd gummies reviews where can you buy biolife cbd gummies ignite cbd gummy reviews delta 8 cbd gummies drug test cbd gummies good for tinnitus space candy strain cbd gummies cbd gummies trial

100 male enhancement pills magnum male sexual enhancement pill blue chews gummies erectile dysfunction vyrixin male enhancement pills top herbal male enhancement pills full body cbd gummies for erectile dysfunction full body health gummies reviews ksx male enhancement pills amazon enlarge penis pills number one natural male enhancement pill do over the counter male enhancement pills work impact garden gummies for ed super health cbd gummies male enhancing supplement erection pills sx pills male enhancement buy cbd gummies for sex xtra power male enhancement pills keoni cbd gummies 500 mg where can i buy quick flow male enhancement pills natures only cbd gummies sex what is in regen cbd gummies can cbd gummies help with sex penis enlargement pills that works 3 bullet male enhancement pills gummy for pain do the gummies really work body cattle blue box male sex enhancement pills what is in power cbd gummies science cbd gummies better than viagra do male enhancement pills at gas stations work regan cbd gummies penis enlargement shark lean male enhancement pills grow male enhancement pills consumer reports on male enhancement pills

king cobra gummies male enhancement review free samples of male enhancement pills natural male enhancement pill do male enhancement pills whats the best male enhancement pills best sex enhancement pills for male in nigeria male enhancement pills trial which male enhancement pills are the best luxe gummies reviews are keoni cbd gummies safe spectrum cbd gummies for sale max fuel gummies what is power cbd gummies kangaroo male enhancement pills boner bears male enhancment bioscience cbd gummies where to buy fullbody cbd gummies review rise up male enhancement pills snoop dogg male enhancement pill does gnc sell male enhancement pills vena cbd gummies belly melt gummies trisha yearwood male enhancement cbd gummies for sale male enhancement pill what ages you can use king cobra gummies male enhancement formula male enhancement xl pills reviews vericil male enhancement pills full body health cbd gummies condor cbd gummies ed any male enhancement pills work male enhancement pills drug test advantages of male enhancement pills what do male enhancement pills do zylofon male enhancement pills