Hello,
I realize that for many of you this has been a long and troublesome process, and I can thankfully say that ShoutWiki will soon be back up and fully functional. Currently, I’m working on recovering all the databases; it’s a slow process, but so far all the databases that have been recovered (there are around 700 in all) are completely intact which means that you did not lose any pages, etc. from this crash.
After all the databases are recovered, we will need to basically reinstall the server, and once that is completed ShoutWiki will back in action.
Thank you all for your patience during this time, and I hope that you continue to choose ShoutWiki!
Regards,
Skizzerz
ShoutWiki Technical Team
UPDATE 1 (31 March 2011, 3:14am UTC): Current status is 125 done, 521 to go. Will continue to update as this progresses.
UPDATE 2 (31 March 2011, 3:52am UTC): Current status is 274 done, 372 to go.
UPDATE 3 (2 April 2011, 6:55pm UTC): There were about 10 wikis that didn’t recover properly, so I’m working on a way to recover them. Once this is done, everything else should go relatively quickly.
UPDATE 4 (4 April 2011, 11:56pm UTC): Clarification on the above: When I said they did not recover properly, this does NOT mean that they can’t be recovered. The issue lies with some stuff wrong with the backend behind the dbs, so certain large wikis with lots of revisions and log entries took too long to dump in one go before MySQL crashed. What I’ll be doing for these 6 wikis (got 4 of them to work normally since the last update) is dumping them table-by-table, so your stuff will not be lost. As for an update on how other stuff is going, we are now waiting for the Head Tech to reinstall the Operating System on the server so that we may begin restoring our backups. Due to the nature of the backup restoring process, wikis will become available as the backups get restored, so check back in a day or three to see if your wiki is up.
UPDATE 5 (9 April 2011): OS is going to be reinstalled sometime later today, and provided all goes well, so will the wiki restorations. Wikis will be restored in alphabetical order and will become available as they get restored (meaning some will be up before others). I’ll post another update when the restorations actually start.
UPDATE 6 (16 April 2011): The tech team has taken backups of databases and important configuration files. The CTO estimates that we might get started on the recovery as soon as this weekend, but it might take a major part of the next week. We’ll keep you posted on this. –Jack Phoenix
UPDATE 7 (19 April 2011): The operating system has now been reinstalled and the backup files are being copied to the server currently. Recovery will begin later today. –Jack Phoenix
UPDATE 8 (19 April 2011): Data recovery proper has begun. During this process the site may display various error messages. –Misza
UPDATE 9 (19 April 2011): The wiki restoration process has begun. Wikis should become available as they get restored. They are going in random order, so we cannot give any ETAs on specific wikis. Please note that just because your wiki is back, it does NOT mean that all the restores are done. As such, if issues arise, please alert a staff member on IRC so that we can look into it. We are also considering rolling out the 1.16 upgrade after the wikis get restored, so you may see some downtime there as well. –Skizzerz
UPDATE 10 (21 April 2011): We have restored the vast majority of the wikis, as people were quick to find out, we ran into a LOT of issues trying to restore Autopedia and Simpsons Fanon. We’re going to continue working on ways to restore these two wikis, but in the meantime, we will be pushing out the 1.16 upgrade to the rest of the wikis. After the upgrade finishes, editing will be re-enabled. –Skizzerz
UPDATE 11 (27 April 2011): We’re working on both issues, the MediaWiki upgrade and restoration of Autopedia and Simpsons Fanon databases, but it turns out that MediaWiki’s default upgrade script is not suitable for our purposes as-is, which means that we have to write our own upgrade script. This is something that we want to do right, without rushing, in order not to mess up anything, which is why it might take a bit longer then expected. Once the techs have finished writing the upgrade script, the actual upgrade process should be relatively quick, after which the database will be unlocked and you can resume editing. We’ll keep you posted on this. –Jack Phoenix
UPDATE 12 (1 May 2011): The update process has begun. In the meantime, a “ShoutWiki is currently undergoing maintenance” page will show until updates are complete. Once complete, editing will be re-enabled. ETA is about two hours from now, provided no issues are found. –Skizzerz
UPDATE 13 (1 May 2011): Updates are complete and editing has been re-enabled. If you run into issues, please let someone know on IRC and open a bug in our bugzilla: http://bugzilla.shoutwiki.com — At some point in time we’ll be re-updating to the newest 1.16 version, but this will not cause any downtime and should be for the most part invisible to you guys.
387 replies on “Update on the System Recovery”
Can we have an update please? Werent wiki restorations supposed to happen on Saturday? Fourth time you’ve missed your dates. This is getting RIDICULOUS.
I agree with that last post by Hockeyben. If you say that something is going to happen today and 3 days later we haven’t even been told why it hasn’t happened yet we kind of feel abandoned.
I hope you guys at Autopedia didn’t lose as much as you think. It might help to search Google for your pages and see if any that might have been lost were cached.
@Escyos We’re at 29 days now, what time zone you’re in doesn’t change how long it’s been down.
@Kirbman I managed to save around 100 of them. The other couple hundred were copied from another car wiki in which Autopedia is merging with.
@kirbman
1. 15th (14th)
2. 16th
3. 17th
4. 18th
5. 19th
6. 20th
7. 21st
8. 22nd
9. 23rd
10. 24th
11. 25th
12. 26th
13. 27th
14. 28th
15. 29th
16. 30th
17. 31st
18. 1st
19. 2nd
20. 3rd
21. 4th
22. 5th
23. 6th
24. 7th
25. 8th
26. 9th
27. 10th
28. 11th
29. 12th
30. 13th
30 days. Pathetic.
See ya’ ShoutWiki.
@Escyos today is the 12th, in times zones where its the 13th, the servers went down on the 15th rather than the 14th
lol at all these people leaving really. I’d hate to see them wait at a restaurant.
It’s not like they wanted the system to fail on them (That’s already Livejournal’s job), and they’ve informed their members of what they were doing. I’d like to see these whiners do better than the team fixing a huge MYSQL database, because I assume it can’t be easy, IT prick or not. Don’t let the door hit you on the way out!
So its a day ahead at the start but NOT at th end?…..you should take basic math again.
If you kept us updated on the server, and alike, people wouldn’t feel like you’ve abandoned them.
Please give us an update soon! People will start to leave if you don’t by Friday…
30 days… now I’m mad!
Jack, Skizzerz, whoever else from ShoutWiki might be reading the blog comments: You guys need to SERIOUSLY get with the program. And I don’t just mean in the technical sense of getting the wikis working; I also mean in the public relations sense.
Each time you say that something will happen by a particular date and it doesn’t, ShoutWiki takes a credibility hit. And each day that goes by with no further updates after a non-happening, you take another credibility hit. You need to get your rears in gear, get onto the blog and say SOMETHING — at this point, even a lame excuse would be better than no communication at all.
If your situation is that you’ve decided to close up shop entirely, that would be unfortunate. But if that’s the case, how about at least making data dumps available so we can take our wikis and go find a host that will provide reliable service and regular communication?
@Mythigator
What’s the point, Mythigator. They’re like Wikia; they’ll just ignore you and do things their way. I’m starting to feel that by time they fix this mess, I’ll be an old lady.
@Everyone: I hope that you realize that our systems did not fail, our host’s did, and as a result, we suffered. We have to fix things that they essentially broke and try not to break anything in the process of doing so — you want your wikis to be fully functional, of course, and that’s what we’re aiming for!
@Mythigator: I personally prefer fixing things as quickly as possible and then announcing “we’ve fixed it!” instead of giving lame excuses to the general public.
And no, we have no plans to “close up shop entirely”, don’t worry about that. Closing the site without giving people a chance to grab a backup of their data would be quite dickish in my opinion.
It is no secret at this point that mistakes were made. We’re hoping to improve the service and naturally prevent disasters of this magnitude. Nevertheless, I think that most of you would appreciate the work we do a bit more if you knew the differences between ShoutWiki and other wiki hosting services (I don’t mean just the differences in features and whatnot, for example).
Thanks for the response Jack. Could you honestly tells us when the site will actually be up? I can’t for another week of this…
Yes, Hopefully by the weekend.
@Jack Phoenix: Thanks for the update, just seeing a new message to confirm that we haven’t been forgotten is nice. Any idea when the restorations will begin?
@Escyos: Nevermind your inability to understand basic math and time zones is hopeless.
Sigh. we are having more problems with wikia, http://deadrisingwiki.com/wiki/Special:AWCforum/st/id81/#post_419 and I would really like to concrete legal and long term strategy here from everyone who left that vindictive manipulative, miserable site.
But since AWA is down, and has been down for sometime, I can’t.
When the site comes back up, I think I would maybe willing to host AWA.
Well, thanks for the update, Jack. I really do hope that ShoutWiki is back up soon. But what’s the progress on re-installing the OS? Any estimations on when the wikis will restore? Thanks again.
@Anno
If AWA moves, can the shoutwiki site stay up? I’m an admin on Awa (however, my rights were lost as they were given on march 12), and I would still like to contribute to the shoutwiki one, as well as the new one.
I would also like a status update on the repairs.
@kirbman – the 14th is the 15th, so the 15th is the 16th, 16th is the 17th…….12th is the 13th, if its a day ahead it stays ahead, even if only by one minute into the next day. And I thought all the idiots stayed behind on wikia….
@everyone else – the site is down and has been for a long time, the site was slow before hand and frequently unaccessable. Over 1 month of down time and no progress is not good. Move to other hosts, Shoutwiki screwed up.
@ hockeyben
Then have two awa’s sigh. That is the same problem again we are all having with wikia.
I probably won’t move to my site, as I don’t know how to install new mediawiki myself. But maybe wikii?
We will see how slow the server is when this returns, then we can all vote on it.
Wikkii doesn’t have Monaco. That’s my main objection to it. I vote for staying with shout.
Monaco is free until July something. Then you’ll have to pay for it which is not a good thing at all.. 🙁
@Escyos As Jack said, Shoutwiki DID NOT screw up, the host did, so stop ranting on this blog and go ranting to them.
And don’t bother prompting everyone to leave Shoutwiki, because they won’t unless they are as unpatient as you.
@Escyos
Hey, give them another chance, Escyos. At least Jack was able to tell us something, y’know.
I’m upset that a month has passed, too, but we got an update from Jack, which is better than nothing, y’know. I understand you hate ShoutWiki now, but some are willing to forgive and forget, y’know.
@UxieLover1994
I understand that it was not their fault, but when something you are paying for (shoutwiki paying for their servers) then you get on it quickly.
They lost some data because they didn’t back it up and thats whats taken so long to get back. I don’t hate them, I am annoyed that they took so long. And one update is not good enough, now if they posted updates everyday then maybe that would be better, but they didn’t.
A one month failure is not good nor is it excusable. They knew that it was taking longer than expected yet only gave up an update once every week or so.
I’m not giving them another chance, I subscribe to a one-shot policy with everything in my life.
@Escyos
Yes, it’s fustrating, but I’m not giving up on them (yet), since they are still under constuction with their community, y’know. (Yeah, I talk like Raijan from Final Fantasy VIII)
I’m planing on moving to Wikkii should this last more than two months. You can move if you want (I’m being friendly), but some will stay, ether because they like it here (WikiSimpsons) of that they cannot afford their own host (Spyro Wiki, which is my homebaise).
@Everyone else
31 days… don’t bring it to 40!
@Escyos Would we listen to Queen, Beatles and other excellent groups, for whom noone gave a crap in their beginnings, if everyone had one-shot policy?
F*CKIN BULLSH*T!!!! I can’t take this anymore!!! Now your pushin’ for 32 days+ Without an update 6. I can’t stand this anymore. I’ve completely lost my respect for ShoutWiki.
What happened to the wikis coming back LAST Saturday. They’re probably not even going to be back by THIS Saturday. Seriously, if you’re gonna say it’s gonna be up by a certain time, it needs to be, or you lose credibility thats kind of what’s happened here. Now, I’m not moving, but if anything like this were to happen again, i will.
@Shoutwiki Staff: If you guys can change your datacenter provider without adding downtime I think that would be best.
All of you above (especially Escyos) need to shut up and get your facts straight before talking crap. Firstly, this crash wasn’t an error on Shoutwiki’s part. FYI, their systems were running better than fine until their hosting company screwed up. Yes folks, that’s right; their HOST COMPANY, not Shoutwiki, screwed up! So if you want to rant at anyone, go rant them. The crash was not in Shoutwiki’s control.
Furthermore after the crash occurred, the Shoutwiki staff indeed (as you would expect) asked their host for details in problems behind the crash so they could know what to do specifically to recover. News for you folks, their host refused to give out any of that info and simply gave them a “your problem, fix it yourself” scenario (why, idk… because they suck, right?). Since the staff had no idea what has gone wrong, they’ve had to restore EVERYTHING from scratch. That should satisfy your query of why the site has been down for so long (yeah, wiki farms aren’t a simple wiki, there’s many wikis that can take ages to recover completely, do some research). Only one staff member has the appropriate access for the re-installation of the server OS, and when you only have one staffer working on such an issue, naturally the process will take a while. Its not (and can’t be) within anybody else’s control. And don’t just think that a re-installation for a server hosting a wiki farm is as simple as your average windows install; there are complicated issues (as well as crash problems) behind it that need to be overcome. Hacks can be done, but is it really worth risking another failure just for the sake of time? If you’re that impatient, write your articles on a text editor or on another wiki for previewing (without saving). Just because your wiki is down doesn’t mean you can’t keep working in the background.
And no, Shoutwiki isn’t closing shop, as per what Jack said above. Suggestion to you is to just wait at least a bit more, even if its been a long time (and its understandable why many of you are upset). I personally find it unfair that you’re ranting Shoutwiki staff for something that wasn’t their fault. Perhaps this might have given you a better insight as to why its been a long wait and open your eyes a bit wider to the situation.
@Nick I agree with just about all of this. But all three techs are working on this. Skizzerz has recovered the databases, and I believe Misza has access to the OS (Not sure, but Just assuming as he is the head tech). If the host basically said that’s not our problem, that leaves the staff in a difficult situation.
@Hockeyben My point exactly. And yes, Misza is the one with access to the OS. Its pretty much up to him to get SW restarted now, except that the underlying issue is that re-installing the OS isn’t as simple as one might think.
@Nick
I am aware that it is not entirely Shoutwikis problem, but they are still at fault, they lost data and have spent too long recovering the system.
Seeing as they are paying for the datacenter and they were the ones to screw up, im not holding them responsible, however 1 month+ is not forgivable. Simple as that.
@Escyos Dude, SHOUTWIKI DIDN’T SCREW UP! In any shape or form!
Blaming Shoutwiki is like shooting a person and when it dies blaming the paramedics, because they couldn’t resuscitate him.
@Nick
I agree with you, Nick. I know that it’s not their fault, but it was not until Jack gave us his update that I regained patience. It seems only one of us (Escyos) gave up on them.
@Escyos
I’m willing to forgive the Staff once the site is back on, but you ”need” to relize that it’s not their fault this crash has happened. Nick’s right, y’know! GROW UP!!! These things happen, and I’m willing to forgive EVEN if it lasts A WHOLE YEAR!!!
FWIW, in this industry relying on your host to keep the server backed up and not having automated daily backups of the entire server IS the company’s fault. It’s inadequate protection of your customer’s data. If ShoutWiki were actually being paid for professional services their customers would have rightfully demanded a refund and found a new host.
FOR THE LAST TIME I AM NOT BLAMING THEM FOR THE CRASH!
I am blaming them for the lack of progress, notification of progress and inadequate back ups.
LEARN TO F**KING READ!
@Escyos
Do you want this to turn into a fight!? Ether get patience or go find a new host and leave your wiki behind! I AM SICK OF YOUR WHINING!!!
@UxieLover1994
I was merely correcting people who are misquoting me. You would do the same.
Im already looking for a new host, and have found one im justing waiting for Shoutwiki to get it together so I can leave.
Intermittent outages and slow times followed by a crash which facilitates the loss of data. There not getting any better and if history has taught us anything its that they won’t. They screwed up by not backing up data. Simple as that.
@Escyos
Just remember: Not everyone is planning on leaving, plus you cannot make everyone leave. I’ll be getting a databaise dump every Sunday, just in case this place crashes again, and it cannot be fixed.
32 days… take your time, but don’t let it reach 60 days.
@UxieLover1994
Take your time!? It may not be a problem for you to wait 60 days, but Autopedia can’t. We’ve lost 32 days of car blogs (New cars, car updates, etc). Autopedia admin Red marquis isn’t even sure if we can recover them all.
@Siver Knight
Who knows; we might get back our wikis on Monday night, but if you start to become as inpatent as Escyos, then just create a new wiki and do the missing pages until ShoutWiki returns; then get your remaining pages. This is a tip for the inpatient.
@UxieLover thats kinda what we’ve been doing on the Wikia Autopedia, making pages to import.
@Escyos
If you have already decided to leave Shoutwiki, then stop commenting and just leave. Stop commenting and find something better to do than wingeing, and once your wikis are back, just go. However, if everyone had the same “policy” as you and left at the first sign of trouble, where would the world be today? Mistakes are always going to be made, but once they are fixed, they are usually never made again. People learn from mistakes, and put measures in place to pervent them, or at least stop them from causing as much damage as they did in the first place. If we all stopped doing something, or gave up on something when it first had a problem, we’d all still be living in caves. It’s people with rationales like yours who stop society from advancing.
33 days… and counting. Some of us have already givin up on ShoutWiki, y’know!
An update would be nice (you don’t have to promise anything).