Categories
General Technical

Downtime since June 17th

Hi everyone,

this is a blog post I wished I never had to write, but here we are again: the server’s down. It’s frustrating for everyone involved, and we’re doing our best to bring it back online, but it seems that it’s going to take a while.

I want to apologize on my own behalf as well as on the behalf of the whole staff team about our poor communication during this downtime. We realize that many of you would like to hear an official word about what’s going on when the site’s down and we were unable to provide that to you in a timely manner, which is not acceptable. Rest assured, we will definitely aim to improve the communication between the staff and the users once the site’s back up again.

What can I say? We believe in our mission and we have no intention of “closing down the shop”, despite what other sources may claim. But it doesn’t change the fact that we don’t have people 24/7 to work on this issue. Downtime is always a critical thing, but this seems to be something only a few select people from the tech team are able to fix. We want to bring the site back up as soon as possible without losing anyone’s important data. Needless to say that it’s not the fastest possible process in the world.

The techs are busy working on our new, more stable server, which will be located in the United States of America. This will replace the Polish server as our main webserver and bring the much-needed stability. This server was planned to be our backup and was close to being ready to be deployed for production when these new issues occurred, we are now changing that plan.

I want you to know that we haven’t given up — and we have no plans to do that!

Jack Phoenix
Head of the Customer Support Team

UPDATE 1: To clarify a few points. This is NOT a repeat of last time. Last time the server itself crashed and needed to be restored from a backup, and said backup was corrupted. This time there was no need of a backup to be restored, and as such there shouldn’t be any data loss, although we cannot confirm this for sure. The main issue here is not recovering databases, but rather moving everything to the new server, which should prove to be much more reliable than our old one. While we’re working on moving everything over, I’ve managed to set up a hack to make MySQL behave properly and actually display the wikis, albeit in read-only mode. Please note that since this is a hack, you might see the same “unable to connect to socket” error as before — this is temporary and if you reload the page in around a minute you should see it. –Skizzerz

By Jack Phoenix

I'm the head of ShoutWiki's Customer Support Team.

151 replies on “Downtime since June 17th”

Thanks for the update!

Glad the site is finally moving to a new server. /crosses fingers and hopes everything will be up and running again by June 30th/

Fantastic. Hopefully it’ll be back up soon, been sitting in the dark for a week without making any edits.

(and hopefully this comment will be posted, for some reason, mine aren’t showing up. Heh.)

Why, oh, why haven’t you backed up the data? If I were your tech I would’ve killed myself out of frustration and humiliation. 🙁
Good luck anyway.

@ngng, it looks to me that they’re saying they have backed it up. It’s taking some time because they need to do some work to bring it back up *without losing anyone’s data.*

@molly
If the data is backed up there would be no need in restoring “without losing” it, they’d just reinstall the system and copy the data back (which would take hours at the most). In my opinion this outage is like the last time when the server has crashed and the only backup available was the actual data inside this crashed server. Besides they have never recovered at least two wikis from the last crash…

How long is a while? You have until July 6 to get everything back up and working, or I’m gone. I just don’t trust you guys anymore. and I really hope no wikis can’t be recovered like autopedia and simpsons fanon.

Something else: Daily or bi-daily updates would go a LONG way to help. Even if it is just saying something like: June 26: Working on new server June 27: Made a little progress. Something like that.

As for the updates; you could always use your Twitter account instead of spamming up the blog. It seems like the Twitter account, as is, is woefully underused and we could use the communication.

I am aware we’re not using twitter as often as we can, the same can be said for our facebook page.

I have recently started using our facebook page more to keep users updated. Apart from that we are also on our IRC server keeping users updated, and answering questions.

The techs are currently working as fast as they can to get our new server ready, this will bring the much needed stability that our old server was clearly missing.

Finally, you did not address whether there was any data loss (ex all edits lost before March 7 last time). Also, will the databases need to be recovered like last time? I hope somoeone can answer my query. Also, I’ve realized I won’t be moving – as my wikia databases dump was 81 MBs. I just hope that my wiki is ok from this, and there is never another outage like this – like everyone else hopes. Good luck and I hope the site’s back by 6 of July (but don’t rush).

There shouldn’t be any data loss, but we do not know for sure at this point in time.

I’ve managed to restore all the wikis to a read-only state, but expect some intermittent MySQL connection errors as I needed to resort to a rather hacky approach to do this.

We realize this is hard on everyone, and we are working as fast as we can to migrate everything. Restoring read-only access to the wiki should help both you and us in the sense that you can at least still see your wikis, and if you encounter problems, you can report them to us so we know what to pay special attention to in the migration process.

Yay! My wikis seem to be up and running again! 😀

Hopefully the switch over to the American server happens before the Polish one gets tempermental again.

Thank you for the information, and best of luck with your new server setup. As much as I dislike the idea of the site being hosted in the US, if it provides more reliable uptime, I suppose it’s something I should be able to live with.

At least [[Special:Export]] is working. You can grab all the contents of the wiki as a Backup.

@Skizzerz Thank you! My wiki is up!

@Anno1404 I decided to give them longer after the blog post. Besides, the wikis are back, and they had till the 6th to do that.

I’ve noiced a bug in Monaco. All tabs (Delete, Protect, Move etc.) are showing double. Is this the place to report it, or Bugzilla?

I’ve search the net for the solution to my urge to be on the safe side and found (and modified) a python script to download all the images from a certain wiki. Here it is (instruction included). Just set your wiki name, login and password prior to running it.

http://depositfiles.com/files/fnb3k45qe

You can export the pages content (with all history) through Special:Export as Ciencia Al Poder has mentioned.
Please do the backup it may (and will) save your data.

May I translate in french the message in the box when you want to edit an article ?
You could put it on .fr wikis ?

Translation :
La migration du serveur est en cours.
Tous les wikis sont en mode lecture seule.
Plus de détails disponibles sur le blog officiel de ShoutWiki.

yaaaay! Good thing it wasnt like april/may. And i think it is very good to replace the old polish server to the new server in the US! Good luck! And thank you! 🙂

Like Skizzerz said, he had to resort to a rather hacky approach to get the wikis back up (in read only mode) while he fixes the issues and works on the migration to the new server. Errors like this are unfortunately to be expected until our new, more stable, home is up and running.

The wikis are still in read only mode, yes. Because some server side issues still need to be fixed and everything still needs to be moved to the new server. This will take a bit of time. But at least now you can visit your wikis, and if you find anything out of the ordinary, you can help us by reporting it.

Since you have access to your wikis, you can just edit in Notepad. Copy and paste your articles onto Notepad, make your changes and when the wiki is able to be edited again, paste your edits and and save. The same goes for new articles.

It would be nice to see an update – even just to let people know your working on it. Communication goes a long way.. Also, Autopedia has moved, so there will be no need to import. I’m just letting you guys know so none of the techs waste their time doing this.

Ditto, it would be nice to get an update even if its just “we’re working on it”. It’s been frustrating the past couple of days not being able to update the Hetalia Archives considering all the developments that have happened for that series the past two days.

Leave a Reply to Nemo Cancel reply

Your email address will not be published. Required fields are marked *