Categories
General Technical

Downtime since June 17th

Hi everyone,

this is a blog post I wished I never had to write, but here we are again: the server’s down. It’s frustrating for everyone involved, and we’re doing our best to bring it back online, but it seems that it’s going to take a while.

I want to apologize on my own behalf as well as on the behalf of the whole staff team about our poor communication during this downtime. We realize that many of you would like to hear an official word about what’s going on when the site’s down and we were unable to provide that to you in a timely manner, which is not acceptable. Rest assured, we will definitely aim to improve the communication between the staff and the users once the site’s back up again.

What can I say? We believe in our mission and we have no intention of “closing down the shop”, despite what other sources may claim. But it doesn’t change the fact that we don’t have people 24/7 to work on this issue. Downtime is always a critical thing, but this seems to be something only a few select people from the tech team are able to fix. We want to bring the site back up as soon as possible without losing anyone’s important data. Needless to say that it’s not the fastest possible process in the world.

The techs are busy working on our new, more stable server, which will be located in the United States of America. This will replace the Polish server as our main webserver and bring the much-needed stability. This server was planned to be our backup and was close to being ready to be deployed for production when these new issues occurred, we are now changing that plan.

I want you to know that we haven’t given up — and we have no plans to do that!

Jack Phoenix
Head of the Customer Support Team

UPDATE 1: To clarify a few points. This is NOT a repeat of last time. Last time the server itself crashed and needed to be restored from a backup, and said backup was corrupted. This time there was no need of a backup to be restored, and as such there shouldn’t be any data loss, although we cannot confirm this for sure. The main issue here is not recovering databases, but rather moving everything to the new server, which should prove to be much more reliable than our old one. While we’re working on moving everything over, I’ve managed to set up a hack to make MySQL behave properly and actually display the wikis, albeit in read-only mode. Please note that since this is a hack, you might see the same “unable to connect to socket” error as before — this is temporary and if you reload the page in around a minute you should see it. –Skizzerz

By Jack Phoenix

I'm the head of ShoutWiki's Customer Support Team.

151 replies on “Downtime since June 17th”

I feel completely left in the dark. Can we have an update?

@Phinbart

We’d be back online and editing last week, the server migration is the reason editing is locked. So I would not call this a ”major crash” or anything. (I’m not saying you did)

WTF!? It’s been 12 workdays now and the best we have is read-only access. If this is the quality of service we can expect from you, then I’m glad I just moved a very tiny personal wiki here, instead of the main wiki. Wikia may be an evil corporate overlord, but at least they are reliable and solve their technical issues on a timely manner.

Moving on to search for another wiki provider…

@Scarbrow, are you an expert? Do you know how long it takes to move 100s of wikis to a new server? Give them a chance. If it gets to a month, then look into leaving.

Just try and be patient guys. We stuck by ShoutWiki when we went through much worse a couple of months ago.

@Scarbrow, Our techs are working as fast as they can to fix the issues and move us to the new servers. Please remember that our techs are all volunteers who donate both their expertise and free time to our project. So the number of “work days” doesn’t really mean much in our case.

@MyDreamName, I am checking with the techs daily to see if there’s any news. When there is no news, there is nothing to report other then saying that there is no news. Whenever our techs have a free moment, they are working to bring us back to full service. Whenever they have an update, I post it here and on our facebook page.

@Jedimca0

Thanks for the update. Really, I’m not upset as I know how much work the migration process must be, but it would be very much appreciated if we were to get an update every few days or so, even if its just to report that there isn’t any news since the last update.

You guys are doing a good job, but people having their wikis here *expect* shoutwiki to be a professional hosting company, not a site run by amateurs on their spare time. Maybe because this fact hasn’t been noticed on a relevant place of the ShoutWiki main page. I suggest ShoutWiki staff to clearly notice that so people coming here for a reliable wiki hosting service don’t get surprises on events like that.

@Ciencia Al Poder: thank you for your comments. I value them, and so do the other staffers.

Your point is a good one, but I can’t totally agree with you. We’re trying to be not just good, but the best — of course this isn’t possible if third parties keep letting us down. We’ve paid them, so we expect them to do their job and not to mess up our data; your data.

While I’m sure that everyone — users and staff alike — would love the idea of multiple ShoutWiki datacenters around the globe with techs maintaining them 24/7, that just isn’t possible yet. For the time being, we unfortunately have to rely on third parties and trust them instead of doing everything ourselves. Even the Wikimedia Foundation has to pay third parties, if not for hosting, then for other things, such as the Internet connection, the actual, physical servers and so on.

I remember paintwiki disappearing. I am not saying shoutwiki is like this at all, but if they do decided to fold up shop anti wikia alliance members, a back up of the site is found on awa.wikkii.com

As said before, ShoutWiki has no intention to “disappear”. The move to our new server was not a planned one, it was decided to move because the Polish server is far from stable and we want to give you all the stability that you deserve. I understand your frustration, and the techs are working on getting us back to editing, but there are only so many hours in a day and there are only so many things our techs can do in the little free time that they have.

Had we planned this move, we would have made sure that our techs had a week, or more, to fully dedicate to this process. But unfortunately the recent failures of the data center have forced us to implement this move at a time where our techs are not available all day. There is currently a lot of fixing and planning going on behind the scenes and the work is progressing. But this will, unfortunately, take its time.

You guys just take your time. Don’t rush. I’m not moving no matter how long it takes.

@Jedimca0

Will the Polish server now act as the backup server if something malfunctioned in the American one?

At this time we do not yet know what the fate of the Polish sever will be, our priority is getting us back to editing. There have been discussions about what to do with that server, but no final decision has been made.

Hockey:

June 20th, 2011 at 6:29 am

Again, is it too much to ask wanting to know why the site’s been down? You have until July 1, or I am gone.

June 25th, 2011 at 10:18 am

You have until July 6 to get everything back up and working, or I’m gone.

July 9th, 2011 at 5:56 pm

You guys just take your time. Don’t rush. I’m not moving no matter how long it takes.

@Anno

I said they had until the first, but changed it to the sixth after the blog post. ”everything up and running” meant getting the wikis back up, which they did by the 26th…

@Jedimca0
The fact that the techs are volunteers was certainly not advertised. I didn’t find that when I first evaluated Shoutwiki, and I searched long and and hard, reading all I found about you.

@Ciencia Al Poder
Exacly my point. When Shoutwiki was advertised as an alternative to Wikia (especially by members of the AWA) I expected something better. Maybe with some small problems, since it was a new company. But certainly not this.

@Jack Phoenix, Jedimca0
Unfortunately for you, the level of service a user expects is set by your competitors. People (me included) are accustomed to Wikipedia’s or Wikia’s reliability, that is, short or very short downtimes. While new users, and specially enthusiasts, may be willing to forgive and stick with you, this sets a very bad record indeed.

Shoutwiki is right now set at “amateur” level for me. An amateur wiki hosting service, currently *down*. It will take quite a while for it to grow into “professional”, if it ever does, and by the look of it, I’m very worried it will never be. Sorry for the harsh words, but almost a month of downtime? (and yes, read-only counts as downtime for me, it’s wikis we’re talking about). Most real-world businesses would be declared dead by now.

With all this said, I certainly wish you a quick recover. I will lose some interesting writings and midly important writings of me if you don’t. Don’t fail us. Please?

Wikia may be more reliable as in server wise but the staff are horrible, the skin is horrible and the new editors they’ve released are horrible. I would much rather have an unstable host over that.

That being said, we all know how bad it is that this is happening. Sure, the techs weren’t advertised as volunteers but many websites run off volunteers. Not many websites actually employ staff.

Another month of downtime is very bad, we know. Hopefully, once the transfer is complete, this should not happen again and if it does, it won’t be as severe.

(I am not staff but this is news I have gotten direct from staff and I’m sure they’ll say the same).

@Hockeyben

As I saw on IRC, it seems they prepare to transfer to the new server, but wait for official confirm as Jedimca0 is waiting news from techs.

@Facy5 : Salut Facy, ce qui s’est passé, c’est que les responsables du serveur polonais ont voulu changer l’adressage IP de ShoutWiki provoquant un nouveau blackout et tout ça sans prévenir l’équipe SW. Par conséquent, le staff a décidé de passer tout SW sur un serveur situé aux US qui devait normalement servir de serveur de secours, mais qui devient le serveur principal. Bien entendu il fallait vérifier que toutes les données n’était pas corrompues (tout va bien de ce côté-là) et maintenant, il doivent se préparer à commencer le transfert, ceci expliquant la perte de com avec le wiki. Ceci est pour l’instant officieux, Jedimca0 attend des nouvelles confirmées avant de publier cela officiellement.

It seems the Polish datacenter was nice enough to update and then restart the server while we were in the process of transferring files to the new server. The file transfer has now resumed and we are on our way to our new, better and more stable, home once again. I hope to have an update soon, verifying that all files were successfully transfered.

I have over 100 pages and pictures to upload, when can we edit again? A move should not take long time.

Were talking about more than 750 wikis here. It may be longer than we think. And the techs have to check if everything is Okay.

Moving 750 wikis is a lot of work, yes. But this is also not just any move, a lot of fixing (issues caused by the Polish datacenter) had and has to take place. And like I said before, had we planned the move, we would have made sure our techs had plenty of time to get it all done. Right now we are unfortunately forced to work around certain other schedules.

We are not waiting with this move because the Polish datacenter we trusted has experienced two “incidents” recently and they have yet to sufficiently explain to us what exactly happened.
And that is something we cannot and will not accept from them. We want to offer you a stable ShoutWiki, which is why we decided to move right away. I know it is all taking longer then we would like it to take, but the techs are working as fast as they can.

Although it’s sad to see the site down for so long, I’m very glad ShoutWiki takes the issue seriously and works to make sure their fixes aren’t just half-assed patches. Despite these issues, I’m still very glad I made the move to ShoutWiki, and look forward to continue using the service.

It’s irritating that this caused ShoutWiki’s downtime – but I am soon to move some wikis to ShoutWiki. I find this wiki farm a great one, and actually, unlike Wikia, provides a service that provides their customers with the best possible experience, while Wikia provides their customers with one skin which reduces reading and editing space for room for more advertisements just for the pleading for money. I hope this issue will be solved ASAP.

”I hope to have an update soon, verifying that all files were successfully transfered.”

One week later. Nothing. Come on!

Patience is the word here. (Would you rather wait for a stable ShoutWiki that would never have any future problems, or get a ShoutWiki RIGHT NOW that would have future problems?)

Leave a Reply

Your email address will not be published. Required fields are marked *