Categories
General Technical

Update on Downtime / F.A.Q

This blog post will serve as a update on the downtime as of 16/04/2014 and an F.A.Q on this and other general downtime and related issues.

The Downtime

So initially, sites were down due to high load on the web server, which causes 503 errors that you see. Earlier that day, it turns out that our Varnish caching software had crashed resulting in about 10 hours of downtime overnight. The downtime you are now seeing is different. Due to problems at our host/datacentre, it appears the hardware that the database server on began having problems, and our host migrated the server to a new “node” without shutting down the server. There appears to have been complications with this that resulted in a sizeable amount of data loss and some database corruptions that were very unlikely and unpredicted when we initially went down.

Our Data Is Gone?

That title is what I assumed might be going through your heads as you read the last section. The answer isn’t a simple yes or no. Our host has a backup from about 12 hours prior to the migration of the node which was during the Varnish crash we believe. In addition to that, the current version of our database server is accessible and has some wikis we should be able to export. It is very important you read the following section thoroughly before clicking the link.

We have logs of all actions that we use to fight vandalism and spam. Using these, we’ve mostly determined the potential loss of data. These “loggable actions” are not all edits. They include things from deletions and protections, to upload and patrol logs. (note: files are not stored on the same server, and are unaffected, they will lose their relevant File/Image namespace page)

Link to list of affected wikis

The Plan

So this part of the post isn’t confirmed, we’re still waiting on a reply from our host but we needed to let you guys know the situation and calm the panic.

As it stands, our host have offered to put up a second instance of our database server running from the backup they took. What we then intend to do is extract as many of the affected wiki’s databases that are not corrupt from the primary instance and import them into the backed up server. This should mitigate as much of the loss as possible.

F.A.Q

When will this be done?

We can’t give an E.T.A at the moment as we have to wait for host to confirm the plans and get things set up. The database server back up is a large file and will take time to transfer. Have you ever performed an action on your computer and the estimated time keeps changing? This is why we can’t tell you for certain, but we’ll be working as hard as we can to get the wikis back up as soon as possible.

Is this the same reason you don’t tell us about other downtimes?

Partially. There are always different factors in each type of errors and sometimes its to do with external factors.

Am I affected?

Unless you contributed to any of the wikis listed after 3AM EDT on Monday the 14th of April, it is very unlikely. However preference changes are not logged so if you changed a setting you may have lost that.

So I should backup my own wiki then/Turn back on DumpsOnDemand?

No. It is very very rare we see data loss and the reason we’ve disabled the features as they are a key cause of the 503 memory issues. It puts high load on the server which ultimately results in downtime. It is understandable for you to see this as the perfect excuse to cry for DumpsOnDemand but it causes downtime, and we get into a vicious cycle of people panicking and backing up their wikis, stopping the service for everyone else.

Why don’t you just buy more servers?

Simple answer, we can’t afford to. We have a former staff member who controls a handful of our servers on his personal account as he gets better deals than our current staff and we can provide the most quality for the little money we have.

Categories
General Technical

Happy Holidays!

It’s December and that means it’s the holiday season! As we prepare for the holidays and spend some quality time with our families, we’d like to share a little present with you — think of it as our way of saying “Thank you for the past year”! But first, let us tell you the story behind it…

It all started with Isarra, ShoutWiki’s designer — though this story predates her ShoutWiki career. In addition to having a university degree in Computer Science, Isarra is a highly skilled individual, as witnessed through several of her creations. And that’s also the very reason why I had offered her a position in ShoutWiki a fair while ago. Before that happened, she designed several graphics for us as a volunteer, as well as several propositions for a new logo and a new skin.

In September 2013, things had changed enough and Isarra was able to join ShoutWiki, which greatly pleased all of us. By that time Uncyclopedia, the content-free encyclopedia, or simply put, the Wikipedia parody, had moved hosting. It’d never have been possible without Isarra, who in addition to her role as an Uncyclopedia administrator helped the entire MediaWiki community by creating and enhancing several tools for moving a wiki without direct database access; these tools include the set of command-line scripts for grabbing a wiki’s contents and the MediaWikiAuth extension. I had the opportunity to help Isarra with the development work, which would eventually benefit not just the Uncyclopedia community, but ShoutWiki and the MediaWiki community at large, too.

Now that you know who the creator is, it’s time to meet the creation: the Aurora skin for MediaWiki! This skin was designed by Isarra exclusively for ShoutWiki and as such, it won’t be available anywhere else. We’ve been working on the skin for quite a while — after all, the initial mockup was written well before Isarra joined ShoutWiki staff, but it wasn’t until 8 December 2013 that we had an almost-ready, feature-complete skin up and running.
As Isarra herself described the Aurora skin, “It has attained this level of threatening usually restricted to things out of myth and legend” — and legendary is just the word that I’d use to describe it. Why? Unlike any other MediaWiki skin, the Aurora skin…

  • …is exclusive to ShoutWiki
  • …supports multiple social networking sites “out-of-the-box”
  • …features two sidebars
  • …has a built-in Coordinated Universal Time (UTC) clock

At this point, we want to emphasize two things: the Aurora skin will become the default skin on ShoutWiki Hub, but as always, you will have the ability to choose what skin you want to use. We at ShoutWiki believe in choices and options, not in restrictions, gates and walled gardens.

Right now Aurora is not available for previewing, but we expect to launch it very soon. Stay tuned!

On behalf of the staff team,
Jack Phoenix
Head of the Customer Support Team