The Current Situation: As it Stands.

As many of you have noticed, as of late ShoutWiki has been displaying more than a few HTTP 503 errors (“Service Unavailable”.)

This blog is going to outline what these issues are and why they are caused, what we are doing and why this blog has been published now. Let’s start with the latter. The reason we’ve left it so long is because we had originally planned to be able to give you all the facts on, in particular a timeline related to the solution. Unfortunately we are unable to give you these dates as of present – although as soon as we are aware we will let you know – and we feel it is wrong to keep our users (you) in the dark for even longer.

To keep it simple, the issues are being thrown by our Varnish caching server as the Apache server (which servers you the pages and images) has been unavailable. In short, this has been caused because the web server has been using excessive amounts of CPU and RAM, and the whole server has been left without memory.

What have we done so far? We’ve temporarily disabled various high CPU processes (unfortunately this includes export functions) in an effort to preserve and extend the uptime of the site. In addition we’ve taken steps to make the infrastructure more scalable. We’ve been making steps towards making it easier for us to add additional web servers to our setup by moving non-wiki sites to their own server, and we’re looking into moving images to the cloud. Obviously this takes a fair amount of planning due to the financial and technical implications.

And finally, what we’re going to do. Simply put, we’re going to reinstall the operating system on the server. We’re fairly certain that there is a problem with the template the web server VPS uses. While we do this, we’re also going to be switching from Apache to Nginx in an attempt to improve performance. When is this downtime, and how long it will it take? We’re not entirely sure – which is what we’ve been waiting on. Our web server VPS is attached to the account of a former staff member, mainly for financial reasons, but also due to the connections this former staff member has with our provider. Unfortunately, we’ve been unable to get a hold of them recently (as soon as they are around again, we’ll be able to get the apache server back online!), and we are still waiting on a reply on when he is available to reinstall the operating system. The length of downtime will be determined on how much data has to be shuffled around, currently images take up a large proportion of the disk space, which is a big reason in why we want to relocate our images.

ShoutWiki is a for profit organisation, however none of our staff members get paid for their work, they are all volunteers and all our income is reinvested into servers and this will be the case until we can establish a fast and stable service.

Again, we apologise for the downtime, and thank you for your continued patience. We understand this is a very frustrating time and we look forward to resolving the issues as soon as possible and allow you to get back to editing your wikis.
On behalf of ShoutWiki staff,

— Lewis Cawte
Chief Technical Officer, ShoutWiki.

5 thoughts on “The Current Situation: As it Stands.”

  1. “ShoutWiki is a for profit organisation, however none of our staff members get paid for their work, they are all volunteers and all our income is reinvested into servers.” Wow, thanks for your efforts! It must be frustrating dealing with all this. Not even Jack Phoenix gets paid a little? He should. Lewis Cawte? Does Shoutwiki have a PayPal account or some other way for people to donate money? Feel free to cover my wiki with ads on all 4 sides.

    I am curious about your server setup. Are all the servers, VPS (virtual private servers), software, and hardware used to host Shoutwiki, on internet hosting sites?
    *https://en.wikipedia.org/wiki/Internet_hosting_service
    *https://en.wikipedia.org/wiki/Virtual_private_server

    If not, why not? Why go through the hassle of trying to buy and maintain hardware? That makes for difficult scalability, especially with bandwidth fluctuations, increasing need for space for images, etc.. Also, why did you increase the allowed kilobyte size for images? It was limited to 500 kilobytes maximum image size before. Now it is more. 500 kilobytes is large enough for almost anything I have ever wanted to do on my wiki, and I have many images on my wiki. I just use IrfanView to reduce the size and/or resolution of images as needed. Wiki farms can not afford to host large print-quality images.

    “The web server has been using excessive amounts of CPU and RAM”. Is this possibly caused by DDOS attacks? That was mentioned in the Facebook comments:
    *https://www.facebook.com/ShoutWiki – something about the possibility of a DDOS attack due to a wiki about Syria. Maybe that wiki can be moved to Wikia, and let them worry about it. 🙂
    *https://en.wikipedia.org/wiki/DDOS

    Maybe super controversial wikis that get all of Shoutwiki shut down should have their pages moved to big free hosts such as Wikia, Google blogs, WordPress.com, etc.. To those sites a DDOS attack is probably just a pinprick in most cases. Those sites get so many page views already.

    1. Hi Timeshifter,

      Firstly, yes, all of our servers are virtual private servers are provided to us by an excellent host. Secondly, I don’t believe your facts are correct in regards to the image sizing. We raised our upload limit from 2MB (2048KB) to 5MB (5120KB) a while back, as some users had been requesting this to upload other resources to complement their wikis (not just images, everybody has different needs!) We were always going to need to change our image storage solution as the more wikis we host, the more images will be uploaded.

      We have seen the posts on our Facebook page suggesting DDoS attacks. We’re not sure that these problems are being caused by a DDoS attack. However, even if this is a DDoS attack, we avoid forcing wiki communities to move, especially when they are fairly discussing important topics, despite controversy. Our motto is “Express yourself” and personally, I feel it would be hypocritical to in effect, censor a wiki like this.

      Finally, in regards to donations – this is not something we are currently doing, as we cannot provide anything additional in return for donations and it be unfair to ask for money from our users who already contribute so much to ShoutWiki. As for additional advertising, Google strongly recommends that we only place at most three advertisements on each page. We’re working on making our paid feature purchases more automated and further developing that revenue stream.

      — Lewis Cawte

  2. I started a thread about site outages in the Shoutwiki forum:
    *http://www.shoutwiki.com/wiki/Forum:Site_outages

    This way the links are clickable, and I can be notified of replies to the discussion. I have my preferences there set to send me email when my watched pages (such as some forum topics) are edited or replied to. Can I copy your reply to that forum? Or you can copy it there.

  3. Hey guys, have you considered migrating to Amazon AWS? I’ve been reading and playing with it and I can imagine a fairly durable solution if you use all the goodness such as automatic load scaling and balancing, dedicated database instances etc. It also tends to be cheaper than traditional VPS solutions, if designed properly.

    Misza

Leave a Reply

Your email address will not be published. Required fields are marked *