Categories
Customer Support Features General

Phabricator: ShoutWiki’s new bug tracker

home

Today we’re announcing public access to our new bug tracking and project management system.

Phabricator is a powerful open source system initially developed by staff at Facebook. We originally moved to Phabricator in 2015 to take advantage of the set of repository hosting tools that are included within the software.

Our move was prompted by the Wikimedia Foundation’s adoption of the software who are also using Phabricator as their bug tracking and code hosting system.

As some users have noticed, our old bug tracking system, powered by Bugzilla has been down for a considerable amount of time. Bugzilla was written in a mix of Python and Perl, whereas Phabricator is written primarily in PHP, the same language as MediaWiki. Over time, Bugzilla has become increasingly hard to maintain with our current software stack, and when we migrated many of our backend tools to our new server in, we made the decision that it was time to migrate to Phabricator.

One of the key features we are excited about in the bug tracking software is the ability to assign multiple projects, whereas in Bugzilla a bug could only be filed under one component and product. A task may be filed under various projects where it affects different features.

Phabricator accounts are currently independent of ShoutWiki.com user accounts, and you can create an account using your email address and password or using a Google or GitHub account. We are currently attempting to implement the MediaWiki OAuth handler developed by the Wikimedia Foundation so you can link your ShoutWiki account with your Phabricator profile.

The Phabricator software is designed to be modular adding different modules for different aspects of functionality. Maniphest is the bug and task tracking module in Phabricator and has a simple interface for filing a bug. When submitting a task, you only have to submit a title and a description which can all be amended later. The projects box can be filled in if a particular feature is causing you issues and will auto-complete project names for you.

create-task

You can securely find Phabricator at https://phabricator.shoutwiki.com.

If you have any questions or issues with Phabricator, you can find us on the Freenode IRC network in #ShoutWiki, email support@shoutwiki.com or comment below.

Categories
General Technical

Update on Downtime / F.A.Q

This blog post will serve as a update on the downtime as of 16/04/2014 and an F.A.Q on this and other general downtime and related issues.

The Downtime

So initially, sites were down due to high load on the web server, which causes 503 errors that you see. Earlier that day, it turns out that our Varnish caching software had crashed resulting in about 10 hours of downtime overnight. The downtime you are now seeing is different. Due to problems at our host/datacentre, it appears the hardware that the database server on began having problems, and our host migrated the server to a new “node” without shutting down the server. There appears to have been complications with this that resulted in a sizeable amount of data loss and some database corruptions that were very unlikely and unpredicted when we initially went down.

Our Data Is Gone?

That title is what I assumed might be going through your heads as you read the last section. The answer isn’t a simple yes or no. Our host has a backup from about 12 hours prior to the migration of the node which was during the Varnish crash we believe. In addition to that, the current version of our database server is accessible and has some wikis we should be able to export. It is very important you read the following section thoroughly before clicking the link.

We have logs of all actions that we use to fight vandalism and spam. Using these, we’ve mostly determined the potential loss of data. These “loggable actions” are not all edits. They include things from deletions and protections, to upload and patrol logs. (note: files are not stored on the same server, and are unaffected, they will lose their relevant File/Image namespace page)

Link to list of affected wikis

The Plan

So this part of the post isn’t confirmed, we’re still waiting on a reply from our host but we needed to let you guys know the situation and calm the panic.

As it stands, our host have offered to put up a second instance of our database server running from the backup they took. What we then intend to do is extract as many of the affected wiki’s databases that are not corrupt from the primary instance and import them into the backed up server. This should mitigate as much of the loss as possible.

F.A.Q

When will this be done?

We can’t give an E.T.A at the moment as we have to wait for host to confirm the plans and get things set up. The database server back up is a large file and will take time to transfer. Have you ever performed an action on your computer and the estimated time keeps changing? This is why we can’t tell you for certain, but we’ll be working as hard as we can to get the wikis back up as soon as possible.

Is this the same reason you don’t tell us about other downtimes?

Partially. There are always different factors in each type of errors and sometimes its to do with external factors.

Am I affected?

Unless you contributed to any of the wikis listed after 3AM EDT on Monday the 14th of April, it is very unlikely. However preference changes are not logged so if you changed a setting you may have lost that.

So I should backup my own wiki then/Turn back on DumpsOnDemand?

No. It is very very rare we see data loss and the reason we’ve disabled the features as they are a key cause of the 503 memory issues. It puts high load on the server which ultimately results in downtime. It is understandable for you to see this as the perfect excuse to cry for DumpsOnDemand but it causes downtime, and we get into a vicious cycle of people panicking and backing up their wikis, stopping the service for everyone else.

Why don’t you just buy more servers?

Simple answer, we can’t afford to. We have a former staff member who controls a handful of our servers on his personal account as he gets better deals than our current staff and we can provide the most quality for the little money we have.