A few days ago, I announced on Facebook that I was working on upgrading ShoutWiki to the newest MediaWiki version, 1.20. For those who don’t know, MediaWiki is the base web application that ShoutWiki is based off, along with Wikipedia, Wikia and many other sites out there on the world wide web. This wasn’t necessary a planned rollout, we just knew that we wanted to get this new version out as soon as possible, already being delayed by real life factors influencing the availability of the staff needed to make this happen. This is why it was not announced here, as the days proceeding the unavailability of our staff were spent on this upgrade.
When I was happy with the upgraded copy, and a new tool or two we put in place internally to smoothen out the upgrade process, I updated the production copy of ShoutWiki, putting the wikis into “Maintenance mode” while the databases upgraded. In theory – and next time, this should be a fairly painless procedure with no more downtime than necessary and no left over bugs to cleanup. This time, that wasn’t the case. We had a series of bugs, some through human error, one or two caused by upstream factors. These bugs and a log of actions are detailed below:
Saturday 17th November, 21:41UTC – Wikis brought back out of maintenance mode. A problem was quickly spotted with captchas not working causing many edits to fail. Another issue with adverts not showing in Vector and Monobook was also discovered, as well as a few smaller non fatal errors leaving messages in our error logs.
Saturday 17th November, 22:10UTC – Adverts in Monobook are fixed.
Saturday 17th November, 22:17UTC – Vector adverts fixed.
Saturday 17th November, 22:41UTC – ConfirmEdit, the extension we use for Captchas is rolled back to an earlier version. We believe this may have been causing us problems because of an incompatible version of the extension being branched for 1.20, however this could be a new feature that is incompatible with some of our systems. We are currently reviewing this breakage to find the root cause and stop this happening again.
Sunday 18th November, 20:33UTC – I am alerted to an error in a wiki on ShoutWiki missing part of its database. Upgrade script is run to correct problem on all wikis in attempt to fix any wikis globally affected by this problem.
Tuesday 20th November 2011, 20:14UTC – After having another report of the same problem from Sunday, the team discovers the problem, rooting from an outdated copy of tables SQL being used to create new wikis. Updated copy of tables deployed and all wikis fixed by script.
Tuesday 20th November 2011, 23:20UTC (approx) – Alerted to error on new wiki creation. Database error.
Wednesday 21st November 2011, 16:17UTC – Team identify problem of additional unneeded tables in the SQL file causing MySQL to throw errors. This is patched fixing problems and allowing new wikis to be created again.
We believe this is everything fixed now. We thank everyone for their patience through this time and if you have any more problems please contact us through the usual channels.
This is the first update that I’ve managed on done through this message, the second or third in total I have managed while here. Through the course of this upgrade, we have learnt a few things and increased our knowledge and toolset to allow us to perform future upgrades with more efficiency and speed and in future releases we should have less bugs while completing this process.
— Lewis Cawte
Chief Technical Officer, ShoutWiki.