Welcome back to the blog! It’s been a while. Since we promised you an update – particularly if you’re not following us on Hub and Facebook – you’re probably overdue. So let’s get to it…
As I’m writing this, about a month later than intended, MediaWiki 1.35 has arrived on ShoutWiki. Not without its challenges, as I’m sure you noticed.
Initially, we planned to deploy MediaWiki 1.35 on our new web server to make the switchover fairly seamless. Turns out ShoutWiki got bigger and busier than we thought, and making that move is nowhere near as simple as we thought. We also finally switched our internal development setups away from Subversion version control to Git. These are all processes that were started way back in 2021, during later waves of lockdowns here in the UK.
Two Years?!
It hasn’t been smooth sailing for the last three years. While after fighting through a rather rough patch for the first wave of COVID-19 lockdowns, we managed to bring a number of long updates to ShoutWiki, including mobile support.
Since then, things have gone pretty quiet. In 2021, I graduated from University (while working most other waking hours in retail) and started a full-time graduate job in a new industry after a month of huge uncertainty. In the 18 months I’ve been doing that, I’ve worked in three different teams in different cities on a whole variety of projects. After relocating during the summer of last year, I’ve finally started to recover and have been able to dedicate some time to ShoutWiki.
As noted, we couldn’t pull off the server move behind the scenes ready for this update. In February, we decided to proceed with the update on the existing production server so we didn’t slip further behind with MediaWiki releases.
The upgrade…
What a mess. Upgrading MediaWiki for something like ShoutWiki has never been a simple process, but this one was pretty spectacular. For once, it wasn’t actually MediaWiki that caused us that many headaches, which is a testament to all the hard work Jack has put in over the period getting our code up to date.
Most people will have noticed the two weeks of really poor performance surrounding the upgrade. This was ultimately resolved by a second operating system upgrade that we had to undertake in production – not the main reason, but one of the reasons we were hoping to switch to the new server. This was required as some of the extensions we use required a newer version of PHP. Long story short, it seems that some of the backported PHP builds for older OS versions have some incompatibilities causing frequent unpredictable crashes to PHP.
Added to this, some people will have seen (and a handful still do – sorry – please keep hitting up the support email!) various errors caused by missing or mismatched versions of extensions. The move to Git meant we had to rebuild our repository from scratch, which is fine; however, this meant we could change how we handle a couple of things that will make building future updates easier for us behind the scenes.
With ShoutWiki now being 16 (and “live” for 14) and over 10000 wikis, it’s fair to say there’s a lot of legacy to account for. We missed a number of extensions, found old ways of including extensions, had to rewrite some of our wikifarm handling code, and removed a few outdated extensions or didn’t meet our standards (more on this later!) We had to clean up a number of these things in production, and the nature of some of this meant it blocked the entire deployment.
Where now?
I’ve spent a while talking about what happened, but I’m sure most of you mostly care about the future. As a team of two volunteers, we’ve got a lot of things to progress, and we need to work on multiple things at once to manage burnout and avoid wasting time/energy. Without further delay –
CreateWiki
No doubt the most requested item since it went horribly wrong, and we had to disable it after the 1.34 upgrade. I’ve undertaken a full review of the PHP and identified a number of areas we need to address.
We’ve also invested some time in a more complete testing environment to allow us to better replicate the full functionality of features like CreateWiki. At this time, there is no timetable for when CreateWiki will be fixed, but we’re working through a number of changes, and announcements will follow when it’s ready to go.
HTTPS
Honestly, this is absolutely something we should already have in this day and age. We started work on this a few years ago. We proved we could do it with a small handful of sites on custom domains. We also identified some challenges with our legacy setup that have prohibited us from deploying this to the rest of ShoutWiki.
The main change required to this is an item in itself, and as such, is covered below. We’re also deploying a new frontend configuration to handle HTTPS connections.
Extension & skin support
One of ShoutWiki’s key features is its support for various MediaWiki extensions and skins. While we don’t want to lose this customizability, supporting a wide number of third-party skins and extensions places a big burden on us as staff, so we’re trying to balance this.
As we mentioned earlier, we removed a number of extensions from our production repositories with this MediaWiki update. We’re in the process of internally formalizing some new guidelines on skins and extensions that we will support, which we’ll publish in due course, accompanied by the full current list. These guidelines will outline minimum requirements, mainly a number of established good practices such as code versioning and security.
Like any good rule, there will always be exceptions, but the drivers for this change are to improve security and privacy while ensuring we can use our limited time to the best use of all our wikis.
Sunsetting language subdomains
Language subdomains (language.topic.shoutwiki.com) have been the primary blocker to the full-scale deployment of HTTPS. We’ve reluctantly taken the decision to sunset this optional setup as the only sustainable and economical solution to allow the rollout of HTTPS (as well as other features) in the near future.
We’ve been slowly reviewing our language subdomain sites to identify the impact. While this work is still underway, language subdomains represented about 9% of ShoutWiki sites (by created wikis) when we started. So far, we’ve removed about 40% of these sites manually.
What does this mean for you, particularly if you use a language sub(sub)domain? If you have an active wiki with content, your wiki URL will likely change. At this moment, we’re not yet in the change stage. When the time comes, we prefer to simply remove the language part of the wiki’s subdomain. If this clashes, we’ve got a number of other options which we will contact the local communities to agree upon. We understand that URL changes are disruptive to users and search engines. We’ll be leaving permanent redirects in place for a number of months to reduce the impact where possible.
We’ve identified another 25% of these sites as English language sites that opted to use the en. subdomain. Where these subdomains don’t clash, we’ll simply be renaming these as there’s an existing fallback for the English language subdomain in the wiki farm code.
Ads
We fully appreciate that adverts are disruptive to the viewing experience. Unfortunately, AdSense has lost a layer of granularity we used to reduce some of the most annoying placements for editors.
These ads pay for ShoutWiki. In order to generate enough revenue to continue to maintain the site and, importantly, allow us to continually improve our offering, we’ve had to opt for allowing “automatic” ad placements, which are more intrusive. We will continue to review this setup and make optimizations where the software and ad network allow us to do so.
Reducing (technical) legacy debt
As we mentioned earlier, there are a lot of things about running a 16-year-old website with 10000+ wikis that are far from perfect or shiny and new.
We’re working on overhauling several things under the hood, implementing newer tools and configurations in use at Wikimedia and other farms, and recovering some of the service levels we had when I first started (albeit with less staff) back in 2012.
Server moves
There’s not one, but two new servers in the works. We identified that in order to grow and provide the additional features that we want to see, as well as those that should be standard now, we needed a little more computing power.
Step one is moving wiki files (uploads) and our bulkier data to a new server. We can do most of this in parallel to keep the service running and allow uploads to continue. At some point, we’ll announce on our social media channels we will have to temporarily disable uploads to allow for a final sync before switching wikis to use the new file server.
The second step is tackling the previous blocker for 1.35, switching our production web server. We’re very much pushing the limits of our existing web server. The main challenge has become how we copy ~600 gigabytes worth of databases (split across 10000 databases) in a way that loses no data and results in minimal downtime.
We’re heavily IO-limited on our existing disks, making transfer times slow. Backups are hard to sync because each database is backed up at different times. We think we might have found a solution – testing is currently taking a back seat to other items listed in the blog but will begin soon.
The next upgrade
The last thing on the list (although there are probably lots of other things going on that I’ve missed) is the next upgrade.
There’ll be continued mini-releases for the points listed above. All being well, these should be unnoticeable and momentary blips at worst. Regarding the next MediaWiki upgrade, we’ll be following the long-term support (LTS) releases for the time being to give us more breathing room to focus on something other than trying to constantly be at the bleeding edge of MediaWiki software.
Jack had already committed most of the 1.39 specific changes to our extensions and skins (prior to the 1.35 deployment), and we’ll be building and testing the new release in the next month or so. We expect to release this sometime during the summer prior to the end of support for 1.35.