Matrix Reloaded

Hey all, hope you’ve been doing alright! I’ve been busy working on a couple of things in my free time this month and I’ll have more specifics to share soon but, for now, I figured that I might as well spend some time talking the process of setting up all the Matrix things.

I’m no newbie when it comes to setting up servers and operating systems. Obviously enough, as I have tended to THP’s operation for over a decade now. Still, there are often multiple ways of approaching things. Not to mention that new technologies and solutions arise with time. So while, yeah, a traditional server deployment on a hosting service with a lot of manual intervention is still possible, the demands of the ever-expanding cloud service providers and businesses have shaken things up quite a bit.

So, partially to see what all the fuss was about, I went out of my way to try something different.

For starters, I decided to try out a “cloud” solution with a more flexible “server” setup. I’ll skip over the more technical distinctions between that and the space where THP is currently hosted (broadly, they are similar) and instead focus on the practical difference when it comes to administering it: the host that I’m using runs OpenStack which allows for a large range of customization when it comes to deploying a server; the exact amount of storage space that I’m paying for, processing power etc can be increased or decreased as needed without needing intervention from the hosting company.

It’s a well-documented API that allows for cool things like customizing images (OS, installed software, users etc) before actually installing them on the server instance. Yes, it’s another thing to learn and not entirely straightforward but it is usually pain-free for casual users who just select a premade image from a drop-down list, add a password or whatever and their OS is all set up for them.

(As a complete aside: finding a host that offers this flexibility, that allows adult content and doesn’t impose limitations on what can be hosted isn’t that easy. A lot of the cloud providers that I looked at are in Europe but anything that involves Germany is difficult to deal with. The French, on the other hand, are fantastic when it comes to balancing privacy, technology, and freedom. I ultimately went with the same US host that runs THP because of a combination of familiarity with their service, attractive pricing and also the fact that I wouldn’t have to overly concern myself with complying with the GDPR. Nothing against the GDPR, btw, just that it’s yet another thing to learn and exporting requested data from Matrix might prove to be a hassle. Anyways, the server is based in Seattle—THP itself is in NYC)

The thing is, however, that due to my adventurous mood, I wasn’t going to settle for something as mundane as Ubuntu Server or Debian. I like both and THP has been using Debian for all of its history but I could also probably write several thousand words of criticism about that ecosystem and my experiences with it. So I decoded that I was going to go with what I run on my desktop: Arch Linux.

Those of you who don’t really understand Linux and why there’s a million different distributions might not get why one might use one distribution over another. I’m going to massively oversimplify things.

Debian is infamously conservative with updates and likes to set up things for the user. It has a 2-3 year release cycle where all software is “frozen” in its current version and any and all updates are just security updates. The intent is to offer a stable experience that doesn’t require any tinkering at the expense of new features and developments. Generally good for a web server, especially for ones that aren’t expected to deploy the latest and shiniest things.

Arch is a rolling release distribution, meaning that there are no versioned releases of the operating system and that software is added as it is released by its creators and tested to be stable by the Arch maintainers. It also requires manual setup of many things because the idea is to keep it simple in terms of having the user customize the system to perform as they like.

Since Matrix and the various implementations of the protocol and its components are continuously developed, I figured I might as well go with a distribution that is guaranteed to have the latest and greatest. Yes, there are ways to install newer software on Debian, especially containerized things, but I figured that, if I was going to make all that effort anyways, I might as well just use a distro that already expects the user to set things up. The familiarity I have with it on my primary computer at home also titled the scales.

While Arch is used on servers and in specific deployments, it is not that common an occurrence. Decades of inertia favor either Debian (and Ubuntu which is ultimately derived from it) or the commercially-backed Fedora/CentOS ecosystem. So the available images on my host’s page did not already include Arch as an option.

No problem! Arch offers its own cloud image compatible with these standards and API. The issue was, however, was figuring out how to get all the supposedly-automagical parts working together. The image would install but it would not set up the default user properly nor the permissions for me to remotely log-into the server from my own box. Despite my reading of documentation and trying more advanced things, I wasn’t able to get everything working the “proper” way.

Luckily, years and years of experience of being a hackerman have taught me to think outside the box when it comes to finding solutions for these types of things. So I decided to completely disregard the initialization API and its annoying YAML syntax and instead modify the misconfigured automatic install of the Arch image I uploaded. I used one of the other available images to load up a commandline rescue environment, log into that, and then mount the install on the server’s storage and modify manually the users and remote access configuration. It was a bit of trial and error to find the the image among the two dozen or so available that had aforementioned rescue environment but after that, things went smoothly.

Now, this deployment should have taken the whole of five minutes. At this point, doing things the “easy” way had required a larger time investment than foreseen (about two hours). But hey, at least I had a working server!

Since the automagical parts that were supposed to configured by the hypervisor weren’t configured properly, the first thing I did with my new server was to cull all of that from the install. That broke some of the automated scripts and I spent about another hour fixing things and optimizing the boot process. In other words, longer than the normal Arch install process. But I got things just the way I wanted them eventually (including hardening network access and security) while managing to learn more about all these new-fangled technology and methods. So not a complete waste of time.

Only then did I get around to setting up Matrix. I thought about doing things manually but an internet acquaintance had recommended I instead use an Ansible playbook. More automation. More magic. My appreciation for all things magical remaining eternally undiminished, I decided to trust the magicians behind the spell-err, I mean, playbook

I’ll provide yet another quick explanation for what was going on with this so-called playbook. Basically, it’s a bunch of scripts that process configuration files to produce a fully-functioning environment without the need to faff about too much. There’s a lot of moving pieces to the Matrix install: the server itself, the backend database, the front end web client that’s being served, the mailer, various bots, proxies, admin tools etc etc. All of which have their own configurations and syntax and potential pitfalls and caveats.

With the playbook a user only needs to flip a variable to on or off, run the script and changes will be made across all interrelated parts. It takes care of updates as well and also exposes more advanced features to users who really know what they’re doing as well. The trade off are some limitations that I’ll get to later.

Things with the playbook went well. I got the basic setup working almost immediately reading the documentation, setting up my domain and subdomains to accommodate for the services I was deploying. I had to wrap my head around a lot of new concepts but I would have had to do a fair amount of that if I had gone the manual route as well. The added bonus of the playbook was that there were relatively good examples for most common use cases and it gave me a better appreciation for how things came together. I’m particularly thankful not to have had to setup the database and the mailing service as those two things are usually a pain to manually set up with sane parameters.

I spent a day and a half tweaking things, familiarizing myself with how to rebuild and upgrade stuff and slowly adding new features as I went along. I ran into a few problems but most of those were upstream limitations. In other words, specific bits of software that were not as easy to customize as I would have liked. Over the following days I spent a lot of time looking up documentation and extending the playbook’s configuration, tweaking as I might have without it but with most of the groundwork taken care for me.

Most of the services set up by the playbook use Docker containers. Briefly, containers are isolated installations for software that can be deployed on any machine regardless of the setup of its host operating system. It has its upsides with reliability and the guarantee of the same environment for all instances being the bigger ones. A big downside, however, is that if you need to customize things for whatever reason, it can be a bit of a pain. Especially when coupled to a playbook that assumes that many of these container images will remain unchanged, only exposing one or two configuration files otherwise.

One concrete example of a pain point would be with the web server parts of the stack. Like, the home page on the domain. On THP I have things set up so there’s all manner of redirects and smart filtering rules (like replace invalid subdomains with www, limit rates for images, kill hotlinking, redirect banned users to certain pages). The playbook exposes some options for some parts but basically urges you to use its automatic reverse-proxy settings (redirecting a lot of matrix API requests from behind the scenes into URLs) or to just set up everything manually. That is to say, without the docker image. So you either don’t modify the generated images much or completely forgo using them. Since there’s a lot of moving parts that can be enabled or disabled, keeping track of required changes is a real chore. Essentially, I had to make a lot of compromises in the service of keeping to the dockerized ecosystem.

It’s a similar story for the web client, Element, that’s served. There’s a lot of things that I would like to change in regards to its appearance and branding but I simply cannot. This isn’t the playbooks fault really either—some values are hardcoded in the source code and I’d basically have to edit those files directly. But that means making the docker image writeable and adding a bunch of utilities within the image in order to be able to work within that environment. And, from then on, maintain those changes myself and make sure that they don’t get overwritten either by upstream Element changes or the playbook’s configurations.

Still, I was impressed with the flexibility of the playbook itself and by how well-integrated things are. It definitely lived up to its hype as being an easy way to deploy everything, small issues notwithstanding.

The rest of the time I spent working on setting up things mostly related to the Matrix service software. I’ll probably talk about some of those things more in the future since it’s a fairly complex protocol and topic. Needless to say that things like who can create rooms and when, how long messages are kept in the database, how long media is kept, and many other things all require careful reading of documentation and understanding of the Matrix API. The administrative tools aren’t all quite there yet either so a lot of it has to be done with specific syntax remotely. Element at least has a few debug tools for admins to change things on the fly but there’s a lot that’s just very much a matter of things the hard way.

The total time it took me to setup the server and its environment, excluding the Matrix server stuff, was a few days. Comparable to doing things manually, mostly because I was learning new concepts and technology as I went along. Still, I’m glad I decided to approach things differently and broaden my horizons. Things have been running well for over a month now and, should push come to shove, I could set up things again in a fraction of the time. With all the scripts tuned to my liking backed up, it might well be the envisioned matter of minutes.

I’ll end this overly-lengthy post by telling you the server’s hostname: Makai. Pandemonium trumps discord, don’t you think?

Until next time, take it easy!