DevOps the fun way

Unless you're a sadist or you get paid to set it up, doing DevOps stuff is the most boring and tedious thing ever. Here's how I made it fun. ...My kind of fun.

Terror

There are many problems in modern society. Many people don’t have homes, many people don’t have food, and many people can’t even afford money itself at this point. And then there’s me, dealing with the age-old first-world problem of “I’m too fucking lazy to compile this code on my own for every platform” and the secondary problem of “I’m too damn lazy to upload the build to Steam when I’m done.” Many programmers like me, and many more unlike me, have had similar versions of this laziness - and that’s how DevOps came to exist.

Why would I waste my time digging out a Windows machine, a Linux machine, a Mac, a copy of SteamCMD, all of my build tools, and a copy of my game’s source code, and slave over a terminal for a couple of minutes banging out commands to build and upload my game? That just sounds like work. So naturally the solution is to set up some really complicated, convoluted automation system to do it for me, and take an entire day out of my life setting it up. That’s just what I did. And naturally, I’m doing it all on my own hardware, completely independent from the cloud…except for uh…the parts that are.

Oh, you thought I meant “fun” as in “fun.” No. I meant “fucking utter nightmare.” That’s more my style.

The Background

A few months ago, I got annoyed at the GitHub UI and decided the best solution to that problem was to take my old gaming tower and turn it into a GitLab server. Most normal people would just use GitLab.com to host their code, but I am not normal. Personally, I like having my own little place on the Internet to host my code. After all, I even get to claim ‘ritchie’ as a username.

So I put a beefy hard drive in that ancient tower of ivory, loaded Linux on it, and now you have https://gitlab.acidiclight.dev/ as a thing that exists. And it genuinely does work well for me, and I feel comfort knowing that I have complete control over where my code is stored. So I moved most of the repositories I care about to it, including Socially Distant.

\Problems came about when two things immediately broke as a result of moving Socially Distant from GitHub. This is because vendor lockin is a very real thing that exists.

  1. The Developer Certificate of Origin wasn’t being enforced, because we did this with a GitHub bot.
  2. Builds of the game were never being made. This is because we never set that up in the first place.

This is where DevOps comes in.

Recreating the GitHub bot

It’s worth noting that GitLab has native support for DCO sign-off checks. You have to pay for it.

The goal is to do this without throwing money at it - remember, some people can’t even afford money.

Thankfully, acknowledging a DCO is so easy that Git has a command-line flag for doing it. As a user, all you have to do is this:

git commit -sm "My commit message"

Git will then sign-off the commit with text like this:

Signed-off-by: Your Name <your-email@example.com>

So all I need to do is write a script that checks to make sure you actually signed off the commit. If you have, then I know you’ve acknowledged the tiniest legal agreement ever!

So I originally came up with this job:

stages:
  - legal
  
check_signoff:
  stage: legal
  script: |
    echo "$CI_COMMIT_MESSAGE" | grep "Signed-off-by: $CI_COMMIT_AUTHOR"

Other than the necessary YAML needed for GitLab to pick up the job, this is a nice bash one-liner. We take the contents of the $CI_COMMIT_MESSAGE variable, run it through grep, checking to see if it contains the value “Signed-off-by: $CI_COMMIT_AUTHOR” anywhere inside.

This works because grep will report a non-zero exit code if no matches are found, causing the job to fail. It also works because, in GitLab, $CI_COMMIT_MESSAGE contains the full message text of the commit going through the pipeline. We are also able to get the user who submitted the contribution via $CI_COMMIT_AUTHOR, which contains the username and email of the user - in the exact format we need (name <email>).

It didn’t work.

You’d think that it’d be just as simple as saving the above code into .gitlab-ci.yml in the repo root, right?

If you did, it’s because you’re used to GitLab.com where they have the scary behind-the-scenes shit set up for you. That’s not the case for my GitLab instance - GitLab has nothing to run the script with.

Giving GitLab a computer to use

For CI/CD to work on self-hosted GitLab, you need to give it a computer to use. This is because your CI job might need to run on Windows, but your GitLab server is most likely not. It also means that you aren’t running someone’s random malicious code on the same machine as GitLab itself.

To give GitLab a computer to use, we need to find a computer and set it up as a GitLab runner. There are many ways to do this, and it depends on your use case and operating system. GitLab has this documented, you can find the documentation on the page that you use to actually create the runner.

The easiest way to give GitLab a computer to use is to install gitlab-runner on any old Linux box, and follow GitLab’s instructions to set it up as a shell runner. Enable the systemd service after registering the runner with your GitLab instance, and awesome. You just gave all your GitLab pipelines full access to a Linux shell prompt on an environment that they can do whatever the hell they want in.

For the DCO check, and because I have control over who has access to my GitLab runners, this was fine. This at least got it working, so no more need for the GitHub bot.

Then it stopped working…

I made the mistake of setting up a shell runner on my main computer. This caused two problems:

  1. Anything installed on the runner is now installed on my system, because it must be installed on my system. The runner shells don’t have root access, but if I run into a situation where I need Rust build tools for example, they’re now permanently installed on my system even though I’m a .NET user and have no interest in writing Rust code.
  2. You can’t contribute to Socially Distant if my computer is off.

This also meant that, when I switched to Windows a few months back, my computer was effectively unusable as a GitLab runner. And when I switched back to Linux, I had to get everything working right all over again. Yuck.

Killing two birds with libvirt

I still want to host the GitLab runner on my main system, because cloud servers are expensive, and I have the hardware. But I’d like it to be isolated from my development Arch install. I should be able to turn off the GitLab runner and move it to another physical computer, or god forbid, a VPS, if I need to. I like self-managing things, but let’s do it the right way so I can recover from hardware failure.

With that in mind, I’ve set up a GitLab runner server as a virtual machine running on my dev PC. I use and work with it in the exact same way you would a cloud server, but it’s running on my hardware. I still have to keep a computer on, but the specific computer in question is irrelevant so long as it can run libvirt.

This also allows me to use GitLab’s Docker executor.

The Ultimate Power of Docker

Setting up GitLab Runner as a Docker executor is just as simple as setting it up as a shell runner, but you now have the ability to use any Linux-based docker image you find or create.

If one of my projects needs a .NET 8 SDK, then .NET 8 it shall have. If it needs Python, then I’m questioning why I’m working on it - but Python it shall have. If it’s on Docker Hub, or another container registry I have access to, then I can write a GitLab pipeline that uses it.

Ultimately, this means I have a dependency on the Internet to get containers from - but I no longer need to install random crap on my main computer to get a build to work. Besides, the runner needs internet to talk to GitLab anyway.

Let’s use it to build Socially Distant

Socially Distant is very hard to build on Linux for a .NET project. This is because it uses MonoGame, and my own fork of it at that. You need to futz with NuGet sources, and you need to setup a Wine environment to use for shader compilation. It is a fucking nightmare to set up if you’re not sure what you’re doing, which is why it’s documented in the README.

However, even with documentation in place, I know that it’s a pain to set this stuff up if all you want to do is try a build of the game. This is where CI/CD comes in again, and where things get annoying as hell to deal with as a GitLab instance admin…but where Docker saves the day.

How you build Socially Distant on Linux

If you want to build Socially Distant for development, it basically boils down to these steps.

  1. Install the shit you need:
    • dotnet-sdk-8.0
    • git
    • curl
    • p7zip-full
    • wine64
    • knowledge of the actual package names for your specific distro
  2. Clone the repo: git clone https://gitlab.acidiclight.dev/sociallydistant/sociallydistant
  3. Add the NuGet source for our custom MonoGame fork, documented in the README, omitted here for brevity
  4. Run the mgfxc-wine-setup.sh script in repo root to set up the D3D compiler in Wine, hoping like hell the script actually sets things up right
  5. Praying to the environment variable gods that dotnet-sdcb is able to find the Wine prefix
  6. Hoping like hell your distro packages Wine properly so that you can actually use wine64 and wine32 together. I’m talking about YOU, Debian. Fix it.
  7. Attempting a self-contained linux-x64 dotnet publish of the game.

Because of flakiness caused by the mgfxc install script and/or Debian, this is really annoying to go through the first time. So you really want to get it right if you’re doing CI/CD, because it’s even harder to debug when it breaks.

It would be really nice if we could just have a pre-configured environment specifically for Socially Distant that just works.

Getting GitLab ready for this

I need to create a custom Docker image for Socially Distant builds. I know I can upload it to Docker Hub, but now I’m relying on Docker Hub to host the image when there’s enough space for it locally.

This is why GitLab is nice, because it can function as a Docker container registry. The problem is, this is something I need to set up as an instance admin. So I did.

My GitLab server is on an ancient gaming PC sitting two inches from my right leg as I write this. To prevent y’all from DDoSing me, and to work around dynamic home IP addresses, I use a cheap VPS as a gateway into my network. All traffic hits the VPS, and gets relayed to the tower over Wireguard using an nginx reverse proxy. This way, traffic also stays encrypted all the way through. This does, however, make things harder to set up.

First, we need a domain to put the GitLab container registry on.

CNAME cr.acidiclight.dev frodomar.direct.acidiclight.dev

Now, in gitlab.rb, I tell GitLab where the registry is. This is so URLs can be formatted right in the UI, but also so GitLab knows where to expect traffic to come from.

registry_external_url 'https://cr.acidiclight.dev/'

To get GitLab OmniBus to actually turn on the container registry in a way that’s reverse-proxy friendly, you also need this in gitlab.rb:

registry_nginx['listen_port'] = 8969
registry_nginx['listen_https'] = false
registry_nginx['proxy_set_headers'] = {
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
}

This gets GitLab OmniBus to pretend that container registry traffic is going over HTTPS, without actually listening with TLS encryption. In my case, TLS is terminated at frodomar.direct.acidiclight.dev, by the reverse proxy. Traffic over a Wireguard tunnel is already encrypted.

Next, because I’m running GitLab EE inside docker-compose, I need to restart the container with the new port exposed on the host. My VPS can now hit Container Registry over Wireguard on port 8969.

Finally, on the VPS running the reverse proxy, it’s just a matter of doing the usual reverse proxy configuration + Let’s Encrypt dance to point cr.acidiclight.dev to gitlab:8969, and boom!

Setting things up on my end

With the help of my friend Zaprit, who actually does this shit for a living, I actually created the two custom containers I need. You can find them at https://gitlab.acidiclight.dev/docker-images/.

So - write the Dockerfile, build the image, tag it, and upload it - right?

Yes, but you missed a step. First we need to log in, which involves creating a GitLab personal access token. Because of how accounts work on my instance, this is something you must do anyway to use Git over HTTPS.

sudo docker login cr.acidiclight.dev
Username: ritchie
Password: Nice try.

After that, it genuinely is as simple as build it, tag it, push it. …We won’t talk about how Cloudflare was rejecting uploads until I turned Cloudflare proxying off. Never mind about that.

With Debian, it’s never that smooth.

And I wanna whine about it.

Here’s how you install Wine properly on Debian.

sudo dpkg --add-architecture i386
sudo apt update 
sudo apt install wine32 wine64

You would think you could then set up a 64-bit Wine prefix like so.

export WINEARCH=win64
export WINEPREFIX=/winemonogame
wine64 wineboot

No. Because this is Debian, and wine64 isn’t a command. Well, it is, it’s just not in the PATH. However, wine is - and this’ll at least appear to work. It’ll set up the prefix.

Socially Distant’s build tools, namely dotnet-sdcb, expect wine64 to be a command though. That’s because that’s how Arch packaged it, and I didn’t even write the scripts - MonoGame did - so I’d wager most other distros package Wine this way as well.

So, hastefully update the container to do a lil ln -s /usr/bin/wine /usr/bin/wine64 and be done with it - right?

Nope.

Debian maintainers have out-smarted you and me.

I looked at the contents of /usr/bin/wine, and noticed that it was a shell script. It was trying to determine whether you are running Wine64 or Wine32, to choose which one to use. Except what it ACTUALLY does is check if Wine32 exists, and if so, uses it. Always. Even if Wine64 is present and that’s the one you actually want. It doesn’t care about the WINEARCH variable either. The mere presence of Wine32 means 64-bit CPUs suddenly do not exist. This caused CI build failures with blank error messages during shader compilation. I swear to God.

You can even verify this:

wine cmd
echo %PROCESSOR_ARCHITECTURE%

If it outputs x86, even though you’re positive you set up a 64-bit prefix, you can go give a Debian maintainer a hug.

FYI, Wine64 is at /usr/lib/wine/wine64. It’s not a library, it’s a program, you can symlink /usr/bin/wine64 to it and bypass the weird shell script. And everything works. Which makes me question why it was packaged this way?

So that was fun.

I even hopped up my DCO check to handle repository maintainers properly, and automated Linux builds of Socially Distant are now available! <3

GitLab Pages is next.