Bitrot in a Devops World

Update: One month later - 2022-09-10 - I rebuilt the container again and again had build breaks, completely different ones, but again involving brew, also docker, (though not llvm). Too tedious to go in to. But it took another entire frustrating night. Can’t wait for October’s reprise…

Original post:

So I’m using gitpod.io to easily do development in the cloud - they spin up a Linux environment for you that comes equipped for most programming tasks, or, you can heavily customize it. You can use a YAML file (.gitpod.yml) to configure your project - install stuff, start it running, etc., and then they’ll prebuild your environment, so that when you spin up your workspace and connect to it with Visual Studio Code (or a JetBrains IDE) you’re ready to go. Or you can really go wild by basing things on a custom Docker container of your own. It works great!

Except when it doesn’t.

Not their fault. Everything is subject to bitrot, even Devops, which is supposed to make your life easier, until it doesn’t.

So I have a custom Docker container I build to base my Gitpod.io workspaces on. It contains a lot of tools that take a long time to build but don’t change often. So I have a project that contains both my custom Dockerfile and my gitpod.yml. My Dockerfile is based on one of the Gitpod.io-provided base containers.

Today I noticed I haven’t rebuilt my custom container for awhile. My tools haven’t changed¹ but Gitpod.io updates its base containers from time to time to include new features of their own as well as newer software versions they’re including in the initial setup. So I fire up my Gitlab pipeline and go do something else for 30 minutes, no problems expected.

Except I get an email saying my pipeline died.

Can’t install llvm-15 (this would be via apt get install on a Ubuntu 20.04 image). WTH? This Dockerfile hasn’t changed, worked a month ago.

Trying things out manually on a plain vanilla Ubuntu 20.04 I find … it doesn’t know from llvm-15. Or clang-15. Etc. (After properly setting up apt with the LLVM package repository, of course.) After quite a bit of investigation I find … nothing really. No blog post about it anywhere that DuckDuckGo knows about, etc.

The instructions at https://apt.llvm.org say nothing particular changed about LLVM 15 - the instructions still point to it.

But eventually I discover this on the front page of llvm.org:

So … my best guess is that while version 15 is in release-prep it somehow isn’t available via the documented method anymore. Trying to confirm this I looked at recent LLVM news - nothing. (Their blog hasn’t been updated since the beginning of the year - I’m good with that, just look at the dates on my blog…)

Ok, I guess I don’t really need to be bleeding edge (though I was using some very recently implemented C++ features, but I can just skip that for now), so I get past this by switching to the documented “automatic installation script” approach - its further up on the page so it must be their preferred approach anyway.

And that installs a bunch of stuff. So, fine. Now I’m good.

Except simple commands like clang and clang-tidy don’t find their executables. Which means the build scripts don’t work since they’re expecting to find clang and clang-tidy and so on.

So, turns out /usr/bin is full of things like clang-14 and clang-tidy-14 and lldb-14 but doesn’t have any of the usual symlinks like clang -> clang-14 or clang-tidy -> clang-tidy-14. They’re nowhere to be found.

Guess the apt packages installs those symlinks but the nice “for convenience” automatic installation script doesn’t. And there isn’t any script laying around after the “automatic installation” script that does either.

So I have to do it manually. I use emacs and some quick keyboard macros to cons up a script from the ls -l listing.

Fine.

Ok, now, I’m good to go. Now I install brew, the same way I’ve always done it, so I can install a couple of other tools I want.

You install brew by sucking their install script off their site and running it. That’s it. No fuss, no muss. It just works, which of course is what you want from a package manager.

So I grab their script and start it running and step away for a few minutes while it installs brew. When I return I can use it.

Except when I return it hasn’t started the install.

Now the install script is prompting me to hit ENTER or RETURN to continue and it is sitting there waiting. Why? Because it wants me to know it is going to create directories, or something, before it starts.

Never did that before. I didn’t care before either: I just assumed that since I was going to install some software some files and, yes, some directories, might need to be created.

Why is it doing it now?

Don’t know, it isn’t in the release notes, and git history shows it has apparently always had a NONINTERACTIVE environment variable to control it, so what changed? Don’t care at this point. I’m installing brew with a NONINTERACTIVE=1 flag now. Moving on …

The next step is to install ccache and bear. And my Dockerfile already knows how to do that too:

brew install ccache && brew install bear

Couldn’t be simpler, so I do it.

And … it can’t install ccache. Tells me why too: There’s no “bottle” for ccache.

Now, I have two gripes here:

1. I have no idea what the hell a “bottle” is to brew. I already know I can’t stand brew because the developers there decided it was real cute to invent an entirely new vocabulary for their stupid package manager. Instead of (binary) packages and scripts and so on they’re all about beer-brewing related terms: “tap” and “formula” and “cask” and now I’m supposed to know what a f—ng “bottle” is too.
2. I wasn’t getting this error before.

Now, helpfully, it does tell me what I need to do: I need to run instead brew install --build-from-source ccache instead. Now I know for a fact that ever since I’ve been building this container from this Dockerfile it has been building ccache from source, no problem. Just now I have to tell it explicitly.

I do that command and it works. Ok, but why? Well, I don’t know. I looked but there’s nothing in any recent news or release notes from brew that mentions this at all.

Fine. ccache is there. Now for bear and, yes, turns out there’s no “bottle” for it (whatever that is) so I do brew install --build-from-source bear and it works.

No. It doesn’t. It stops because there’s no “bottle” for abseil. That’s a C++ library. One of the dependencies of bear I guess. I’ve got to build that from source too, so I do, and now the install of bear works.

No. It doesn’t. It stops because there’s no “bottle” for grpc. So apparently I have to discover the dependencies for bear one at a time.²

Fortunately grpc is the last such dependency.

And now, finally, I am done. My Dockerfile builds. I’m ready to go.

And an entire night has been burnt on this.

And someday, perhaps soon, like next month, I’ll have to go through some nonsense like this again. And there still won’t be any release notes or blog posts on the any official site or anything else that tells me to expect a problem, or how to fix it, and that’s what really chaps my hide.