Tech

When Fedora make some bad design decisions, people suffer.

Background

If you've used Linux at all in the past 2 decades, chances are you might have heard about systemd at least once.

In a nutshell, systemd is a component that's meant to handle background services on a Linux box. It offers a centralized means of configuring, controlling and sand-boxing services, timers and other very useful resources that administrators of a Linux system might be interested in.

Normally, one would think that this is only necessary on servers, since that's where the bulk of background services would go such as databases, web servers, torrent servers and so on.

And it's true, background services (also known as daemons), are very useful on servers but they aren't used just there. They are also heavily used on desktop Linux as well, on distros such as Ubuntu, Fedora and many others.

In fact, systemd is the norm nowadays. One would have to go out of their way to find a distro that doesn't include it out of the box.

Examples of distros that don't contain systemd out of the box are OpenWrt, LibreCMC and DD-WRT (which are mainly used on router devices), GrapheneOS, Replicant and LineageOS (which are Android custom ROMs), Chrome OS and Chromium OS, Artix Linux, Puppy Linux, Hyperbola GNU/Linux-libre, Slackware and others.

The reasons for using or not using systemd are mainly philosophical in nature, that being the argument on whether systemd follows the Unix philosophy of “Do just one job, but do it well”. This philosophy is very common among Unix-like system administrators because it is both intuitive and has generally meant that software is written with less bugs when it is followed from the development phase all the way to the end.

systemd breaks this philosophy because it doesn't do just one job, it does multiple jobs: it's both a daemon monitor (allowing the administrator to monitor the status of each and every daemon, along with searching for them), a daemon manager (since it allows you to start, stop, restart, enable and disable daemons on an individual basis), configurator (since it allows you to configure both daemons and daemon dependencies), it determines the order in which daemons need to run in order to satisfy all of their dependencies, it also monitors timers and allows for timers to trigger daemons and so on.

All of these tasks, unsurprisingly, make systemd a very complex piece of software which made many people hesitant in introducing it into their distributions. However, it is undeniable that systemd is very useful. It does a lot of things but, once you learn how to use it, it is indeed a very powerful tool.

Even on a desktop computer that's not used as a server, systemd allows administrators fine grained control over the system to such an extent that confers a lot of power.

For example, on Fedora KDE Plasma, PipeWire, which is the system service that handles all things audio, is controlled by systemd.

There were times when, soon after being introduced in Fedora, that PipeWire would stop working. And by saying this I mean, you would hear no audio on your system while playing YouTube videos, even though the volume would be cranked to the max.

And so I would have to go my terminal to do a systemctl --user restart pipewire pipewire-pulse wireplumber and voila! Everything would work once again. Neat, right?

Or how about power management? If you also use Fedora Linux and want to enter sleep mode from a script, you'd have to do systemctl suspend as root and you would achieve just that.

I'm not sure why you have to use systemd to use this but hey, it works.

The issue

Now we get to the sad state of affairs. Keep in mind that this story is old, very old.

This probably happened around 2018 or 2019, which was around the days of Fedora 30. By now, what I'm about to describe has likely been fixed.

But, long story short, systemd ended up having a serious issue with some daemon that I had installed on my Fedora Linux system. Or, maybe it wasn't a daemon but just a regular system process that would run in the background. Honestly, I don't know which it is.

Again, this was during the Fedora 30 days, back when I used GNOME Shell.

I'll try to be as explicit as possible with the details but, sadly, even I don't know much about what happened.

Basically, one day, I noticed something peculiar about my system. I would use Fedora on my laptop which was an Asus Zenbook that I had back then and, when I decided to shutdown or restart my laptop, I noticed that it would hang.

It wouldn't hang for very long, mind you, just about 1 minute.

In fact, it was exactly 1 minute.

Strange, right?

Well, a neat thing about Fedora is that, during the shutdown splash screen, if you press on the escape key, you get the full system log (you can also do this during the boot splash screen as well) if you're curious. On this screen, I saw that Fedora was reporting that some daemon or process was running in the background and hanging the full system, preventing it from shutting down in a timely manner.

Fedora had a max timeout configured to 1 full minute back then so that, in case any process still hadn't shut down after 1 minute since the moment I pressed the Shut down button and sent the signal to all the running processes on the system to prepare for winding down, the system would effectively timeout and kill them so as to not wait indefinitely for a process that was likely stuck in some kind of infinite loop.

Now, you might say something along the lines of “So what? You had to wait 1 minute to shutdown your laptop. Boo hoo, cry me a river. Are you really that impatient?” and that's what I thought too.

It wasn't a big deal, right? Just 1 minute once.

Well, as it turns out, it wasn't just that one time. It happened the next time I tried to shut down too. And then the next. And the next. And every time I wanted to restart as well. Every single time I wanted to shutdown or restart, I'd have to wait 1 minute.

Have a kernel update and want to test it out? Wait 1 minute. Changed a configuration file in /etc and wanted to verify that it worked? Wait 1 minute. Had some kind of system breaking bug and needed to restart to return to a sane system state? You guessed it, wait 1 minute.

After a while, it started to grind on me. Usually I don't mind small inconveniences since, as always, they tend to happen a lot on Linux.

But this was starting to annoy me more than it had to.

And the thing is, Linux was being praised at the time for things that Windows lacked: modularity, transparency and, most of all, efficiency. Linux was supposed to boot faster than Windows and shut down in an instant while Windows would lag behind, doing a lot of unnecessary cleanup and move at the pace of a snail.

And yet here I was, on Linux, having to wait at every restart 1 minute for no discernible reason, feeling like my laptop was taking a coffee break for itself.

Naturally, this didn't sit well with me.

And so I did the only sane thing any system admin would do and started to investigate.

“So” I naively said to myself. “A daemon is preventing the system from shutting down in time. The solution, then, would have to be that I kill that daemon before I shut down so that it doesn't get the chance to hang anything anymore”.

So yeah, just like that, the problem would be solved. And I knew enough about systemd to know that just opening a terminal and writing the magical sudo systemctl stop <daemon-name> would be enough to solve this problem once and for all. If the daemon in question was not doing any useful work at all, I might even consider uninstalling it entirely.

Mind you, I was assuming that the culprit was a systemd daemon but this could very well have been false. Like I said, this could have been just a regular background process, not necessarily managed by systemd. I just want to make it clear that, despite my introduction of systemd in this blog post, it could very well not have been the issue here.

“Simple!” I would say to myself.

Well, as many things in Linux, simplicity is just a theoretical concept that only exists in the eyes of the naive beginner. For any Linux admin veteran, there is no such thing as simplicity.

The journey I embarked on, as easy as it promised to be, would turn into a rabbit hole that I would have to dig through with the equivalent of a spoon in my hand, because the shovel store ran out of stock.

How does one find the culprit? How does one find out which daemon is responsible for delaying the system at every shutdown or restart? Well, back then at least, this was no easy feat.

Because, for whatever reason that to this day I cannot fathom, systemd did not keep logs over which daemon is responsible for system hangs; nor did Fedora as a whole. You just had to kind of guess which process was the culprit.

No, I wish I was kidding. I could not believe this when I read it on some Linux tech forum back in the day, either.

And I wasn't the only one with this problem back then. I stumbled upon forums with people complaining about the same thing: how systemd would prevent their systems from shutting down in a timely manner, all because of some mysterious daemon that went rogue and would simply refuse to stop running. They had to wait until Fedora's configured timeout elapsed for the system to decide “OK, I waited long enough. Time to pull the plug” and finally do the deed.

And trust me, I searched far and wide for my answer.

Nobody knew for an easy way to identify the culprit. The only way people would figure things out is by listing all the possible daemons running on the faulty systems and trying to guess which of them was the culprit, stop them manually, attempt the shutdown and see if the computer would wait for a minute again or if it would shutdown instantly this time. If it wouldn't wait then congratulations, you've found the bad daemon. If not, take another guess.

That's it. That's the brilliant solution that people had to rely on.

And, of course, nobody was happy with this.

I vividly remember, to this day, how one forum user (not me), after many back and forths with some forum admin there about possible strategies he would have to take, eventually had a break down and began hurling insults at systemd, how it was a terribly designed piece of garbage, how he would wonder how anyone ever designed such a badly thought out piece of software with no useful logs, how everything made no sense and how he hated it all.

It was a long blog post in which he had a full meltdown. And you know what? I sympathized with him.

Maybe it wasn't systemd's fault, maybe it was Fedora's. Maybe the Fedora developers should have been the ones to log such important things. Or I don't know, someone in the chain had to have been clever enough to say “Maybe this is an important bit to log”. But, for whatever reason, they hadn't done that.

And now, because of them, so many admins lost hours upon hours on forums, trying to figure out a way to fix this madness.

And, after 3 excruciating days of digging through forums, I gave up. I said to myself “this isn't worth it” and I just stopped caring.

One of the many skills that seasoned Linux administrators eventually learn sooner or later is that, at some point, you have to stop going through the rabbit hole. Whatever system issue you're facing, it's simply not worth your time after a certain point.

My limit is usually 3 days. If I don't figure out a solution to my problem in 3 days of searching, investigating and posting, I simply say “It's not worth it anymore”.

There weren't many times in my life when I had to employ this skill thankfully but this was one of the few times when I had to since it really seemed like there was no end in sight.

The conclusion

After my futile attempts at trying to resolve the issue, I decided to move on with my life.

Waiting for 1 minute after issuing a shutdown became normal for me. It was simply the cost of using Linux, I thought.

Soon afterwards, a new version of Fedora Linux came out, the next one and I did a full system upgrade as soon as I could.

The sad thing about it is that, this next version bumped up the shutdown timeout from 1 minute to 1 minute and 30 seconds, prolonging my suffering, almost like a spit in my face.

I can't recall which exact version of Fedora introduced this but, if anyone knows, then they will know at which point in time this issue had occurred to me, since it was between those two versions when this problem was taking place.

So, anyways, I thought that this will only become more painful, since they bumped up the timeout. I was already bracing myself for an even rougher future ahead but, I'm happy to say, something, somehow, somewhere, decided to give me a merciful conclusion.

Some developer pushed an update to some piece of software that I had installed on my system, the process in question stopped hanging on shutdown.

Why? How?

I don't know. I really don't know. I don't know what happened, I don't know who solved the issue or what daemon or background process was the culprit to begin with.

I never found out for myself.

But it just goes to show that even the most difficult problems in system administration can go away, if you simply wait long enough.

Time is the healer of all necessary evils – Menander, 300 B.C. (according to this)

Blog post by Alexandru Pentilescu.

You may contact me at alexandru.pentilescu@disroot.org

Optionally, you may also encrypt your emails to me using the following PGP key: 0xFF49E5748BD42A6A6A7DECFDD38B28DF9F7497A2

Download that key from any keyserver you wish