Let's see how something seemingly completely unrelated to audio can affect someone's listening experience.
Greetings, everyone. It has been a while, huh?
I have been meaning to write this post a couple weeks ago, but I never found the energy to do so (yay for working myself to the bone!).
Today we'll see how I managed to find the source of a very annoying issue with my laptop's audio, the debug process and my extreme surprise at the result.
The problem
Since switching to Pipewire 0.3.x I have been having an extremely annoying issue: every time I was using anything that was based on the WebRTC technology (that includes Jitsi, Google Meet, Discord, Telegram...) my audio would start breaking. That means that I would end up with the last letter of an earlier phrase being repeated, popping, crackling, skipping: the whole circus.
It. was. vexing .
The non-solution
I contacted the people at Freedesktop via their GitLab to see what could be done, because I didn't notice anything like this on the 0.2.x branch of Pipewire, nor on Pulseaudio.
I tried following all the standard troubleshooting steps: deleting any custom configuration, installing rtkit and enabling realtime privileges, nothing worked.
The Pipewire team suggested I increase the minimum allowed quantum, which improved the situation a lot at first, but ended up being host of a whole other set of issues.
Increasing the minimum quantum meant that the programs had more latency, which brought WebRTC programs to break down after 10 or so minutes of call: I couldn't hear anyone after a while.
The debugging
I have been fighting with this issue for almost a year, on and off, with a period in between (lasting about 4 months) where I just gave up and surrender to my crackly future.
Until about 3 weeks ago, when I became "maximum over-annoyed" at this issue and decided to contact the Pipewire team again: I got suggested to try different hardware and/or a different distro. It was going to be either my system, or my laptop's hardware.
I tried to come up with the most easily reproducible test I could, which was essentially playing a song via MPV while keeping
pavucontrol
opened. That showed the "ERR" column in
pw-top
rising quite often along with the crackling.
I tried using different hardware (with a small mixer used as external soundcard) to no avail, then I tried using Ubuntu Live from a USB stick and realized there was no crackling.
My hardware was ok, that left my distro, or my configuration.
At that point my work colleague throws an idea on the field: close everything that is not useful for audio and the system to "survive" and see what happens.
I turned off everything : services, daemons, panels... I was left with only me, Pipewire and my window manager (openbox). No crackling.
That was my lead.
The cause
While still in a call with my colleague (using my mobile phone), he complained about his Conky misbehaving. That reminded me to turn Conky back on (among other things, for testing) and suddenly... the crackling is back.
I kill Conky, the crackling goes away, I restart it and the crackling comes back.
We have found the cause of the crackling, but why though? Why would a system monitor affect the audio so much?
Diving deeper
I decided to restart my laptop to go back to a more usable machine, killed Conky and made sure that the audio was indeed working as it should.
It was time to understand what was going on.
I tried changing some basic settings that could have put my laptop in a bit of a bind, like transparency or double-buffering, without result. I changed the interval used by Conky to update itself from every 2 seconds to every second. The crackling got a lot worse .
Conky was executing a command that was causing the audio issue. It was time for a good old "binary search": I commented half of my Conky configuration and checked.
Commenting the first half was not enough for the problem to disappear, so I restored it and commented the second half. The crackling problem went away.
My Conky config was small enough that from there on, I could just comment each command separately. Until I got to
${i8k_cpu_temp}
. My Conky configuration checking the CPU temperature was causing the audio to break.
To make sure of it, I closed Conky and checked its manual, where it says that the
${i8k_cpu_temp}
command reads the temperature from
/proc/i8k
. I decided to try it myself using
cat /proc/i8k
while playing music: the audio was breaking every time I triggered that command.
I found the culprit.
The reason
But why checking the CPU temperature would break my audio? The answer was hypothesized by the Pipewire team when I brought them my findings and subsequently confirmed by myself when I read what i8k is (from https://github.com/lochotzke/i8k ):
Linux driver for accessing the SMM BIOS on Dell Laptops
SMM stands for "System Management Mode" (sometimes known as "Ring -2"), which is a mode where all normal execution including the OS is suspended while a software in the computer's firmware is executed.
That means that every time my Conky checked for the CPU temperature (every 2 seconds), I was freezing my entire OS for just the time needed to get the temperature.
I was floored.
That is what I get for using a command that was supposed to get my CPU temperature without checking for any side effects, back when I changed my laptop.
After all of this kerfuffle I even realized the temperature reported was wrong anyway!
The aftermath
I posted my solution on the Pipewire GitLab, as well as my Linux distro's forums (where it seems to have solved someone else's audio issues too!), thanking everyone for their patience and helpful input.
Now my audio is working great and I can enjoy music without issues, even when I have a WebRTC call open waiting for people to join.
Hopefully this can be helpful to some other people who ran into a similar issue.
Thank you everyone for reading.
Penaz.