Introduction

I am using a Raspberry Pi 3 Model B+ as my media center. Libreelec, Kodi and HDMI-CEC (using the remote of the TV to navigate through the content) are very nice, and I’m even planning to buy a DVD-T adapter so that I won’t have to use the TV as a source but always stay in Kodi.

What was bothering me was a high CPU load when Kodi was supposed to be idling. The 15min CPU load average was hovering at around 0.15 when it should have been around 0.00 – as no media was played, no Weather/RSS updates were active, so was causing this Kodi CPU load ? It turned out to be the way the PlexKodiConnect plugin integrates with Kodi and works around Kodi limitations. But this is not immediately obvious on why.

strace and gdb were extremely helpful and this post is mostly a log of the debugging session. I was able to identify a bug in PKC and also delve deep into decompiling and modifying live code.

strace-ing the Kodi process

Before we start: strace is not provided on current libreelec distribution (version 8.2.4), only the GNU debugger is – so you have to statically cross-compile your own for armv7-a 32bit architecture (for example, using a buildroot environment).

1. Obtaining the PID:

2. A simple strace shows a LOT of noise (partially because the system is not on 64bit and the clock_gettime is not hidden from strace by Linux vDSO). Anyway, trying to cleanup the output there are many grep excludes and one thing caught my eye, a read from file descriptor 4 that happened more than once a second:

Investigating /proc/374/fd/4 surprise surprise, Kodi is reading /proc/stat very often.

Why would Kodi do that ? I needed to get the stack trace upon the reads and then to check the source code to find out. Enter gdb.

gdb – getting the stack trace (registry, breakpoints)

There were few challenges with gdb since I have never used it in this scenario:

  1. Can it break onto syscall ?
  2. If so, how can it break into a specific syscall (where first argument is 4) ?
  3. Can it write memory (so that I can NOP the instruction) ?
  4. Can it write specific memory from a batch script (so that I don’t do the manual steps every time Kodi starts) ?

Google answered ‘yes’ to all these questions.

One thing to mention, to find out how the names of the registers we need to use the info reg command – but for that, gdb needs to be already at a breakpoint. So I’ll skip ahead and show the output (luckily, it seemed to have stopped at a read from the file descriptor 4 (found in register r0)).

The summary of the gdb commands is below:

  1. gdb -p 374 – attach to Kodi pid.
  2. break read if $r0 == 4 – break point on the read syscall if the file descriptor (r0 register on Cortex-A53 (ARMv8) of RPI3B+)
  3. c – is to continue (gdb will start paying attention to the process)
  4. bt – to get the stack trace.

Let’s see them in action:

Cool. It seems that CApplication::Process() calls to CCPUInfo::getUsedPercentage(). Checking github of Application.cpp, I found this code:

and indeed within CCPUInfo.cpp there is a call to readProcStat():

The trouble I see is see is that the result value of getUsedPercentage() is not used. I would understand to call it if the Debug is on (when the UI is supposed to show the CPU load) but why during Application main loop ? It does seem to update the values internally, but at a quick glance with Github search (without having the code locally) I was not able to find who uses them (beside a test and GUIWindowDebugInfo.cpp, but that is also calling the getUsedPercentage() just at the beginning of the CGUIWindowDebugInfo::Process method, which makes the Application call useless).

Anyway, I wanted the comment out the call, but how, without recompiling ? It seemed to have no configuration option, and recompiling Kodi … so let’s go low level.

Disassembly

If I can identify the assembly call to CCPUInfo::getUsedPercentage() within CApplication::Process() it will be easier just to NOP the instruction. The address of CApplication::Process() is 0x006fa1e4 according to the listing above. gdb allows disassembling any memory location using the disassemble [address] command:

And scrolling few pages through the code, it is immediately visible … the branch instruction 0x006fa1e0 <+332>: bl 0x570938 <_ZN8CCPUInfo17getUsedPercentageEv>  which is the actual call to the getUsedPercentage(). Replacing it with NOP provided a challenge, what is the instruction code for NOP ?! After some searching, it seems that the Cortex CPU has a NOP instruction code of 0xE320F000.

gdb also allows to write any address of the memory using the set [address] = [val] so let’s replace the bl with a nop, effectively commenting out the call to read/parse /proc/stat:

Permanent fix

A nice feature of gdb allows executing command scripts in batch mode. It also seems that the address of the code is not relocatable, and once Kodi is up, the jump to getUsedPercentage() is always at the same position, 0x006fa1e0. By creating a small file with the set command, gdb can execute it after Kodi starts:

And then the .config/autostart.sh has to be (created or) edited to have gdb applying the fix:

Observations

  1. gdb breakpoints is causing Kodi to restart.
  2. Kodi does so many clock_gettime calls that trigger kernel context switches. Linux vDSO comes to mind.
  3. no stability issues encountered disabling the bl instruction at boot. Debug mode works correctly once activated.

As a side node: you can actually manipulate the file descriptors and make Kodi read /dev/zero when accessing file descriptor 4, but still, it feels like CPU cycles will still be wasted trying to do a syscall just to get noting out of it. Few links describing the approach:

https://groups.google.com/forum/#!topic/alt.hackers/0ZMsMc5DvUw

http://ingvar.blog.redpill-linpro.com/2010/07/10/changing-a-process-file-descriptor-on-the-fly/

http://users.linpro.no/ingvar/fdswap.sh.txt