embedded-web-interaction:uclibc-ng-problems []

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== gst1-rpicamsrc and uclibc-ng ======
What started off as a simple wish to install the Raspberry Pi camera plugin for GStreamer1 on Buildroot, became a week long saga of infinite trials and errors. Below are the conclusions:

===== The symptom =====
When trying to load the plugin, for example via the command:
<code>gst-inspect-1.0 rpicamsrc</code>
The load fails with a series of errors such as this:
<code>
/usr/libexec/gstreamer-1.0/gst-plugin-scanner: symbol 'mmal_port_pool_create': can't resolve symbol
</code> 

At first these were symbols that were actually missing from the //libgstrpicamsrc.so// library, and for example adding the linker flag **-lmmal_component** would solve the problem. But the issue became vastly more complex when the unresolved symbols actually belonged to **libmmal_util.so** A file that is already linked to the library! 

==== Background reading ====
These are the two best readings I have found for the subject:
  * [[http://www.kaizou.org/2015/01/linux-libraries/|Better understanding Linux secondary dependencies solving with examples]]
  * [[https://flameeyes.blog/2010/09/01/your-worst-enemy-undefined-symbols/|Your worst enemy: undefined symbols]].

===== Analysis =====
The first step was to figure out at what stage the undefined symbols are observed. The symbol **mmal_port_pool_create** is owned by //libmmal_util.so//. This is found using:
<code>
./host/bin/arm-buildroot-linux-uclibcgnueabihf-nm -D target/usr/lib/libmmal_util.so | grep port_pool_create
</code>
  * The ARM version of //nm// is used, even though it's not necessary for this output. 
  * The -D parameter is used, since this is a dynamic shared library.

The output:
<code>
000041bc T mmal_port_pool_create 
</code>
The **T** symbol means this is a globally defined symbol that is available to outside links.

The second step was to figure out what libraries are using those symbols. This command could help:
<code>
nm -D target/usr/lib/libmmal*.so
</code>
And searching for **port_pool_create**, we find this only under //libmmal_core.so//:
<code>
U mmal_port_pool_create
</code>
The **U** symbol (undefined) means the library is using this symbol from another library.
Further examination shows that these libraries are **co-dependent**. //mmal_core// is using symbols from //mmal_util// and //mmal_util// is using symbols from //mmal_core//.

Normally it should be enough that our own //libgstrpicamsrc.so// is loading //libmmal_core.so// and //libmmal_util.so// for everything to work. We can verify this in 3 different ways:
<code>
readelf -d target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so
</code>
Or
<code>
objdump -x target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so
</code>
Would both show this line:
<code>
 0x00000001 (NEEDED)                     Shared library: [libmmal_core.so]
 0x00000001 (NEEDED)                     Shared library: [libmmal_util.so]
</code>
We can also see that //libmmal_core.so// is requested before //libmmal_util.so//. This is actually due to linkage order when **-lmmal_core** appears before **-lmmal_util**
We can also run dependency analysis with the **ldd** tool:
<code>
LD_LIBRARY_PATH=./target/usr/lib ./host/bin/arm-buildroot-linux-uclibcgnueabihf-ldd target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so
</code>
And find the dependent libraries.
<code>
	libmmal_core.so => ./target/usr/lib/libmmal_core.so (0x00000000)
	libmmal_util.so => ./target/usr/lib/libmmal_util.so (0x00000000)
</code>
So if these libraries are supposed to be loaded, why aren't they? 
I started looking at the next level of dependency, could it be that //libmmal_core// doesn't mention needing //libmmal_util//? Running **ldd** for //libmmal_util// shows that indeed //libmmal_util// is **not NEEDED**. This flag called **DT_NEEDED** should appear when this library is linked, meaning when building the raspberry pi userland. Sometimes this is caused by a more recent compiler optimization called **as-needed**. There is more information about this at link in the reading section, but basically it means that the compiler/linker drops the NEEDED requirement from a linked library if it doesn't find that it is actually being used. This flag can be overridden using a counter flag called **no-as-needed**, and this was actually a problem at some point regarding rpi-userland [[https://github.com/raspberrypi/userland/issues/178|here]], but **this is a decoy!** This is not the issue at hand:  Checking the CMake file [[https://github.com/raspberrypi/userland/blob/master/interface/mmal/core/CMakeLists.txt|here]] confirms that //mmal_util// is not present there at all:
<code>
target_link_libraries (mmal_core vcos)
</code>
But then how come this library works at all? Apparently (and I also confirmed this with my own addhoc test) it is fine to not declare this dependency as long as the parent library/executable would link against both co-depending libraries. So why doesn't it work now?

Apparently this is related to the fact buildroot is using a different dynamic linker from a different base library. Instead of the standard linux libc, it uses the [[https://uclibc-ng.org/|uclibc-ng micro library]].

After some investigation, I found out how to debug the library loading process. First support for **LD_DEBUG** must be enabled in buildroot. In the defconfig:
<code>
BR2_UCLIBC_CONFIG="softbot-uClibc-ng.config"
</code>
and in that custom file:
<code>
SUPPORT_LD_DEBUG=y
</code>
Now on the Pi, before running the command, I can run **export LD_DEBUG=all** or usually just **export LD_DEBUG=** is enough data. The results are {{ :thesis:work-journal:ld_debug.txt |here}}.
The dynamic loading process goes through (I think) two stages. First, all of mentioned libraries are loaded and dependencies are checked according to symbols. There I noticed something strange. This line appeared for every library, including //mmal_core and mmal_util//:
<code>
do_dlopen():438: Circular dependency, skipping '/usr/lib//libvcos.so'
</code>
But I don't think there really is a circular dependency there! This is very strange.

At the next stage it seems the DT_NEEDED headers are examined. And of course the ones we need are missing:
<code>
lib: /usr/lib//libmmal_core.so has deps:
 /lib//libc.so.0  /usr/lib//libvcos.so 
lib: /usr/lib//libmmal_util.so has deps:
 /lib//libc.so.0  /usr/lib//libvcos.so 
</code>
I believe that they should have been loaded in the previous step and I am trying to find out what happened at **#uclibc-ng** on **freenode**.

===== Solution =====
Without knowing what is wrong with uclibc-ng, the first attempt at fix was the use a utility called //patchelf// to add the missing //DT_NEEDED// header:
<code>
(post-image.sh script)
patchelf --add-needed libmmal_core.so ${TARGET_DIR}/usr/lib/libmmal_util.so
patchelf --add-needed libmmal_util.so ${TARGET_DIR}/usr/lib/libmmal_core.so
</code>
But apparently that tool is not very stable, and using it corrupted the entire file.

OK, let's try to fork rpi userland and compile is so it does have //DT_NEEDED// headers where we need them. Trying to modify the commands as such:
<code>
(libmmal_core CMakelists.txt)
target_link_libraries (mmal_core vcos mmal_util)
(libmmal_util CMakelists.txt)
target_link_libraries (mmal_util vcos mmal_core)
</code>
Actually doesn't work, because CMake complains on the circular dependency:
<code>
CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
  "mmal_core" of type SHARED_LIBRARY
    depends on "mmal_util" (weak)
  "mmal_util" of type SHARED_LIBRARY
    depends on "mmal_core" (weak)
At least one of these targets is not a STATIC_LIBRARY.  Cyclic dependencies are allowed only among static libraries
</code>
Ok.. but why? We only want the headers that tell compiler to load both of these libraries, the dynamic linker should handle the cyclic dependency!

Apparently there is a way around this that I found pretty much by guessing. You can tell CMake to add the //DT_NEEDED// flag without actually adding the library to its precious dependncy graph like so:
<code>
(libmmal_core)
target_link_libraries (mmal_core vcos -lmmal_util)
(libmmal_util)
target_link_libraries (mmal_util vcos -lmmal_core)
</code>
Recompiling userland this way and then try to run //gst-inspect// **actually works!**
Congratulations.
User Tools

Site Tools

Page Tools
