What started off as a simple wish to install the Raspberry Pi camera plugin for GStreamer1 on Buildroot, became a week long saga of infinite trials and errors. Below are the conclusions:

When trying to load the plugin, for example via the command:

gst-inspect-1.0 rpicamsrc

The load fails with a series of errors such as this:

/usr/libexec/gstreamer-1.0/gst-plugin-scanner: symbol 'mmal_port_pool_create': can't resolve symbol

At first these were symbols that were actually missing from the libgstrpicamsrc.so library, and for example adding the linker flag -lmmal_component would solve the problem. But the issue became vastly more complex when the unresolved symbols actually belonged to libmmal_util.so A file that is already linked to the library!

These are the two best readings I have found for the subject:

• Better understanding Linux secondary dependencies solving with examples
• Your worst enemy: undefined symbols.

The first step was to figure out at what stage the undefined symbols are observed. The symbol mmal_port_pool_create is owned by libmmal_util.so. This is found using:

./host/bin/arm-buildroot-linux-uclibcgnueabihf-nm -D target/usr/lib/libmmal_util.so | grep port_pool_create

• The ARM version of nm is used, even though it's not necessary for this output.
• The -D parameter is used, since this is a dynamic shared library.

The output:

000041bc T mmal_port_pool_create 

The T symbol means this is a globally defined symbol that is available to outside links.

The second step was to figure out what libraries are using those symbols. This command could help:

nm -D target/usr/lib/libmmal*.so

And searching for port_pool_create, we find this only under libmmal_core.so:

U mmal_port_pool_create

The U symbol (undefined) means the library is using this symbol from another library. Further examination shows that these libraries are co-dependent. mmal_core is using symbols from mmal_util and mmal_util is using symbols from mmal_core.

Normally it should be enough that our own libgstrpicamsrc.so is loading libmmal_core.so and libmmal_util.so for everything to work. We can verify this in 3 different ways:

readelf -d target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so

Or

objdump -x target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so

Would both show this line:

 0x00000001 (NEEDED)                     Shared library: [libmmal_core.so]
 0x00000001 (NEEDED)                     Shared library: [libmmal_util.so]

We can also see that libmmal_core.so is requested before libmmal_util.so. This is actually due to linkage order when -lmmal_core appears before -lmmal_util We can also run dependency analysis with the ldd tool:

LD_LIBRARY_PATH=./target/usr/lib ./host/bin/arm-buildroot-linux-uclibcgnueabihf-ldd target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so

And find the dependent libraries.

	libmmal_core.so => ./target/usr/lib/libmmal_core.so (0x00000000)
	libmmal_util.so => ./target/usr/lib/libmmal_util.so (0x00000000)

So if these libraries are supposed to be loaded, why aren't they? I started looking at the next level of dependency, could it be that libmmal_core doesn't mention needing libmmal_util? Running ldd for libmmal_util shows that indeed libmmal_util is not NEEDED. This flag called DT_NEEDED should appear when this library is linked, meaning when building the raspberry pi userland. Sometimes this is caused by a more recent compiler optimization called as-needed. There is more information about this at link in the reading section, but basically it means that the compiler/linker drops the NEEDED requirement from a linked library if it doesn't find that it is actually being used. This flag can be overridden using a counter flag called no-as-needed, and this was actually a problem at some point regarding rpi-userland here, but this is a decoy! This is not the issue at hand: Checking the CMake file here confirms that mmal_util is not present there at all:

target_link_libraries (mmal_core vcos)

But then how come this library works at all? Apparently (and I also confirmed this with my own addhoc test) it is fine to not declare this dependency as long as the parent library/executable would link against both co-depending libraries. So why doesn't it work now?

Apparently this is related to the fact buildroot is using a different dynamic linker from a different base library. Instead of the standard linux libc, it uses the uclibc-ng micro library.

After some investigation, I found out how to debug the library loading process. First support for LD_DEBUG must be enabled in buildroot. In the defconfig:

BR2_UCLIBC_CONFIG="softbot-uClibc-ng.config"

and in that custom file:

SUPPORT_LD_DEBUG=y

Now on the Pi, before running the command, I can run export LD_DEBUG=all or usually just export LD_DEBUG= is enough data. The results are here. The dynamic loading process goes through (I think) two stages. First, all of mentioned libraries are loaded and dependencies are checked according to symbols. There I noticed something strange. This line appeared for every library, including mmal_core and mmal_util:

do_dlopen():438: Circular dependency, skipping '/usr/lib//libvcos.so'

But I don't think there really is a circular dependency there! This is very strange.

At the next stage it seems the DT_NEEDED headers are examined. And of course the ones we need are missing:

lib: /usr/lib//libmmal_core.so has deps:
 /lib//libc.so.0  /usr/lib//libvcos.so 
lib: /usr/lib//libmmal_util.so has deps:
 /lib//libc.so.0  /usr/lib//libvcos.so 

I believe that they should have been loaded in the previous step and I am trying to find out what happened at #uclibc-ng on freenode.

Without knowing what is wrong with uclibc-ng, the first attempt at fix was the use a utility called patchelf to add the missing DT_NEEDED header:

(post-image.sh script)
patchelf --add-needed libmmal_core.so ${TARGET_DIR}/usr/lib/libmmal_util.so
patchelf --add-needed libmmal_util.so ${TARGET_DIR}/usr/lib/libmmal_core.so

But apparently that tool is not very stable, and using it corrupted the entire file.

OK, let's try to fork rpi userland and compile is so it does have DT_NEEDED headers where we need them. Trying to modify the commands as such:

(libmmal_core CMakelists.txt)
target_link_libraries (mmal_core vcos mmal_util)
(libmmal_util CMakelists.txt)
target_link_libraries (mmal_util vcos mmal_core)

Actually doesn't work, because CMake complains on the circular dependency:

CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
  "mmal_core" of type SHARED_LIBRARY
    depends on "mmal_util" (weak)
  "mmal_util" of type SHARED_LIBRARY
    depends on "mmal_core" (weak)
At least one of these targets is not a STATIC_LIBRARY.  Cyclic dependencies are allowed only among static libraries

Ok.. but why? We only want the headers that tell compiler to load both of these libraries, the dynamic linker should handle the cyclic dependency!

Apparently there is a way around this that I found pretty much by guessing. You can tell CMake to add the DT_NEEDED flag without actually adding the library to its precious dependncy graph like so:

(libmmal_core)
target_link_libraries (mmal_core vcos -lmmal_util)
(libmmal_util)
target_link_libraries (mmal_util vcos -lmmal_core)

Recompiling userland this way and then try to run gst-inspect actually works! Congratulations.