Differences

This shows you the differences between two versions of the page.

--- thesis:work-journal [2019/02/27 14:52]
avnerus
+++ thesis:work-journal [2019/04/13 15:25] (current)
avnerus
@@ Line 121: / Line 121: @@
 I might be able to live with that.
-===== Thu 21/02/2019 =====
+===== Fri 21/02/2019 =====
   * Ordered some new parts (Valves and tube connectors) - Made a new page to remember the [[soft-robotics:finnish-terminology|Finnish terminology]].
   * With my new resistors, connected two more pressure sensors. Now I have two neck muscles and two arms connected and working.
   * Checked the arm sensors and they are still fine.
-===== Wed 27/02/2019 =====
+===== Thu 27/02/2019 - End of the gst1-rpicamsrc Saga =====
-Long time since the last update.
+What started off as a simple wish to install the Raspberry Pi camera plugin for GStreamer1 on Buildroot, became a week long saga of infinite trials and errors. Below are the conclusions:
-==== GST-RpiCamSrc Linkage issues ====
+==== The symptom ====
-For now just keeping track of links:
+When trying to load the plugin, for example via the command:
+<code>gst-inspect-1.0 rpicamsrc</code>
+The load fails with a series of errors such as this:
+<code>
+/usr/libexec/gstreamer-1.0/gst-plugin-scanner: symbol 'mmal_port_pool_create': can't resolve symbol
+</code>
+At first these were symbols that were actually missing from the //libgstrpicamsrc.so// library, and for example adding the linker flag **-lmmal_component** would solve the problem. But the issue became vastly more complex when the unresolved symbols actually belonged to **libmmal_util.so** A file that is already linked to the library!
+==== Background reading ====
+These are the two best readings I have found for the subject:
   * [[http://www.kaizou.org/2015/01/linux-libraries/|Better understanding Linux secondary dependencies solving with examples]]
-  * [[https://flameeyes.blog/2010/09/01/your-worst-enemy-undefined-symbols/|Your worst enemy: undefined symbols]]
+  * [[https://flameeyes.blog/2010/09/01/your-worst-enemy-undefined-symbols/|Your worst enemy: undefined symbols]].
+==== Analysis ====
+The first step was to figure out at what stage the undefined symbols are observed. The symbol **mmal_port_pool_create** is owned by //libmmal_util.so//. This is found using:
+<code>
+./host/bin/arm-buildroot-linux-uclibcgnueabihf-nm -D target/usr/lib/libmmal_util.so | grep port_pool_create
+</code>
+  * The ARM version of //nm// is used, even though it's not necessary for this output.
+  * The -D parameter is used, since this is a dynamic shared library.
+The output:
+<code>
+bc T mmal_port_pool_create
+</code>
+The **T** symbol means this is a globally defined symbol that is available to outside links.
+The second step was to figure out what libraries are using those symbols. This command could help:
+<code>
+nm -D target/usr/lib/libmmal*.so
+</code>
+And searching for **port_pool_create**, we find this only under //libmmal_core.so//:
+<code>
+U mmal_port_pool_create
+</code>
+The **U** symbol (undefined) means the library is using this symbol from another library.
+Further examination shows that these libraries are **co-dependent**. //mmal_core// is using symbols from //mmal_util// and //mmal_util// is using symbols from //mmal_core//.
+Normally it should be enough that our own //libgstrpicamsrc.so// is loading //libmmal_core.so// and //libmmal_util.so// for everything to work. We can verify this in 3 different ways:
+<code>
+readelf -d target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so
+</code>
+Or
+<code>
+objdump -x target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so
+</code>
+Would both show this line:
+<code>
+x00000001 (NEEDED)                     Shared library: [libmmal_core.so]
+x00000001 (NEEDED)                     Shared library: [libmmal_util.so]
+</code>
+We can also see that //libmmal_core.so// is requested before //libmmal_util.so//. This is actually due to linkage order when **-lmmal_core** appears before **-lmmal_util**
+We can also run dependency analysis with the **ldd** tool:
+<code>
+LD_LIBRARY_PATH=./target/usr/lib ./host/bin/arm-buildroot-linux-uclibcgnueabihf-ldd target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so
+</code>
+And find the dependent libraries.
+<code>
+	libmmal_core.so => ./target/usr/lib/libmmal_core.so (0x00000000)
+	libmmal_util.so => ./target/usr/lib/libmmal_util.so (0x00000000)
+</code>
+So if these libraries are supposed to be loaded, why aren't they?
+I started looking at the next level of dependency, could it be that //libmmal_core// doesn't mention needing //libmmal_util//? Running **ldd** for //libmmal_util// shows that indeed //libmmal_util// is **not NEEDED**. This flag called **DT_NEEDED** should appear when this library is linked, meaning when building the raspberry pi userland. Sometimes this is caused by a more recent compiler optimization called **as-needed**. There is more information about this at link in the reading section, but basically it means that the compiler/linker drops the NEEDED requirement from a linked library if it doesn't find that it is actually being used. This flag can be overridden using a counter flag called **no-as-needed**, and this was actually a problem at some point regarding rpi-userland [[https://github.com/raspberrypi/userland/issues/178|here]], but **this is a decoy!** This is not the issue at hand:  Checking the CMake file [[https://github.com/raspberrypi/userland/blob/master/interface/mmal/core/CMakeLists.txt|here]] confirms that //mmal_util// is not present there at all:
+<code>
+target_link_libraries (mmal_core vcos)
+</code>
+But then how come this library works at all? Apparently (and I also confirmed this with my own addhoc test) it is fine to not declare this dependency as long as the parent library/executable would link against both co-depending libraries. So why doesn't it work now?
+Apparently this is related to the fact buildroot is using a different dynamic linker from a different base library. Instead of the standard linux libc, it uses the [[https://uclibc-ng.org/|uclibc-ng micro library]].
+After some investigation, I found out how to debug the library loading process. First support for **LD_DEBUG** must be enabled in buildroot. In the defconfig:
+<code>
+BR2_UCLIBC_CONFIG="softbot-uClibc-ng.config"
+</code>
+and in that custom file:
+<code>
+SUPPORT_LD_DEBUG=y
+</code>
+Now on the Pi, before running the command, I can run **export LD_DEBUG=all** or usually just **export LD_DEBUG=** is enough data. The results are {{ :thesis:work-journal:ld_debug.txt |here}}.
+The dynamic loading process goes through (I think) two stages. First, all of mentioned libraries are loaded and dependencies are checked according to symbols. There I noticed something strange. This line appeared for every library, including //mmal_core and mmal_util//:
+<code>
+do_dlopen():438: Circular dependency, skipping '/usr/lib//libvcos.so'
+</code>
+But I don't think there really is a circular dependency there! This is very strange.
+At the next stage it seems the DT_NEEDED headers are examined. And of course the ones we need are missing:
+<code>
+lib: /usr/lib//libmmal_core.so has deps:
+ /lib//libc.so.0  /usr/lib//libvcos.so
+lib: /usr/lib//libmmal_util.so has deps:
+ /lib//libc.so.0  /usr/lib//libvcos.so
+</code>
+I believe that they should have been loaded in the previous step and I am trying to find out what happened at **#uclibc-ng** on **freenode**.
+==== Solution ====
+Without knowing what is wrong with uclibc-ng, the first attempt at fix was the use a utility called //patchelf// to add the missing //DT_NEEDED// header:
+<code>
+(post-image.sh script)
+patchelf --add-needed libmmal_core.so ${TARGET_DIR}/usr/lib/libmmal_util.so
+patchelf --add-needed libmmal_util.so ${TARGET_DIR}/usr/lib/libmmal_core.so
+</code>
+But apparently that tool is not very stable, and using it corrupted the entire file.
+OK, let's try to fork rpi userland and compile is so it does have //DT_NEEDED// headers where we need them. Trying to modify the commands as such:
+<code>
+(libmmal_core CMakelists.txt)
+target_link_libraries (mmal_core vcos mmal_util)
+(libmmal_util CMakelists.txt)
+target_link_libraries (mmal_util vcos mmal_core)
+</code>
+Actually doesn't work, because CMake complains on the circular dependency:
+<code>
+CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
+  "mmal_core" of type SHARED_LIBRARY
+    depends on "mmal_util" (weak)
+  "mmal_util" of type SHARED_LIBRARY
+    depends on "mmal_core" (weak)
+At least one of these targets is not a STATIC_LIBRARY.  Cyclic dependencies are allowed only among static libraries
+</code>
+Ok.. but why? We only want the headers that tell compiler to load both of these libraries, the dynamic linker should handle the cyclic dependency!
+Apparently there is a way around this that I found pretty much by guessing. You can tell CMake to add the //DT_NEEDED// flag without actually adding the library to its precious dependncy graph like so:
+<code>
+(libmmal_core)
+target_link_libraries (mmal_core vcos -lmmal_util)
+(libmmal_util)
+target_link_libraries (mmal_util vcos -lmmal_core)
+</code>
+Recompiling userland this way and then try to run //gst-inspect// **actually works!**
+Congratulations.
+===== Fri 01/03/2019 =====
+==== Continuing the gst-rpicamsrc saga ====
+I finally got an answer from someone in the #uclibc-ng channel. He wasn't from the actual development team, but he had very good insights. The log of the chat is here: {{ :thesis:work-journal:uclibc-ng.txt |}}. To summarize the points from the conversation:
+  - Uclibc-ng has plenty of flaws. The dynamic linker is a big mess.
+  - All of the embedded/micro linux distributions, including OpenWRT, Yocto and Alpine Linux have moved from uclibc to another faster and cleaner library called [[https://www.musl-libc.org/|musl]].
+  - Buildroot supports musl, but it still maintains uclibc-ng as the default libc implementation, the main reason being that the developers of both platforms are closesly related.
+To sum it up from the words of sh4rm4^bnc:
+<code>
+< sh4rm4^bnc> if you want a slim, fast, memory-efficient OS, musl should be the first choice
+< sh4rm4^bnc> if you want the kitchen sink and be able to run all kinds of unportable crap, then you should use glibc
+< sh4rm4^bnc> uclibc is for when you have no other choice
+< sh4rm4^bnc> glibc is the bloated stuff running on your desktop linux
+</code>
+So I already built a new image using musl. I had to update and apply [[https://github.com/WebPlatformForEmbedded/meta-wpe/blob/master/recipes-wpe/wpewebkit/wpewebkit/0001-Fix-build-with-musl.patch|this patch]] for WPE, but I'm excited to see how gst-rpicamsrc works, **with the original rpi-userland**, stay tuned. I also posted my problem to the uclibc-ng mailing list [[https://mailman.uclibc-ng.org/pipermail/devel/2019-March/001824.html|here]].
+==== Fried Teensy and Motor drivers ====
+Over the past week, while mostly working on the uclibc issue, I also managed to fry my Teensy. I was setting up the pressure board when suddenly I heard one of the solenoids flip, and after that the Teensy would not respond anymore and was declared **bricked**.
+Since I am still using a breadboard with tons of wires, it may have been that a loose wire from somewhere touched something that it shouldn't be touching, but Janne suggested that it may have been [[https://en.wikipedia.org/wiki/Counter-electromotive_force|"Back EMF"]] - a transient counter electromagnetic force that goes the wrong way, and fries your components. I am still not sure about it, but the manual for TB6612FNG suggests that it is something worth considering in your circuit, and the solution is pretty simple - Diods. As described [[https://www.instructables.com/id/Using-the-Sparkfun-Motor-Driver-1A-Dual-TB6612FNG-/|here]], if your motor is driving in just one direction, all it takes is one diode that is parralel to the motor's +/- , with the direction (line) of the diode going from (-) to (+).
+With great delight I installed diodes on all of the valves and the pumps, and when turning on the power, **3 motor driverss were fried**, one of them even burst into a flame, because it burned the paper label that was on it:
+{{:thesis:work-journal:burned_motor_driver_label.jpeg?400|}}
+Apparently, the black/red wires going from the valves were actually reversed, because what actually determines which pin of the driver is plus and which is minus, is how I set the HIGH and LOW on the input pins. So I basically diverted all of the current right back at the motor driver. Damn. **Next time try one motor driver first **. Luckily the teensy survived and the motor driver is not too expensive. After organizing and reversing all of the wires, I now have a working setup again, which is also protected from Back EMF, and //apart from 1 teensy and 3 motor drivers //, everything else seems fine.
+===== Tue 05/03/2019 =====
+For just adding some task notes so I won't forget:
+  * Chcek out the WPE Webkit performance tips for RPi [[https://github.com/Igalia/meta-webkit/wiki/PerformanceTips|here]] and [[https://github.com/Igalia/meta-webkit/wiki/RPi|here]].
+  * Add GSTREAMER_GL setting.
+  * Modify GPU memory settings.
+===== Thu 07/03/2019 =====
+Again a reminder:
+  * **Patch gst-omx to remove the hdmi / analogue audio sink plugins from gstomx.conf**
+===== Mon 18/03/2019 =====
+Reminder:
+  * File a bug for Webkit WPE, audio elements are looping unsolicitedly, otherwise audio is starting to work great!
+  * Check this link for real time manipulation of the voice: https://github.com/urtzurd/html-audio
+===== Sun 24/03/2019 =====
+A long overdue update.
+The work on Hitodama has been more sporadic in the last couple of weeks, mainly due to congestion of tasks coming from my other work. Nevethelss progress, has been made, and work was mainly about setting up a foundatoin for Hitodama's audio in WPE.
+==== Toolkits ====
+In the last update regarding the gstreamer saga I hinted at the possibility of using a different toolkit than uclibc, namely musl, to solve the dynamic linking issues I had with gstrpicamsrc. Since then the following developments unfolded:
+**TLDR** : I am now more involved in WPE development, know a lot more about toolkit possibilities, running the latest WPE and looking toward 64bit support.
+  * I switched from uclibc to musl, and indeed it solved the dynamic linking problem.
+  * In order to compile wpewbkit with musl, I had to migrate some patches that I found are being used in [[https://github.com/WebPlatformForEmbedded/meta-wpe|here]]. My patches are now located [[https://github.com/Avnerus/softbot-buildroot/tree/softbot/package/wpewebkit/musl-patch-backup|here]].
+  * Since then I got more involved in WPE, and when compiling the latest unstable version I had some issues with compiling JIT (Optimizied Javascript native compiler) support for RPi3 - ARMv7/8. While JIT compiles all of the javascript code into native code, the CLoop backend is more of an interpreter. There is some explanation and comparison of the backends in [[https://community.arm.com/developer/tools-software/tools/b/tools-software-ides-blog/posts/comparison-of-javascript-execution-modes-of-the-webkit-browser-engine|here]].
+  * To avoid more issues with toolkit compatibility - I ended up switching to the 'bloated' GLIBC. It should be noted that although glibc is bloated, it is still very fast and optimized, and in many cases performs faster than musl. More about this [[http://www.etalabs.net/compare_libcs.html|here]].
+  * The new Webkit supports JIT on ARM in two types of insturction sets - ARM64 or ARM THUMB2.
+  * For this project I am compiling for 32bit - This is because the Raspberri Pi userland kernel (the linux drivers for RPi specific hardware) doesn't yet fully support 64bit. There is an open issue about it [[https://github.com/raspberrypi/userland/issues/314|here]].
+  * There are two main usages in my project for the userland. One is the Raspicam - using an RPi driver called MMAL and another is hardware graphics accelration driver called VC4.
+  * MMAL does not yet support 64bit, BUT - the camera can be accessed via an open linux driver V4L2 which does support 64bit.
+  * VC4 closed source drivers by RPi do not support 64bit, but there is an open source driver VC4 KMS. I'm not sure what's the compatilibity of this driver with the WPE backend I'm using - RDK. However there is another more standard backend called FDO which uses Mesa and should support 64bit, but is probably not as fast and robust.
+  * Currently I'm not pursuing 64bit further, but this may start to be an issue with support for WebAssembly. Apparently WebAssembly requires 64 bit - which I think is a requirement of the B3 JIT. There is more about this [[https://webkit.org/blog/7691/webassembly/|here]].
+  * For compiling the new webkit I still had issues with supporting THUMB2. The support for the instruction set was not detected. I solved it by adding the following config to Buildroot and then recompiling the entire toolkit:
+<code>
+BR2_TARGET_OPTIMIZATION="-mthumb -mfpu=neon-fp-armv8 -mtune=cortex-a53"
+</code>
+==== Audio / Voice ====
+I have been working on Hitodama's web based voice. The current strategy for implementation is as follows:
+  - Translate the controller's text if requested - Using Google Cloud Translate (Which I belive is the best translate API).
+  - Generate the voice using Microsoft's Cloud Speech (Which supports much more langauges than Google, including Hebrew, Arabic and Finnish ).
+  - Apply Web Audio filters on the voice to make it gender neutral and nonhuman.
+This was fairly easy to make for the desktop, currently using Granular pitch shifting for Web Audio from [[https://github.com/urtzurd/html-audio|here]]. However, as what is becoming a pattern, this became much more complicated when trying to run on WPE:
+  * The first problem was that there is a bug with streaming //audio src>// elements, having the audio loop indefinitely. After discussing on #webkit I submitted a report [[https://bugs.webkit.org/show_bug.cgi?id=196000|here]],  which was then marked as a duplicate of [[https://bugs.webkit.org/show_bug.cgi?id=189471|this bug]]. Now Phillipe made a patch and I was able to apply it, and waiting to test it.
+  * However I did realized that I probably don't need streaming anyway because it's good enough to buffer the audio first and then pass it through Web Audio.
+  * Web-audio support in cog / WPE is apparently not enabled by default, even if you compile WPE with support. It is enabled using command line arguments when running Cog, and you can see my current command line arguments in [[https://github.com/Avnerus/softbot-buildroot/blob/softbot/softbot-scripts/softbot-browser.sh#L4|here]].
+  * I have Web Audio working now in WPE, but for some reason in my latest attempt, I couldn't play a blob that was converted to an arraybuffer node. It was just silent. This requires further investigation.
+  * NOTE: In order to remotely inspect the latest WPE, one needs the latest WebKitGTK based browser. Easiest way to install it in a sandboxed ([[http://flatkill.org/|maybe]]) environment would be using Flatpak. You can install the latest Epiphany from [[https://webkitgtk.org/epiphany-tech-preview|here]].
+===== Sat 13/04/2019 (Returned from Israel) =====
+Something really strange is happening with Teensy.
+I am experiencing the following, though it needs more tests:
+If "Optimization" is Faster and **without** LTO , and **using any vprintf** function, SPI gets messed up. The readings are bad and for example to pressure drops instead of rising when pushing. It is super strange.

User Tools

Site Tools

Differences

Page Tools