This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
thesis:work-journal [2019/02/20 15:13] avnerus |
thesis:work-journal [2019/03/23 23:14] avnerus |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== Work Journal ====== | ====== Work Journal ====== | ||
- | Decided to try and keep this daily work jour - nal so I don't lose track of the work. | + | Decided to try and keep this daily work journal so I don't lose track of the work. |
===== Sat 02/02/2019 - Robotics Day (retrospect) ===== | ===== Sat 02/02/2019 - Robotics Day (retrospect) ===== | ||
Line 113: | Line 113: | ||
The second thing I am realizing is the non-linearity of PWM (especially on the solenoids, but also on motors). Each valve has its own configuration of a PWM value that is low enough for it not to open completely. Looks like I have to provide PWM values for each actuator separately. | The second thing I am realizing is the non-linearity of PWM (especially on the solenoids, but also on motors). Each valve has its own configuration of a PWM value that is low enough for it not to open completely. Looks like I have to provide PWM values for each actuator separately. | ||
- | However, it seems that there is room to play with the way PWM is sent. The control board in [[https://softroboticstoolkit.com/book/control-board|here]] references [[https://softroboticstoolkit.com/files/sorotoolkit/files/srt_controlboardcode.txt|this code]], where it seems like they are modifying the Arduino's registers to change the form of the PWM! I need to learn more about this, but there is also a guide for Teensy in [[https://www.pjrc.com/teensy/td_pulse.html|here]]. | + | However, it seems that there is room to play with the way PWM is sent. The control board in [[https://softroboticstoolkit.com/book/control-board|here]] references [[https://softroboticstoolkit.com/files/sorotoolkit/files/srt_controlboardcode.txt|this code]], where it seems like they are modifying the Arduino's registers to change the form of the PWM! I need to learn more about this, but there is also a guide for Teensy in [[https://www.pjrc.com/teensy/td_pulse.html|here]]. **TODO: Learn about the concept of Duty Cycle, which is also used to describe the operation of [[https://en.wikipedia.org/wiki/Duty_cycle#Biological_systems|biological muscles]]** |
+ | With the advice of Otso I tried to implement a mini-envelope for sending the PWM wave. Basically I opened the valve entirely for some cycles and only then closed it partially. This was supposed to get over the "friction" for opening a solenoid partially. However it didn't really do anything, there should be a smarter solution. Anyway the code for this version of valve is [[https://github.com/Avnerus/softbot/blob/master/softcontrol/circuit_trials/valve_pwm_envelope.cpp|here]]. What I did end up doing was increase the resolution of the PWM, to get a more stable value. I used 10 bits: | ||
+ | <code> | ||
+ | analogWriteResolution(10); | ||
+ | </code> | ||
+ | I might be able to live with that. | ||
+ | |||
+ | ===== Fri 21/02/2019 ===== | ||
+ | * Ordered some new parts (Valves and tube connectors) - Made a new page to remember the [[soft-robotics:finnish-terminology|Finnish terminology]]. | ||
+ | * With my new resistors, connected two more pressure sensors. Now I have two neck muscles and two arms connected and working. | ||
+ | * Checked the arm sensors and they are still fine. | ||
+ | |||
+ | ===== Thu 27/02/2019 - End of the gst1-rpicamsrc Saga ===== | ||
+ | What started off as a simple wish to install the Raspberry Pi camera plugin for GStreamer1 on Buildroot, became a week long saga of infinite trials and errors. Below are the conclusions: | ||
+ | |||
+ | ==== The symptom ==== | ||
+ | When trying to load the plugin, for example via the command: | ||
+ | <code>gst-inspect-1.0 rpicamsrc</code> | ||
+ | The load fails with a series of errors such as this: | ||
+ | <code> | ||
+ | /usr/libexec/gstreamer-1.0/gst-plugin-scanner: symbol 'mmal_port_pool_create': can't resolve symbol | ||
+ | </code> | ||
+ | |||
+ | At first these were symbols that were actually missing from the //libgstrpicamsrc.so// library, and for example adding the linker flag **-lmmal_component** would solve the problem. But the issue became vastly more complex when the unresolved symbols actually belonged to **libmmal_util.so** A file that is already linked to the library! | ||
+ | |||
+ | ==== Background reading ==== | ||
+ | These are the two best readings I have found for the subject: | ||
+ | * [[http://www.kaizou.org/2015/01/linux-libraries/|Better understanding Linux secondary dependencies solving with examples]] | ||
+ | * [[https://flameeyes.blog/2010/09/01/your-worst-enemy-undefined-symbols/|Your worst enemy: undefined symbols]]. | ||
+ | |||
+ | ==== Analysis ==== | ||
+ | The first step was to figure out at what stage the undefined symbols are observed. The symbol **mmal_port_pool_create** is owned by //libmmal_util.so//. This is found using: | ||
+ | <code> | ||
+ | ./host/bin/arm-buildroot-linux-uclibcgnueabihf-nm -D target/usr/lib/libmmal_util.so | grep port_pool_create | ||
+ | </code> | ||
+ | * The ARM version of //nm// is used, even though it's not necessary for this output. | ||
+ | * The -D parameter is used, since this is a dynamic shared library. | ||
+ | |||
+ | The output: | ||
+ | <code> | ||
+ | 000041bc T mmal_port_pool_create | ||
+ | </code> | ||
+ | The **T** symbol means this is a globally defined symbol that is available to outside links. | ||
+ | |||
+ | The second step was to figure out what libraries are using those symbols. This command could help: | ||
+ | <code> | ||
+ | nm -D target/usr/lib/libmmal*.so | ||
+ | </code> | ||
+ | And searching for **port_pool_create**, we find this only under //libmmal_core.so//: | ||
+ | <code> | ||
+ | U mmal_port_pool_create | ||
+ | </code> | ||
+ | The **U** symbol (undefined) means the library is using this symbol from another library. | ||
+ | Further examination shows that these libraries are **co-dependent**. //mmal_core// is using symbols from //mmal_util// and //mmal_util// is using symbols from //mmal_core//. | ||
+ | |||
+ | Normally it should be enough that our own //libgstrpicamsrc.so// is loading //libmmal_core.so// and //libmmal_util.so// for everything to work. We can verify this in 3 different ways: | ||
+ | <code> | ||
+ | readelf -d target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so | ||
+ | </code> | ||
+ | Or | ||
+ | <code> | ||
+ | objdump -x target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so | ||
+ | </code> | ||
+ | Would both show this line: | ||
+ | <code> | ||
+ | 0x00000001 (NEEDED) Shared library: [libmmal_core.so] | ||
+ | 0x00000001 (NEEDED) Shared library: [libmmal_util.so] | ||
+ | </code> | ||
+ | We can also see that //libmmal_core.so// is requested before //libmmal_util.so//. This is actually due to linkage order when **-lmmal_core** appears before **-lmmal_util** | ||
+ | We can also run dependency analysis with the **ldd** tool: | ||
+ | <code> | ||
+ | LD_LIBRARY_PATH=./target/usr/lib ./host/bin/arm-buildroot-linux-uclibcgnueabihf-ldd target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so | ||
+ | </code> | ||
+ | And find the dependent libraries. | ||
+ | <code> | ||
+ | libmmal_core.so => ./target/usr/lib/libmmal_core.so (0x00000000) | ||
+ | libmmal_util.so => ./target/usr/lib/libmmal_util.so (0x00000000) | ||
+ | </code> | ||
+ | So if these libraries are supposed to be loaded, why aren't they? | ||
+ | I started looking at the next level of dependency, could it be that //libmmal_core// doesn't mention needing //libmmal_util//? Running **ldd** for //libmmal_util// shows that indeed //libmmal_util// is **not NEEDED**. This flag called **DT_NEEDED** should appear when this library is linked, meaning when building the raspberry pi userland. Sometimes this is caused by a more recent compiler optimization called **as-needed**. There is more information about this at link in the reading section, but basically it means that the compiler/linker drops the NEEDED requirement from a linked library if it doesn't find that it is actually being used. This flag can be overridden using a counter flag called **no-as-needed**, and this was actually a problem at some point regarding rpi-userland [[https://github.com/raspberrypi/userland/issues/178|here]], but **this is a decoy!** This is not the issue at hand: Checking the CMake file [[https://github.com/raspberrypi/userland/blob/master/interface/mmal/core/CMakeLists.txt|here]] confirms that //mmal_util// is not present there at all: | ||
+ | <code> | ||
+ | target_link_libraries (mmal_core vcos) | ||
+ | </code> | ||
+ | But then how come this library works at all? Apparently (and I also confirmed this with my own addhoc test) it is fine to not declare this dependency as long as the parent library/executable would link against both co-depending libraries. So why doesn't it work now? | ||
+ | |||
+ | Apparently this is related to the fact buildroot is using a different dynamic linker from a different base library. Instead of the standard linux libc, it uses the [[https://uclibc-ng.org/|uclibc-ng micro library]]. | ||
+ | |||
+ | After some investigation, I found out how to debug the library loading process. First support for **LD_DEBUG** must be enabled in buildroot. In the defconfig: | ||
+ | <code> | ||
+ | BR2_UCLIBC_CONFIG="softbot-uClibc-ng.config" | ||
+ | </code> | ||
+ | and in that custom file: | ||
+ | <code> | ||
+ | SUPPORT_LD_DEBUG=y | ||
+ | </code> | ||
+ | Now on the Pi, before running the command, I can run **export LD_DEBUG=all** or usually just **export LD_DEBUG=** is enough data. The results are {{ :thesis:work-journal:ld_debug.txt |here}}. | ||
+ | The dynamic loading process goes through (I think) two stages. First, all of mentioned libraries are loaded and dependencies are checked according to symbols. There I noticed something strange. This line appeared for every library, including //mmal_core and mmal_util//: | ||
+ | <code> | ||
+ | do_dlopen():438: Circular dependency, skipping '/usr/lib//libvcos.so' | ||
+ | </code> | ||
+ | But I don't think there really is a circular dependency there! This is very strange. | ||
+ | |||
+ | At the next stage it seems the DT_NEEDED headers are examined. And of course the ones we need are missing: | ||
+ | <code> | ||
+ | lib: /usr/lib//libmmal_core.so has deps: | ||
+ | /lib//libc.so.0 /usr/lib//libvcos.so | ||
+ | lib: /usr/lib//libmmal_util.so has deps: | ||
+ | /lib//libc.so.0 /usr/lib//libvcos.so | ||
+ | </code> | ||
+ | I believe that they should have been loaded in the previous step and I am trying to find out what happened at **#uclibc-ng** on **freenode**. | ||
+ | |||
+ | ==== Solution ==== | ||
+ | Without knowing what is wrong with uclibc-ng, the first attempt at fix was the use a utility called //patchelf// to add the missing //DT_NEEDED// header: | ||
+ | <code> | ||
+ | (post-image.sh script) | ||
+ | patchelf --add-needed libmmal_core.so ${TARGET_DIR}/usr/lib/libmmal_util.so | ||
+ | patchelf --add-needed libmmal_util.so ${TARGET_DIR}/usr/lib/libmmal_core.so | ||
+ | </code> | ||
+ | But apparently that tool is not very stable, and using it corrupted the entire file. | ||
+ | |||
+ | OK, let's try to fork rpi userland and compile is so it does have //DT_NEEDED// headers where we need them. Trying to modify the commands as such: | ||
+ | <code> | ||
+ | (libmmal_core CMakelists.txt) | ||
+ | target_link_libraries (mmal_core vcos mmal_util) | ||
+ | (libmmal_util CMakelists.txt) | ||
+ | target_link_libraries (mmal_util vcos mmal_core) | ||
+ | </code> | ||
+ | Actually doesn't work, because CMake complains on the circular dependency: | ||
+ | <code> | ||
+ | CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle): | ||
+ | "mmal_core" of type SHARED_LIBRARY | ||
+ | depends on "mmal_util" (weak) | ||
+ | "mmal_util" of type SHARED_LIBRARY | ||
+ | depends on "mmal_core" (weak) | ||
+ | At least one of these targets is not a STATIC_LIBRARY. Cyclic dependencies are allowed only among static libraries | ||
+ | </code> | ||
+ | Ok.. but why? We only want the headers that tell compiler to load both of these libraries, the dynamic linker should handle the cyclic dependency! | ||
+ | |||
+ | Apparently there is a way around this that I found pretty much by guessing. You can tell CMake to add the //DT_NEEDED// flag without actually adding the library to its precious dependncy graph like so: | ||
+ | <code> | ||
+ | (libmmal_core) | ||
+ | target_link_libraries (mmal_core vcos -lmmal_util) | ||
+ | (libmmal_util) | ||
+ | target_link_libraries (mmal_util vcos -lmmal_core) | ||
+ | </code> | ||
+ | Recompiling userland this way and then try to run //gst-inspect// **actually works!** | ||
+ | Congratulations. | ||
+ | |||
+ | ===== Fri 01/03/2019 ===== | ||
+ | |||
+ | ==== Continuing the gst-rpicamsrc saga ==== | ||
+ | I finally got an answer from someone in the #uclibc-ng channel. He wasn't from the actual development team, but he had very good insights. The log of the chat is here: {{ :thesis:work-journal:uclibc-ng.txt |}}. To summarize the points from the conversation: | ||
+ | - Uclibc-ng has plenty of flaws. The dynamic linker is a big mess. | ||
+ | - All of the embedded/micro linux distributions, including OpenWRT, Yocto and Alpine Linux have moved from uclibc to another faster and cleaner library called [[https://www.musl-libc.org/|musl]]. | ||
+ | - Buildroot supports musl, but it still maintains uclibc-ng as the default libc implementation, the main reason being that the developers of both platforms are closesly related. | ||
+ | |||
+ | To sum it up from the words of sh4rm4^bnc: | ||
+ | <code> | ||
+ | < sh4rm4^bnc> if you want a slim, fast, memory-efficient OS, musl should be the first choice | ||
+ | < sh4rm4^bnc> if you want the kitchen sink and be able to run all kinds of unportable crap, then you should use glibc | ||
+ | < sh4rm4^bnc> uclibc is for when you have no other choice | ||
+ | < sh4rm4^bnc> glibc is the bloated stuff running on your desktop linux | ||
+ | </code> | ||
+ | |||
+ | So I already built a new image using musl. I had to update and apply [[https://github.com/WebPlatformForEmbedded/meta-wpe/blob/master/recipes-wpe/wpewebkit/wpewebkit/0001-Fix-build-with-musl.patch|this patch]] for WPE, but I'm excited to see how gst-rpicamsrc works, **with the original rpi-userland**, stay tuned. I also posted my problem to the uclibc-ng mailing list [[https://mailman.uclibc-ng.org/pipermail/devel/2019-March/001824.html|here]]. | ||
+ | |||
+ | ==== Fried Teensy and Motor drivers ==== | ||
+ | |||
+ | Over the past week, while mostly working on the uclibc issue, I also managed to fry my Teensy. I was setting up the pressure board when suddenly I heard one of the solenoids flip, and after that the Teensy would not respond anymore and was declared **bricked**. | ||
+ | |||
+ | Since I am still using a breadboard with tons of wires, it may have been that a loose wire from somewhere touched something that it shouldn't be touching, but Janne suggested that it may have been [[https://en.wikipedia.org/wiki/Counter-electromotive_force|"Back EMF"]] - a transient counter electromagnetic force that goes the wrong way, and fries your components. I am still not sure about it, but the manual for TB6612FNG suggests that it is something worth considering in your circuit, and the solution is pretty simple - Diods. As described [[https://www.instructables.com/id/Using-the-Sparkfun-Motor-Driver-1A-Dual-TB6612FNG-/|here]], if your motor is driving in just one direction, all it takes is one diode that is parralel to the motor's +/- , with the direction (line) of the diode going from (-) to (+). | ||
+ | |||
+ | With great delight I installed diodes on all of the valves and the pumps, and when turning on the power, **3 motor driverss were fried**, one of them even burst into a flame, because it burned the paper label that was on it: | ||
+ | |||
+ | {{:thesis:work-journal:burned_motor_driver_label.jpeg?400|}} | ||
+ | |||
+ | Apparently, the black/red wires going from the valves were actually reversed, because what actually determines which pin of the driver is plus and which is minus, is how I set the HIGH and LOW on the input pins. So I basically diverted all of the current right back at the motor driver. Damn. **Next time try one motor driver first **. Luckily the teensy survived and the motor driver is not too expensive. After organizing and reversing all of the wires, I now have a working setup again, which is also protected from Back EMF, and //apart from 1 teensy and 3 motor drivers //, everything else seems fine. | ||
+ | |||
+ | ===== Tue 05/03/2019 ===== | ||
+ | |||
+ | For just adding some task notes so I won't forget: | ||
+ | |||
+ | * Chcek out the WPE Webkit performance tips for RPi [[https://github.com/Igalia/meta-webkit/wiki/PerformanceTips|here]] and [[https://github.com/Igalia/meta-webkit/wiki/RPi|here]]. | ||
+ | * Add GSTREAMER_GL setting. | ||
+ | * Modify GPU memory settings. | ||
+ | |||
+ | ===== Thu 07/03/2019 ===== | ||
+ | |||
+ | Again a reminder: | ||
+ | |||
+ | * **Patch gst-omx to remove the hdmi / analogue audio sink plugins from gstomx.conf** | ||
+ | |||
+ | |||
+ | |||
+ | ===== Mon 18/03/2019 ===== | ||
+ | Reminder: | ||
+ | * File a bug for Webkit WPE, audio elements are looping unsolicitedly, otherwise audio is starting to work great! | ||
+ | * Check this link for real time manipulation of the voice: https://github.com/urtzurd/html-audio | ||
+ | ===== Sun 24/03/2019 ===== | ||
+ | A long overdue update. | ||