Decided to try and keep this daily work journal so I don't lose track of the work.

This was an attempt to showcase Hitodama after ~3 days of work with the help of Joaquin. Joaquin was able to arrange the valves wiring in a tidy box. It looks great (for a breadboard setup), but I'm worried on whether it will scale up when I need more valves (soon). I got good feedback during the show, as most people haven't seen this type of robot before (some even named it as their favorite of the show!), and the kids really enjoyed playing with the pumps, but it was also not stable or presentable enough to get the full experience. Faults:

• Didn't have time to setup the pressure senors, or tune the inflation/deflation speeds so was basically limited to quick inflation and deflation - this movement is not natural and doesn't really show the potential of soft robotics.
• Was only able to setup the two arms and 2/3 neck actuators.
• The valves were all leaking - I didn't know that NPT connectors are meant to be wrapped by PTFE tape (This is the first time I use these valves on display, previous 'generaton' had the smaller valves).
• In the last minutes before the show I could get the arms kind of working, but they both broke during the show - being hung down when deflated hurt he wires. During the show one arm sensor stopped responding completely and the other was unstable.

I spent this day unpacking the robot and cleaning the desk at the Mechatronics workshop.

Checked the left arm which broke completely at the Robotics day. The problem was at the “shoulder” area, the connection between the conductive fabric threads and my soldering to the ribbon cable. One cable got disconnected. Fixing the conductive threads is difficult because they tear apart and become inaccesisble, I had to poke a wire with some solder into the silicone and heat it up to try and connect with the fiber, eventually I could do it and was able to create a bridge to the ribbon cable. It remained relatively clean because the “shoulder” area has more space to playing around with, but it's still not super stable. TODO: Needs an isolation solution. Perhaps with the help of Joaquin.

Just when I thought the arms are fine now, the other arm (right one) started having issues and these ones were worse, because there was a disconnection in the sensor area. At that point I decided to not use conductive threads again for this task, even though I'm still not sure if the problem is because of my bad soldering. It might be that having the threads submerge in silicone damages their durability. Note that I'm using the rare and best kind of fibers by Mitsufuji, graciously gifted to me by the woman at Kobakant Berlin. There was no choice, I had to do surgery and cut open the sensor. While examining, literally all of the conductive threads tore apart. It took many hours, but eventually I could do the poking solution with all of the pins, but it was probably a mistake to bridge from the button of the sensor and not from above, because now the hand won't close up:

However surprisingly pressing the palm still works pretty well, so at least it's a temporary solution. Meanwhile I am continuing the production of two new arms, this time trying to avoid the threads all-together by gluing the ribbon cable directly, basically having it as both the conductive and the fabric actuation restraining layer:

The thickness of the cable is 1mm so there shouldn't be an issue with the skin casting, hopefully. As for the temporary arm, I believe I can fix it later on by re-soldering the bridges to the sensor from the top instead from from the bottom. I can't really make the new arms yet anyway, my Ecoflex is too old. I ordered a new pack from Kaupo.

I moved on from the arms to debugging and developing the air-flow to the actuators. This is a new setup compared to the one I had in Japan, and not thoroughly tested. The first issue was leakage. All of the valves were leaking. I performed some tests and read about the NPT connections, finally realizing that these connections are designed to be sealed by PTFE tape (Joaquin was actually suggesting this earlier, but we didn't know it's a must, well it is). I moved on to testing the pressure sensors. I thought these are the same ones that I had in Japan, but I was wrong. They are actually the MPS20N0040D or a very similar clone. Turns out the pinout is different and that these sensors require an Operational amplifier. I spend the day trying to understand what that even mean and ordered some from Matti.

Went to Motonet and IKH, got PTFE tape(s) and some more NPT connectors for testing. TODO: Need to buy at Etra more of the larger silicone tube for the large NPT connector, and converters between the large and the small. Also buy even smaller ones for the connection to the actuator. I put the PTFE on the valves and it did stop the leak, but last time I tried there was still something coming out, I think it might be other valves that I forget that are connected and I didn't seal yet TODO: Check that. Janne had some LM358 Op-Amps so I went and tried that with the pressure sensors, couldn't get it to work. I got a constant output coming out of the Op-Amp. TODO: Now I found this page that shows how they connect resistors to the LM358, could be that is what was missing.* But anyway, eventually I analyzed the signal with the Oscilloscope and then with the Arduino and I found one pin that was changing the voltage in response to pressure. It might be that that's all I need. I implemented a pressure reporting component to the software but it wasn't working yet how I wanted it to. TODO: Report the pressure using 2 bytes instead of 1!

Went and sealed with PTFE all of the valve connectors. Wasn't very good at it, and had to redo some of the wrappings. Eventually managed to have leak-proof arm actuation, but after about one hour it started leaking again. Did some tests and ended up with the conclusion that one valve died. It wasn't 'normally-closed' anymore and it kept transmitting air. I wonder if using PWM on the valve is abusing it? Anyway, I replaced the valve with another one and marked it as “Broken”. Then moved on to experimenting with the pressure sensors. The software component is fine, but I am starting to reach the conclusion that I do need the Op-Amp. The range I get is very small and seems unstable, making it very hard to determine the minimum and maximum pressure on the software. Even if I calibrate the pressure as a base pressure value every run, it can still be unpredictable. I need to try the resistor setup with the LM358.

I spent the day learning about electronic circuits and Op-Amps, but still don't fully grasp all of the concepts. At the end of the day I was able to get meaningful, but not amplified output from the LM358 chip. It seems that the sensors I have are not exactly the same as the MPS20N0040D-D mentioned in all of the documentation, which causes inconsistent results. This is either way not a very scalable solution, requiring an Op-Amp (probably dual, not to mention filters etc) setup for each pressure sensor. I could probably use this now for testing, but shouldn't spend more than this week to get it stabilized. I think I should just find a way to order the pressure sensors that I had in Japan. For example:Through here?

After the last couple of days I can finally say that I have an adequate understanding of Op-Amps. One website that enlightened me after getting lost in the mathematical jargon was this one. I was able to amplify the Chinese sensor sufficiently to get a good range of values and it also seems more stable now that I'm wiring all of the pins on the sensor into the differential amp.

After manual testing I believe that I can work with this sensor, but it seems I should still to a little bit of averaging on the software side because opening and closing valves seems to cause sudden and very transient drops in voltage. The final test would be whether I can run my “Pressure oscillation” code (inflating up and down) in a stable manner. If that works, I will document my setup in the Wiki and continue hooking up more pressure sensors. TODO: I need more Op-Amps and resistors though. Simultaneously I've still been searching for other sensors that I can buy, or a vendor that sells that Metrodyne sensors I used in Japan. Eventually I contacted the company itself in Taiwan and got a quick positive reply asking me for the quantity and application. I responded but haven't heard from them back so far.

Work in the recent days is divided into two categories:

In the weekend I ran more tests on the amplified, cheap pressure sensors. My goal was to have “Oscillations” working correctly. I ended up putting a bunch of optimizations to the pressure calculator. Primarily:

1. Averaging the pressure calculation - currently averaging over a 100 read window.
2. Accounting for fluctuations - I have noticed that when closing the valves after deflating or inflating, there is a sudden fluctuation. For example, inflating until I reach 200, then closing the valves, would result in a quick drop to 190. The opposite effect would happen when deflating. To account for that, I am now 'overshooting' the inflation/deflation, inflating for example up to 210 and then letting it drop back to 200.

I have also documented the setup and gathered the material in here.

Now I have started to arrange the pressure sensors on a new breadboard, utilizing the LM358s with 2 sensors per chip. But I ran out of the resistors to use, so I have ordered some from PartCo and they should arrive soon.

The interaction of Hitodama is Web-Powered and runs on Raspberry Pi.

About a year ago I found out about the project WPE - "Webkit Platform for Embedded". At that time it was fairly immature, but I did manage to generate a 'buildroot' image that runs WPE, and even my Rust controller. Now, coming back to it, a few things have changed and I began researching again about the best way to run WPE. After a lot of confusions and attempts, I ended up on the #webkit IRC channel on FreeNode asking for help, and was super lucky to get advised by one of the creators of WPE. I have linked my chat with him in the page: webkit-wpe_.log.txt

Now I am in the process of building another buildroot with all of the latest stuff, it's pretty exciting!

Yesterday I started arranging the pressure sensors on the breadboard and connecting them to the main circuit. It looks like this:

Overall it seems nice, but I am worried about scaling. Especially with the valves. So far I have connected two of the neck muscles.

I am spending time figuring out the valves and their quirkiness. I am using PWM outout to try and open the valve only partially. It seemed to work with the first actuator (Neck Right), but then it didn't work with the left muscle. It seemed as if closing the entry valve disabled my ability to run PWM on the release valve. I have no idea why, but I managed to solve this by changing the IN2 pin of the entry valve. Here is how the configuration was when it didn't work:

    Chamber("LeftNeck",
            new Valve(2,4,5,25),
            new Valve(2,3,5,25),
            A16, 
            120, 
            300
    )

And here is how it does work:

    Chamber("LeftNeck",
            new Valve(2,26,5,25),
            new Valve(2,3,5,25),
            A16, 
            120, 
            300
    ),

I updated the Teensy 3.6 Hitodama pin assigns Google Doc that Joaquin made.

The second thing I am realizing is the non-linearity of PWM (especially on the solenoids, but also on motors). Each valve has its own configuration of a PWM value that is low enough for it not to open completely. Looks like I have to provide PWM values for each actuator separately.

However, it seems that there is room to play with the way PWM is sent. The control board in here references this code, where it seems like they are modifying the Arduino's registers to change the form of the PWM! I need to learn more about this, but there is also a guide for Teensy in here. ​TODO: Learn about the concept of Duty Cycle, which is also used to describe the operation of biological muscles

With the advice of Otso I tried to implement a mini-envelope for sending the PWM wave. Basically I opened the valve entirely for some cycles and only then closed it partially. This was supposed to get over the “friction” for opening a solenoid partially. However it didn't really do anything, there should be a smarter solution. Anyway the code for this version of valve is here. What I did end up doing was increase the resolution of the PWM, to get a more stable value. I used 10 bits:

analogWriteResolution(10);

I might be able to live with that.

• Ordered some new parts (Valves and tube connectors) - Made a new page to remember the Finnish terminology.
• With my new resistors, connected two more pressure sensors. Now I have two neck muscles and two arms connected and working.
• Checked the arm sensors and they are still fine.

What started off as a simple wish to install the Raspberry Pi camera plugin for GStreamer1 on Buildroot, became a week long saga of infinite trials and errors. Below are the conclusions:

When trying to load the plugin, for example via the command:

gst-inspect-1.0 rpicamsrc

The load fails with a series of errors such as this:

/usr/libexec/gstreamer-1.0/gst-plugin-scanner: symbol 'mmal_port_pool_create': can't resolve symbol

At first these were symbols that were actually missing from the libgstrpicamsrc.so library, and for example adding the linker flag -lmmal_component would solve the problem. But the issue became vastly more complex when the unresolved symbols actually belonged to libmmal_util.so A file that is already linked to the library!

These are the two best readings I have found for the subject:

• Better understanding Linux secondary dependencies solving with examples
• Your worst enemy: undefined symbols.

The first step was to figure out at what stage the undefined symbols are observed. The symbol mmal_port_pool_create is owned by libmmal_util.so. This is found using:

./host/bin/arm-buildroot-linux-uclibcgnueabihf-nm -D target/usr/lib/libmmal_util.so | grep port_pool_create

• The ARM version of nm is used, even though it's not necessary for this output.
• The -D parameter is used, since this is a dynamic shared library.

The output:

000041bc T mmal_port_pool_create 

The T symbol means this is a globally defined symbol that is available to outside links.

The second step was to figure out what libraries are using those symbols. This command could help:

nm -D target/usr/lib/libmmal*.so

And searching for port_pool_create, we find this only under libmmal_core.so:

U mmal_port_pool_create

The U symbol (undefined) means the library is using this symbol from another library. Further examination shows that these libraries are co-dependent. mmal_core is using symbols from mmal_util and mmal_util is using symbols from mmal_core.

Normally it should be enough that our own libgstrpicamsrc.so is loading libmmal_core.so and libmmal_util.so for everything to work. We can verify this in 3 different ways:

readelf -d target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so

Or

objdump -x target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so

Would both show this line:

 0x00000001 (NEEDED)                     Shared library: [libmmal_core.so]
 0x00000001 (NEEDED)                     Shared library: [libmmal_util.so]

We can also see that libmmal_core.so is requested before libmmal_util.so. This is actually due to linkage order when -lmmal_core appears before -lmmal_util We can also run dependency analysis with the ldd tool:

LD_LIBRARY_PATH=./target/usr/lib ./host/bin/arm-buildroot-linux-uclibcgnueabihf-ldd target/usr/lib/gstreamer-1.0/libgstrpicamsrc.so

And find the dependent libraries.

	libmmal_core.so => ./target/usr/lib/libmmal_core.so (0x00000000)
	libmmal_util.so => ./target/usr/lib/libmmal_util.so (0x00000000)

So if these libraries are supposed to be loaded, why aren't they? I started looking at the next level of dependency, could it be that libmmal_core doesn't mention needing libmmal_util? Running ldd for libmmal_util shows that indeed libmmal_util is not NEEDED. This flag called DT_NEEDED should appear when this library is linked, meaning when building the raspberry pi userland. Sometimes this is caused by a more recent compiler optimization called as-needed. There is more information about this at link in the reading section, but basically it means that the compiler/linker drops the NEEDED requirement from a linked library if it doesn't find that it is actually being used. This flag can be overridden using a counter flag called no-as-needed, and this was actually a problem at some point regarding rpi-userland here, but this is a decoy! This is not the issue at hand: Checking the CMake file here confirms that mmal_util is not present there at all:

target_link_libraries (mmal_core vcos)

But then how come this library works at all? Apparently (and I also confirmed this with my own addhoc test) it is fine to not declare this dependency as long as the parent library/executable would link against both co-depending libraries. So why doesn't it work now?

Apparently this is related to the fact buildroot is using a different dynamic linker from a different base library. Instead of the standard linux libc, it uses the uclibc-ng micro library.

After some investigation, I found out how to debug the library loading process. First support for LD_DEBUG must be enabled in buildroot. In the defconfig:

BR2_UCLIBC_CONFIG="softbot-uClibc-ng.config"

and in that custom file:

SUPPORT_LD_DEBUG=y

Now on the Pi, before running the command, I can run export LD_DEBUG=all or usually just export LD_DEBUG= is enough data. The results are here. The dynamic loading process goes through (I think) two stages. First, all of mentioned libraries are loaded and dependencies are checked according to symbols. There I noticed something strange. This line appeared for every library, including mmal_core and mmal_util:

do_dlopen():438: Circular dependency, skipping '/usr/lib//libvcos.so'

But I don't think there really is a circular dependency there! This is very strange.

At the next stage it seems the DT_NEEDED headers are examined. And of course the ones we need are missing:

lib: /usr/lib//libmmal_core.so has deps:
 /lib//libc.so.0  /usr/lib//libvcos.so 
lib: /usr/lib//libmmal_util.so has deps:
 /lib//libc.so.0  /usr/lib//libvcos.so 

I believe that they should have been loaded in the previous step and I am trying to find out what happened at #uclibc-ng on freenode.

Without knowing what is wrong with uclibc-ng, the first attempt at fix was the use a utility called patchelf to add the missing DT_NEEDED header:

(post-image.sh script)
patchelf --add-needed libmmal_core.so ${TARGET_DIR}/usr/lib/libmmal_util.so
patchelf --add-needed libmmal_util.so ${TARGET_DIR}/usr/lib/libmmal_core.so

But apparently that tool is not very stable, and using it corrupted the entire file.

OK, let's try to fork rpi userland and compile is so it does have DT_NEEDED headers where we need them. Trying to modify the commands as such:

(libmmal_core CMakelists.txt)
target_link_libraries (mmal_core vcos mmal_util)
(libmmal_util CMakelists.txt)
target_link_libraries (mmal_util vcos mmal_core)

Actually doesn't work, because CMake complains on the circular dependency:

CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
  "mmal_core" of type SHARED_LIBRARY
    depends on "mmal_util" (weak)
  "mmal_util" of type SHARED_LIBRARY
    depends on "mmal_core" (weak)
At least one of these targets is not a STATIC_LIBRARY.  Cyclic dependencies are allowed only among static libraries

Ok.. but why? We only want the headers that tell compiler to load both of these libraries, the dynamic linker should handle the cyclic dependency!

Apparently there is a way around this that I found pretty much by guessing. You can tell CMake to add the DT_NEEDED flag without actually adding the library to its precious dependncy graph like so:

(libmmal_core)
target_link_libraries (mmal_core vcos -lmmal_util)
(libmmal_util)
target_link_libraries (mmal_util vcos -lmmal_core)

Recompiling userland this way and then try to run gst-inspect actually works! Congratulations.

I finally got an answer from someone in the #uclibc-ng channel. He wasn't from the actual development team, but he had very good insights. The log of the chat is here: uclibc-ng.txt. To summarize the points from the conversation:

1. Uclibc-ng has plenty of flaws. The dynamic linker is a big mess.
2. All of the embedded/micro linux distributions, including OpenWRT, Yocto and Alpine Linux have moved from uclibc to another faster and cleaner library called musl.
3. Buildroot supports musl, but it still maintains uclibc-ng as the default libc implementation, the main reason being that the developers of both platforms are closesly related.

To sum it up from the words of sh4rm4^bnc:

< sh4rm4^bnc> if you want a slim, fast, memory-efficient OS, musl should be the first choice
< sh4rm4^bnc> if you want the kitchen sink and be able to run all kinds of unportable crap, then you should use glibc
< sh4rm4^bnc> uclibc is for when you have no other choice
< sh4rm4^bnc> glibc is the bloated stuff running on your desktop linux

So I already built a new image using musl. I had to update and apply this patch for WPE, but I'm excited to see how gst-rpicamsrc works, with the original rpi-userland, stay tuned. I also posted my problem to the uclibc-ng mailing list here.

Over the past week, while mostly working on the uclibc issue, I also managed to fry my Teensy. I was setting up the pressure board when suddenly I heard one of the solenoids flip, and after that the Teensy would not respond anymore and was declared bricked.

Since I am still using a breadboard with tons of wires, it may have been that a loose wire from somewhere touched something that it shouldn't be touching, but Janne suggested that it may have been "Back EMF" - a transient counter electromagnetic force that goes the wrong way, and fries your components. I am still not sure about it, but the manual for TB6612FNG suggests that it is something worth considering in your circuit, and the solution is pretty simple - Diods. As described here, if your motor is driving in just one direction, all it takes is one diode that is parralel to the motor's +/- , with the direction (line) of the diode going from (-) to (+).

With great delight I installed diodes on all of the valves and the pumps, and when turning on the power, 3 motor driverss were fried, one of them even burst into a flame, because it burned the paper label that was on it:

Apparently, the black/red wires going from the valves were actually reversed, because what actually determines which pin of the driver is plus and which is minus, is how I set the HIGH and LOW on the input pins. So I basically diverted all of the current right back at the motor driver. Damn. Next time try one motor driver first . Luckily the teensy survived and the motor driver is not too expensive. After organizing and reversing all of the wires, I now have a working setup again, which is also protected from Back EMF, and apart from 1 teensy and 3 motor drivers , everything else seems fine.

For just adding some task notes so I won't forget:

• Chcek out the WPE Webkit performance tips for RPi here and here.
• Add GSTREAMER_GL setting.
• Modify GPU memory settings.

Again a reminder:

• Patch gst-omx to remove the hdmi / analogue audio sink plugins from gstomx.conf

Reminder:

• File a bug for Webkit WPE, audio elements are looping unsolicitedly, otherwise audio is starting to work great!
• Check this link for real time manipulation of the voice: https://github.com/urtzurd/html-audio

A long overdue update.

The work on Hitodama has been more sporadic in the last couple of weeks, mainly due to congestion of tasks coming from my other work. Nevethelss progress, has been made, and work was mainly about setting up a foundatoin for Hitodama's audio in WPE.

In the last update regarding the gstreamer saga I hinted at the possibility of using a different toolkit than uclibc, namely musl, to solve the dynamic linking issues I had with gstrpicamsrc. Since then the following developments unfolded:

TLDR : I am now more involved in WPE development, know a lot more about toolkit possibilities, running the latest WPE and looking toward 64bit support.

• I switched from uclibc to musl, and indeed it solved the dynamic linking problem.
• In order to compile wpewbkit with musl, I had to migrate some patches that I found are being used in here. My patches are now located here.
• Since then I got more involved in WPE, and when compiling the latest unstable version I had some issues with compiling JIT (Optimizied Javascript native compiler) support for RPi3 - ARMv7/8. While JIT compiles all of the javascript code into native code, the CLoop backend is more of an interpreter. There is some explanation and comparison of the backends in here.
• To avoid more issues with toolkit compatibility - I ended up switching to the 'bloated' GLIBC. It should be noted that although glibc is bloated, it is still very fast and optimized, and in many cases performs faster than musl. More about this here.
• The new Webkit supports JIT on ARM in two types of insturction sets - ARM64 or ARM THUMB2.
• For this project I am compiling for 32bit - This is because the Raspberri Pi userland kernel (the linux drivers for RPi specific hardware) doesn't yet fully support 64bit. There is an open issue about it here.
• There are two main usages in my project for the userland. One is the Raspicam - using an RPi driver called MMAL and another is hardware graphics accelration driver called VC4.
• MMAL does not yet support 64bit, BUT - the camera can be accessed via an open linux driver V4L2 which does support 64bit.
• VC4 closed source drivers by RPi do not support 64bit, but there is an open source driver VC4 KMS. I'm not sure what's the compatilibity of this driver with the WPE backend I'm using - RDK. However there is another more standard backend called FDO which uses Mesa and should support 64bit, but is probably not as fast and robust.
• Currently I'm not pursuing 64bit further, but this may start to be an issue with support for WebAssembly. Apparently WebAssembly requires 64 bit - which I think is a requirement of the B3 JIT. There is more about this here.
• For compiling the new webkit I still had issues with supporting THUMB2. The support for the instruction set was not detected. I solved it by adding the following config to Buildroot and then recompiling the entire toolkit:

BR2_TARGET_OPTIMIZATION="-mthumb -mfpu=neon-fp-armv8 -mtune=cortex-a53"

I have been working on Hitodama's web based voice. The current strategy for implementation is as follows:

1. Translate the controller's text if requested - Using Google Cloud Translate (Which I belive is the best translate API).
2. Generate the voice using Microsoft's Cloud Speech (Which supports much more langauges than Google, including Hebrew, Arabic and Finnish ).
3. Apply Web Audio filters on the voice to make it gender neutral and nonhuman.

This was fairly easy to make for the desktop, currently using Granular pitch shifting for Web Audio from here. However, as what is becoming a pattern, this became much more complicated when trying to run on WPE:

• The first problem was that there is a bug with streaming audio src> elements, having the audio loop indefinitely. After discussing on #webkit I submitted a report here, which was then marked as a duplicate of this bug. Now Phillipe made a patch and I was able to apply it, and waiting to test it.
• However I did realized that I probably don't need streaming anyway because it's good enough to buffer the audio first and then pass it through Web Audio.
• Web-audio support in cog / WPE is apparently not enabled by default, even if you compile WPE with support. It is enabled using command line arguments when running Cog, and you can see my current command line arguments in here.
• I have Web Audio working now in WPE, but for some reason in my latest attempt, I couldn't play a blob that was converted to an arraybuffer node. It was just silent. This requires further investigation.
• NOTE: In order to remotely inspect the latest WPE, one needs the latest WebKitGTK based browser. Easiest way to install it in a sandboxed (maybe) environment would be using Flatpak. You can install the latest Epiphany from here.