Differences

This shows you the differences between two versions of the page.

--- thesis:work-journal [2019/03/23 23:14]
avnerus
+++ thesis:work-journal [2019/03/24 11:11]
avnerus
@@ Line 317: / Line 317: @@
 A long overdue update.
+The work on Hitodama has been more sporadic in the last couple of weeks, mainly due to congestion of tasks coming from my other work. Nevethelss progress, has been made, and work was mainly about setting up a foundatoin for Hitodama's audio in WPE.
+==== Toolkits ====
+In the last update regarding the gstreamer saga I hinted at the possibility of using a different toolkit than uclibc, namely musl, to solve the dynamic linking issues I had with gstrpicamsrc. Since then the following developments unfolded:
+**TLDR** : I am now more involved in WPE development, know a lot more about toolkit possibilities, running the latest WPE and looking toward 64bit support.
+  * I switched from uclibc to musl, and indeed it solved the dynamic linking problem.
+  * In order to compile wpewbkit with musl, I had to migrate some patches that I found are being used in [[https://github.com/WebPlatformForEmbedded/meta-wpe|here]]. My patches are now located [[https://github.com/Avnerus/softbot-buildroot/tree/softbot/package/wpewebkit/musl-patch-backup|here]].
+  * Since then I got more involved in WPE, and when compiling the latest unstable version I had some issues with compiling JIT (Optimizied Javascript native compiler) support for RPi3 - ARMv7/8. While JIT compiles all of the javascript code into native code, the CLoop backend is more of an interpreter. There is some explanation and comparison of the backends in [[https://community.arm.com/developer/tools-software/tools/b/tools-software-ides-blog/posts/comparison-of-javascript-execution-modes-of-the-webkit-browser-engine|here]].
+  * To avoid more issues with toolkit compatibility - I ended up switching to the 'bloated' GLIBC. It should be noted that although glibc is bloated, it is still very fast and optimized, and in many cases performs faster than musl. More about this [[http://www.etalabs.net/compare_libcs.html|here]].
+  * The new Webkit supports JIT on ARM in two types of insturction sets - ARM64 or ARM THUMB2.
+  * For this project I am compiling for 32bit - This is because the Raspberri Pi userland kernel (the linux drivers for RPi specific hardware) doesn't yet fully support 64bit. There is an open issue about it [[https://github.com/raspberrypi/userland/issues/314|here]].
+  * There are two main usages in my project for the userland. One is the Raspicam - using an RPi driver called MMAL and another is hardware graphics accelration driver called VC4.
+  * MMAL does not yet support 64bit, BUT - the camera can be accessed via an open linux driver V4L2 which does support 64bit.
+  * VC4 closed source drivers by RPi do not support 64bit, but there is an open source driver VC4 KMS. I'm not sure what's the compatilibity of this driver with the WPE backend I'm using - RDK. However there is another more standard backend called FDO which uses Mesa and should support 64bit, but is probably not as fast and robust.
+  * Currently I'm not pursuing 64bit further, but this may start to be an issue with support for WebAssembly. Apparently WebAssembly requires 64 bit - which I think is a requirement of the B3 JIT. There is more about this [[https://webkit.org/blog/7691/webassembly/|here]].
+  * For compiling the new webkit I still had issues with supporting THUMB2. The support for the instruction set was not detected. I solved it by adding the following config to Buildroot and then recompiling the entire toolkit:
+<code>
+BR2_TARGET_OPTIMIZATION="-mthumb -mfpu=neon-fp-armv8 -mtune=cortex-a53"
+</code>
+==== Audio / Voice ====
+I have been working on Hitodama's web based voice. The current strategy for implementation is as follows:
+  - Translate the controller's text if requested - Using Google Cloud Translate (Which I belive is the best translate API).
+  - Generate the voice using Microsoft's Cloud Speech (Which supports much more langauges than Google, including Hebrew, Arabic and Finnish ).
+  - Apply Web Audio filters on the voice to make it gender neutral and nonhuman.
+This was fairly easy to make for the desktop, currently using Granular pitch shifting for Web Audio from [[https://github.com/urtzurd/html-audio|here]]. However, as what is becoming a pattern, this became much more complicated when trying to run on WPE:
+  * The first problem was that there is a bug with streaming //audio src>// elements, having the audio loop indefinitely. After discussing on #webkit I submitted a report [[https://bugs.webkit.org/show_bug.cgi?id=196000|here]],  which was then marked as a duplicate of [[https://bugs.webkit.org/show_bug.cgi?id=189471|this bug]]. Now Phillipe made a patch and I was able to apply it, and waiting to test it.
+  * However I did realized that I probably don't need streaming anyway because it's good enough to buffer the audio first and then pass it through Web Audio.
+  * Web-audio support in cog / WPE is apparently not enabled by default, even if you compile WPE with support. It is enabled using command line arguments when running Cog, and you can see my current command line arguments in [[https://github.com/Avnerus/softbot-buildroot/blob/softbot/softbot-scripts/softbot-browser.sh#L4|here]].
+  * I have Web Audio working now in WPE, but for some reason in my latest attempt, I couldn't play a blob that was converted to an arraybuffer node. It was just silent. This requires further investigation.
+  * NOTE: In order to remotely inspect the latest WPE, one needs the latest WebKitGTK based browser. Easiest way to install it in a sandboxed ([[http://flatkill.org/|maybe]]) environment would be using Flatpak. You can install the latest Epiphany from [[https://webkitgtk.org/epiphany-tech-preview|here]].

User Tools

Site Tools

Differences

Page Tools