20 January 2017

What Have I Been Doing Lately?

My last posting was in October of 2016, and since then I have been working on moving the code and lessons I learned from the razmq project to a new project called Python Banyan. The code has been complete for some time now, but I have been struggling to get it documented. I am pleased to say that the documentation is well under way and hopefully will be complete in about 2 weeks time (hopefully shorter). You can view a draft of  the User's Guide here, but be mindful it is subject to change. 


Python Banyan is a lightweight, reactive framework used to create flexible, non-blocking, event driven, asynchronous applications. It was designed primarily to implement physical computing applications for devices such as the Raspberry Pi and Arduino, but it is not limited to just that domain, and may be used to create application in any domain.
Most traditional physical computing libraries or frameworks use an object oriented model that results in a single, tightly coupled, monolithic executable image. Python Banyan uses an extension of the object oriented model, called the component based model. A component based application is comprised of a set of independent, loosely coupled modules. Functionality is easily added to a Python Banyan application, and in fact it may be added to a fully running system without the need to recompile or reboot.
Because each module is a self contained entity, applications can be created with a set of modules that use different versions of Python, and in fact, you can even add modules written in other computer languages.
In addition, the modules may be run on a single computer, or may be distributed across multiple  computers running different operating systems, without having to change a single line of code.

A Little More Detail

The Python Banyan Framework consists of a single, simple base class. All Banyan compatible components inherit from this class. When a Banyan component is first invoked, it automatically connects to a common shared software backplane. All of the complexity of managing connections is hidden within and handled by the base class. All Banyan modules exchange information with each another by sending or publishing user defined protocol messages via the backplane. All routing and message buffering is automatically handled by the Framework. Each Banyan component can "publish" messages, "subscribe" to receive specific messages or both publish and subscribe messages. A Banyan component is not limited as to how many types of messages it may subscribe to.
Because Banyan messages are not computer language specific, components written in other computer Languages, such as JavaScript, can be used within a Python Banyan application. A simple JavaScript demo is provided in the examples section below.
Python Banyan takes full advantage of the ZeroMQ networking library that not only provides connectivity, but in addition acts as a concurrency framework. All of this is handled transparently by the Python Banyan base class. If your application requires additional concurrency support, you are free to choose whatever works best for your application, such as a multi-threading or a Python asyncio approach. User defined messages are prepared for transport across the network by wrapping them in theMessage Pack format.
The complexities of MessagePack are handled transparently by the base class, both for transmitting and receiving messages.

A User's Guide Is Provided. Here is what the guide covers:

Chapter 1 is an introduction to developing with Python Banyan. An application tailored specifically for this purpose will be presented in detail.
In chapter 2, a demonstration of, and a discussion about creating physical computing components with Python Banyan.
To demonstrate Python Banyan's flexibility, applications will be built from Python 2 components, Python 3 components and even a JavaScript component, all cooperating and communicating with each other within a single Python Banyan application.
Python Banyan applications may be distributed across multiple computers, all without changing a single line of code. Using the components created in this guide, all of the components will first be launched to run on a Raspberry Pi, and then then the components will be re-distributed across a Linux Unbuntu PC, a Windows PC and the Raspberry Pi without having to change a single line of code.
In Chapter 3, we will add components to control an 8x8 bicolor LED matrix connected to a Raspberry Pi. A demonstration of using Python Banyan to control an i2c device will be presented.
If you need to control multiple i2c devices sharing the same i2c pins, Python Banyan provides the concurrency support for you to do so.
Chapter 4 will discuss how to use Python's setup tools to convert a Banyan component into an executable file and have it automatically installed on the execution path.

2 October 2016

Writing The Software To Control The Raspberry RedBot 

Part 2

I have completed the bulk of the coding for razmq and the results are in!

I wanted to share this information now. The code will be published in a short while, after some cleanup and additional testing.

The following screen shots were taken on a Raspberry Pi 3 running the latest Raspbian image, Raspbian Jessie with Pixel

The first is "quiescent" mode. The pigpio daemon, pigpiod,  the xrdp  daemon and htop  are running.

CPU utilization is at 4.2% and Memory utilization is 129 of 925 MB. CPU utilization varies and can go as high as 6.1% for this scenario.

Starting all the processes that monitor and control the Raspberry RedBot we see the following:

CPU utilization rises to 6.4% and Memory utilization is 294 of 925 MB. CPU utilization varies from about 4.5 to 6.5%.

The Accelerometer (ADXL345) and the  3 Line Followers (PCF8591) are continuously streaming data via i2c in this scenario.

Finally, we run the motors at full speed. This causes the encoders to add their streaming information into the mix. The motors at full speed run at 140 RPM. Each encoder sensor generates 192 "ticks" with each wheel revolution. Since this level of resolution was not necessary for the purposes of this project, the number of ticks was throttled  by a factor of 6. So now each wheel revolution is limted to generating 32 "tick" notifications per wheel (but 192 are still being generated). This yields a 1/8 inch or 3.175 mm distance resolution that is reported on the GUI screen. Here are the results when the motors are running full speed and all other sensors and actuators are active:

CPU utilization varies from about 16% to 26% and there is no additional memory utilization while the motors are running.

Here is a screen shot of the Kivy based GUI running on my Ubuntu based desktop PC.

Let's summarize what is running when the "motor run" numbers were captured:

  1. Data is continuously being streamed by the accelerometer, 3 line followers and 2 wheel encoders.
  2. All other actuators (LED, Buzzer) and sensors (Push Button Switch, two mechanical bumpers) are ready to be activated or provide reports in a non-blocking fashion.
  3. There are currently 12 separate processes running to monitor and control the Raspberry RedBot.

Some Important Facts About The Framework Used to Create razmq

  1. Everything is written using Python 3.4.2, the version that comes with the Raspberry Pi image.
  2. All processes are activated using the lunch process launcher - simplifies the start up process.
  3. Processes may be started, stopped, added and removed at will, with no system crashes. Effectively providing a software backplane with hot swapping capabilities.
  4. Most of the processes were built using a base class that encapsulates and hides the complexities (and there really are very little) of ZeroMQ and MessagePack.
  5. For the purposes of this article, all processes, with the exception of the GUI, were run on a single Raspberyy Pi 3. However, during development some processes were run on a Pycharm IDE in Ubuntu on my desktop.  The pigpio dependent processes were remotely debugged using the same PyCharm session. 
  6. Without any coding changes, any process may be run on any computer residing on  a single LAN.
  7. Module interfaces are defined as language independent protocol messages. The messages are asynchronously published to the network and may be subscribed to by any other process. This is done by transporting the MessageQueue messages using ZeroMQ.  The result of using this mechanism is a simplified, non-blocking IPC (inter process communication) scheme with no data needed to be protected or locked. A lot more about this will also follow in a future article.
  8. The system is totally non-blocking without the need of mechanisms like the Python asyncio or threading or multiprocessing modules.
  9. Testing new modules is easy to do. Their published protocol messages may be observed in real-time using a simple monitor process. Building drivers for these modules is also quick and easy as a result of using the encapsulation base class (more on this will follow in future articles as well).
  10. All modules use a consistent and common coding pattern.

So What Does This All Mean?

The razmq project was built for a specific set of hardware actuators and sensors. However, the framework is easily adaptable to be used for other projects. New features can easily be added to existing projects and enabled on an already executing system. New projects can be created using a simple and consistent coding pattern.

The next several articles will explain in detail how razmq works and how you can use its framework in your own projects. 

Stay tuned!

Writing The Software To Control The Raspberry RedBot

18 September 2016

Now that the hardware has been completed and is in place, it is time to start writing the software. When working on a new project I like to take a "Mise en place" approach - that is getting everything organized and in place and ready to go. First I create a Github page for the project and update the Wiki pages as the project progresses. For this project, I created a Github repository called razmq

Next, I identify any additional software packages I might need and install them. The additional software requirements for razmq are identified here

After that, I  write a set of routines to exercise the hardware.- a hardware baseline so to speak. In this way, I am able to verify hardware functionality at any time.

An Experimental Software Architectural Approach

As I have done in several of my previous projects, Python 3 will be used. I believe it is better to look towards the future of the language rather than have to worry about maintaining the legacy of Python 2. 

I will still maintain an asynchronous event driven design approach, however, I would like to simplify things and provide concurrency without the use of either the Python asyncio module or the threading module. To do this, I will be dividing support for a given hardware functionality into 2 separate processes. The first process will present a "user" view of the hardware. For example, for motor control, this process will describe motor motion in abstractions such as direction and speed. These abstractions will be translated into MessagePack messages and then "published" using the ZeroMQ publisher/subscriber pattern. Another process will be created and it will be a subscriber for these messages. It will translate the abstractions into pigpio calls to directly control a motor controller device. In this way, I can maintain an abstract view of the world and swap in specific hardware at any time to implement that abstraction.

To simplify the networking connections between all of these processes, a ZeroMQ forwarder will be used. The forwarder uses a "well-known" IP address/port for all publishers to publish their messages, and another "well-known" IP address/port for subscribers to subscribe to receive messages. There are no configuration files to maintain and each process uses a consistent methodology to both publish and subscribe to messages. To make the coding easier, a base class will be provided to encapsulate many of the ZeroMQ and MessagePack details. This base class can be inherited by the classes that provide either the user or hardware view of the system. 

The experimental part of this exercise is to determine if the memory and CPU utilization is viable while having multiple concurrent processes running.

If this all seems a little abstract, in my next posting I will be describing the details of supporting a simple hardware device, the LED, using these methods. Hopefully that will demystify the described approach. It really is simpler than it sounds and provides a very flexible development environment.

Until next time ....
9 September 2016

It's Alive!

Brain Transplant A Success.

I've successfully replaced the original Arduino based processor board that comes with the RedBot with a Raspberry Pi 3.

It looks like a chaotic mess of wires, but is actually quite organized, and no GPIO pins were harmed in the process. A Pi Ez-Connect was used so that I can easily remove the Pi and use it in another project at any time. All solder connections were made on a solderable breadboard, so the Ez-Connect can be reused as well.

All of the original RedBot connectors were maintained so that the Arduino board can be restored if need be. The new brain supports all of the original features:
  • Dual Motor Drive
  • Encoders for each motor
  • Left and right mechanical bumpers.
  • On board LED.
  • On board push button switch
  • Piezo buzzer
  • Line followers
  • Accelerometer
In addition a 5 volt regulator was added to keep the Raspberry Pi 3 happy for as long as possible when running on battery power.

Current limiting resistors were also added to the bumper and push button switches. This helps protect the Pi's GPIO pins.

All in all, I am quite pleased with the outcome. 

Next up, the software!

2 September 2016

I'm Back!

I haven't posted anything in a while and thought would do a quick post about what I have been up to and what I will be posting about in the coming days and weeks.

I have been working on replacing the Arduino based controller board on my RedBot with a Raspberry Pi. Yeah, I know, RPi based robots are nothing new and maybe even boring at this point in time, but my focus has always been on software, and a RedBot with a Linux powered brain gives me the perfect platform for me to try out my software musings.

Whatever happened to Wire Wrapping?            #bringBackWireWrap

Back in the late 1970's, when I did a lot of hardware prototyping, a prototyping technique that was all the rage was wire wrapping. It seems to have disappeared off the face of the earth and I have no idea why. For those not familiar with this technique, it predated wireless breadboards and allowed one to connect components together without soldering. You simply placed a wire wrap terminal or socket on a perf board, and then using a special tool, wrapped a lead around the terminal. The connection was easy to make, easy to remove, and gave great conductivity. Here is a picture:

There is a great article on about the technique.

Now don't get me wrong, for quick a proof of concept, a wireless breadboard can't be beat, but when it comes to actually building a board that is not a printed circuit board, transferring the prototype to working design can be a challenge.

For the RedBot Raspberry Pi brain transplant, I decided to use a solder-able breadboard. Sounds like a great idea in theory, and I am using it, but for my needs it is far from perfect. First, you need to solder. The solder holes are getting smaller and my eyes are getting weaker - not a great combination. When I first started prototyping back in the vacuum tube days - 100 watt irons, massive connectors with equally massive connection points, which were all the rage, soldering was really simple. Not so today.

Also, using a wireless breadboard layout is not ideal. There is a lot of wasted real estate on the board to accommodate the connection redundancy. Again great when testing out a design or creating a temporary one, but horrible for a real world permanent solution.

In addition, I am a software guy. When I screw up the software, I rewrite a few lines and fix the problem. When I solder something incorrectly or decide there is a better way to do things, I need to find my solder sucker and pray I don't destroy the board or components.

Wire wrapping alleviated all these problems. It was simple and quick to do and just as easy to remove an errant connection. Sparkfun, Adafruit I am calling on you to  #bringBackWireWrap.

RedBot Brain Surgery

Here is a picture of my creation taken about a week ago. I have since added an LED and push button switch.

I will be connecting to the RPi using a Pi EZ-Connect.

Why a Pi?

The Raspberry Pi 3 is a cost effective platform for me to play with. First it offers a mainstream Linux OS (Debian). It has integrated WiFi and bluetooth, a fairly hefty 4 core processor, and plenty of room for additional software on the SD card.

The stability of the Pi, and its community support in my opinion, is unsurpassed. 

Using a Pi allows me to write everything in Python 3, my language of preference.

GPIO Support

I have selected the pigpio library to be the basis of low level GPIO control. This may be controversial, but I think pigpio is the best RPi GPIO control library available for Python. It supports Python 3 (as well as 2), is actively supported by its author (joan - your the best!), and its API is concise and simple to understand (even for a simpleton like me.)

RedBot Supported Features and Hardware Improvements

I will be maintaining the features of the Arduino based RedBot. I decided not to reuse the RedBot accelerometer board since that device requires a fairly complicated i2c solution. Instead, I am using the ADXL345, which allows for much simpler software.

By creating my own interface board, I also added some hardware "improvements". For example, I have added current limiting for both the push button switch and bumpers.

ZeroMQ Pub/Sub Based software Modules

The software will revolve around a ZeroMQ Pub/Sub communication pattern. This effort will not be using Python asyncio, but instead a simpler multi-process approach that uses custom protocols implemented in MessagePack- stay tuned and all will be revealed. 

A Kivy GUI

Eventually, when all else is working I will be implementing a Kivy based GUI. Kivy is a Python based GUI library that I found compelling both visually and from a programming standpoint.
Currently, I have Kivy working with ZeroMQ for the PC.

Kivy comes with a sophisticated tool called Buildozer, that ports the GUI to an Android .apk file.
Unfortunately, I have not been able to get ZeroMQ and Buildozer to play nice together. Hopefully that will be solved in the near future. 

More To Come Later ...

Version 2.15 pymata-aio


This version fully implements the StandardFirmata 2.5.3 protocol. It also includes a new version of pymata_iot. We are no longer using autobahn for websocket support, but have moved to the websockets package.

To install from pip:

pip install pymata-aio

To upgrade from an older version using pip:

pip install pymata-aio --upgrade

You can also download the .zip file, unpack and run: install

Updates Available for pymata-aio, s2aio and FirmataPlus

Pymata-aio version 2.14 and s2aio version 2.9 are now available on Pypi.

To upgrade s2aio to the latest version, in a command window type:

pip install s2aio --upgrade

This should update both s2aio and pymata_aio.

To upgrade pymata-aio, in a command window type:

pip install pymata-aio --upgrade

FirmataPlus and its variants have been updated and are baselined against StandardFirmata 2.5.3.

These libraries now support both PyMata and pymata-aio. You can download the zip file here and installation instructions can be found here.