Inspiration for a Dataflow Language

2013 Aug 10

Inspiration

A few years ago I was starting a project to create a data-logging and control system for my motorcycles. I didn't have a single specific purpose in mind; I wanted a flexible system that would help a rider understand and control his machine and his riding. This project was an outgrowth of both my passion and my profession; therefore, I had high standards by which I would judge my own work. I wanted a system that would be configurable/programmable in an elegant way appropriate to the types of applications for which it would be used.

A few characteristics of these types of systems are:

  • Multiple sensors provide data in parallel
  • Most sensor data is sampled at a constant rate
  • These systems run indefinitely, constantly sensing and reacting to their environments

I wanted a programming language suited to describing this type of real-time reactive systems in a concise way. I also wanted to avoid having to write procedural code, as this to me is only suited to batch processing systems.

Goals

A few goals of the language were:

  • Simple, Expressive, Beautiful
  • No Boiler-Plate Code
  • Predictable Run-Time Behavior and Timing
  • Platform Independence
  • Modularity and Composability

Some additional "nice to haves":

  • Automatically Parallelizable
  • An Isomorphic Visual Representation

In general I wanted to build something that could be used for a simple system with a few sensors, or scale up to a high-speed signal-processing system.

A Note on Surface Syntax

I spent a little time thinking about surface syntax before I stopped worrying and learned to love lisp s-expressions. This fit both my technical needs and personal aesthetic preferences perfectly. While this may not suit everybody, this has been one design decision that I have never looked back on.

Result

After some work I ended up designing a synchronous dataflow language. A program in this language is effectively a set of equations expressing the value of discrete-time output signals in terms of discrete-time input signals.

Here is what it looks like:

(imports a b)
(export c)
(def c (+ a b))

This program will consume two input signals a and b and produce an exported signal c which will be sum at each point in time of the input signals a and b.

The dataflow program itself is side-effect free. Any interaction between the dataflow program and the world happens through imports and exports. There are various ways that this dataflow language could be embedded within a larger system, but the idea is to make the system configurable/programmable much in the way Lua has been used by game programmers to make their games and game engines configurable/programmable to various degrees.

And now for a slightly more interesting example:

(include units)

(imports speed dt)
(export acceleration)

(def speed-mps
  (convert speed :mph :mps))

(def acceleration-mps2
  (/ (- speed-mps (pre speed-mps))
     dt))

(def acceleration
  (convert acceleration-mps2 :mps2 :gravity))

Here we compute acceleration from speed by looking at the difference between the current speed and the speed at the previous time-step divided by the time-step. We also do some unit conversion so that everything comes out right. The pre operator used here is a keyword that references the previous value of a signal and is part of the magic automatically handled by the system.

Here is the above program at work on some sample data from my truck:

speed | acceleration
------+-------------
 0.00 |
 0.00 |         0.00
 1.86 |         0.08
 4.35 |         0.11
 6.22 |         0.08
 8.08 |         0.08
10.57 |         0.11
14.92 |         0.20
16.16 |         0.06
18.65 |         0.11
22.37 |         0.17
26.10 |         0.17
26.72 |         0.03
26.72 |         0.00
28.59 |         0.08
30.45 |         0.08
31.08 |         0.03
31.70 |         0.03
32.32 |         0.03
32.94 |         0.03
32.94 |         0.00
33.56 |         0.03
33.56 |         0.00
33.56 |         0.00
34.18 |         0.03
35.43 |         0.06
36.05 |         0.03
36.67 |         0.03
37.29 |         0.03
37.29 |         0.00
37.29 |         0.00

So far I have been very pleased with the results. It is a fun language to develop these types of systems in. I'm not sure if anyone else has any interest in a language like this. If so, let me know.

Update 2013 Oct 30: Be sure to check out the next post in this series.