What is BACnet? A video experience

A quick and dirty explanation of what is BACnet

You now have the chance to enjoy my beautiful voice, along with my strong (and hopefully sexy) french accent.

In my opinion, the most important part of BACnet is how it opens up a whole lot of possibilities by giving access to all these sweet sensors. It’s not for nothing that I’m actively archiving gigabytes of data from various HVAC systems…

I say in this video that BACnet offers a way to free the building owners. It does, but mostly for updrades. As I’ve already mentioned, vendors can still inflict a non-negligible switch cost with their proprietary programming language. Vendors might do this totally unwillingly, but the burden is still there and must be paid.

I had a discussion recently with fellow BACneteers about this very subject. Many possible solutions are currently studied. I might summarize it on my mailing list… Meanwhile let’s just say that the statu quo might not persist.

My Video experience

I spent way too much time doing this video… at least for what I consider to be a so-so result. The next time I’ll try to jack up the audio a little, so that my voice is loud and clear.

I could also slack on the drawing board. The effect is nice, but boy is it time consuming! Each drawings must be accelerated ~8 times to have an interesting look. This means that if I want 2 minutes of doodling, I must do it for 16 minutes! That’s if I have everything right the first time (which I didn’t). Another downside of the drawings is to make sure the audio is in sync. Here again, that was quite long.

Next time I might give a try to screen capture. I have a tutorial to do… and as everyone who has written a tutorial before knows, nobody RTFM. A video might be what it takes to capture the attention.


The Importance of Data Visualization

As I’ve mentioned in previous post, I’m working on bacnethelp.com, a website dedicated to help those with a BACnet network.

The current main service is about recording the state of the network for future consultation. If you don’t really know what is a BACnet network, let’s just say that this service is comparable to recording the life signs of an individual, but for buildings. You can then see how it reacted to various stimuli, or even if it’s still alive.

However obtaining the data is only a small part of the work. The biggest part is giving it meaning. “Does the temperature changed because someone opened the window? Is this because my boiler is out of fuel?” One should seek to maximize the speed at which he can troubleshoot.

Here is a minuscule example of what a raw trend log can look like:

{:value "1159.0", :time "2009-03-02T15:45:00.000-05:00"} 
{:value "1160.0", :time "2009-03-02T16:00:00.000-05:00"} 
{:value "1161.0", :time "2009-03-02T16:15:00.000-05:00"} 
{:value "1162.0", :time "2009-03-02T16:30:00.000-05:00"} 

Quite dry isn’t it? Obviously this is not to be used directly.

Now look at how everything is instantly easier when you can see it in a graphical form.

https://frozenlock.files.wordpress.com/2012/10/wpid-objects-overview.png

Overview of multiple analog objects – bacnethelp.com

Oh my! If I had a something not working in there, I can almost guarantee it’s the first one. Did I look at the numbers? Nope, the simple shape of the graph was enough.

Of course, any children could tell you that. But why is it easier to use a graph than numeric data?

Let’s say you are looking at the number “3”. In reality your are seeing a particular shape on your screen. Your brain must then analyze it step by step to realize it is indeed the number “3”. In order, the steps taken:

  • Shape
  • Analyze
  • Realize

We shall call it the mental process.

What if you had to make a relation between many numbers? Say for example that you want to determine if a series of 20 numbers is exponential. Well now it gets more complicated, because with the realization of numbers, you will probably try to draw a graph in your head. Once you can visualize it, you will analyze it.

  • Shape
  • Analyze
  • Realize (multiplied by the 20 characters)
  • Visualize shape
  • Analyze
  • Realize

Even if we suppose that that the visualization part is effortless, we still have a total of 21 mental processes. And that’s only for 20 numbers.

If a computer were to draw the series for you, you would only be seeing a shape.

  • Shape
  • Analyze
  • Realize

A single mental process, versus 21 for the raw data. The visualization tools offers the possibilities of simplifying the data back to a single mental process regardless of the data size. The more data you have, the more the advantageous this becomes.

The same principle applies if I were to compare multiple graphs. Instead of having the previous 4 individual graphs, it could be easier to find relations by merging them together. Once again, this reduce the work our brain have to do.

https://frozenlock.files.wordpress.com/2012/10/wpid-comparison.png

Comparison of multiple analog objects – bacnethelp.com

This video (at around 2m40s) really shows the difference between the data merely ‘dumped’, versus a carefully designed representation. You will see that even if the data is the same, the difference for the user is staggering.


Leveraging the Org Mode Structure with CSS

Org-mode is an incredible tool for editing text. May it be for HTML, Latex, doc, or even plain text format.

I’ve already told that I use it to edit this blog, but I also use it for almost every text file I have to do. For example, all my readme on Github are in .org. (Even if they don’t support the org format very well. Screw Markown, org-mode is superior!)

Without any surprises, I also use it if I have to write long texts for my commercial solution BACnet Help. I can write thousands of words without having to think once about any HTML tags (DIV, SPAN, BR…). This is all handled automatically when exporting to HTML.

This is all very unsurprising for any Org-mode user. However while inspecting a page in the Chromium browser, I noticed how an Org-mode document is so well structured when exported to HTML.

Let ‘s take the Table of Contents as an example. You get it for free, with all the links and anchors. But the more important part is how the TOC is made: with successive stages of <UL> and <LI>. The structure itself is giving information. With it, we can easily determine what is a subset of a section, and what is a subset of that subset.

Here is how the plain TOC looks: https://frozenlock.files.wordpress.com/2012/10/wpid-toc.png

Now let’s leverage the <UL> and <LI> structure with a pinch of CSS.

 #text-table-of-contents ul > li > ul {
    margin-left: 0.5em;
    border-left: 1px dotted #779898;
    border-bottom-left-radius: 1em;
    padding: 0.5em;
 }

If you don’t read CSS, just know this: We select the element identified by “text-table-of-contents”, find any children that’s an unlisted list (<UL>), inside a list item (<LI>), inside another unlisted list. Then we apply some simple decorations to those.

https://frozenlock.files.wordpress.com/2012/10/wpid-toc2.png

Such a simple change, and yet a tremendous impact!


BACnet lock-in and the program object (16)

The BACnet marketing

You are in the HVAC business and trying to sell some BACnet devices. How will you market your product? Well, you’ll probably talk about how this protocol protect the user from being victim of a vendor lock-in.

Switching manufacturer

How wonderful! You can tell the customer how easy it will be for him to switch to another company if you ever let him down! But will it be so easy? Besides the various BACnet variants that might be a compatibility problem, what could be a kick in the teeth? Let’s make a quick check list of what you need in order to perform a vendor transfer:

  • Access to the controller I/Os? Yes;
  • Access the trend logs? Yes;
  • Access to alarms? Yes;
  • Access to the programs? NO!

Wait what? So the user can access every I/Os in the building, but can’t change his system behaviour. So, besides monitoring, what can he do?

“Sure, you can reprogram your controllers, but you’ll need to use our closed source interface application in order to change anything in your programs. We will gladly let you buy one for the modest amount of <insert stupid locked-in price>$ and your soul for eternity.”

This, my friends, is a lock-in.

It is also the moral fight that the Free Software Foundation and the open source projects under the GNU licence are trying to win. You should not be a slave of your programs (nor should your clients).

Priority is power

Not only can’t the user change the programs, he is most likely to be unable to change the programs priorities. This means that even if the user were to try to do a tug of war with the existing controllers by adding a new one with a whole new program in it, the new commands could be completely ignored! If the old devices have higher priorities than your the new one, it’s game over!

Towards a BACnet language standard?

The obvious solution (but less obvious to implement) is to have a language standard which could be read and edited by everyone. Obviously I would go for a lisp-like language, because I think it’s easier to grasp for the everyman. Anyone who have done primary school mathematics can understand how the parentheses are evaluated.

A language standard, while possible, is highly unlikely. The amount of work and cooperation between the many manufacturers would be non-trivial to say the least. In addition, if the program standard was ill-designed, it could drag down the entire BACnet standard with it.

Plain text program

There is however one little change in the standard that could help greatly. What if the programs were plain text and accessible through the BACnet network? Then every workstation would be able to at least open the program and let the user study it. Bonus point if they can also modify it.

The Program object type defines a standardized object whose properties represent the externally visible characteristics of an application program. In this context, an application program is an abstract representation of a process within a BACnet Device, which is executing a particular body of instructions that act upon a particular collection of data structures. The logic that is embodied in these instructions and the form and content of these data structures are local matters. – BACnet Standard 135-2004

Instead of making the content of the programs “local matters”, it could be a property, such as the description.

I can understand how, when the BACnet draft was elaborated, the controllers couldn’t have hold the plain text version of their programs, but today there’s no excuse.

Towards a truly open solution!


Emacs: Unit Converter for the Engineer

How many cubic centimeters in a liter? How many feet in a kilometer? What about gallons per minute and liters per second?

Every single engineer needs to convert units. What may change is how he does it.

  1. Find an online converter / ask Google;
  2. Use a desktop application, such as convert; https://frozenlock.files.wordpress.com/2012/08/wpid-convert.png
  3. Or do it on-the-fly with Emacs!

Now, you may ask:

“But why on earth would I ever consider Emacs to convert my units?”

Because Emacs has an amazing hidden gem inside it: Calc. Fear not, there is no need to learn yet another tool only to convert some units; I’ve package everything needed in a little .el file.

Once this is installed, you can ask Emacs, on-the-fly, to convert some units. Here is an example:

Old value and unit (type C-z for unit list) :

100oz RET

to unit (type C-z for unit list) : g

g RET

100oz converted to g = 2834.9523125 g

Pretty nice, isn’t it?

But this is far from it. What’s really handy is how you can input usual prefixes, such as k (kilo), m (milli), y (yocto)… without any special care.

Old value and unit (type C-z for unit list) :

kg RET

to unit (type C-z for unit list) :

g RET

kg converted to g = 1000 g

Tada!

What if you need units not yet defined in Calc? Simply add your own!

Here are some units I’ve added to be more helpful in a HVAC context:

;;; Add some custom units often used in HVAC
(eval-after-load "calc-units"
  '(progn
     (setq math-additional-units
           '((gpm "gal / min" "Gallons per minute")
             (ls "l / s" "Liters per second")
             (cfm "ft^3 / min" "Cubic feet per minute"))
           math-units-table nil)))

Finally, to make sure you take Emacs and Calc seriously, here is the unit list (obtained with M-x calc-view-units-table):

Calculator Units Table:

(All definitions are exact unless marked with an asterisk (*).)

Unit    Type  Definition                  Description

 gpm      U   gal / min                   Gallons per minute
 ls       U   l / s                       Liters per second
 cfm      U   ft^3 / min                  Cubic feet per minute

 m            m                           Meter (base unit)
 in           2.54 cm                     Inch
 ft           12 in                       Foot
 yd           3 ft                        Yard
 mi           5280 ft                     Mile
 au           149597870691 m (*)          Astronomical Unit
 lyr          c yr                        Light Year
 pc           3.0856775854 10^16 m (*)    Parsec
 nmi          1852 m                      Nautical Mile
 fath         6 ft                        Fathom
 fur          660 ft                      Furlong
 mu           1 um                        Micron
 mil          (1/1000) in                 Mil
 point        (1/72) in                   Point (1/72 inch)
 Ang          10^-10 m                    Angstrom
 mfi          mi + ft + in                Miles + feet + inches
 texpt        (100/7227) in               Point (TeX conventions)
 texpc        12 texpt                    Pica
 texbp        point                       Big point (TeX conventions)
 texdd        (1238/1157) texpt           Didot point
 texcc        12 texdd                    Cicero
 texsp        (1/65536) texpt             Scaled TeX point

 hect         10000 m^2                   Hectare
 a            100 m^2                     Are
 acre         (1/640) mi^2                Acre
 b            10^-28 m^2                  Barn

 L            10^-3 m^3                   Liter
 l            L                           Liter
 gal          4 qt                        US Gallon
 qt           2 pt                        Quart
 pt           2 cup                       Pint
 cup          8 ozfl                      Cup
 ozfl         2 tbsp                      Fluid Ounce
 floz         2 tbsp                      Fluid Ounce
 tbsp         3 tsp                       Tablespoon
 tsp          4.92892159375 ml            Teaspoon
 vol          tsp+tbsp+ozfl+cup+pt+qt+gal Gallons + ... + teaspoons
 galC         galUK                       Canadian Gallon
 galUK        4.54609 L                   UK Gallon

 s            s                           Second (base unit)
 sec          s                           Second
 min          60 s                        Minute
 hr           60 min                      Hour
 day          24 hr                       Day
 wk           7 day                       Week
 hms          wk + day + hr + min + s     Hours, minutes, seconds
 yr           365.25 day                  Year (Julian)
 Hz           1 / s                       Hertz

 mph          mi / hr                     Miles per hour
 kph          km / hr                     Kilometers per hour
 knot         nmi / hr                    Knot
 c            299792458 m / s             Speed of light

 ga           9.80665 m / s^2             "g" acceleration

 g            g                           Gram (base unit)
 lb           16 oz                       Pound (mass)
 oz           28.349523125 g              Ounce (mass)
 ton          2000 lb                     Ton
 tpo          ton + lb + oz               Tons + pounds + ounces (mass)
 t            1000 kg                     Metric ton
 tonUK        1016.0469088 kg             UK ton
 lbt          12 ozt                      Troy pound
 ozt          31.10347680 g               Troy ounce
 ct           0.2 g                       Carat
 u            1.660538782 10^-27 kg (*)   Unified atomic mass

 N            m kg / s^2                  Newton
 dyn          10^-5 N                     Dyne
 gf           ga g                        Gram (force)
 lbf          ga lb                       Pound (force)
 kip          1000 lbf                    Kilopound (force)
 pdl          0.138254954376 N            Poundal

 J            N m                         Joule
 erg          10^-7 J                     Erg
 cal          4.1868 J                    International Table Calorie
 calth        4.184 J                     Thermochemical Calorie
 Cal          1000 cal                    Large Calorie
 Btu          1055.05585262 J             International Table Btu
 eV           ech V                       Electron volt
 ev           eV                          Electron volt
 therm        105506000 J                 EEC therm
 invcm        h c / cm                    Energy in inverse centimeters
 Kayser       invcm                       Kayser (inverse centimeter energy)
 men          100 / invcm                 Inverse energy in meters
 Hzen         h Hz                        Energy in Hertz
 Ken          k K                         Energy in Kelvins
 Wh           W hr                        Watt hour
 Ws           W s                         Watt second

 W            J / s                       Watt
 hp           550 ft lbf / s              Horsepower
 hpm          75 m kgf / s                Metric Horsepower

 K        K   K                           Degree Kelvin (base unit)
 dK       K   K                           Degree Kelvin
 degK     K   K                           Degree Kelvin
 dC       C   K                           Degree Celsius
 degC     C   K                           Degree Celsius
 dF       F   (5/9) K                     Degree Fahrenheit
 degF     F   (5/9) K                     Degree Fahrenheit

 Pa           N / m^2                     Pascal
 bar          10^5 Pa                     Bar
 atm          101325 Pa                   Standard atmosphere
 Torr         (1/760) atm                 Torr
 mHg          1000 Torr                   Meter of mercury
 inHg         25.4 mmHg                   Inch of mercury
 inH2O        2.490889 10^2 Pa (*)        Inch of water
 psi          lbf / in^2                  Pounds per square inch

 P            (1/10) Pa s                 Poise
 St           10^-4 m^2 / s               Stokes

 A            A                           Ampere (base unit)
 C            A s                         Coulomb
 Fdy          ech Nav                     Faraday
 e            ech                         Elementary charge
 ech          1.602176487 10^-19 C (*)    Elementary charge
 V            W / A                       Volt
 ohm          V / A                       Ohm
 mho          A / V                       Mho
 S            A / V                       Siemens
 F            C / V                       Farad
 H            Wb / A                      Henry
 T            Wb / m^2                    Tesla
 Gs           10^-4 T                     Gauss
 Wb           V s                         Weber

 cd           cd                          Candela (base unit)
 sb           10000 cd / m^2              Stilb
 lm           cd sr                       Lumen
 lx           lm / m^2                    Lux
 ph           10000 lx                    Phot
 fc           lm / ft^2                   Footcandle
 lam          10000 lm / m^2              Lambert
 flam         (1 / pi) cd / ft^2          Footlambert

 Bq           1 / s                       Becquerel
 Ci           37 10^9 Bq                  Curie
 Gy           J / kg                      Gray
 Sv           Gy                          Sievert
 R            258 10^-6 C / kg            Roentgen
 rd           (1/100) Gy                  Rad
 rem          rd                          Rem

 mol          mol                         Mole (base unit)

 rad          rad                         Radian (base unit)
 circ         2 pi rad                    Full circle
 rev          circ                        Full revolution
 deg          circ / 360                  Degree
 arcmin       deg / 60                    Arc minute
 arcsec       arcmin / 60                 Arc second
 grad         circ / 400                  Grade
 rpm          rev / min                   Revolutions per minute

 sr           sr                          Steradian (base unit)

 h            6.62606896 10^-34 J s (*)   Planck's constant
 hbar         h / 2 pi                    Planck's constant
 mu0          4 pi 10^-7 H / m            Permeability of vacuum
 eps0         1 / mu0 c^2                 Permittivity of vacuum
 G            6.67428 10^-11 m^3/(kg s^2) (*) Gravitational constant
 Nav          6.02214179 10^23 / mol (*)  Avogadro's constant
 me           9.10938215 10^-31 kg (*)    Electron rest mass
 mp           1.672621637 10^-27 kg (*)   Proton rest mass
 mn           1.674927211 10^-27 kg (*)   Neutron rest mass
 mmu          1.88353130 10^-28 kg (*)    Muon rest mass
 Ryd          10973731.568527 /m (*)      Rydberg's constant
 k            1.3806504 10^-23 J/K (*)    Boltzmann's constant
 alpha        7.2973525376 10^-3 (*)      Fine structure constant
 muB          927.400915 10^-26 J/T (*)   Bohr magneton
 muN          5.05078324 10^-27 J/T (*)   Nuclear magneton
 mue          -928.476377 10^-26 J/T (*)  Electron magnetic moment
 mup          1.410606662 10^-26 J/T (*)  Proton magnetic moment
 R0           8.314472 J/(mol K) (*)      Molar gas constant
 V0           22.710981 10^-3 m^3/mol (*) Standard volume of ideal gas


Unit Prefix Table:

 Y     10^24      Yotta
 Z     10^21      Zetta
 E     10^18      Exa
 P     10^15      Peta
 T     10^12      Tera
 G     10^9       Giga
 M     10^6       Mega
 k K   10^3       Kilo
 h H   10^2       Hecto
 D     10^1       Deka
 d     10^-1      Deci
 c     10^-2      Centi
 m     10^-3      Milli
 u     10^-6      Micro
 n     10^-9      Nano
 p     10^-12     Pico
 f     10^-15     Femto
 a     10^-18     Atto
 z     10^-21     zepto
 y     10^-24     yocto


BACnet’s trend log is dead, long live the trend log!

Trend log

BACnet’s object #20, the trend log, is an object in which you can store another object present value at regular time interval, or on change-of-value trigger. It is invaluable when the need to debug a system arises (or to prove to a client that, no, it’s not cold. He simply is hungry and should eat something).

https://frozenlock.files.wordpress.com/2012/07/wpid-trendlog.png

A typical trend log, as shown on bacnethelp.com

Unfortunately, there’s something really bad with trend logs. Really, really really bad. You need to configure them before a problem occurs. For every IOs you want to monitor, you have to create a trend log object and make sure it’s logging the correct IO. (Unless of course you are using my quickly hacked together bacnet-io-creator, which creates a trend log for every IOs.) Another downfall is that logging an object takes non-negligible space in a controller. Thus, you are often really limited in the number of object you can monitor, in addition to the length of time you can monitor it.

There is of course some workaround, such as external databases, which will download the trend logs data at a given interval. Still, the trend log needs to exist for this to work. In addition, this simple step isn’t free even if you use your own SQL database. Again, BACnet proves itself to be very costly!

Continuous logging

With all the disk space available today, why should we limit ourselves with a couple hundred datapoints? Why choose between a temperature and a power consumption? Can’t we just log every friggin thing? Well yes, we can.

Introducing the BACnet Help Logger Vigilia. It’s simple. At every given interval of time, it takes a snapshot of the BACnet network and saves everything1. Now, when a problem arises, you can check the historical data of every BACnet objects regardless of the existence of a trend log object!

Everything is sent to the webserver, which means there’s nothing coming in the user network; there is absolutely no security risk for the user!

The bandwidth usage is also minimum, as the data is compressed and the logger only connects once every hour to send the last hour records2.

And of course, the necessary fail-safes are in place to insure that no data is lost even in case of Internet connection problems. As long as the logger is unable to connect to the server, it keeps the data locally. When the connection is re-established, it then sends everything back to the server. Chances are that the user will never notice anything!

With all this, the trend log object is rendered obsolete! Yet, its spirit lives on. We now have the opportunity to log everything. Long live the trend log!

Footnotes:

1 except trend logs; they are made redundant by this feature

2 by default there’s 6 scans per hour


Url-retrieve and JSON API

Update 2012-10-25 Thu

Philipp Moeller posted a better version of the code in the comments. You might want to take a look.

JSON API

This thing is everywhere. Perhaps one day it will be Clojure maps, which would be quite an improvement. But until then, we have to cope with it.

But what is it really?

An application programming interface (API) is a specification
intended to be used as an interface by software components to
communicate with each other.
Wikipedia

Here is an example to help you wrap your head around it. Say I have an awesome website which offers you to scan a BACnet network and keep the data in a nice database (shameless plug https://bacnethelp.com). Not everyone want to go to the site to consult their data. Some want to be able to manipulate it as they wish, which is quite hard when the data is stuck in an html format. The solution is to offer some URLs intended only for machines. No pretty formatting, no page layout, no nothing. Simply the pure data (which, to programmers, is like pron).

JSON is the format in which this data is represented. You can find more on this website, but this is mostly irrelevant for this post, because we will discover that Emacs already has everything it needs!

Emacs can get data directly from URLs

Yes, I know, this is common knowledge… and yet when I researched the url-retrieve function recently, I found my blog in the top results, only because I mentioned it when I spoke about ELPA.

With this limited information, it is possible that some poor souls are missing on this great Emacs’ feature. Say you want to get some info from google.com… no, let’s strech it a little, https://google.ca (that’s right, with encrypted connection)! An easy way would be to do this:

(url-retrieve-synchronously "https://google.ca")

This function will go to the given URL, retrieve the data and stuff it in a buffer; in this case #<buffer *http http://www.google.ca:443*>. Of course, we rarely want to leave this buffer alone, we need this data. To catch the returned buffer, simply use a let function

  (let ((buffer (url-retrieve-synchronously url))
        ...

Now as long as we remain in this let function, we can get the google buffer with the symbol buffer. Let’s skip some steps and go to the interesting part:

(defun get-json (url)
  (let ((buffer (url-retrieve-synchronously url))
        (json nil))
    (save-excursion
      (set-buffer buffer)
      (goto-char (point-min))
      (re-search-forward "^$" nil 'move)
      (setq json (buffer-substring-no-properties (point) (point-max)))
      (kill-buffer (current-buffer)))
    json))

This function will return any content from a given url.

Ok, we have the JSON formatted data, what can we do with it? Of course, Emacs already has a JSON parser. Simply make sure you require ‘json before going any further.

(defun get-and-parse-json (url)
  (let ((json-object-type 'plist))
    (json-read-from-string 
     (get-json url))))

Here I’ve choosen to parse it as a PLIST, be it could as well be an ALIST, or even an hash-table!

And because we are in Emacs, we loooove lists. Here is a little function from my bhelp.el to convert any vectors into lists:

(defun bhelp-listify (arg)
  "Convert any remaining vector into a list"
  (mapcar (lambda (x) (if (or (vectorp x) (listp x))
                          (bhelp-listify x)
                        x)) arg))

That’s it! All the tools required to get data from a JSON API from inside Emacs! You can look to the source code of bhelp.el if you need to see how these functions are used in a real live application, but there really isn’t much more. Getting the data from within Emacs is really straightforward.

Another shamless plug: I use this to get info from https://bacnethelp.com. I can then use the data in various Emacs mode (such as Org-mode) and compare the BACnet networks with the initial designs. I can also make customized reports to send to the client, or the IT department (they might want to make sure we didn’t take any IP address).

Emacs make me feel like a have a humongous reproductive instrument.