Url-retrieve and JSON API

Update 2012-10-25 Thu

Philipp Moeller posted a better version of the code in the comments. You might want to take a look.


This thing is everywhere. Perhaps one day it will be Clojure maps, which would be quite an improvement. But until then, we have to cope with it.

But what is it really?

An application programming interface (API) is a specification
intended to be used as an interface by software components to
communicate with each other.

Here is an example to help you wrap your head around it. Say I have an awesome website which offers you to scan a BACnet network and keep the data in a nice database (shameless plug https://bacnethelp.com). Not everyone want to go to the site to consult their data. Some want to be able to manipulate it as they wish, which is quite hard when the data is stuck in an html format. The solution is to offer some URLs intended only for machines. No pretty formatting, no page layout, no nothing. Simply the pure data (which, to programmers, is like pron).

JSON is the format in which this data is represented. You can find more on this website, but this is mostly irrelevant for this post, because we will discover that Emacs already has everything it needs!

Emacs can get data directly from URLs

Yes, I know, this is common knowledge… and yet when I researched the url-retrieve function recently, I found my blog in the top results, only because I mentioned it when I spoke about ELPA.

With this limited information, it is possible that some poor souls are missing on this great Emacs’ feature. Say you want to get some info from google.com… no, let’s strech it a little, https://google.ca (that’s right, with encrypted connection)! An easy way would be to do this:

(url-retrieve-synchronously "https://google.ca")

This function will go to the given URL, retrieve the data and stuff it in a buffer; in this case #<buffer *http http://www.google.ca:443*>. Of course, we rarely want to leave this buffer alone, we need this data. To catch the returned buffer, simply use a let function

  (let ((buffer (url-retrieve-synchronously url))

Now as long as we remain in this let function, we can get the google buffer with the symbol buffer. Let’s skip some steps and go to the interesting part:

(defun get-json (url)
  (let ((buffer (url-retrieve-synchronously url))
        (json nil))
      (set-buffer buffer)
      (goto-char (point-min))
      (re-search-forward "^$" nil 'move)
      (setq json (buffer-substring-no-properties (point) (point-max)))
      (kill-buffer (current-buffer)))

This function will return any content from a given url.

Ok, we have the JSON formatted data, what can we do with it? Of course, Emacs already has a JSON parser. Simply make sure you require ‘json before going any further.

(defun get-and-parse-json (url)
  (let ((json-object-type 'plist))
     (get-json url))))

Here I’ve choosen to parse it as a PLIST, be it could as well be an ALIST, or even an hash-table!

And because we are in Emacs, we loooove lists. Here is a little function from my bhelp.el to convert any vectors into lists:

(defun bhelp-listify (arg)
  "Convert any remaining vector into a list"
  (mapcar (lambda (x) (if (or (vectorp x) (listp x))
                          (bhelp-listify x)
                        x)) arg))

That’s it! All the tools required to get data from a JSON API from inside Emacs! You can look to the source code of bhelp.el if you need to see how these functions are used in a real live application, but there really isn’t much more. Getting the data from within Emacs is really straightforward.

Another shamless plug: I use this to get info from https://bacnethelp.com. I can then use the data in various Emacs mode (such as Org-mode) and compare the BACnet networks with the initial designs. I can also make customized reports to send to the client, or the IT department (they might want to make sure we didn’t take any IP address).

Emacs make me feel like a have a humongous reproductive instrument.

5 responses to “Url-retrieve and JSON API

  • Philipp Moeller

    thanks for the code.

    I made it a little bit more performant and elegant:

    (defun get-and-parse-json (url)
    (with-current-buffer (url-retrieve-synchronously url)
    (goto-char (point-min))
    (re-search-forward “^$”)

    You can avoid most of the clutter and setq’ing by using with-current-buffer and json-read.


  • Frozenlock

    Hello Philipp,

    I wasn’t even aware of the function ‘json-read’. A very economical function indeed!

    As for ‘with-current-buffer’, I have no excuses… I’ve already written that one should user ‘with-temp-buffer’ as often as he can. Bad me!

    Thanks for keeping me on the right path!

  • Louis Dion-Marcil

    Thanks for this. Definitely helped!

  • ricardo

    Normally I do not read post on blogs, but I would like to say that this write-up very compelled me to take a look at and do so! Your writing style has been surprised me. Thanks, quite great article.

  • Finding The Closest Item From A List Of Options – Part 2 | rjfoy

    […] two functions from here, we have a way to get and parse JSON from a given […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: