Showing posts from 2014

Three Simple Rules for Escaping Callback Hell

A lot of newcomers to Node.JS complain about "callback hell" and the "pyramid of doom" when they're getting started with the callback-driven continuation passing style.  It's confusing, and a lot of people reach for an async / flow-control module right away.  Many people have settled on using Promises, a solution that brings some unfortunate problems along with it (performance, error-hiding anti-patterns, and illusory behavior, for example).

I prefer using some simple best practices for working with callbacks to keep my code clean and organized. These techniques don't require adding any extra modules to your code base, won't slow your program down, don't introduce error-hiding anti-patterns, and don't convey a false impression of synchronous execution. Best of all, they result in code that is actually more readable and concise, and once you see how simple they are, you might want to use them, too.

Here they are:
use named functions for callba…

How Wolves Change Rivers

A beautifully filmed short video from Yellowstone National Park that reminds us of the importance of wildlife for the health of the whole planet:

How Wolves Change Rivers from Sustainable Man on Vimeo.

doxli - a help utility for node modules on command line

Quite often I fire up the node REPL and pull in some modules I've written to use on the command line. Unfortunately I often forget the exact way to call the various functions in those modules (there are a lot) and end up doing something like foo.dosomething.toString() to see the source code and recall the function signature.

In the interest of making code as "self-documenting" as possible,  I wrote a small utility that uses dox to provide help for modules on the command line. It adds a help() function to a module's exported methods so you can get the dox / jsdoc comments for the function on the command line.

So now will return the description, parameters, examples and so on for the method based on the documentation in the comments.

It's still a bit of a work in progress, but it works nicely - provided you actually document your modules with jsdoc-style comments.

All the info is here:

REST API Best Practices 4: Collections, Resources and Identifiers

Other articles in this series:
REST API Best Practices: A REST Cheat SheetREST API Best Practices: HTTP and CRUDREST API Best Practices: Partial Updates - PATCH vs. PUT RESTful APIs center around resources that are grouped into collections. A classic example is browsing through the directory listings and files on a website like When you browse the directory listing, you can click through a series of folders to download files.  The folders are collections of CentOS resource files.

In Rest, collections and resources are accessed via HTTP URI's in a similar way:

members/ -- a collection of members
members/1 -- a resource representing member #1
members/2 -- a resource representing member #2

It may help to think of a REST collection as a directory folder containing files, although its highly unlikely that the member data is stored as literal JSON files on the server. The member data should be coming from a database, but from the perspective of a REST API, it look…

Defensive Shift - Turning the Tables on Surveillance

Like many people lately, I've been pondering the implications of pervasive surveillance, "big data" analysis, state-sponsored security exploits, and the role of technology in government. For one thing, my work involves a lot of the same technology: deep packet inspection, data analysis, machine learning and even writing experimental malware. However, instead of building tools that enable pervasive government surveillance, I've built a product that tells mobile smartphone users if their device, or a laptop connected to it, has been infected with malware, been commandeered into a botnet, or come under attack from a malicious website, and so on.  I'm happy to be working on applying some of this technology in a way that actually benefits regular people. It feels much more on the "good side" of technology than on the bad side we've been hearing so much about lately.

Surveillance of course has been in the news a lot lately, so we're all familiar with…

Repackaging node modules for local install with npm

If you need to install an npm package for nodejs from local files, because you can't or prefer not to download everything from the repo, or you don't even have a network connection, then you can't just get an npm package tarball and do `npm install <tarball>`, because it will immediately try to download all it's dependencies from the repo.

There are some existing tools and resources you can try:

npmbox - gist - npm issue -
I found all of these a bit over-wrought for my taste. So if you prefer a simple DIY approach, you can simply edit the module's package.json file, and copy all of its dependencies over to the "bundledDependencies" array, and then run npm pack to build a new tarball that includes all the dependencies bundled inside.

Using `forever` as an example:
make a directo…

JavaScript's Final Frontier - MIDI

JavaScript has had an amazing last few years. Node.JS has taken server-side development by storm. First person shooter games are being built using HTML and JavaScript in the browser. Natural language processing and machine learning are being implemented in minimalist JavaScript libraries. It would seem like there's no area in which JavaScript isn't set blow away preconceptions about what it can't do and become a major player.

There is, however, one area in which JavaScript - or more accurately the web stack and the engines that implement it - has only made a few tentative forays.  For me this represents a final frontier; the one area where JavaScript has yet to show that it can compete with native applications. That frontier is MIDI.

I know what you're probably thinking. Cheesy video game soundtracks on your SoundBlaster sound card. Web pages with blink tags and bad music tracks on autoplay. They represent one use case where MIDI was applied outside of its original in…

REST API Best Practices 3: Partial Updates - PATCH vs PUT

This post is a continuation of REST API Best Practices 2: HTTP and CRUD, and deals with the question of partial updates.

REST purists insist that PATCH is the only "correct" way to perform partial updates [1], but it hasn't reached "best-practice" status just yet, for a number of reasons.

Pragmatists, on the other hand, are concerned with building mobile back-ends and APIs that simply work and are easy to use, even if that means using PUT to perform partial updates [2].

The problems with using PATCH for partial updates are manifold:
Support for PATCH in browsers, servers and web application frameworks is not universal. IE8, PHP, Tomcat, django, and lots of other software has missing or flaky support for it. So depending on your technology stack and users, it might not even be a valid option for you.Using the PATCH method correctly requires clients to submit a document describing the differences between the new and original documents, like a diff file, rather th…

REST API Best Practices 2: HTTP and CRUD

This post expands a bit further on the REST API Cheat Sheet regarding HTTP operations for Create / Read / Update / Delete functionality in REST APIs.

APIs for data access and management are typically concerned with four actions (the so-called CRUD operations):
Create - the ability to create a resourceRead - the ability to retrieve a resourceUpdate - the ability to modify a resourceDelete - the ability to remove a resource
CRUD operations don't have a perfect, 1-to-1 mapping to HTTP methods, which has led to different opinions and implementations, but the following list represents best practice as I see it in the industry today, and follows the HTTP specification:

CRUD Operation    HTTP MethodCreatePOSTReadGETUpdatePUT and/or PATCHDeleteDELETE
To reiterate, HTTP methods can be used to implement CRUD oprations as follows:
POST - create a resourceGET - retrieve a resourcePUT - update a resource (by replacing it with a new version)*PATCH - update part of a resource (if available and ap…

REST API Best Practices: a REST Cheat Sheet

I'm interested in REST API design and identifying the best practices for it. Surprisingly, a lot of APIs that claim to be RESTful, aren't. And the others all do things differently. This is a popular area, though, and some best practices are starting to emerge.  If you're interested in REST, I'd like to hear your thoughts about best practices.

REST is not simply JSON over HTTP,  but most RESTful APIs are based on HTTP. Request methods like POST, GET, PUT and DELETE are used to implement Create, Read, Update and Delete (CRUD) operations. The first question is how to map HTTP methods to CRUD operations.

To start, here's a "REST API Design Cheat Sheet" that I typed up and pinned to my wall. Its based on the book "REST API Design Rulebook", and the HTTP RFC. I think it reflects standard practice. There are newer and better books on the subject now, but this list covers the basics of HTTP requests and response codes used in REST APIs.

Request Methods…


Kasa  © 2010 Darren DeRidder.
Umbrella detail, Kinosaki, Japan

When Agile Went Off the Rails

Whenever I hear a company say "We follow an agile development process", I can't help but wince a little. The core ideas of agile development are excellent, but somewhere along the way it accumulated quite a lot of codified process, and became its own formal methodology - almost the same thing the Agile Manifesto was trying to counteract. It's not too surprising, since the agile manifesto didn't prescribe any particular project management methodology for implementing its guidelines. So naturally it wasn't long before management professionals began to formalize agile philosophy into a methodology of their own.

Now one of the original authors of the Agile Manifesto has come out with a piece, originally titled "Time to Kill Agile", in which he makes this point that a formal methodology runs counter to the original goals of the agile development concept. Dave Thomas has been hugely influential in the software development field. Aside from being one of t…

Itsukushima Jinja

UNESCO World Heritage Site, Itsukushima Shrine, Hatsukaichi, Hiroshima Prefecture, Japan.

Itsukushima Jinja
© 2014 Darren DeRidder. Originally uploaded by 73rhodes