Thursday, August 14, 2014

Repackaging node modules for local install with npm

If you need to install an npm package for nodejs from local files, because you can't or prefer not to download everything from the repo, or you don't even have a network connection, then you can't just get an npm package tarball and do `npm install <tarball>`, because it will immediately try to download all it's dependencies from the repo.

There are some existing tools and resources you can try:

  • npmbox -
  • bundle.js gist -
  • relevant npm issue -

I found all of these a bit over-wrought for my taste. So if you prefer a simple DIY approach, you can simply edit the module's package.json file, and copy all of its dependencies over to the "bundledDependencies" array, and then run npm pack to build a new tarball that includes all the dependencies bundled inside.

Using `forever` as an example:
  1. make a directory and run `npm init; npm install forever` inside of it
  2. cd into the node_modules/forever directory
  3. edit the package.json file
  4. look for the dependencies property
  5. add a bundledDependencies property that's an array
  6. copy the names of all the dependency modules into the bundledDependencies array
  7. save the package.json file
  8. now run `npm pack`. It will produce a forever-<version>.tgz file that has all it's dependencies bundled in.

Thursday, May 29, 2014

JavaScript's Final Frontier - MIDI

JavaScript has had an amazing last few years. Node.JS has taken server-side development by storm. First person shooter games are being built using HTML and JavaScript in the browser. Natural language processing and machine learning are being implemented in minimalist JavaScript libraries. It would seem like there's no area in which JavaScript isn't set blow away preconceptions about what it can't do and become a major player.

There is, however, one area in which JavaScript - or more accurately the web stack and the engines that implement it - has only made a few tentative forays.  For me this represents a final frontier; the one area where JavaScript has yet to show that it can compete with native applications. That frontier is MIDI.

I know what you're probably thinking. Cheesy video game soundtracks on your SoundBlaster sound card. Web pages with blink tags and bad music tracks on autoplay. They represent one use case where MIDI was applied outside of its original intent. MIDI was made for connecting electronic musical instruments, and it is still very much alive and well. From lighting control systems to professional recording studios to GarageBand, MIDI is a key component of arts performance and production. MIDI connects sequencers, hardware, software synthesizers and drum machines to create the music many people listen to everyday. The specification, though aging, shows no signs of going away anytime soon. It's simple and effective and well crafted.

It had to be. Of all applications, music could be the most demanding. That's because in most applications, even realtime ones, the exact timing of event processing is flexible within certain limits. Interactive web applications can tolerate latency on their network connections. 3D video games can scale down their frames per second and still provide a decent user experience. At 30 frames per second, the illusion of continuous motion is approximated. The human ear, on the other hand, is capable of detecting delays as small as 6 milliseconds. For a musican, latency of 20ms between striking a key and hearing a sound, would be a show-stopper. Accurate timing is essential for music performance and production.

There's been a lot of interest and some amazing demos of Web Audio API functionality.  The Web MIDI API, on the other hand, hasn't gotten much support.  Support for Web MIDI has landed in Chrome Canary, but that's it for now.  A few people have begun to look at the possibility of adding support for it in Firefox.  Until the Web MIDI API is widely supported, interested people will have to make due with the JazzSoft midi plugin and Chris Wilson's Web MIDI API shim.

I remain hopeful that support for this API will grow, because it will open up doors for some truly great new creative and artistic initiatives.

Wednesday, May 7, 2014

REST API Best Practices 3: Partial Updates - PATCH vs PUT

This post is a continuation of REST API Best Practices 2: HTTP and CRUD, and deals with the question of partial updates.

REST purists insist that PATCH is the only "correct" way to perform partial updates [1], but it hasn't reached "best-practice" status just yet, for a number of reasons.

Pragmatists, on the other hand, are concerned with building mobile back-ends and APIs that simply work and are easy to use, even if that means using PUT to perform partial updates [2].

The problems with using PATCH for partial updates are manifold:
  1. Support for PATCH in browsers, servers and web application frameworks is not universal. IE8, PHP, Tomcat, django, and lots of other software has missing or flaky support for it. So depending on your technology stack and users, it might not even be a valid option for you.
  2. Using the PATCH method correctly requires clients to submit a document describing the differences between the new and original documents, like a diff file, rather than a straightforward list of modified properties. This means the client has to do a lot of extra work - keep a copy of the original resource, compare it to the modified resource, create a "diff" between the two, compose some type of document showing the differences, and send it to the server. The server also has more work to apply the diff file. 
  3. There's no specification that says how the changes in the diff file should be formatted or what it should contain, exactly. The RFC simply says:
    "With PATCH, however, the enclosed entity contains a set of instructions describing how a resource currently residing on the origin server should be modified to produce a new version."
    There are some interesting recommendations emerging like JSON Patch [3], but at this point it seems mainly up to each developer to figure out their own way of using PATCH.
Using PUT for partial updates, however, is pretty simple, even if it doesn't conform strictly to the concept of Representational State Transfer.  So a fair number of programmers happily use it to implement partial updates on back-end mobile API servers. It's fair to say that when developing an API, a pragmatic approach that focuses on the needs of mobile client applications is completely reasonable.

So what are the current "best practices" when using PUT for partial updates, for those who choose practicality over purity? As I see it, basically this: When you PUT the update, include the properties you want to update, leave out the properties you don't want to update, and for any properties you want to delete, set them null.


 Pragmatic partial updates with PUT

  1. Include properties to be updated
  2. Don't include properties not to be updated
  3. Set properties to be 'deleted' to null
The reality is that most data is going to be stored in a database that, even if it's a NoSQL database, has an implicit or explicit schema that describes what sort of data your application is expecting. If you're using a relational database, this will end up being columns in your database tables, some of whose values may be null. In this scenario it makes perfect sense to "delete" properties by setting them null, since the database columns are not going to disappear in any case. And for those who use a document database, its not a stretch to delete nullified properties.

Further reading


Monday, April 7, 2014

REST API Best Practices 2: HTTP and CRUD

This post expands a bit further on the REST API Cheat Sheet regarding HTTP operations for Create / Read / Update / Delete functionality in REST APIs.

APIs for data access and management are typically concerned with four actions (the so-called CRUD operations):
  • Create - the ability to create a resource
  • Read - the ability to retrieve a resource
  • Update - the ability to modify a resource
  • Delete - the ability to remove a resource

CRUD operations don't have a perfect, 1-to-1 mapping to HTTP methods, which has led to different opinions and implementations, but the following list represents best practice as I see it in the industry today, and follows the HTTP specification:

CRUD Operation    HTTP Method
UpdatePUT and/or PATCH

To reiterate, HTTP methods can be used to implement CRUD oprations as follows:
  • POST - create a resource
  • GET - retrieve a resource
  • PUT - update a resource (by replacing it with a new version)*
  • PATCH - update part of a resource (if available and appropriate)*
  • DELETE - remove a resource

Although PATCH is considered the officially correct and "RESTful" way to do partial updates, it has yet to gain wide adoption. Many popular web application frameworks don't support the PATCH method yet, so in practice, it is not uncommon to use PUT for partial updates even though its not strictly "RESTful". The decision to use PUT vs. PATCH for partial updates is driven by the capabilities of your framework of choice (Rails only recently introduced PATCH, for example) and by the practical requirements of building web/mobile back-end services that actually work and are easy to use, even if they don't satisfy REST purists. More on this in the next post.


Safe and Idempotent Methods


The HTTP 1.1 specification defines "safe" and "idempotent" methods [1].  Safe methods don't modify data on the server no matter how many times you call them. Idempotent methods can modify data on the server the first time you call them, but repeating the same call over and over again won't make any difference. Here's a partial list:

Method    Safe    Idempotent

The safe and/or idempotent nature of these HTTP methods provides some further insight into how they ought to be used. Notice that POST is neither safe, nor idempotent. A successful POST should create new data on the server, and repeating the same call should create even more copies on the server. GET, on the other hand, is safe and idempotent, so no matter how many times you call it, the data on the server shouldn't be affected.

GET - use it to fetch resources, but don't "tunnel" request parameters through to the server as a way to alter the state of data on the server - as a "safe" method, calling GET shouldn't have side effects.

PUT - use it to update an existing resource by replacing it with a new representation. The data you PUT to the server should be a complete replacement for the specified resource. Although PUT can in theory be used to insert new resources, in practice it's not advisable. Note that after the first PUT request, repeatedly calling the same PUT method with the same data won't change the data on the server more than it already has been (a condition of idempotent methods).

PATCH - if this method is available and well supported in both your client and server side technology stack (ie. Rails 4), consider using it to update part of an existing resource by changing some of it's properties, following the recommendations of the framework for how to submit the change descriptions. The PATCH method isn't supported everywhere and not common enough to be considered a current best practice, but the industry seems to be moving this way and technically it's the correct way to provide partial updates according to the HTTP spec [2].

If your server, framework or client user base (IE8, etc) doesn't support PATCH, rest assured that many developers take the pragmatic approach and simply bend the rules to use PUT for partial updates [3]. I'll cover this in the next post in more detail. Note that, no matter how you do your partial update, it should be atomic, that is once the update has started, it should not be possible to retrieve a copy of the resource until the update has been fully applied.

POST - use it to create new resources. The server should create a unique identifier for each newly created resource. Return a 201 Created response if the request was successful. Best practice appears to be to return the unique ID in the response. POST is also frequently used to trigger actions on the server which technically aren't part of RESTful API, but provide useful functionality for web applications.

DELETE - use it to delete resources; it's pretty self-explanatory.


Friday, March 21, 2014

REST API Best Practices: a REST Cheat Sheet

I'm interested in REST API design and identifying the best practices for it. Surprisingly, a lot of APIs that claim to be RESTful, aren't. And the others all do things differently. This is a popular area, though, and some best practices are starting to emerge.  If you're interested in REST, I'd like to hear your thoughts about best practices.

REST is not simply JSON over HTTP,  but most RESTful APIs are based on HTTP. Request methods like POST, GET, PUT and DELETE are used to implement Create, Read, Update and Delete (CRUD) operations. The first question is how to map HTTP methods to CRUD operations.

To start, here's a "REST API Design Cheat Sheet" that I typed up and pinned to my wall. Its based on the book "REST API Design Rulebook", and the HTTP RFC. I think it reflects standard practice. There are newer and better books on the subject now, but this list covers the basics of HTTP requests and response codes used in REST APIs.

Request Methods

  • GET and POST should not be used in place of other request methods
  • GET is used to retrieve a representation of a resource
  • HEAD is used to retrieve response headers
  • PUT is used to insert or update a stored resource
  • POST is used to create a new resource in a collection
  • DELETE is used to remove a resource

Response Status Codes

  • 200 "OK" indicates general success
  • 200 "OK" shouldn't be used to return error messages
  • 201 "Created" indicates a resource was successfully created
  • 202 "Accepted" indicates that an asynch operation was started
  • 204 "No Content" indicates success but with an intentionally empty response body
  • 301 "Moved Permanently" is used for relocated resources
  • 303 "See Other" tells the client to query a different URI
  • 304 "Not Modified" is used to save bandwidth
  • 307 "Temporary Redirect" means resubmit the query to a different URI
  • 400 "Bad Request" indicates a general failure
  • 401 "Unauthorized" indicates bad credentials
  • 403 "Forbidden" denies access regardless of authentication
  • 404 "Not Found" means the URI doesn't map to a resource
  • 405 "Method Not Allowed" means the HTTP method isn't supported
  • 406 "Not Acceptable" indicates the requested format isn't available
  • 409 "Conflict" indicates a problem with the state of the resource
  • 412 "Precondition Failed" is used for conditional operations
  • 415 "Unsupported Media Type" means the type of payload can't be processed
  • 500 "Internal Server Error" indicates an API malfunction
A note about the PATCH method. There are good reasons to consider using the HTTP PATCH method for partial updates of resources, but because it's not supported everywhere, and because there are workarounds, I haven't added it to my cheat sheet yet.

Wednesday, March 19, 2014


Kasa  © 2010 Darren DeRidder.
Umbrella detail, Kinosaki, Japan

Tuesday, March 11, 2014

When Agile Went Off the Rails

Whenever I hear a company say "We follow an agile development process", I subconsciously wince a little. Agile methodology is something of an oxymoron. Oh, the core ideas of it were (are) excellent, but somewhere along the way it turned into a bunch of codified management processes, and became exactly the sort of thing the Agile Manifesto was trying to eliminate. As soon as the management folk came along and tried to formalize agile philosophy into a methodology, it stopped being agile.

Now one of the original authors of the Agile Manifesto has come out with a piece, originally titled "Time to Kill Agile", in which he makes this point. Dave Thomas has been hugely influential in the software development field. Aside from being one of the authors of the agile manifesto, he's written a lot of other stuff, and he's the guy who coined the phrases "code kata" and "DRY" (Don't Repeat Yourself - the maxim developers follow to effectively organize their code).  Evidently, Dave considered that "Agile" was probably already dead and renamed the piece "Agile Is Dead (Long Live Agility)!"

The problem with being a critic of Agile, or anything else that's become ingrained in a big software development organization, is that you'll come up against folks who've been drinking the kool-aid for long enough that if you call into question the value of Agile (or Oracle, or Java, or whatever) you'll likely be viewed as an idiot.

So this - having one of the original authors of the Agile Manifesto come out and make the same kind of criticism of "Agile" methodology that I've been skeptical of for years - is gold.

It all boils down to people over processes. Agile software development cannot be implemented as a set of methodologies, and the managers, consultants and companies that have turned it into a set of rules, processes and formalities, have demonstrated that they totally don't get what the authors of the Agile Manifesto intended in the first place.  In my humble opinion.

Dave Thomas has some good advice for teams that want to develop software with agility. He advocates an iterative approach to development, and choosing options that enable future change. He recommends thinking of "agile" in the form of an adverb (agilely, or "with agility"). Programming with agility. Teams that execute with agility.

I've found that when it comes to managing a project, simple is usually better. What's worked best in my experience is, in a nutshell, to simply encourage communication. Make sure everyone understands the overall objective, how they can contribute to it, what progress has been made and what challenges remain, and importantly, give everyone the opportunity to have their work fully recognized and appreciated on a regular basis. Given the opportunity to work on a challenging project and the chance to have their contributions seen and appreciated by colleagues, most developers will bend over backwards to do their best work.

There's no right or wrong way to do this, but one technique I found effective was a brief (set-a-timer with a hard cut-off) meeting at the beginning of the week where we laid out the objectives for the week ahead, a quick information-gathering hike around the office at the end of the week, and an email re-cap on Friday afternoon highlighting the teams progress. Showing the percentage-towards-completion of major tasks was also a big motivator, as developers began to take pride in seeing their areas of responsibility make steady, visible progress towards completion.  We didn't formalize or get locked into one way of doing it, so when our company got acquired and our management structure changed, we adapted pretty easily.

  • People over processes.
  • Working software over documentation.
  • Collaboration over contracts.
  • Adaptability over planning.