Sunday, December 13, 2015

More Node.JS Module Patterns

The post on Node.JS module patterns and the slideshow from the talk I did at OttawaJS keep getting a lot of mentions. Those are really simple examples, but in practice most modules have a bit more substance to them.

There are a couple of patterns that I've seen used in Express.JS apps a lot.  One simply exports a bunch of functions that are used as route handlers. Another passes an object to the module, and the module attaches things to it.  And the last one exports an Express router object that the main app can use to define more specific routes on a base URL.

Let's look at these with some simple examples.

Exporting route handler functions

This is a fairly common pattern where the module simply exports a number of functions. In this case they're route handler functions that an Express app can use to handle the various routes it declares.


exports.getUser = function (req, res, next) {
  res.send('respond with a user');

exports.updateUser = function (req, res, next) {
  res.send('update user and respond');


var users = require('users.js');

app.get('/user/:userid', users.getUser);
app.put('/user/:userid', users.updateUser);

Passing in and enhancing an object

This pattern has been used in some Express example apps and shows how you can pass variables into a module, either to use them in the module or to "enhance" an object by attaching things onto it. Of course you have to know what you're doing and not have two different modules that try to do the same thing.

module.exports = function (app) {
  app.get('/user/:userid', function (req, res, next) {
    res.send('respond with a user');


var users = require('users.js')(app);

Express router modules

The most recent version of express-generator creates a sample app using this pattern. Route modules are created under the routes/ directory, and they require an instance of the Express router. They add their routes to the router instance, and set module.exports as the router object. The main app.js file then attaches these router objects via app.use('/path', router) as you can see below. It's a nice clean way to organize route modules in Express.


var express = require('express');
var router = express.Router();

router.get('/:userid', function (req, res, next) {
  res.send('respond with a user');

module.exports = router;

var express = require('express');
var users = require('users.js');
var app = express();
app.use('/users', users);

Friday, November 13, 2015

Reacting to React

While I'm on the blog here, I figured I'd take a minute to write my thoughts on React / React Native.

Recently I've had occasion to play with React Native a bit. This post is not so much about first impressions, though, but about my perspective and some preconceptions about things like React in general.

I run a big JavaScript meetup group called OttawaJS. I get to see lots of interesting presentations on the latest and greatest technology, and sometimes give talks myself. Over the past few years I've seen an absolute deluge of new frameworks and libraries for web and mobile development, and like a lot of people in the tech community, have suffered from "new framework fatigue".  Perhaps this is one reason why, when React came around, I didn't get immediately excited about it. Beyond that, here are some general observations, personal biases, and preconceptions.

1. When Mark Zuckerberg famously said "HTML5 isn't ready", many people felt they hadn't given it a fair shake. The developers at Sencha proved the point by building an HTML5 clone of the Facebook app that outperformed the native one.  So Facebook hasn't been a big proponent of using the web stack for mobile apps, historically. Although I think there are some really good ideas underlying React, having a large company behind something doesn't mean it's the right solution for everybody.

2. I'm a little wary of frameworks built and promoted by large companies. Enterprises don't usually build open source frameworks without some benefit to themselves, and having more developers on a project tends to add complexity. The frameworks and tools that I usually prefer, and the ones that have generally proven most successful over time, are often written by a single author out of personal interest, and have slowly built up a following.

3. There seemed to be a lot of marketing behind React, and that can be a bad sign. It had barely been introduced and there were conferences about it and a flood of videos and articles. Paul Graham said something similar about Java a long time ago. About how good languages and frameworks don't need to be marketed, and how anything with a big marketing engine behind it just smells funny.

4. There are some cool functional programming concepts in React but they're mixed in with the classical object model. This kinda points to confusion about what the designers thought they were trying to build.

5. Generating HTML programmatically actually does suck. Other languages and framework have tried this, and they also sucked. I think Ember's Glimmer engine has taken an approach that minimizes DOM updates for great performance, without re-inventing the way HTML is written.

6. React seems to have done a lot to make people aware of cool things like immutability and components. And that's really great. However I think those concepts have value outside of React and can be applied with smaller, bespoke libraries like Redux and Riot.js.

7. My gut feeling is that it just doesn't feel like React is the holy grail of component-oriented web frameworks. It brings some cool concepts to the table and has helped shift developers' focus towards interesting alternatives to two-way binding and REST to thinks like one-way data flow and GraphQL. But I think web components and the native web stack are heading in the right direction and will ultimately replace a lot of what client-side MVC frameworks currently do.

So that's why I've told people I'm not betting on React becoming the defacto standard way to build web/mobile apps. At best, I think some of it's most useful concepts will be extracted, cleaned up and incorporated in smaller standalone implementations.

KCJS keep calm and javascript. :-)

Successful Software Projects

Successful software projects don't happen by accident. A recent post by Jeffrey Ventrella on Slow Programming resonates with developers who've been through enough projects to know the pros and cons of the old top-down design approaches versus the popular new "iterative development" and agile approaches.  The author wrote about his team's fast, iterative development process:

At the job, we were encouraged to work in the same codebase, as if it were a big cauldron of soup, and if we all just kept stirring it continuously and vigorously, a fully-formed thing of wonder would emerge. It did not.

The fallacy of fungible engineers - viewing engineers as resources that can be assigned interchangeably to various projects, is at the heart of the problem. Unleashing a bunch of developers on a project to "vigorously stir the cauldron" without a solid high-level design is clearly not a great idea. Iterative development may be a more realistic approach to software development, but you can't do away with up-front planning altogether.  As Ventrella says, "You can't wish away the design process."

So is iterative development a bad thing?  I don't think so. It's important to note that new paradigms are usually reactionary - they introduce a different approach to fix the failings of the old one.  Out of necessity, they're usually overly reactionary. The new approach condemns the old practices in order to get people to change, but it often fails to recognize that underneath there's probably a core of ideas still worth considering.

There are valuable aspects of both the "slow programming" approach and the "iterative" approach. It may not be necessary to define all the detailed requirements up front, but projects can benefit from asking a few simple questions to lay the groundwork before development starts.  These are things like the optimal technology stack for your problem space, documentation, and what you want your testing to look like.  Too often, these decisions are done in an ad hoc manner, late in the development cycle, or simply ignored altogether.  I feel that addressing these questions at the start of project is important for success. Here they are, roughly in the order of importance, as I see them.

  • Design: design is often the worst part of software products created by large enterprises. Almost no thought goes into it; a product gets built starting with the data model; layers of code are built up on top of it until the data is finally vomited onto the user's screen in a great pile of clutter. Good design is not part of the corporate DNA of a large company usually. This makes visual design an issue of low hanging fruit for smaller competitors, who can quite easily win over potential clients with the usability and elegance of their product, even if it lacks the feature set of a more mature offering.  Conversely, great design can't be done by a non-technical designer without a deep technical understanding of the product; it's a mistake to view design as a separate activity that can take place in isolation from the technical implementation. Graphic designers are not great at understanding the technical details of a project, and sometimes force engineers into making poor compromises for the sake of preserving a look and feel.  Engineers are generally bad at user interface design, unless they also have a background and strong aptitude in the visual arts. Good engineers engineers who are also artists are rare. Unless you have one of them, you'll need a design / engineer team who can work in close collaboration to define the initial product vision. It is not going to magically emerge from a set of requirements from Product Level Managers. Ask yourself: Is there a strong vision for the aesthetic sense of your product? Is it informed by a strong technical understanding as well as a good aesthetic sensibility?  Can it be demonstrated that the design decisions are grounded in good usability guidelines and not just the whim of a non-technical "graphic designer"? Are you leveraging existing best-in-class frameworks for UI design? Can you point to the examples that inspire the aesthetic vision of the product? Do you know what you want your product to look like?  Is it beautiful? Does it inspire people to want to use it?
  • Testability: how do you want to be able to test your code? Do you want unit tests?  End to end functional testing? What do you want the testing process to look like?  Will it be the same in the developer environment and the build environment? What tools will you consider? Do you want continuous integration? What's considered bad enough to break the build? Do you value test coverage reporting? How much test coverage is enough?
  • Documentation: it can make or break a project and often it is left as an afterthought, to be completed by technical writers who have little knowledge of how the low level works. Can you make it "self documenting"? How? What do you want the documentation to look like? In what formats will it be available? Will you have a developer's guide? An install guide? A user guide? How will you keep it up to date? How will you know if it's out of date?
  • Surface area: how do you define the points at which clients interface with your product (it's surface area)? A lot of this boils down to good API design. What are the best practices that you wish to implement? Which existing products / services exemplify what you want to achieve? What are the existing successful APIs, trends, industry best practices you want to emulate. Against what do you measure the usability of your API? Who are the current best-in-class leaders and how do you build upon their examples?
  • Technology stack: Quite often the default in large enterprises is Java. This is unfortunate because Java is rarely if ever the best technological solution to a particular problem - it's just the one that large numbers of programmers happen know. Think outside the box - one developer with a killer tool-set is better than a team of programmers with poor tools. Have you checked out ThoughtWorks Technology Radar? How do you place bets on which technologies will succeed going forward?  Do you have a strong sense of picking technology winners that comes from being in the trenches? Or are you a manager trying to make a safe choice?  Are your choices based on cargo cult or bandwagon tendencies?  Does your framework of choice have a track record of solid bets, critical mass of users, a core team with a clear focus and commitment to keeping things small - aka. the "do one thing well" philosophy? Will you ask for permission or for forgiveness? As a lead developer, are you willing to tell your managers that the technology and technical design decisions are ultimately your decisions to make?
  • Tooling: will you standardize on set of tools or let developers use whatever ad hoc collection of tools they prefer? Will you utilize linters, and static code analyzers like PMD, Findbugs, and Flow? Where in the development process where they be used - in the developer environment by triggering these tools as each file is edited? Will you trigger build failures after committing flawed code? Will you depend on a monolithic IDE like Eclipse or use a lightweight code editor like Sublime?
  • Code organization / architecture: The default is the Big Ball of Mud design pattern.  Are you a devotee of Object Oriented design? Are you enlightened by Functional Programming? Prefer an Abstraction Oriented architecture? What is your high level view of how it all fits together with a view towards delegating responsibilities? How to you define the boundaries between components? Can you specify, in draft form, an APIs to act as a  'contract' between components that can then be delegated off to different developers? How will you communicate your architectural vision and get buy-in from team members?
  • Prototype: have you built a proof of concept? Will you? How? Do you intend to productize the prototype, or is it meant to serve only as a reference for what you want the final product to be like? If you plan to evolve your prototype into a production application, do you have a clear path for doing so? For example, have you steered clear of design decisions in the prototype that will be difficult to back out of, like introducing dependencies on libraries or techniques that you won't be able to release in a production environment? As a real-world example: we created a mobile security demo using a relatively new technology that allowed rapid development, but made sure the API specification was well defined, knowing that it would not be difficult to re-implement the API in any server side language we chose.
  • Delegation: how can the project can be split up into sensible areas of responsibility as it grows? If the project has started out with one full-stack developer, will it be easy to carve off the API, UI, mobile app, etc. into standalone projects that another developer or team can take ownership of? A lot of this comes down to having well defined and documented APIs as a contract between components.
  • Security: Its not enough to trust that your product is going to be secure because it uses Framework X.  Upon inspection, our initial subscriber-facing web interface turned out to be susceptible to session hijacking, cross-site scripting, and SQL insertion, and security issues continued to be present despite being overseen by senior developers.  Security audit tools revealed even more weaknesses. What tools will you use to audit the security of your product (Skipfish, Nessus, OpenVAS, etc.)? Will you carry out manual penetration testing on your product? Will security audits be performed automatically, on a regular basis? Will they be integrated into your build environment or QA environment?

The questions you need to ask may vary, but the important thing is to have a plan, communicate the plan well, empower people to own their part of it, make the design objectives clear (including all of the points above), and make sure everyone knows how their piece fits into the overall product vision.

Thursday, January 22, 2015

Fluent 2014 Talk Summaries 3

"Speed, Performance and Human Perception"

In this performance-related talk, Ilya Grigorik explains that performance is not only a function of speed, but of meeting user expectations in a way that allows them to complete tasks effectively, with insightful examples.


My rating: 4/5. Provides useful insight on usability.

"Delivering the Goods"

Paul Irish discusses optimization in this keynote talk, describing the "critical path" and how requests impact page load times. Chrome developer tools are used to explain page load sequences and timing. Recommendations include: eliminate render-blocking JS; minimize render-blocking CSS; serve content in the original HTML response and use gzip compression. Google Page Speed Test is a tool to automatically recommend such optimizations.


My rating: 5/5. Beneficial recommendations for all developers of web-based software.

Tuesday, January 20, 2015

Fluent 2014 Talk Summaries, continued

"Reading, Writing, Arithmetic... and JavaScript?"

Pamela Fox, the person behind the JavaScript-based programming curriculum at Khan Academy, discusses how age affects a person's ability to learn programming and says that most 13 year olds are capable of learning basic programming skills that will help them explore other fields like art, history and language.  Visit Khan Academy for more information on their Computer Programming curriculum.

My rating: 3/5.  Introduces a great educational resource for young people. Plus, I've been a fan of Pamela's work ever since I read her articles on JavaScript widgets at Coursera.


"Virtual Machines, JavaScript and Assembler"

Popular podcaster Scott Hanselman delivers a humorous talk describing how the basic features of an operating system exist in both the cloud and the browser.

My rating: 3/5. Entertaining for programmers, though light on technical take-aways.


"The Humble Border-Radius"

In this talk Lea Verou demonstrates how border radii can be used to create a variety of shapes and animations, as well as upcoming CSS specifications for different corner shapes. While interesting, I think what web designers really want is complete flexibility to design arbitrary shapes, and border-radius seems like a very awkward mechanism for accomplishing that.

My rating: 3/5. Interesting CSS hacks.


Monday, January 19, 2015

Fluent Talk Summary: Brendan Eich, "JavaScript: the High and Low Roads"

In this lighthearted talk, Brendan Eich (inventor of JavaScript) discusses high and low-level improvements in upcoming versions of JavaScript. Acknowledging that Web development is hard ("like having a chain-saw in place of a hand"), Eich says the upcoming version of JavaScript, ES6 "Harmony", will address many difficulties facing web developers, with improvements like a built-in module system, observable objects, and many other features that will make "transpilers" and other "syntactic sugar" libraries unnecessary.

The future ES7 version of JavaScript aims to bring many low-level improvements for high-performance and scientific computing, including new value objects, operator overloading, and SIMD intrinsics. New value objects proposed in ES7 include: int64, uint64, int32x4 and int32x8 (SIMD), float32 (useful for GPUs), float32x4 and float32x8 (SIMD), bignum, decimal, rational, and complex. ES7 will also introduce operator overloading and new literals like 0L (int64), 0UL (uint64), etc. Support for SIMD (Single Instruction, Multiple Data) will lead to native performance for applications like game programming and signal processing. To demonstrate further, Eich reveals the first-ever public demo of the Unreal Engine 4 from Epic Games, showing stunning 3D graphics running at a full 60 frames per second, with no plugins, in a Firefox build with special SIMD capabilities.

Guided by the Extensible Web Manifesto, the high road of developer-friendly features and the low road of safe, low-level language improvements will converge in a virtual machine that offers native performance while being very developer friendly.

In a nutshell - "Always bet on JS!"

My rating: 4/5. Informative, insightful and entertaining for programmers.

Sunday, January 18, 2015

Dynamic Types FTW

There's a belief among some programmers, particularly the OO classical inheritance folks, that static typing is a bulwark of security against all kinds of disasters that may happen in your code.  They feel that the compiler will ensure their programs work correctly, by generating errors and warnings and refusing to compile their program until all the mistakes are fixed.

Some developers mistakenly speak in terms of "strong typing" and "weak typing". Put that way, who wouldn't prefer "strong" over "weak" typing? In reality the terms "strong typing" and "weak typing" are undefined in the language of computer science. They're misnomers that people sometimes use to talk about strict (explicit) type definition with static type checking, versus dynamic (runtime) type checking with implicit type conversion.

It's unfortunate that this line of reasoning causes some people to avoid languages that offer dynamic types. Dynamic types can be very useful, and static typing actually offers very little in the way of ensuring program correctness.  I finally sat down and watched Eric Elliott's presentation "Static Types are Overrated: The Dynamic Duo - Loose Types and Object Extension" and was treated to a bang-on description why static types aren't all they're cracked up to be.  Eric does a good job of explaining and clarifying things about JavaScript and programming in general that I intuitively believed but had a difficult time putting into words or backing up with solid examples.

I borrowed a bunch of points from Eric's talk when I spoke at a recent Ottawa JavaScript meetup. I was presenting on Flow, a static type checker for JavaScript, but I wanted people to see it as something that could be added to their regular code linting process rather than something to enforce a statically typed programming style. Here are some of them.
  • It's a myth that dynamic loosely typed languages like JavaScript aren't suitable for building large, complex applications.  There are dozens of examples from Facebook to 37signals, Dow Jones, Adobe, Flickr, LinkedIn, Walmart, etc).
  • Type correctness does not guarantee program correctness. This is one of the biggest cargo-cult myths out there: thinking that static, strict type checking will save you. It won't.
  • When languages lack dynamic types, people fake it with ugly hacks - void pointers, variadic functions, abusing array accessors, type-casting of all sorts, generics and ugly template classes, to name a few. I don't know any C or Java programmers who haven't used and abused at least a few of these techniques.
  • "Any sufficiently advanced C / Fortran [Java, etc] program contains an ad hoc, informally specified, bug-ridden, slow implementation of half of common Lisp" - Greenspun's Tenth Rule. (There are no rules 1 through 9, by the way. Greenspun's Rules of Programming start and end with number 10.)
  • Older statically typed languages are now introducing dynamic types because they really are useful. Objective-C added them a long time ago; more recently C++11 has added dynamic types, Java has "generics", and libraries like cppscript and boost provided ways to do dynamic types in earlier versions of C++.
  • Functional programming benefits from functions that can operate on any type which implements its requirements (for example, arguments have a valueOf method). Otherwise known as duck typing. This permits things like map, filter, forEach etc. to be used generically.
  • Correctness can only really be assured through proper unit and integration testing. 
  • Code that is well organized in small, simple modules, linted, unit tested, peer reviewed, and integration tested, is very unlikely to contain type errors. 

The last thing I would add is that while static types are over-rated, static code analysis tools like JSLint, Tern.js and Flow are mostly underrated and underutilized. If you use something like Flow or Tern.js to provide hints and insights into potential type mismatches, the chances of type errors become essentially zero. Ideally they should be as up-front as possible, i.e. integrated directly into your code editor. I wrote a JSLint plugin for SublimeText and recently wrote a Flow plugin as well. They provide immediate static code analysis warnings as I edit my code, so I can fix problems before I commit anything to the source code repository.