Productivity and Note-taking

I told a friend of mine that I wasn't really happy with the amount of time that gets taken up by Slack and "communication and scheduling" in general, and that on one particularly "noisy" week it had taken up around 7 hours. He said that was "hardly anything" and that most of his work was done via Slack.  He has a support role, so I guess that makes sense. So it definitely can be a useful tool, but in the past few weeks I've managed to keep Slack usage down to about 3 hours per week.

On a related topic, I've tried a bunch of different task management techniques over the years and none of them ever stuck. I've always ended up with a fragmented collection of things to do, scattered between various Notes apps, email-based task lists, Trello boards, and hand-written notes.  The problem with a lot of the software-based task management options for me is that they're not always in front of me and they take a conscious effort to open and use.

A notebook on the other hand is always on the desk beside me, usually open to the last page of notes. There's no effort getting to it, and I can easily glance over without disrupting whatever I was doing on the computer.

There's a system called a bullet journal for keeping and managing lists.  The website bulletjournal.com explains the system and has an online store with their BuJo journals that are designed to work with the bullet journal system. There's also an app, so for people who really want to get into it, there are a variety of ways. You don't need a special BuJo journal to start using the technique, however.

I've switched my list of tasks from an online document to my pen-and-paper bullet journal now. I'm not fastidious about keeping the journal up to date on a daily or even weekly basis, but I find that its just a lot easier to take notes this way than it was to type things into a digital form.  The only downside I can see is that it's harder to share than digital notes which can be copy and pasted. But frankly most of the tasks on my list are too detailed and boring for anybody else to care about. They just want to know how a feature is coming along, etc.  So, check out the bullet journal.

Slack is a major productivity drain

For the past few years I've been using RescueTime time management software.  It helps keep track of the different activities I've spent time on during the work week and stay focused on things that contribute to my productivity.  Another application that's become a big part of many workplaces is Slack. We use it for communication in my current team. A lot of times, it's really helpful. You can fire off a quick message and get a quick reply without really interrupting your main task. A lot of times you can ask a question, get valuable opinions from everyone concerned, and come up with a consensus that works for everybody, in a way that would've been impossible with traditional email and meetings.

Recently though I was shocked to see that Slack accounted for a full 7 hours of a recent work week. We've been doing a lot of planning and discussion for new feature work, and also working through some issues related to customer documentation, deployments, and things that are not specifically coding-related, so it's understandable that there's been more planning and discussion activity than usual, and less actual code-writing. But 7 hours!  My gosh.  That is a real time sink.

This was a wake-up call into something that I already intuitively knew, that Slack - as helpful as instant messaging can be - has the potential to be a black-hole where productivity gets sucked into a terminal death-spiral.

The challenge is how to reign in this Slack tyranny... if you don't monitor the conversations going on in various Slack channels, you're liable to miss out on vital information. A lot of people seem to think that broadcasting a message out on a Slack channel is "job complete" when it comes to communication - a surrogate for the old-school email.  That's not really the case; messages easily get lost in the backlog of noisy, run-on chatroom conversations.

I'm going to be keeping an eye on Slack usage and trying to figure the best way to keep it from sucking up a large percentage of my productive working hours without losing the benefit of instant communication with the broader team.  Maybe just being respectful of people's time and being a little less cavalier about using Slack to post casual commentary, understanding that Slack can definitely become a drain on my own and other people's time, and approaching it as a tool to be used with a certain level of professional self-restraint, might be a start.

REST API Best Practices 5: Further Reading

Since I started writing on REST API Best Practices there have been some interesting new developments. Going forward we'll take a look at some of them, covering things like documenting APIs, how to define relationships between different resources, and various tools that - while not specifically REST-related - are useful for working with JSON as a data interchange format.

In the meantime, here's a list of articles that provide more information on a lot of the concepts that were outlined in the first four posts on REST API Best Practices. No doubt a lot more has been written about REST APIs in the last few years, but I think these resources are a pretty good window into some of the original sources that shaped the best practices used for REST API design today.

If you want to suggest other articles please feel free to comment below (note that comments are moderated and won't appear immediately).

Tutorials
http://www.restapitutorial.com/
http://obeautifulcode.com/API/Learn-REST-In-18-Slides/

General best practices
http://www.restapitutorial.com/
https://zapier.com/learn/apis/
https://s3.amazonaws.com/tfpearsonecollege/bestpractices/RESTful+Best+Practices.pdf
http://apigee.com/about/api-best-practices
http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api
http://www.slideshare.net/mario_cardinal/best-practices-for-designing-pragmatic-restful-api
http://apigee.com/about/api-best-practices/restful-api-design-second-edition
http://devproconnections.com/web-development/restful-api-development-best-practices
http://www.restapitutorial.com/
http://madhatted.com/2013/3/19/suggested-rest-api-practices

HATEOAS
http://restcookbook.com/Basics/hateoas/
http://timelessrepo.com/haters-gonna-hateoas

HAL
https://en.wikipedia.org/wiki/Hypertext_Application_Language

Documentation best practices
http://bocoup.com/weblog/documenting-your-api/

Partial updates:
http://stackoverflow.com/questions/232041/how-to-submit-restful-partial-updates
http://restful-api-design.readthedocs.org/en/latest/methods.html

Misc
http://www.wekeroad.com/2012/02/28/someone-save-us-from-rest/
http://docs.couchdb.org/en/latest/api/basics.html#api-basics

Auth
http://www.thebuzzmedia.com/designing-a-secure-rest-api-without-oauth-authentication/
http://en.wikipedia.org/wiki/OAuth
http://restcookbook.com/Basics/loggingin/
http://broadcast.oreilly.com/2009/12/principles-for-standardized-rest-authentication.html
http://stackoverflow.com/questions/630538/designing-a-web-api-how-to-authenticate
http://en.wikipedia.org/wiki/Session_hijacking#Methods
http://apiux.com/2013/03/21/authentication-dont-be-clever/
https://developer.github.com/v3/auth/
https://github.com/blog/1509-personal-api-tokens
http://stackoverflow.com/questions/7999295/rest-api-authentication
https://www.google.com/search?client=ubuntu&channel=fs&q=api+authentication&ie=utf-8&oe=utf-8

More Node.JS Module Patterns

The post on Node.JS module patterns and the slideshow from the talk I did at OttawaJS keep getting a lot of mentions. Those are really simple examples, but in practice most modules have a bit more substance to them.

There are a couple of patterns that I've seen used in Express.JS apps a lot.  One simply exports a bunch of functions that are used as route handlers. Another passes an object to the module, and the module attaches things to it.  And the last one exports an Express router object that the main app can use to define more specific routes on a base URL.

Let's look at these with some simple examples.

Exporting route handler functions

This is a fairly common pattern where the module simply exports a number of functions. In this case they're route handler functions that an Express app can use to handle the various routes it declares.

users.js

exports.getUser = function (req, res, next) {
  res.send('respond with a user');
};

exports.updateUser = function (req, res, next) {
  res.send('update user and respond');
};

app.js

var users = require('users.js');

app.get('/user/:userid', users.getUser);
app.put('/user/:userid', users.updateUser);

Passing in and enhancing an object

This pattern has been used in some Express example apps and shows how you can pass variables into a module, either to use them in the module or to "enhance" an object by attaching things onto it. Of course you have to know what you're doing and not have two different modules that try to do the same thing.

users.js

module.exports = function (app) {
  app.get('/user/:userid', function (req, res, next) {
    res.send('respond with a user');
  });
};

app.js

var users = require('users.js')(app);

Express router modules

The most recent version of express-generator creates a sample app using this pattern. Route modules are created under the routes/ directory, and they require an instance of the Express router. They add their routes to the router instance, and set module.exports as the router object. The main app.js file then attaches these router objects via app.use('/path', router) as you can see below. It's a nice clean way to organize route modules in Express.

users.js

var express = require('express');
var router = express.Router();

router.get('/:userid', function (req, res, next) {
  res.send('respond with a user');
});

module.exports = router;

app.js

var express = require('express');
var users = require('users.js');
var app = express();
app.use('/users', users);

Reacting to React

While I'm on the blog here, I figured I'd take a minute to write my thoughts on React / React Native.

Recently I've had occasion to play with React Native a bit. This post is not so much about first impressions, though, but about my perspective and some preconceptions about things like React in general.

I run a big JavaScript meetup group called OttawaJS. I get to see lots of interesting presentations on the latest and greatest technology, and sometimes give talks myself. Over the past few years I've seen an absolute deluge of new frameworks and libraries for web and mobile development, and like a lot of people in the tech community, have suffered from "new framework fatigue".  Perhaps this is one reason why, when React came around, I didn't get immediately excited about it. Beyond that, here are some general observations, personal biases, and preconceptions.

1. When Mark Zuckerberg famously said "HTML5 isn't ready", many people felt they hadn't given it a fair shake. The developers at Sencha proved the point by building an HTML5 clone of the Facebook app that outperformed the native one.  So Facebook hasn't been a big proponent of using the web stack for mobile apps, historically. Although I think there are some really good ideas underlying React, having a large company behind something doesn't mean it's the right solution for everybody.

2. I'm a little wary of frameworks built and promoted by large companies. Enterprises don't usually build open source frameworks without some benefit to themselves, and having more developers on a project tends to add complexity. The frameworks and tools that I usually prefer, and the ones that have generally proven most successful over time, are often written by a single author out of personal interest, and have slowly built up a following.

3. There seemed to be a lot of marketing behind React, and that can be a bad sign. It had barely been introduced and there were conferences about it and a flood of videos and articles. Paul Graham said something similar about Java a long time ago. About how good languages and frameworks don't need to be marketed, and how anything with a big marketing engine behind it just smells funny.

4. There are some cool functional programming concepts in React but they're mixed in with the classical object model. This kinda points to confusion about what the designers thought they were trying to build.

5. Generating HTML programmatically actually does suck. Other languages and framework have tried this, and they also sucked. I think Ember's Glimmer engine has taken an approach that minimizes DOM updates for great performance, without re-inventing the way HTML is written.

6. React seems to have done a lot to make people aware of cool things like immutability and components. And that's really great. However I think those concepts have value outside of React and can be applied with smaller, bespoke libraries like Redux and Riot.js.

7. My gut feeling is that it just doesn't feel like React is the holy grail of component-oriented web frameworks. It brings some cool concepts to the table and has helped shift developers' focus towards interesting alternatives to two-way binding and REST to thinks like one-way data flow and GraphQL. But I think web components and the native web stack are heading in the right direction and will ultimately replace a lot of what client-side MVC frameworks currently do.

So that's why I've told people I'm not betting on React becoming the defacto standard way to build web/mobile apps. At best, I think some of it's most useful concepts will be extracted, cleaned up and incorporated in smaller standalone implementations.

KCJS keep calm and javascript. :-)

Successful Software Projects

Successful software projects don't happen by accident. A recent post by Jeffrey Ventrella on Slow Programming resonates with developers who've been through enough projects to know the pros and cons of the old top-down design approaches versus the popular new "iterative development" and agile approaches.  The author wrote about his team's fast, iterative development process:

At the job, we were encouraged to work in the same codebase, as if it were a big cauldron of soup, and if we all just kept stirring it continuously and vigorously, a fully-formed thing of wonder would emerge. It did not.

The fallacy of fungible engineers - viewing engineers as resources that can be assigned interchangeably to various projects, is at the heart of the problem. Unleashing a bunch of developers on a project to "vigorously stir the cauldron" without a solid high-level design is clearly not a great idea. Iterative development may be a more realistic approach to software development, but you can't do away with up-front planning altogether.  As Ventrella says, "You can't wish away the design process."

So is iterative development a bad thing?  I don't think so. It's important to note that new paradigms are usually reactionary - they introduce a different approach to fix the failings of the old one.  Out of necessity, they're usually overly reactionary. The new approach condemns the old practices in order to get people to change, but it often fails to recognize that underneath there's probably a core of ideas still worth considering.

There are valuable aspects of both the "slow programming" approach and the "iterative" approach. It may not be necessary to define all the detailed requirements up front, but projects can benefit from asking a few simple questions to lay the groundwork before development starts.  These are things like the optimal technology stack for your problem space, documentation, and what you want your testing to look like.  Too often, these decisions are done in an ad hoc manner, late in the development cycle, or simply ignored altogether.  I feel that addressing these questions at the start of project is important for success. Here they are, roughly in the order of importance, as I see them.

• Design: design is often the worst part of software products created by large enterprises. Almost no thought goes into it; a product gets built starting with the data model; layers of code are built up on top of it until the data is finally vomited onto the user's screen in a great pile of clutter. Good design is not part of the corporate DNA of a large company usually. This makes visual design an issue of low hanging fruit for smaller competitors, who can quite easily win over potential clients with the usability and elegance of their product, even if it lacks the feature set of a more mature offering.  Conversely, great design can't be done by a non-technical designer without a deep technical understanding of the product; it's a mistake to view design as a separate activity that can take place in isolation from the technical implementation. Graphic designers are not great at understanding the technical details of a project, and sometimes force engineers into making poor compromises for the sake of preserving a look and feel.  Engineers are generally bad at user interface design, unless they also have a background and strong aptitude in the visual arts. Good engineers engineers who are also artists are rare. Unless you have one of them, you'll need a design / engineer team who can work in close collaboration to define the initial product vision. It is not going to magically emerge from a set of requirements from Product Level Managers. Ask yourself: Is there a strong vision for the aesthetic sense of your product? Is it informed by a strong technical understanding as well as a good aesthetic sensibility?  Can it be demonstrated that the design decisions are grounded in good usability guidelines and not just the whim of a non-technical "graphic designer"? Are you leveraging existing best-in-class frameworks for UI design? Can you point to the examples that inspire the aesthetic vision of the product? Do you know what you want your product to look like?  Is it beautiful? Does it inspire people to want to use it?
• Testability: how do you want to be able to test your code? Do you want unit tests?  End to end functional testing? What do you want the testing process to look like?  Will it be the same in the developer environment and the build environment? What tools will you consider? Do you want continuous integration? What's considered bad enough to break the build? Do you value test coverage reporting? How much test coverage is enough?
• Documentation: it can make or break a project and often it is left as an afterthought, to be completed by technical writers who have little knowledge of how the low level works. Can you make it "self documenting"? How? What do you want the documentation to look like? In what formats will it be available? Will you have a developer's guide? An install guide? A user guide? How will you keep it up to date? How will you know if it's out of date?
• Surface area: how do you define the points at which clients interface with your product (it's surface area)? A lot of this boils down to good API design. What are the best practices that you wish to implement? Which existing products / services exemplify what you want to achieve? What are the existing successful APIs, trends, industry best practices you want to emulate. Against what do you measure the usability of your API? Who are the current best-in-class leaders and how do you build upon their examples?
• Technology stack: Quite often the default in large enterprises is Java. This is unfortunate because Java is rarely if ever the best technological solution to a particular problem - it's just the one that large numbers of programmers happen know. Think outside the box - one developer with a killer tool-set is better than a team of programmers with poor tools. Have you checked out ThoughtWorks Technology Radar? How do you place bets on which technologies will succeed going forward?  Do you have a strong sense of picking technology winners that comes from being in the trenches? Or are you a manager trying to make a safe choice?  Are your choices based on cargo cult or bandwagon tendencies?  Does your framework of choice have a track record of solid bets, critical mass of users, a core team with a clear focus and commitment to keeping things small - aka. the "do one thing well" philosophy? Will you ask for permission or for forgiveness? As a lead developer, are you willing to tell your managers that the technology and technical design decisions are ultimately your decisions to make?
• Tooling: will you standardize on set of tools or let developers use whatever ad hoc collection of tools they prefer? Will you utilize linters, and static code analyzers like PMD, Findbugs, and Flow? Where in the development process where they be used - in the developer environment by triggering these tools as each file is edited? Will you trigger build failures after committing flawed code? Will you depend on a monolithic IDE like Eclipse or use a lightweight code editor like Sublime?
• Code organization / architecture: The default is the Big Ball of Mud design pattern.  Are you a devotee of Object Oriented design? Are you enlightened by Functional Programming? Prefer an Abstraction Oriented architecture? What is your high level view of how it all fits together with a view towards delegating responsibilities? How to you define the boundaries between components? Can you specify, in draft form, an APIs to act as a  'contract' between components that can then be delegated off to different developers? How will you communicate your architectural vision and get buy-in from team members?
• Prototype: have you built a proof of concept? Will you? How? Do you intend to productize the prototype, or is it meant to serve only as a reference for what you want the final product to be like? If you plan to evolve your prototype into a production application, do you have a clear path for doing so? For example, have you steered clear of design decisions in the prototype that will be difficult to back out of, like introducing dependencies on libraries or techniques that you won't be able to release in a production environment? As a real-world example: we created a mobile security demo using a relatively new technology that allowed rapid development, but made sure the API specification was well defined, knowing that it would not be difficult to re-implement the API in any server side language we chose.
• Delegation: how can the project can be split up into sensible areas of responsibility as it grows? If the project has started out with one full-stack developer, will it be easy to carve off the API, UI, mobile app, etc. into standalone projects that another developer or team can take ownership of? A lot of this comes down to having well defined and documented APIs as a contract between components.
• Security: Its not enough to trust that your product is going to be secure because it uses Framework X.  Upon inspection, our initial subscriber-facing web interface turned out to be susceptible to session hijacking, cross-site scripting, and SQL insertion, and security issues continued to be present despite being overseen by senior developers.  Security audit tools revealed even more weaknesses. What tools will you use to audit the security of your product (Skipfish, Nessus, OpenVAS, etc.)? Will you carry out manual penetration testing on your product? Will security audits be performed automatically, on a regular basis? Will they be integrated into your build environment or QA environment?

The questions you need to ask may vary, but the important thing is to have a plan, communicate the plan well, empower people to own their part of it, make the design objectives clear (including all of the points above), and make sure everyone knows how their piece fits into the overall product vision.

Fluent 2014 Talk Summaries 3

"Speed, Performance and Human Perception"

In this performance-related talk, Ilya Grigorik explains that performance is not only a function of speed, but of meeting user expectations in a way that allows them to complete tasks effectively, with insightful examples.

[video]

My rating: 4/5. Provides useful insight on usability.

"Delivering the Goods"

Paul Irish discusses optimization in this keynote talk, describing the "critical path" and how requests impact page load times. Chrome developer tools are used to explain page load sequences and timing. Recommendations include: eliminate render-blocking JS; minimize render-blocking CSS; serve content in the original HTML response and use gzip compression. Google Page Speed Test is a tool to automatically recommend such optimizations.

[video]

My rating: 5/5. Beneficial recommendations for all developers of web-based software.