Productivity and Note-taking

I told a friend of mine that I wasn't really happy with the amount of time that gets taken up by Slack and "communication and scheduling" in general, and that on one particularly "noisy" week it had taken up around 7 hours. He said that was "hardly anything" and that most of his work was done via Slack.  He has a support role, so I guess that makes sense. So it definitely can be a useful tool, but in the past few weeks I've managed to keep Slack usage down to about 3 hours per week.

On a related topic, I've tried a bunch of different task management techniques over the years and none of them ever stuck. I've always ended up with a fragmented collection of things to do, scattered between various Notes apps, email-based task lists, Trello boards, and hand-written notes.  The problem with a lot of the software-based task management options for me is that they're not always in front of me and they take a conscious effort to open and use.

A notebook on the other hand is always on the desk beside me, usually open to the last page of notes. There's no effort getting to it, and I can easily glance over without disrupting whatever I was doing on the computer.

There's a system called a bullet journal for keeping and managing lists.  The website bulletjournal.com explains the system and has an online store with their BuJo journals that are designed to work with the bullet journal system. There's also an app, so for people who really want to get into it, there are a variety of ways. You don't need a special BuJo journal to start using the technique, however.

I've switched my list of tasks from an online document to my pen-and-paper bullet journal now. I'm not fastidious about keeping the journal up to date on a daily or even weekly basis, but I find that its just a lot easier to take notes this way than it was to type things into a digital form.  The only downside I can see is that it's harder to share than digital notes which can be copy and pasted. But frankly most of the tasks on my list are too detailed and boring for anybody else to care about. They just want to know how a feature is coming along, etc.  So, check out the bullet journal.

Slack is a major productivity drain

For the past few years I've been using RescueTime time management software.  It helps keep track of the different activities I've spent time on during the work week and stay focused on things that contribute to my productivity.  Another application that's become a big part of many workplaces is Slack. We use it for communication in my current team. A lot of times, it's really helpful. You can fire off a quick message and get a quick reply without really interrupting your main task. A lot of times you can ask a question, get valuable opinions from everyone concerned, and come up with a consensus that works for everybody, in a way that would've been impossible with traditional email and meetings.

Recently though I was shocked to see that Slack accounted for a full 7 hours of a recent work week. We've been doing a lot of planning and discussion for new feature work, and also working through some issues related to customer documentation, deployments, and things that are not specifically coding-related, so it's understandable that there's been more planning and discussion activity than usual, and less actual code-writing. But 7 hours!  My gosh.  That is a real time sink.

This was a wake-up call into something that I already intuitively knew, that Slack - as helpful as instant messaging can be - has the potential to be a black-hole where productivity gets sucked into a terminal death-spiral.

The challenge is how to reign in this Slack tyranny... if you don't monitor the conversations going on in various Slack channels, you're liable to miss out on vital information. A lot of people seem to think that broadcasting a message out on a Slack channel is "job complete" when it comes to communication - a surrogate for the old-school email.  That's not really the case; messages easily get lost in the backlog of noisy, run-on chatroom conversations.

I'm going to be keeping an eye on Slack usage and trying to figure the best way to keep it from sucking up a large percentage of my productive working hours without losing the benefit of instant communication with the broader team.  Maybe just being respectful of people's time and being a little less cavalier about using Slack to post casual commentary, understanding that Slack can definitely become a drain on my own and other people's time, and approaching it as a tool to be used with a certain level of professional self-restraint, might be a start.

Reacting to React

While I'm on the blog here, I figured I'd take a minute to write my thoughts on React / React Native.

Recently I've had occasion to play with React Native a bit. This post is not so much about first impressions, though, but about my perspective and some preconceptions about things like React in general.

I run a big JavaScript meetup group called OttawaJS. I get to see lots of interesting presentations on the latest and greatest technology, and sometimes give talks myself. Over the past few years I've seen an absolute deluge of new frameworks and libraries for web and mobile development, and like a lot of people in the tech community, have suffered from "new framework fatigue".  Perhaps this is one reason why, when React came around, I didn't get immediately excited about it. Beyond that, here are some general observations, personal biases, and preconceptions.

1. When Mark Zuckerberg famously said "HTML5 isn't ready", many people felt they hadn't given it a fair shake. The developers at Sencha proved the point by building an HTML5 clone of the Facebook app that outperformed the native one.  So Facebook hasn't been a big proponent of using the web stack for mobile apps, historically. Although I think there are some really good ideas underlying React, having a large company behind something doesn't mean it's the right solution for everybody.

2. I'm a little wary of frameworks built and promoted by large companies. Enterprises don't usually build open source frameworks without some benefit to themselves, and having more developers on a project tends to add complexity. The frameworks and tools that I usually prefer, and the ones that have generally proven most successful over time, are often written by a single author out of personal interest, and have slowly built up a following.

3. There seemed to be a lot of marketing behind React, and that can be a bad sign. It had barely been introduced and there were conferences about it and a flood of videos and articles. Paul Graham said something similar about Java a long time ago. About how good languages and frameworks don't need to be marketed, and how anything with a big marketing engine behind it just smells funny.

4. There are some cool functional programming concepts in React but they're mixed in with the classical object model. This kinda points to confusion about what the designers thought they were trying to build.

5. Generating HTML programmatically actually does suck. Other languages and framework have tried this, and they also sucked. I think Ember's Glimmer engine has taken an approach that minimizes DOM updates for great performance, without re-inventing the way HTML is written.

6. React seems to have done a lot to make people aware of cool things like immutability and components. And that's really great. However I think those concepts have value outside of React and can be applied with smaller, bespoke libraries like Redux and Riot.js.

7. My gut feeling is that it just doesn't feel like React is the holy grail of component-oriented web frameworks. It brings some cool concepts to the table and has helped shift developers' focus towards interesting alternatives to two-way binding and REST to thinks like one-way data flow and GraphQL. But I think web components and the native web stack are heading in the right direction and will ultimately replace a lot of what client-side MVC frameworks currently do.

So that's why I've told people I'm not betting on React becoming the defacto standard way to build web/mobile apps. At best, I think some of it's most useful concepts will be extracted, cleaned up and incorporated in smaller standalone implementations.

KCJS keep calm and javascript. :-)

Successful Software Projects

Successful software projects don't happen by accident. A recent post by Jeffrey Ventrella on Slow Programming resonates with developers who've been through enough projects to know the pros and cons of the old top-down design approaches versus the popular new "iterative development" and agile approaches.  The author wrote about his team's fast, iterative development process:

At the job, we were encouraged to work in the same codebase, as if it were a big cauldron of soup, and if we all just kept stirring it continuously and vigorously, a fully-formed thing of wonder would emerge. It did not.

The fallacy of fungible engineers - viewing engineers as resources that can be assigned interchangeably to various projects, is at the heart of the problem. Unleashing a bunch of developers on a project to "vigorously stir the cauldron" without a solid high-level design is clearly not a great idea. Iterative development may be a more realistic approach to software development, but you can't do away with up-front planning altogether.  As Ventrella says, "You can't wish away the design process."

So is iterative development a bad thing?  I don't think so. It's important to note that new paradigms are usually reactionary - they introduce a different approach to fix the failings of the old one.  Out of necessity, they're usually overly reactionary. The new approach condemns the old practices in order to get people to change, but it often fails to recognize that underneath there's probably a core of ideas still worth considering.

There are valuable aspects of both the "slow programming" approach and the "iterative" approach. It may not be necessary to define all the detailed requirements up front, but projects can benefit from asking a few simple questions to lay the groundwork before development starts.  These are things like the optimal technology stack for your problem space, documentation, and what you want your testing to look like.  Too often, these decisions are done in an ad hoc manner, late in the development cycle, or simply ignored altogether.  I feel that addressing these questions at the start of project is important for success. Here they are, roughly in the order of importance, as I see them.

• Design: design is often the worst part of software products created by large enterprises. Almost no thought goes into it; a product gets built starting with the data model; layers of code are built up on top of it until the data is finally vomited onto the user's screen in a great pile of clutter. Good design is not part of the corporate DNA of a large company usually. This makes visual design an issue of low hanging fruit for smaller competitors, who can quite easily win over potential clients with the usability and elegance of their product, even if it lacks the feature set of a more mature offering.  Conversely, great design can't be done by a non-technical designer without a deep technical understanding of the product; it's a mistake to view design as a separate activity that can take place in isolation from the technical implementation. Graphic designers are not great at understanding the technical details of a project, and sometimes force engineers into making poor compromises for the sake of preserving a look and feel.  Engineers are generally bad at user interface design, unless they also have a background and strong aptitude in the visual arts. Good engineers engineers who are also artists are rare. Unless you have one of them, you'll need a design / engineer team who can work in close collaboration to define the initial product vision. It is not going to magically emerge from a set of requirements from Product Level Managers. Ask yourself: Is there a strong vision for the aesthetic sense of your product? Is it informed by a strong technical understanding as well as a good aesthetic sensibility?  Can it be demonstrated that the design decisions are grounded in good usability guidelines and not just the whim of a non-technical "graphic designer"? Are you leveraging existing best-in-class frameworks for UI design? Can you point to the examples that inspire the aesthetic vision of the product? Do you know what you want your product to look like?  Is it beautiful? Does it inspire people to want to use it?
• Testability: how do you want to be able to test your code? Do you want unit tests?  End to end functional testing? What do you want the testing process to look like?  Will it be the same in the developer environment and the build environment? What tools will you consider? Do you want continuous integration? What's considered bad enough to break the build? Do you value test coverage reporting? How much test coverage is enough?
• Documentation: it can make or break a project and often it is left as an afterthought, to be completed by technical writers who have little knowledge of how the low level works. Can you make it "self documenting"? How? What do you want the documentation to look like? In what formats will it be available? Will you have a developer's guide? An install guide? A user guide? How will you keep it up to date? How will you know if it's out of date?
• Surface area: how do you define the points at which clients interface with your product (it's surface area)? A lot of this boils down to good API design. What are the best practices that you wish to implement? Which existing products / services exemplify what you want to achieve? What are the existing successful APIs, trends, industry best practices you want to emulate. Against what do you measure the usability of your API? Who are the current best-in-class leaders and how do you build upon their examples?
• Technology stack: Quite often the default in large enterprises is Java. This is unfortunate because Java is rarely if ever the best technological solution to a particular problem - it's just the one that large numbers of programmers happen know. Think outside the box - one developer with a killer tool-set is better than a team of programmers with poor tools. Have you checked out ThoughtWorks Technology Radar? How do you place bets on which technologies will succeed going forward?  Do you have a strong sense of picking technology winners that comes from being in the trenches? Or are you a manager trying to make a safe choice?  Are your choices based on cargo cult or bandwagon tendencies?  Does your framework of choice have a track record of solid bets, critical mass of users, a core team with a clear focus and commitment to keeping things small - aka. the "do one thing well" philosophy? Will you ask for permission or for forgiveness? As a lead developer, are you willing to tell your managers that the technology and technical design decisions are ultimately your decisions to make?
• Tooling: will you standardize on set of tools or let developers use whatever ad hoc collection of tools they prefer? Will you utilize linters, and static code analyzers like PMD, Findbugs, and Flow? Where in the development process where they be used - in the developer environment by triggering these tools as each file is edited? Will you trigger build failures after committing flawed code? Will you depend on a monolithic IDE like Eclipse or use a lightweight code editor like Sublime?
• Code organization / architecture: The default is the Big Ball of Mud design pattern.  Are you a devotee of Object Oriented design? Are you enlightened by Functional Programming? Prefer an Abstraction Oriented architecture? What is your high level view of how it all fits together with a view towards delegating responsibilities? How to you define the boundaries between components? Can you specify, in draft form, an APIs to act as a  'contract' between components that can then be delegated off to different developers? How will you communicate your architectural vision and get buy-in from team members?
• Prototype: have you built a proof of concept? Will you? How? Do you intend to productize the prototype, or is it meant to serve only as a reference for what you want the final product to be like? If you plan to evolve your prototype into a production application, do you have a clear path for doing so? For example, have you steered clear of design decisions in the prototype that will be difficult to back out of, like introducing dependencies on libraries or techniques that you won't be able to release in a production environment? As a real-world example: we created a mobile security demo using a relatively new technology that allowed rapid development, but made sure the API specification was well defined, knowing that it would not be difficult to re-implement the API in any server side language we chose.
• Delegation: how can the project can be split up into sensible areas of responsibility as it grows? If the project has started out with one full-stack developer, will it be easy to carve off the API, UI, mobile app, etc. into standalone projects that another developer or team can take ownership of? A lot of this comes down to having well defined and documented APIs as a contract between components.
• Security: Its not enough to trust that your product is going to be secure because it uses Framework X.  Upon inspection, our initial subscriber-facing web interface turned out to be susceptible to session hijacking, cross-site scripting, and SQL insertion, and security issues continued to be present despite being overseen by senior developers.  Security audit tools revealed even more weaknesses. What tools will you use to audit the security of your product (Skipfish, Nessus, OpenVAS, etc.)? Will you carry out manual penetration testing on your product? Will security audits be performed automatically, on a regular basis? Will they be integrated into your build environment or QA environment?

The questions you need to ask may vary, but the important thing is to have a plan, communicate the plan well, empower people to own their part of it, make the design objectives clear (including all of the points above), and make sure everyone knows how their piece fits into the overall product vision.

Defensive Shift - Turning the Tables on Surveillance

Like many people lately, I've been pondering the implications of pervasive surveillance, "big data" analysis, state-sponsored security exploits, and the role of technology in government. For one thing, my work involves a lot of the same technology: deep packet inspection, data analysis, machine learning and even writing experimental malware. However, instead of building tools that enable pervasive government surveillance, I've built a product that tells mobile smartphone users if their device, or a laptop connected to it, has been infected with malware, been commandeered into a botnet, or come under attack from a malicious website, and so on.  I'm happy to be working on applying some of this technology in a way that actually benefits regular people. It feels much more on the "good side" of technology than on the bad side we've been hearing so much about lately.

Surveillance of course has been in the news a lot lately, so we're all familiar with the massive betrayal of democratic principles by governments, under the guise of hunting the bogeyman. It's good that people are having conversations about reforming it, but don't expect the Titanic to turn around suddenly. There's far too much money and too many careers on the line to just shut down the leviathan of pervasive surveillance overnight. It will take time, and a new generation of more secure networking technologies.

Big data has also been in the news in some interesting ways: big data analysis has been changing the way baseball is played! CBC's David Common presents the story [1]:

Not everyone is happy with the "defensive shift" - the process of repositioning outfield players based on batting stats that tell coaches how likely a batter is to hit left or right, short or long.  Longtime fans feel it takes away from the human element of the game and turns it into more of a science experiment.

I tend to agree.  And to be honest, until now deep traffic inspection, big data analysis, surveillance, and definitely state-sponsored hacking, have quite justifiably earned a reputation as, well, repugnant to any freedom-loving, democracy-living, brain-having person. Nevertheless, as powerful as big data analytics, machine learning, and network traffic analysis are, and as much as they have been woefully abused by our own governments, I don't think we've yet begun to see the potential for good that these technologies could have, particularly if they are applied in reverse to the way they're being used now.

Right now we're in a position where a few privileged, state-sponsored bad actors are abusing their position of trust and authority to turn the lens of surveillance and data analysis upon ordinary people, foreign business competitors[2], jilted lovers [3], etc.  The sea change that will, I think, eventually come is when the lens of technology slowly turns with relentless inevitability onto the government itself, and we have the people observing and monitoring and analyzing the effectiveness of our elected officials and public servants and their organizations.

How do we begin to turn the tables on surveillance?

Secure Protocols

As I see it, this "defensive shift" will happen due to several factors. First, because the best and brightest engineers - the ones who design the inner workings of the Internet and write the open-source software used for secure computing - are on the whole smart enough to know that pervasive surveillance is an attack and a design flaw [4], are calling for it to be fixed in future versions of Internet protocols [5], and are already working on fixing some of the known exploits [6].

One of the simplest remedial actions available right now for pervasive surveillance attacks is HTTPS, with initiatives like HTTPS Now[9] showing which web sites follow good security practices, and tools like HTTPS Everywhere[10], a plugin for your web browser that helps you connect to websites securely. There is still work to be done in this area, as man-in-the-middle attacks and compromised cryptographic keys are widespread at this point - a problem for which perfect forward secrecy[11] needs to become ubiquitous. We should expect future generations of networking protocols to be based on these security best practices.

Some people say that creating a system that is totally secure against all kinds of surveillance, including lawful intercept, will only give bad people more opportunity to plan and carry out their dirty deeds.  But this turns out not to be true when you look at the actual data of how much information has been collected, how much it all costs, and how effective it's actually been.  It yields practically nothing useful and is almost always a "close the barn door, the horse is out!" scenario. This, coming from an engineer who actually works in the area of network-based threat analysis, by the way.

Open Data

Second, the open data movement. Its not just you and I who are producing data-trails as we mobe and surf and twit around the Interwebs.  There's a lot of data locked up in government systems, too.  If you live in a democracy, who owns that data? We do. It's ours. More and more of it is being made available online, in formats that can be used for computerized data analysis.  Sites like the Center for Responsive Politics' Open Secrets Database [8], for example, shed a light on money in politics, showing who's lobbying for what, how much money they're giving, and who's accepting the bribes, er, donations.

One nascent experiment in the area of government open data analysis is AnalyzeThe.US, a site that let's you play with a variety of public data sources to see correlations. Warning - it's possible for anyone to "prove" just about anything with enough graphs and hand-waving. For real meaningful analysis, having some background in mathematics and statistics is a definite plus, but the tool is still super fun and provides a glimpse of where things could be going in the future with open government.

Automation

Third, automation. There's still a long way to go in this area, but even the slowness and inefficiency of government will eventually give way to the relentless march of technology as more and more systems that have traditionally been mired in bureaucratic red tape become networked and automated, all producing data for analytics. Filling in paper forms for hours on end will eventually be as absurd for the government to require as it would be for buying a book from Amazon.

With further automation and data access, the ability to monitor, analyze and even take remedial action on bureaucratic inefficiencies should be in the hands of ordinary people, turning the current model of Big Brother surveillance on its head. Algorithms will be able to measure the effectiveness of our public services and national infrastructures, do statistical analysis, provide deep insight and make recommendations. The business of running a government, which today seems to be a mix of guesswork, political ideology and public relations management, will start to become less of a religion and more of a science, backed up with real data. It won't be a technocracy - but it will be leveraging technology to effectively crowd-source government.  Which is what democracy is all about, after all.


[1] http://www.cbc.ca/news/world/how-the-defensive-shift-and-big-data-are-changing-baseball-1.2739619
[2] http://www.cbc.ca/news/politics/why-would-canada-spy-on-brazil-mining-and-energy-officials-1.1931465
[3] http://www.cnn.com/2013/09/27/politics/nsa-snooping/
[4] http://tools.ietf.org/html/rfc7258
[5] http://techcrunch.com/2013/10/11/icann-w3c-call-for-end-of-us-internet-ascendancy-following-nsa-revelations/
[6] https://www.fsf.org/blogs/community/gnu-hackers-discover-hacienda-government-surveillance-and-give-us-a-way-to-fight-back
[7] AnalyzeThe.US
[8] https://www.opensecrets.org/
[9] https://www.httpsnow.org/
[10] https://www.eff.org/https-everywhere
[11] http://en.wikipedia.org/wiki/Forward_secrecy#Perfect_forward_secrecy

Parable of the Banana Leaf

Charles and Ray Eames were brilliant designers. For them, design wasn't about style, it was a philosophy and a way of living and thinking. This is from a lecture by Charles Eames.

There's sort of a parable I'd like to . . . In India . . . I guess it's a parable: In India, sort of the lowest, the poorest, the, those, those without and the lowest in caste, eat very often--particularly in southern India--they eat off of a banana leaf. And those a little bit up the scale, eat off of a sort of a un . . . a low-fired ceramic dish. And a little bit higher, why, they have a glaze on--a thing they call a "tali"--they use a banana leaf and then the ceramic as a tali upon which they put all the food. And there get to be some fairly elegant glazed talis, but it graduates to--if you're up the scale a little bit more--why, a brass tali, and a bell-bronze tali is absolutely marvelous, it has a sort of a ring to it. And then things get to be a little questionable. There are things like silver-plated talis and there are solid silver talis and I suppose some nut has had a gold tali that he's eaten off of, but I've never seen one. But you can go beyond that and the guys that have not only means, but a certain amount of knowledge and understanding, go the next step and they eat off of a banana leaf. And I think that in these times when we fall back and regroup, that somehow or other, the banana leaf parable sort of got to get working there, because I'm not prepared to say that the banana leaf that one eats off of is the same as the other eats off of, but it's that process that has happened within the man that changes the banana leaf. And as we attack these problems--and I hope and I expect that the total amount of energy used in this world is going to go from high to medium to a little bit lower--the banana leaf idea might have a great part in it.

The Global Power Shift

Paddy Ashdown's fantastic speech at TEDx Brussels, which was published today.  This is well worth watching for anyone interested in what is happening in global power structures and the end of the American century.  Absolutely brilliant talk, at once troubling and still inspirational.

Checking In

Hard to believe the month of November flew by without so much as a peep out of 51 Elliot.  Between getting my boat put away for the winter, removing stain from my porch and cleaning the wood, spending too many hours on floorplanner.com and various architectural websites, and raking up about 87 bags of autumn leaves, it just happened.

And, instead of posting my usual techo / philosophical points of view, the last couple of months have been mainly a time to sit back and observe. The Occupy movement has been in the news a lot, the OpenMedia.ca campaign in Canada has been building momentum, and businesses of the digital economy are speaking out against the abuse of copyright by media conglomerates as manifest in the much-hated SOPA and Protect-IP proposals. I'm looking forward to seeing how it all plays out, and I hope it results in a better world for everyone.  A Christmas wish, a bit early.

I did manage to accumulate a fairly big back log of interesting articles, so here to make up for my recent silence is the second ever web-dump of nifty things I found interesting:

US Big Brother wholesale spying aka. the PATRIOT Act is taking a bite out of the US cloud computing market. Business are choosing not to use the products and services of companies like Microsoft and Amazon, who are bound by the laws of a government that has created what is considered an "indemic surveillance society".

Fabian Pascal has a very good paper on database normalization, integrity and performance.

The first in a series of posts on damn cool algorithms... this one's on BK Trees.

This was the year for Javascript. Javascript on the client, Javascript on the server, everywhere you look Javascript. This is great! Why? Because JavaScript Is Not Java.  Thank goodness.

This year Node.js, the server-side Javascript engine based on Google's blazing-fast V8 Javascript engine, took off like wildfire. Node.js modules you should know about is a series of posts on... um, Node.js modules you should know about.

I know that Chrome is what all the cool kids are using. But, given the aforementioned reference to Big Brother, I still use Firefox frequently.  For web developers, there are a ton of great web developer plugins for Firefox, and this post covers a half-gazillion of them.

Paper.js is a neat little Javascript graphics library for programmers of neat little Javascript graphics.

Or perhaps you want to try doing face detection in Javascript?

One of my biggest complaints about any library-heavy language is poor curation. Java suffers terribly from this, and so do jQuery and Node.js to a degree. That's why I thought The Hand-picked jQuery Plugins Repo was pretty interesting.

Mozilla Popcorn hasn't been generating the same amount of buzz as some of the other web debutantes, but it looks interesting for the future of web video. In particular the National Film Board's project "One Millionth Tower" is... really cool to watch and play with.

It was about a year ago I predicted a rise in popularity of darknets or mesh networks. So I was delighted to find a solitary article detailing the rise in popularity of darknets. Well, there were a couple, actually.  This is going to happen sooner or later because politicians and corporate fascists are too technologically illiterate to understand the way their old-school strategies are pushing new-school innovation forward. As a colleague of mine says, the Internet views censorship as a failure and routes around it.

The JavaScript PC Emulator by Fabrice Bellard is mind-numbingly jaw-droppingly cool. If you're into that sorta thing. Bellard is one amazing researcher.

I really like the Crescent Road House. It's beautiful.

We have it pretty good in Canada. Canadians are officially richer than Americans. We have better privacy laws. We have a great health care system.  But, what keeps high tech innovation from happening here on a larger scale. According to this post, it might be because Canadian laws would have made starting up Facebook or Youtube impossible in Canada.

Do you feel like taking a course in Machine Learning from Stanford University?  It's free. As should all education be in this information age... and will be.  Oops, I guess that's my prediction for this post.  Watch for a movement around free higher education based on meritocratic principles.

Oh, here's that other article on the rise of darknets / mesh networks that I was talking about.  Why are people doing this? Fear of repression.  Yes, even in America, once known as "the land of the free".

How many mobile frameworks could a woodchuck chuck, if a woodchuck could chuck mobile frameworks? He'd chuck all the mobile frameworks on this page and then probably end up using Phonegap.

I can't resist one last JavaScript item, a truly incredible and beautiful work by Mr. Doob... yep, it's three.js. I used to say that web technology was poised to do everything traditional desktop software programming environments could do, with the exception of 3D video games.  That is no longer the case.  I for one, welcome our new JavaScript overlords!

Happy reading and have a great week.

Japanese Craftsmanship

In his wonderful and comprehensive book, "Japanese Homes and Their Surroundings", first published in 1885 and now available from Google Books, Edward S. Morse writes the following delightful description of Japanese home-building craftsmanship.

Within these houses there are often to be seen marvels of exquisite carving, and the perfection of cabinet work; and surprise follows surprise, as one becomes more fully acquainted with the interior finish of these curious and remarkable dwellings.

... as to altering the present plan of housebuilding... If such changes are effected, then will perish many of the best features of true Japanese art, which has been the surprise and admiration of Western nations, and of which in the past they have been the unwitting cause of modification and degradation it has already undergone.

A somewhat extended experience with the common everyday carpenter at home leads me to say, without fear of contradiction, that in matters pertaining to their craft the Japanese carpenters are superior to American. Not only do they show their superiority in their work, but in their versatile ability in making new things. It is a notorious fact that most of the carpenters in our smaller towns and villages are utterly incompetent to carry out any special demand made upon them, outside the building of the conventional two-storied house and ordinary roof. They stand bewildered in the presence of a window-projection or cornice outside the prescribed ruts with which they and their fathers were familiar. Indeed, in most cases their fathers were not carpenters, nor will their children be; and herein alone the Japanese carpenter has an immense advantage over the American, for his trade, as well as other trades, have been perpetuated through generations of families. The little children have been brought up amidst the odor of fragrant shavings, -- have with childish hands performed the duties of an adjustable vice or clamp; and with the same tools which when children they have handed to their fathers, they have in later days earned their daily rice.

When I see one of our carpenters' ponderous tool-chests, made of polished woods, inlaid with brass decorations, and filled to repletion with several hundred dollars' worth of highly polished and elaborate machine-made implements, and contemplate the work done with them, -- with everything binding that should go loose, and everything rattling that should be tight, and much work that has to be done twice over, with an indication everywhere of a poverty of ideas, -- and then recall the Japanese carpenter with his ridiculously lights and flimsy tool-box containing a meager assortment of rude and primitive tools, -- considering the carpentry of the two people, I am forced to the conviction that civilization and modern appliances count as nothing unless accompanied with a moiety of brains and some little taste and wit.

It is a very serious fact that now-a-days no one in our country is acquiring faithfully the carpenter's trade. Much of this lamentable condition of things is no doubt due to the fact that machine-work has supplanted the hand-work of former times. Doors, blinds, sashes, mouldings are now turned out by the cord and mile, and all done in such greedy haste, and with the greenest of lumber, that if it does not tumble to pieces in transportation it is sure to do so very soon after entering into the house-structure. Nevertheless, the miserable truth yet remains that any man who has nailed up a few boxes, or stood in front of a circular saw for a few months, feels competent to exercise all the duties of that most honorable craft, -- the building of a house.

Windows 8 gets on the Web Stack Bandwagon

I'm a Mac / Linux guy, but it's good to see Microsoft putting some serious work into the user experience in the Windows 8 preview.  What's cool about the video on that link is that they've also seen the light when it comes to the developer experience, paving the way for rapid application development by leveraging the web stack of HTML5 and JavaScript.  A long way back I wrote about  "Gödel, Escher, Bach" to explore the link between artistic creativity and great engineering work - how a love of technical complexity drives some developers, compared to a love of elegance and simplicity that drives another kind of developer.  While technical complexity enamors the uber-geek, artists are inspired by technology that gets out of the way and allows them to do what they do best - create stuff.  Even though a web stack wouldn't be my first choice for some kinds of apps, it's just fine for a whole lot of them, and when it comes to rapid application development its way, way up high on the list of best options.

It seems like in the midst of Apple's incredible growth in the mobile, laptop, and tablet space there's been a lot of soul-searching going on at companies like Nokia, Microsoft, RIM, and many others.  A dominant theme that keeps coming up is "simplicity".  For example, see "Why Dropbox beat Syncplicity".  By embracing web technologies for app development  Microsoft is taking a forward-looking step towards simplicity in application development.  Hopefully this means standards-based web technologies and not the embrace-and-extend, proprietary model we have come to expect from Microsoft.

Things all seem to be moving in this direction.  RIM has a platform for building apps on web standards for the Blackberry.  The iPhone has had "web apps" for a long time.  And recently I've run into more people who are excited about Nitobi's PhoneGap, a technology I reviewed a few months ago (PhoneGap and jQuery Mobile First Impressions) which let's you build apps for 6 different smartphone platforms using standard web technologies

I think I just set a record for cross-linking to my own posts!  To make up for it next time I'll post a bunch of external links to cool stuff from the interwebs.

Omar

Prayer for Omar by darrenderidder

Here's a short composition I wrote using Omnisphere, a virtual synthesizer. It's basically a meditation on making a choice between two different mindsets: forgiveness, reconciliation, and respect for human dignity... versus fear, bitterness, vindictiveness and disregard for human rights. The song is meant to be a musical expression of the first mindset.  

Tell CRTC to Reverse UBB

The CRTC is soliciting input from Canadians on anti-competitive Usage Based Billing practices by Bell, Rogers, and other incumbent ISPs.  The Open Media group has a convenient online form letter that you can submit to the CRTC.

http://openmedia.ca/crtc

Go the Open Media page and let your voice be heard by the CRTC!

When I wrote that "dumb is the new smart" in December, predicting that consumers would demand dumb pipes, it was before the FCC ruling on the merger of Comcast & NBC and the CRTC 2011-44 decision.

That CRTC ruling and subsequent public uproar has brought the issue to a head faster than I imagined.  Alas it seems that big ISPs like Bell and Rogers don't get it yet, but a lot of Canadians seem to think that dumb pipes are a smart choice. Yay! They're demanding it loudly, even if the big incumbent ISPs were too slow to see it coming.

Petition Anti-competitive Internet Fees in Canada

Canadian ISPs want to charge you so much for Internet usage that it would be cheaper and faster to buy the most expensive solid-state hard drives and ship them across the country than it would be to download the same data online.  Sign the petition against Usage Based Billing.

Why Dropbox Beat Syncplicity

If you follow this blog you know I'm a big fan of simplicity in design.  Most software development is rife with complexity, and a lot of developers just love to wallow in it.  It's a good recipe for Epic Fail.

Isaac Hall addresses this in a great post on why Dropbox beat it's competitors, summarized by Graham Wetzler.  It echoes similar stories on the reasons for Facebook's displacement of MySpace.  The idea that "features equals power" is a myth.  As Paul Graham said, power is the result of conciseness.  More programmers need to get this.

Syncplicity lacked simplicity.   Anyways, good read.

[via HN]

Dumb Is The New Smart

Source: http://i.imgur.com/5RrWm.png

Quite a while ago I wrote a post called "Channelization" and predicted that carriers would attempt to slice-and-dice Internet access in a myriad of ways in an attempt to bilk customers for accessing web-based content and services available on the open Internet.  Carriers are chafing at being nothing more than "dumb pipes" which dutifully carry packets of information. Though they are paid to be the mail-carriers of the Internet, entrusted to deliver packets of information like the post office delivers any letter with a postage stamp, for some of these unscrupulous ISP's the temptation to discard or delay the packages they don't profit as much from, charge extra for receiving packages from your favourite senders (even though the postage has already been paid), and give preferential treatment to their own services is too much to resist.

This prediction is now getting a lot closer to reality, as outlined in the following article on Techdirt:
DPI Firms Trying to Turn Net Neutrality Satire Into Reality.

The basic idea is that your Internet and wireless providers would like to monitor everything you do online, and charge you extra every time you view a page on Facebook, watch a Youtube video, or stream a movie from Netflix.

This is a bad idea.  Most people realize this type of coercive monopoly, bundling and tying are  anti-competitive maneuvers that are sure to run afoul of competition law, provided the government hasn't completely sold out to Big Corp.

Indeed the FCC is set to vote on December 21st on guidelines to protect Net Neutrality (the notion that your Internet Service Provider shouldn't be able to block certain websites while giving preferential treatment to their own products and services).  Their proposal has drawn criticism from all sides, however, and comes with some caveats, particularly exemptions for mobile service providers. Even the UN is setting up a committee on Internet governance.  They've decided that only governments can have a seat on this committee, a decision that Google reasonably rejects.

So as this plays out, here's a new prediction:

"Dumb Is the New Smart".

Wireline and wireless carriers have a responsibility to deliver packets from a sender to a receiver in the same way that a mail carrier delivers letters.  The measure of a carrier, snail-mail, electronic, wireless or otherwise, should be how reliably, efficiently, impartially and cost-effectively they deliver the goods.

This type of discriminatory "channelization" is going to be a hard sell.  Carriers who choose to go down this path will find themselves at a competitive disadvantage to those who continue to offer unfettered Internet access.  In a competitive landscape defined by greedy nickle-and-dime service offerings and anti-competitive favoritism, the companies that come out waving a banner that says "We're A Dumb Pipe And Proud Of It!" are going to get the most customers.

The reason is that all the complaints carriers are making about the costs of carrying all that bandwidth is mostly BS.  There's lots of dark fiber in the ground - the bottleneck is the last mile.  The only reason they're getting up to these shenanigans is because its a money grab, and they think they're in a position to abuse their network monopoly.

In the next year or so watch for service providers to start offering net neutrality - unmonitored, unregulated open access -  as a competitive differentiator, regardless of whether or not their competitors have erected Internet toll booths.  See for example this page touting unlimited mobile browsing.  The bottom line is that once you've offered something to customers, it's very difficult to claw it back and try to slap a bunch of limitations on it.

The way forward for carriers is not through taking fair and open service and imposing restrictive, expensive policies on it.  I think the way forward will be to create value-added services that leverage their network infrastructure in ways that are independent of content delivery.  Its not about creating smarter pipes.  Its about creating better faucets.  Location, social awareness, convergence, and security hold potential in this area.

Update: Read Al Franken's post on The Most Important Free Speech Issue of Our Time.
It delves more into the spat between Comcast and Level 3 over Netflix traffic.

Webdump

Cleaning out the bookmarks folder... random interesting stuff from around the Interwebs.

• Cinder is a C++ library for doing some really cool graphical things.
• Issuu - You Publish You can read different magazines online here.   Apple won't let them have their app on the iPhone.
• http://www.khanacademy.org/  Pretty much anything you want to learn about, you can learn about in 10 minutes here.  Khan Academy rocks.
• You can read all about the dirty secrets of Canadian conservatives at "Pushed to the Left".
• Surprise, surprise, the US spies on their citizens.  A lot.  Moving along...
• Sharebuilder from ING looks like a not bad way to get going with a stock portfolio for not a lot of cash up front.
• Whoo hoo, an SSH port forwarding cheat sheet.  And here are 25 cool SSH tricks you can try at home!
• Ever thought those cables at the electronics store are stupidly over priced?  They are.  Go here for cheap cables (in Canada).
• If you haven't seen it yet, Arcade Fire has a cool HTML5 web video.
• Aviary.  Cool online tools for creating art.
• Maplight.org attempts to draw the connections between money and politics.  hmm...
• Martin Bailey's photography podcast is great for photography buffs.
• Stuck in your ways? A study finds people avoid views that contradict their existing ones.

LynchMob 2.0

So, Amazon, Paypal, Visa and Mastercard cave in to pressure from the State Department and bypass due legal process altogether  by shutting down Wikileaks accounts.  No trial, judgement, no conviction, no nothing, they just get pressure from Lieberman and that's it.  Judge jury and hangman all in one.

Gizmodo just released a piece, "Why the reaction of governments to Wikileaks should scare the hell out of you".  It echoes my previous post, and it's nice to know there are other people who see things the same way.

Facebook and Twitter joined the lynch mob today by shutting down Anonymous, the decentralized hive mind who's Low Orbit Ion Canon has taken down Mastercard, just days after an unknown botnet targeted Wikileaks servers. Meanwhile a lot of people are wondering what's in the insurance.aes256 file, and when the key will drop.

(Picture of the "free speech zone" at the 2004 Democratic National Convention)

Secrets and Democracy

I've written at length about this before, but the furor over Wikileaks prompted me to write again.  I've been thinking over privacy, freedom, governance, responsibility, and transparency for a while and trying to come up with a nice model of how they work together.  That's still a work in progress, but it goes something like this:

As responsibility increases, so does the requirement for accountability and transparency.  For governments, who hold great responsibility, the need for accountability and transparency is high.  For individuals in a free democratic society, they are mainly responsible for themselves, and the need for accountability and transparency is minimal.   The more people you're responsible to, the more you need to be accountable and transparent to them.  In my mind, responsibility and privacy are on opposite sides of the spectrum.

Privacy belongs to individuals.  Your privacy is inseparable from your freedom in a democratic society.  On the other side of the spectrum, you have transparency, which is inseparable from accountability.  And responsibility for good governance in a democratic society needs accountability and transparency.

Governments should be transparent.  They shouldn't really have a whole lot of secrets to keep from the public.  Individuals, on the other hand, should have the right to privacy in order to exercise their democratic rights and freedoms.

This whole system has been turned upside down in the name of security.  The argument is that, in order to have security, you have to give up your privacy.  Now people are getting their phones tapped, their cards traced, and being sexually violated in airport security line ups.  And at the same time governments start screaming bloody murder over the Wikileaks pat-down they've received. They don't like it when people touch their junk.

Democracy isn't supposed to work that way.  When privacy is taken away from individuals and given to those in authority, the basis for democratic freedom is destroyed. This is what we're seeing in America today.

"They who can give up essential liberty to obtain a little temporary safety, deserve neither liberty nor safety."  - Benjamin Franklin

Further reading:
Why Wikileaks is Good for America
http://www.wired.com/threatlevel/2010/12/wikileaks-editorial/
Wikileaks and the Long Haul
http://www.shirky.com/weblog/2010/12/wikileaks-and-the-long-haul/

Happy Thanksgiving

If you live in the US, which most of my readers do, I'd like you to know that up here in Canada, and in a lot of the rest of the world, we're looking at the unprecedented loss of civil liberties and freedoms evidenced by the unbelievable violations of personal privacy and basic human dignity propagated by the Transportation Security Administration, and we are thankful today that we live in free and democratic societies that still protect the rights and freedoms of their citizens.

There's a purple heart and a letter from President Franklin D. Roosevelt in my home to commemorate the sacrifice made in defense of freedom when my uncle died as a paratrooper in the 101st Airborne Division.  I know deep down that he and many others like him did not die so that innocent Americans could be strip searched, humiliated and assaulted in their own homeland by a government gone insane with security paranoia. That's not freedom. 

Happy Thanksgiving.

Keynote Notes

Jobs is giving his keynote, introducing the next generation of iPods, iTunes and Apple TV.  One of the things I like, probably everybody likes, about Apple is their continual innovation.  One takeaway from the keynote is Jobs' description of the driving factor behind Apple's innovation and success:

We listened to what customers wanted...  This is what customers were telling us...  Our customers wanted this.

Apple delivers.  Cha-ching.