Successful Software Projects

Successful software projects don't happen by accident. A recent post by Jeffrey Ventrella on Slow Programming resonates with developers who've been through enough projects to know the pros and cons of the old top-down design approaches versus the popular new "iterative development" and agile approaches.  The author wrote about his team's fast, iterative development process:

At the job, we were encouraged to work in the same codebase, as if it were a big cauldron of soup, and if we all just kept stirring it continuously and vigorously, a fully-formed thing of wonder would emerge. It did not.

The fallacy of fungible engineers - viewing engineers as resources that can be assigned interchangeably to various projects, is at the heart of the problem. Unleashing a bunch of developers on a project to "vigorously stir the cauldron" without a solid high-level design is clearly not a great idea. Iterative development may be a more realistic approach to software development, but you can't do away with up-front planning altogether.  As Ventrella says, "You can't wish away the design process."

So is iterative development a bad thing?  I don't think so. It's important to note that new paradigms are usually reactionary - they introduce a different approach to fix the failings of the old one.  Out of necessity, they're usually overly reactionary. The new approach condemns the old practices in order to get people to change, but it often fails to recognize that underneath there's probably a core of ideas still worth considering.

There are valuable aspects of both the "slow programming" approach and the "iterative" approach. It may not be necessary to define all the detailed requirements up front, but projects can benefit from asking a few simple questions to lay the groundwork before development starts.  These are things like the optimal technology stack for your problem space, documentation, and what you want your testing to look like.  Too often, these decisions are done in an ad hoc manner, late in the development cycle, or simply ignored altogether.  I feel that addressing these questions at the start of project is important for success. Here they are, roughly in the order of importance, as I see them.

• Design: design is often the worst part of software products created by large enterprises. Almost no thought goes into it; a product gets built starting with the data model; layers of code are built up on top of it until the data is finally vomited onto the user's screen in a great pile of clutter. Good design is not part of the corporate DNA of a large company usually. This makes visual design an issue of low hanging fruit for smaller competitors, who can quite easily win over potential clients with the usability and elegance of their product, even if it lacks the feature set of a more mature offering.  Conversely, great design can't be done by a non-technical designer without a deep technical understanding of the product; it's a mistake to view design as a separate activity that can take place in isolation from the technical implementation. Graphic designers are not great at understanding the technical details of a project, and sometimes force engineers into making poor compromises for the sake of preserving a look and feel.  Engineers are generally bad at user interface design, unless they also have a background and strong aptitude in the visual arts. Good engineers engineers who are also artists are rare. Unless you have one of them, you'll need a design / engineer team who can work in close collaboration to define the initial product vision. It is not going to magically emerge from a set of requirements from Product Level Managers. Ask yourself: Is there a strong vision for the aesthetic sense of your product? Is it informed by a strong technical understanding as well as a good aesthetic sensibility?  Can it be demonstrated that the design decisions are grounded in good usability guidelines and not just the whim of a non-technical "graphic designer"? Are you leveraging existing best-in-class frameworks for UI design? Can you point to the examples that inspire the aesthetic vision of the product? Do you know what you want your product to look like?  Is it beautiful? Does it inspire people to want to use it?
• Testability: how do you want to be able to test your code? Do you want unit tests?  End to end functional testing? What do you want the testing process to look like?  Will it be the same in the developer environment and the build environment? What tools will you consider? Do you want continuous integration? What's considered bad enough to break the build? Do you value test coverage reporting? How much test coverage is enough?
• Documentation: it can make or break a project and often it is left as an afterthought, to be completed by technical writers who have little knowledge of how the low level works. Can you make it "self documenting"? How? What do you want the documentation to look like? In what formats will it be available? Will you have a developer's guide? An install guide? A user guide? How will you keep it up to date? How will you know if it's out of date?
• Surface area: how do you define the points at which clients interface with your product (it's surface area)? A lot of this boils down to good API design. What are the best practices that you wish to implement? Which existing products / services exemplify what you want to achieve? What are the existing successful APIs, trends, industry best practices you want to emulate. Against what do you measure the usability of your API? Who are the current best-in-class leaders and how do you build upon their examples?
• Technology stack: Quite often the default in large enterprises is Java. This is unfortunate because Java is rarely if ever the best technological solution to a particular problem - it's just the one that large numbers of programmers happen know. Think outside the box - one developer with a killer tool-set is better than a team of programmers with poor tools. Have you checked out ThoughtWorks Technology Radar? How do you place bets on which technologies will succeed going forward?  Do you have a strong sense of picking technology winners that comes from being in the trenches? Or are you a manager trying to make a safe choice?  Are your choices based on cargo cult or bandwagon tendencies?  Does your framework of choice have a track record of solid bets, critical mass of users, a core team with a clear focus and commitment to keeping things small - aka. the "do one thing well" philosophy? Will you ask for permission or for forgiveness? As a lead developer, are you willing to tell your managers that the technology and technical design decisions are ultimately your decisions to make?
• Tooling: will you standardize on set of tools or let developers use whatever ad hoc collection of tools they prefer? Will you utilize linters, and static code analyzers like PMD, Findbugs, and Flow? Where in the development process where they be used - in the developer environment by triggering these tools as each file is edited? Will you trigger build failures after committing flawed code? Will you depend on a monolithic IDE like Eclipse or use a lightweight code editor like Sublime?
• Code organization / architecture: The default is the Big Ball of Mud design pattern.  Are you a devotee of Object Oriented design? Are you enlightened by Functional Programming? Prefer an Abstraction Oriented architecture? What is your high level view of how it all fits together with a view towards delegating responsibilities? How to you define the boundaries between components? Can you specify, in draft form, an APIs to act as a  'contract' between components that can then be delegated off to different developers? How will you communicate your architectural vision and get buy-in from team members?
• Prototype: have you built a proof of concept? Will you? How? Do you intend to productize the prototype, or is it meant to serve only as a reference for what you want the final product to be like? If you plan to evolve your prototype into a production application, do you have a clear path for doing so? For example, have you steered clear of design decisions in the prototype that will be difficult to back out of, like introducing dependencies on libraries or techniques that you won't be able to release in a production environment? As a real-world example: we created a mobile security demo using a relatively new technology that allowed rapid development, but made sure the API specification was well defined, knowing that it would not be difficult to re-implement the API in any server side language we chose.
• Delegation: how can the project can be split up into sensible areas of responsibility as it grows? If the project has started out with one full-stack developer, will it be easy to carve off the API, UI, mobile app, etc. into standalone projects that another developer or team can take ownership of? A lot of this comes down to having well defined and documented APIs as a contract between components.
• Security: Its not enough to trust that your product is going to be secure because it uses Framework X.  Upon inspection, our initial subscriber-facing web interface turned out to be susceptible to session hijacking, cross-site scripting, and SQL insertion, and security issues continued to be present despite being overseen by senior developers.  Security audit tools revealed even more weaknesses. What tools will you use to audit the security of your product (Skipfish, Nessus, OpenVAS, etc.)? Will you carry out manual penetration testing on your product? Will security audits be performed automatically, on a regular basis? Will they be integrated into your build environment or QA environment?

The questions you need to ask may vary, but the important thing is to have a plan, communicate the plan well, empower people to own their part of it, make the design objectives clear (including all of the points above), and make sure everyone knows how their piece fits into the overall product vision.