ntschutta.com

After listening to an OOPSLA podcast about a workshop on Fred Brooks‘ widely read No Silver Bullets, I was inspired to reread his seminal piece. Though 20 years old, I was struck by just how applicable NSB is today and while there are a few things that place it in time, as I’ve said before, the more things change, the more they remain the same. Heck, I even decided to assign it at dynamic language camp. Much of what Brooks writes about relates to accidental vs. essential complexity, a topic that’s echoed by Neal Ford in this post and Reg Braithwaite here. Stu Halloway touched on this at Code Freeze this year though he rephrased the concept as essence vs. ceremony. More and more, we’re finally heeding the message found in this C. A. R. Hoare quote:

“Programmers are always surrounded by complexity; we cannot avoid it. … If our basic tool, the language in which we design and code our programs, is also complicated, the language itself becomes part of the problem rather than part of its solution.”

Anyway, on to Brooks. In the spirit of a number of Ted Neward’s posts, I’ll take snippets of NSB and inject my thoughts. Let’s start near the top with this gem:

“[Germ theory] told workers that progress would be made stepwise, at great effort, and that a persistent, unremitting care would have to be paid to a discipline of cleanliness. So it is with software engineering today.”

Though perhaps not what he intended, I see this as yet another call for continuous integration as well as fixing broken windows. It isn’t easy, it takes a great deal of work, but when we fail to be diligent, our “patients” get sick. And anyone that’s ever worked on decaying software knows how much fun that is…

This quote should be endlessly fed to those that think programmers are essentially typists:

“I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.”

This work is made up of thought stuff - and anything we do to disrupt flow will ultimately hurt our chances of successfully developing software.

To those that think some tool or modeling language will make software so easy anyone can do, I’d counter with this:

“The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence.”

In other words, software is hard…though we often make it harder. Sometimes that’s related to our organizations:

“Much of the complexity that he must master is arbitrary complexity, forced without rhyme or reason by the many human institutions and systems to which his interfaces must conform. These differ from interface to interface, and from time to time, not because of necessity but only because they were designed by different people, rather than by God.”

Further confirming that the problems in software are largely people oriented, one can practically hear Brooks’ echo in the agile manifesto:

“The central question in how to improve the software art centers, as it always has, on people.”

Technologies don’t kill projects, people do and better programmers really are better, a point made by Neal as well as Paul Graham:

“The differences between the great and the average approach an order of magnitude.”

These days, I’m asked often about how we’re going to “scale up” our development teams which is really just management speak for “off shore 80% of the work.” Now, fundamentally, I don’t have any issues with taking advantage of vast labor pools, but I’d much rather have a small team of top notch developers than a large team of, well, less than average ones. I’m not sure if it’s just the fiefdom complex or the overriding dictate of distant management, but big teams are usually problematic. With a few great developers, I can move the world. And let’s never forget garbage in, garbage out.

Back to Brooks - he’s more entertaining than I am. He touches on something near and dear to my heart praising higher level languages:

“Surely the most powerful stroke for software productivity, reliability, and simplicity has been the progressive use of high-level languages for programming. Most observers credit that development with at least a factor of five in productivity, and with concomitant gains in reliability, simplicity, and comprehensibility.”

Seems to echo with what people say about Rails, Ruby and a host of other languages these days. Expressiveness matters - a lot.

I’d argue this is largely what Stu was getting at in his Ending Legacy Code talk:

“[Abstract types and hierarchical types] removes yet another accidental difficulty from the process, allowing the designer to express the essence of the design without having to express large amounts of syntactic material that add no information content. For both abstract types and hierarchical types, the result is to remove a higher-order kind of accidental difficulty and allow a higher-order expression of design.”

Getting rid of boiler plate and focusing on the problem at hand is key. I know a lot of developers who defensively shout “but my tool handles all that for me.” Sure. See above. And don’t forget, you or someone coming in behind you will still have to read all those excess symbols.

Though I wish buying new hardware would solve all our woes, Brooks reaffirms what we already know:

“More powerful workstations we surely welcome. Magical enhancements from them we cannot expect.”

Bummer. Guess I’ll need a better reason to get that new MBP.

There’s quite an agile flavor to NSB and a number of Brooks’ comments speak very well of what is becoming a more and more common approach to writing software. I’m not sure about you, but I haven’t seen much success with waterfall…but iteratively developing solutions seems to work, a point he makes quite clearly:

“Therefore, the most important function that the software builder performs for the client is the iterative extraction and refinement of the product requirements. For the truth is, the client does not know what he wants. The client usually does not know what questions must be answered, and he has almost never thought of the problem in the detail necessary for specification.”

Despite what some think, you just can’t get it all right up front. This isn’t some fundamental failing, it’s a feature not a bug. Rather than attempt to fight this, just work with it; instead of trying to write it all on the plan before you break ground, iterate. Software types tend to be good abstract thinkers, but our customers often aren’t - thus why getting working products in front of them early is so important:

“I would go a step further and assert that it is really impossible for a client, even working with a software engineer, to specify completely, precisely, and correctly the exact requirements of a modern software product before trying some versions of the product.”

The value in this approach seems to obvious yet it still isn’t “the norm.” I honestly cannot understand why customers don’t demand this process.

I try not be as wordy as Steve Yegge but Brooks summarizes my post on house building in less than fifty words:

“Much of present-day software-acquisition procedure rests upon the assumption that one can specify a satisfactory system in advance, get bids for its construction, have it built, and install it. I think this assumption is fundamentally wrong, and that many software-acquisition problems spring from that fallacy.”

He goes on to discuss growing software, an analogy that really speaks to me. I remember mentioning something along these lines to a former co-worker only to be rather rudely dismissed:

“Incremental development—grow, don’t build, software.”

When I discuss agile with skeptics, I really try to hammer on this point:

“One always has, at every stage in the process, a working system. I find that teams can grow much more complex entities in four months than they can build.”

All I can say is, I’m no Fred Brooks. But what does he know right?

Systems that spin off for months (or years) without that iterative review tend to fail - often rather expensively. I remember one project many years ago where the client eventually ran out of money (remember the dot com implosion?) Due to decisions made outside my influence, we really didn’t have anything he could use. Sure, he could “demo” the product, but he certainly couldn’t sell it. We’d spent a great deal of time designing everything in the system - all the screens, all the interactions…too bad we hadn’t spent more time building. Had we worked more iteratively, he would have had *something*. Oh well, lesson learned.

As I mature in this industry, I become more aware of our pioneers; as I find my path I discover how far out they saw. Though recent times have seen amazing advances, we have much to learn from our past.

A few years back, my wife and I built a house. OK, so *we* didn’t actually swing a hammer; if you’ve ever seen me attempt a project around the casa, you know hiring the project out was the sanest course of action. This was my second go round with home construction, a process that many say they’d never repeat. But hey, I’m a glutton for punishment so in we plunged! I’d mostly walled off the entire experience in that place we put painful events like child birth and two-a-day practices, but last week one of my projects got me thinking that maybe, just maybe, there’s a comparison to me be made between house construction and building software.

Before I get started, I have to consider whether I’m on shaky ground here - many people have written pieces questioning the whole construction analogy. For starters, my friends Neal Ford and Glen Vanderburg have their takes with building bridges without engineering and bridges and software. Jack W. Reeves’ classic What Is Software Design? is a must read and Reg Braithwaite has a great post about what he admires about engineers and doctors which has this money quote:

Try this: Employ an Engineer. Ask her to slap together a bridge. The answer will be no. You cannot badger her with talk of how since You Hold The Gold, You Make The Rules. You cannot cajole her with talk of how the department needs to make its numbers, and that means getting construction done by the end of the quarter.

Considering the shared experience behind that impressive collection of wise words, I’m questioning my sanity to even think a construction analogy might fit software. But, even though I’m conflicted about the whole thing, I’m still going to share. Heck, maybe I’ll learn something in the process.

When we built our house, we spent several weeks looking at model houses, poring over floor plans, looking at carpet samples - all sorts of fun stuff. Now, with the last “home project,” during the requirements gathering phase we were just focused on the big issues: do you want a built in here? Would you like a fireplace? How about a skylight here? You know, the stuff you’ve got to get right before the foundation is set and the walls are up. At various stages, I’d get a call from the project manager (yep, that was his title, I’m feeling more confident already!) and he’d setup a time for me to come out to the site and work with one of the trades on issues like outlet placement. With my sample size of one, I expected a similar (iterative) process this go round - alas I was wrong.

You see, this builder had a different approach. They believed heavily in making all (and I mean ALL) the decisions up front (can you say BDUF?) Being a software geek, I think I do a pretty good job of thinking abstractly but needless to say, it can be quite a challenge to figure out where you want your phone jacks when all you have to go on is a 2D model of your future dwelling. I pushed back on the builder and was told they did this for a reason - they felt that if everything was on the plan, I could go on (as they put it) a four month vacation and come back to a completed house *exactly* as I intended it to be.

As much as I wanted to believe the people I was about to give a very large check to, I wasn’t convinced and as you might expect, my wife and I were on site pretty much every other day keeping track of what was going on. It was a good thing we were vigilant customers constantly running our acceptance tests. Nearly every visit revealed something that needed to be fixed, a story to be added to the backlog (or punch list in this case.) Some things were minor - a switch not controlling the proper light or a misunderstanding about what the plumbing code would allow. But others were, well, of the show stopper category. For example, despite a very clear floor plan showing where the washer and dryer were to be, the plumber decided he’d just put the washer where it was in every other house. Thank goodness we caught it early but this whole “we get it on the plan thing” certainly didn’t work in practice.

So what the heck does this have to do with software? Well, one of my projects has a customer group that thinks like my builder - they want to give us all the requirements, we give them an estimate, then they don’t talk to us until the project goes live. Obviously, this doesn’t work too well. I’m not sure why people ever thought this approach worked, heck, if we can’t get it right with houses (something we’ve been building, oh, forever) how can we possibly get close with a discipline as new as software? No, the answer is found in an agile approach where we work closely with our customers. Does that mean we need to see them for eight hours a day everyday? I sure hope not, but if they aren’t willing to commit some time to the project, how important could it actually be?

Needless to say, we’re trying to get the customers to think different, to embrace a more collaborative approach and I hope we succeed. Otherwise, I have a pretty good idea what will happen after that four month vacation.

Ours is a very young profession - we don’t have hundreds or thousands of years of experience. But we are blessed with some amazing pioneers that did truly groundbreaking work - discovering things that are still amazingly powerful today. Sometimes I think we forget our history or at a minimum fail to understand it. When you think about the essential paradigm of mainframe computing, it isn’t that different from web applications, something I mentioned at Code Freeze (and something Douglas Crockford covers in his State of Ajax talk.)

Of course it isn’t just raw technology that echoes its past - no, our development languages do as well. Look at the features we are trying to add to mainstream languages today - despite what some developers think, these aren’t new ideas. Languages exist on a spectrum, and many believe Lisp, invented fifty years ago, is the most powerful language around (though it’s hard to get paid to write it.) Usually, things improve with time; I don’t think any of us would trade in our MacBook Airs for an Altair but apparently not so with languages - I’m not really sure what to make of that.

Anyway, back to history. On the way home from the Code Freeze speakers dinner the other night I was listening to the radio when I stumbled upon A Technical Camelot, the story of Jean Bartik whose first job was as a “computer.” Let me repeat that - her title was literally human computer (which makes more sense than some of the titles I’ve held frankly.) In the mid 40s, she found herself in the midst of history as one of the first programmers on the ENIAC and listening to her describe how she became a programmer and the kinds of things she was working on was quite something.

My point here is simple: history matters. With the advances in hardware, we’ve done some amazing things, but let’s not forget where we came from and the people that propelled us to where we are today.

As the year comes to a close, I wanted to note a few interesting things I’ve read of late. I’m sure I could come up with a few hundred words about each but the kid is napping so I’ll keep this short and to the point. I first ran across this quote on Daring Fireball (a must read BTW) but seeing it again on Reg Braithwaite’s Raganwald spurred me to post it here:

Any sufficiently advanced incompetence is indistinguishable from malice.

Well, that explains an awful lot.

Mike Nygard had an interesting post about some of the common pathologies we see in this silly industry. I found the part about negotiating particularly fitting but if you haven’t seen budgetculture, I’d love to know where you work. My friend Ted Neward has his annual prediction piece - lots of very interesting fodder hiding in there. All I have to say is Granny Schottenwilkins has been a proud Mac user for years but I’m sure there are some Windows 98 users in Wisconsin.

Jeff Atwood’s Coding Horror reminds us that we all write shitty code. I’ve run into some amazingly arrogant developers that were convinced everything they wrote was pure gold, too bad they’re the types that generally don’t read, well, anything. Speaking of Jeff, I liked his take on registration keys though I’d point out that some companies trust their users…

Anyway, just a few good reads to finish off 2007. Happy New Year to all three of my readers!

In case you’re looking for a great one day software symposium, UMSEC will once again be hosting Code Freeze January 17th from 8 to 5 at the McNamara Alumni Center. The price is right especially if you’re a MSSE alum. Our topic is innovation and we’ve got a great lineup of speakers (if I do say so myself) - hope to see you there!

Congrats to local tech guru Michael Nygard - his book Release It! has been named a Jolt Finalist. Way to go Mike! In case you haven’t noticed, Pragmatic Bookshelf has some awfully good titles…

I had lunch with my friend Dan on Monday (and finally got to a RUM meeting…) and he mentioned that local guru Mike Nygard was on a recent Pragmatic Podcast. As you might expect, I pointed iTunes at the feed and was not disappointed. Mike wrote Release It, a very important book on making sure your code is production ready. Anyway, well worth a listen. Speaking of podcasts, my friends Neal Ford and Andy Glover’s chat on JRuby et al is up on JavaWorld. Good stuff guys! BTW, Andy, if you’re looking for other people for podcasts, I *cough* know a guy.

Just a heads up that I’ll be speaking at Twin SPIN next week - here are the details. I’ll be giving one of this season’s No Fluff talks. Hope to see you there! Unfortunately that means I won’t be at the latest edition of Minnedemo but I think that just gives you an idea of just how much tech goodness is happening in the Twin Cities these days.

In my Test Infecting talk, I do my best to counter a number of myths that I (and others) have encountered when introducing testing into an organization. One of the most persistent misconception revolves around time - or rather the lack thereof. Many a developer has claimed they don’t have time to test to which I generally reply with a Pat Parelli quote from this post on Kathy Sierra’s blog:

“Take the time it takes so it takes less time.”

Kathy was talking about multitasking but my point is simple: forgo testing and you’ll pay that price plus more later when the defects start rolling in. While I *think* this is persuasive, Dean Wampler went one better by using charts which we all know makes for a better argument Dean makes some great points in Why you have time for TDD (but may not know it yet…) though the part about moving unscheduled project end time up earlier into the project really hit home.

Ranges are fine and the key to success is frequent milestones; as we learn more about the problem domain and the technology we are using, the more accurate our estimates. But most organizations take a random guess (with, I’d say, a wind’s spittle of support) and turn that into a concrete date around which the world turns. They then ignore all the little milestones (if they track them at all) or they green shift the project status. The result is failure, though sometimes we redefine that word to mean something else entirely…

Like any industry, ours likes to pretend that today’s problems are new and unique - it’s part of how we justify our salaries. And while the demands on what we are expected to do with technology continues to exceed the gains made with improved languages and hardware, the more things change, the more they stay the same. For example, take collaborative work environments. With today’s focus on outsourcing, far flung offices, and distributed supply chains, using technology to help distributed teams work together is a must. While some people might think this is a relatively new phenomena, in truth, Douglas Engelbart (inventor of the mouse) was working on these issues back in the early 60’s (a point Doug Crockford made in his RWE keynote). A while back, I stumbled upon this video of a presentation from Alan Kay circa 1988.

During the talk, Kay shows a video of Engelbart demonstrating some really amazing stuff. In the presentation, Engelbart works with a colleague located about 30 miles away - along with sharing the desktop, they’ve got full audio and video connectivity. Needless to say, I was pretty wowed.

Further along, you see some amazing applications developed by school children! While some of this may seem quite simple today, you have to consider when this was filmed. I’ve long been fascinated by novel ways technology is applied in the classroom - something beyond using a word processor or doing some research on the web. I have to wonder what has become of programs like those discussed in this talk. Of course now we have things like Yahoo! Teachers but still, the focus seems to be on attendance and grades not on enhanced teaching methods.

These days you can’t swing a short iron without running into some discussion of patent law and how it applies to software (or more often doesn’t apply) but I was really surprised to hear it come up in the Q&A of Kay’s talk! To paraphrase Alan, while he feels that protecting inventions is good, there is “enormous confusion between what’s an idea and what’s an invention”. He argues that there should be some protection but that it is and can be abused. A little later, he argues for more than just a technical education and as a graduate of a small liberal arts college, I was very pleased to hear him urge computer scientists to get a liberal arts background arguing that the challenge is to find the aesthetic. Frankly, I think my BA in computer science makes me a far more well rounded individual and has allowed me to perform a wider variety of duties.

Anyway, a very fascinating presentation by a giant of computing. I highly recommend you take the time to watch these; despite their age, they are still quite timely.