Thursday, July 09, 2009

Agile Programming: I'm Stuck in the Middle

I found out yesterday, that at least according to Xmarks, my blog is far more popular than even I thought. However, before you think my head has gotten too big, let me tell you about two companies I interviewed with recently that turned me down. Twitter turned me down a couple years ago, and Aardvark turned me down a couple weeks ago (which is really a shame because I think Aardvark is awesome!).

Both of them said pretty much the same thing. They said I was too methodical and relied too heavily on documentation. More importantly, I'm sure, was that I didn't yet embrace test-driven development, TDD.

In TDD, you write your tests first, and then you write your code. However, ever since college, I've written my docstrings first, and then I write the code. I generally write the tests after I've written some code, and then I grow the two together. Although, I don't practice TDD, I am test infected, which means I feel uncomfortable about code that I've written unless I've also written tests.

I guess I've always been influenced by Knuth's literate programming. Apparently, someone forgot to tell me that literate programming is no longer cool.

My buddy Dan Aronson introduced me to a great programmer at Pivotal named Kelly Felkins. Pivotal is famous for being one of the most agile shops around. They practice TDD, they peer program pretty much all the time, etc., and they write good software. When I explained to Kelly my style of "docstring-driven development", he said I'd never make it at Pivotal. The other engineers would just delete all my docstrings. I think the idea is that in TDD, comments are to be distrusted, whereas tests are the true way of documenting code.

In real life, comments usually should be distrusted, but that's not the case with my comments. That's because in my style of docstring-driven development, I refactor the comments before I refactor the code. In the Python world, there's even doctest which lets you embed your tests right in your docstrings, which ensures that they are more likely to be current.

Until yesterday, I had never considered there to be a contradiction between literate programming and test-driven development. I just figured most engineers were too lazy to write both tests and documentation. As we all know, many engineers are too lazy to write either tests or documentation ;) Nonetheless, Kelly's arguments for TDD were pretty convincing. For instance, one of his best arguments was that TDD leads to much better test coverage.

It's interesting to note that not everyone likes agile or extreme programming (extreme programming is a specialized form of agile programming). For instance, Donald Knuth, the godfather of computer programming, said:
Let me just say that almost everything I’ve ever heard associated with the term "extreme programming" sounds like exactly the wrong way to go...with one exception. The exception is the idea of working in teams and reading each other’s code. That idea is crucial, and it might even mask out all the terrible aspects of extreme programming that alarm me.
Adam Wolff, who worked for several years on OpenLaszlo wrote a particularly insightful anti-agile piece:
Without even looking at the telltale URL, you can see these apps from a mile away. They still have the smell of the original auto-generated scaffolding, but more importantly, it's pretty clear that they were developed piecemeal...If you step back a bit, it's pretty easy to see the whole Web 2.0 thing as giant circlejerk of technology creators making it easier for mediocre developers to ship the next marginally useful application, while they struggle to implement a user interface that has the quality of a typical '80s PowerBuilder app...My second point is potentially more important: piecemeal approaches yield piecemeal solutions...In fact, many of the agile best practices reinforce short-sightedness and lock-in, even as they claim to emphasize nimbleness...The whole point of design is to incorporate the big picture in the little details, but when you're pursuing a "let's try it and see how it goes" approach, there is no big picture; there's just a bunch of stories.
If you've ever used an OpenLaszlo app, you know that he has a point. An OpenLaszlo app generally does a better job conveying the big picture. In general, OpenLaszlo apps put Ajax-based Web 2.0 apps to shame.

Perhaps user interfaces are one thing that shouldn't be designed in an agile way. One of the best user experience people I know, Stephanie Hornung, really drove this point home for me. Consider Amazon.com. It leaves me feeling overwhelmed and confused because there are too many piecemeal features vying for my attention. It's like an experiment in agile programming gone wrong. Stephanie argued that good user experience people aren't agile. They have to understand and plan for the big picture. I think this is one thing that Apple is particularly good at.

At the risk of using an overused metaphor, I don't think you can use agile techniques to build Gothic architecture:


(from Wikipedia)

However, most of us don't live in Gothic cathedrals. My own house is 100 years old, and has had at least five addons. It's a total mess, but unlike most Gothic cathedrals, it has central heating and air ;)

And just to be fair, even though Pivotal is an agile shop, their tool Pivotal Tracker doesn't feel like a piecemeal solution. It's easy to use, and the big picture is obvious.

The question remains, are agile programming and literate programming at odds with each other? If you're writing a Rails application, documenting the big picture is pointless. There's too much magic (aka convention) going on for in-line documentation to be helpful. The fact of the matter is, if you're building a Rails app, you need to read either a book or a tutorial ahead of time. Once you've done that, there's not much point in writing documentation that covers what you've already read.

However, not everything is a Rails app. For instance, Structure and Interpretation of Computer Programs, SICP, talks about the challenges of writing a memory manager for Lisp. Either you get it right, or you're completely screwed. Imagine trying to write a memory manager from scratch on a new hardware architecture. If you mess it up, your entire object graph is going to be toast. In such cases, I think it does make sense to embrace literate programming. I also think that working it out on paper ahead of time and heavy code review are necessary. Of course, just like Gothic cathedrals, not many people have to write memory managers for Lisp these days. Banging out Rails apps for Web 2.0 is far more the norm.

While I'm on the subject of SICP, perhaps it's telling that M.I.T. has switched from Scheme to Python for its first computer science course that students must face. Gerry Sussman said, "Nowadays, a real engineer is given a big software library, with a 300-page manual that’s full of errors...The engineer must learn to perform scientific experiments to find out how the software and hardware actually work, at least enough to accomplish the job at hand." Sussman pointed out that we may not like it this way (”because we’re old fogies”), but that’s the way it is, and M.I.T. has to take that into account. When you put it that way, TDD makes all the sense in the world.

Futhermore, Kelly pointed out that if you only write as much code as need to make the tests pass, you avoid overengineering a bunch of code that may not match the real world.

Where does that leave me? I'm going to use TDD for my Rails app, but I still maintain that a truly brilliant user interface must be developed like a Gothic cathedral. If you're interested in doing TDD for building a Web app in Python, I suggest some mix of Twill, Nose, and Selenium. If you're using TDD for building a Web app in Rails, Kelly suggested some mix of Cucumber, RSpec, and Selenium.

Happy Hacking!

20 comments:

Sergey Kishchenko said...

> I suggest some mix of Twill, Nose, and Selenium

I think, Windmill would be a better choice than Selenium. And I also recommend tddspry, if you use django.

Shannon -jj Behrens said...

This video is a trip: http://media.railscasts.com/videos/155_beginning_with_cucumber.mov

Cucumber lets you write your tests in pseudo-English. Since Cucumber tests are executable, but they are also readable by non-engineers, I can see how they might be move valuable than normal docstrings.

Juergen Brendel said...

Good article and observations. I also feel that Agile is suitable for some, but not all projects. The good UI as well as the memory manager are good examples.

I believe that Agile works best in environments where all developers are clear on the overall vision and direction the project has to take and we are just in the process of adding more features.

However, you need a more unified big-picture approach for things like the UI, but also consistent error handling and other less visible aspects of the system. Design those in a more traditional, up-front manner, and allow a more Agile approach to adding the various features once these things are settled on.

Agile for architecture development is less suitable than for just feature addition. If you work only in frameworks such as Django or Rails then you don't need to worry much about the architecture side of things, since that's pretty much dictated to you. For non framework-based apps, though, I'd be hesitant to go with Agile from the beginning.

Martin Aspeli said...

The word "agile" is tricky, because so many people mean so many different things by it, and even more people who've never done an "agile" project, *assume* so much about it.

To me, "agile" is a set of principles and values that say something about how to deliver good software projects for customers. It's about focusing on delivering value and quality, in a world where some traditional techniques adopted from other disciplines have proven ill-suited to the rather messy world of software development.

Crucially, it's not an excuse for skimping on documentation or design, and it doesn't dictate that things have to be done piecemeal. Some people who claim to be doing agile projects do that, but that's only a valid approach if it is the most appropriate thing for the project (e.g. the customer doesn't value a highly consistent UI, or this is a rapid prototype).

You're agile if you apply the *right* amount of design for the project, and if your designs and plans are malleable enough to be able to absorb change in requirements and external factors.

I think Juergen is right that agile works best in environments with a clear vision. I'd add it also works best if you have "good" programmers who can think for themselves and bring valuable insight or opinions to the frequent discussions that tend to take place in agile projects. With a team that is less experienced (or with outsourced 'commodity' programming resources), you need to adopt a lot more structure and a more waterfall-like approach. That dosen't mean you can't be agile, but it does mean your process has to change if you're used to managing co-located, small, highly skilled teams.

TDD is an approach that is often employed in agile projects. Not all TDD projects are agile and not all agile projects use TDD. It's quite an interesting and good technique. In my experience, it works well when you're writing small components against a dedicated API. Doing TDD at the UI level (e.g. with Selenium/Windmill) seems utter madness to me, though. There's simply no way that can replace doing proper UI design and human click testing. Having some integration tests that drive the UI is often valuable, but not as a substitute.

I also fully agree with your approach to write docstrings first, and refactor them with the code. Good documentation is hugely important, and saying that the code is self-documenting or the tests document the code is an excuse used by pepole who don't read other people's code.

Martin

Shannon -jj Behrens said...

Great points, Juergen.

Shannon -jj Behrens said...

> Crucially, it's not an excuse for skimping on documentation or design, and it doesn't dictate that things have to be done piecemeal.

Martin, thanks for your comment. It very much matches what I thought before Kelly sat me down and gave me a talkin' too.

I'd hesitate to say that Pivotal isn't an agile company. They do TDD, peer programming, some version of Scrum (I think), etc. I definitely don't know of any company that is *more* agile than they are.

However, they seemed to be the ones who objected most to my excess of documentation. Kelly was definitely advocating against designing too much up front because he said it lead to overengineering.

He preferred to describe a requirement using a testing tool like Cucumber and then attack that test as directly as possible.

You said, "doing TDD at the UI level (e.g. with Selenium/Windmill) seems utter madness to me," but that's exactly what Kelly was advocating for. Clearly, there's a divide between the two points of view.

By the way, I don't think Kelly would be against some real user testing. After all, I never have figured out how to write a test that says, "This UI must not be ugly."

Martin Aspeli said...

Hi Shannon,

I'm *sure* pivotal are agile. I use Pivotal Tracker and really like it, too. ;-)

But for *their use cases* documentation may just not be deemed valuable enough. So they're making the right choice for their use case. That doesn't mean you can't do TDD, follow "agile" principles, and still have good documentation.

Maybe I should temper my comment though: I've never been able to make TDD of the UI work. I've seen a lot of developers write a lot of Selenium tests, trying to guess in their minds what the output HTML will look like, and then having to rewrite their tests again and again because they are very brittle. I'm sure there are better ways (like testing mockups), but I don't see the greatest cost/benefit tradeoff in UI TDD. I see a much better cost/benefit ratio for TDD of business logic functions.

Clearly, there is a risk of over-engineering. To me, you're agile if you're doing the right amount of design for your project. Progress in an agile project is measured in terms of business value delivered, and there's no intrinsict business value in a big design document. Design in models, mock-ups and the like will often be more appropriate since they are (a) more easily changed if required and (b) a lot less work to produce and maintain.

Martin

Shannon -jj Behrens said...

Good conversation. Thanks.

Shannon -jj Behrens said...

By the way, Kelly said he prefers Cucumber over the integration testing that Rails has built-in. Cucumber can run Webrat which can use Selenium, Watir, etc.

Shannon -jj Behrens said...

Is it just me, or does Cucumber remind you of Prolog. I wonder if it's possible to use a Prolog-like logic language to drive Cucumber-like tests.

Kent Johnson said...

The construction of Notre Dame may have been more agile than you think. According to Wikipedia, it took almost 200 years to build and "over the construction period, numerous architects worked on the site, as is evidenced by the differing styles at different heights of the west front and towers."

Florian said...

I often feel like the whole TDD/Doctest/Up Front design etc. debate in regards to making bug-free software is slightly off-topic.

One of the things I see even in well tested life systems is that no amount of coverage, no number of tests, no documentation and certainly no amount of up-front design is able to prevent bugs from creeping in.

The bugs you discover after you've done all you could to produce solid software are usually those which make most of the maintenance efford.

I've come to practise a style of programming that: 1) makes any error noisy 2) logs away contextual data relevant to reproduce errors (such as inputs etc.)

When an error happens in a system of mine, I have an easy time tracking it down and writing a test for it much more often then simply relying on a suite of pre-fabricated tests that do not cover that error.

It's not really TDD, though tests are involved, but only post-mortem. However I find that this is faster then a large efford in up-front Tests and Design to erradicate bugs. I Design to squash bugs fast and painlessly in operation.

Shannon -jj Behrens said...

> The construction of Notre Dame may have been more agile than you think. According to Wikipedia, it took almost 200 years to build and "over the construction period, numerous architects worked on the site, as is evidenced by the differing styles at different heights of the west front and towers."

Nice comment! Perhaps the differing styles is proof that an agile design leads to lack of conceptual integrity? Perhaps this wasn't the case with other cathedrals?

Shannon -jj Behrens said...

Florian, great comment!

I think your style of development matches Titus Brown's approach to testing. I think he calls it stupid testing or something like that. He adds tests for code that has proven it needs it. Titus is a total test buff who has to cope with legacy systems, so he has room to argue.

I agree that testing and designing up front doesn't prevent bugs. Most of my bugs come from situations that I never even thought of. Hence, neither the code nor the tests addressed those situations. A TDD person would say that once you find a bug, you should write a test for it, and once you fix a bug, your existing tests keep you from breaking something else. It's a useful viewpoint, even more so if you have to fix someone else's bugs, and they haven't written any tests.

I think I'd like to be like Knuth with TeX. Once you find a bug, the author owes you an ever-increasing amount of money ;)

I also agree with your sentiments which I like to call "fail fast" and "plan for debugging". There's nothing worse than code that results in nil, NULL, or None in places where it shouldn't be, and then some other, innocent part of the code crashes. I also hate it when things crash, and I'm left with absolutely no leverage to fix the problem.

kellyfelkins said...

JJ,

Thanks for the great article.
First, let me clarify: I was trying to make a point about test driven development and comments in code -- from what I can tell you are a smart guy and Pivotal would be lucky to have you.

I'm a strong believer in agile software development. I know XP is not right for all software projects but I do think it is right for the majority of what I do today, which is building web and mobile applications.

Your cathedral analogy is useful. Some cathedrals took more than a hundred years to build. A plan is necessary to make that happen. Keep in mind that some cathedrals had design changes before the building was complete. Once complete a cathedral is relatively static for hundreds of years.

Unlike cathedrals, application requirements change almost continuously to meet ever evolving requirements and changes in technology, business and culture. Few applications will be around for more than, say, 10 years. And far too many never go into production.

Agile practices attempt to improve on the software practices that cause software projects to fail or lead to inflexible software that cannot evolve as the requirements change.

Automated testing is one of those practices. Automated test suites allow you to change your code with confidence. Without an automated test suite code becomes brittle while development schedules and testing personnel expand. You eventually reach the point where you can't afford to change it. Then, if anyone still cares about the application, the answer becomes "Let's rewrite it".

Comments in *application* code are generally a code *smell* -- it's a sign that the code is obtuse, and if it's obtuse it should probably be changed.

Commenting a public API is another thing altogether. An API is a contract and you need to be careful when changing that contract. Providing documentation on that contract helps consumers of that API get their work done.

It's cool to be discussing these issues. My opinions are constantly moving and have moved a great distance since I've been at Pivotal and practicing XP every day.

Shannon -jj Behrens said...

Kelly, thanks for the great comment.

> from what I can tell you are a smart guy and Pivotal would be lucky to have you.

Thanks. That's good because I'm going to be keeping Pivotal in mind whenever I need to find another job ;)

> I'm a strong believer in agile software development.

Actually, so am I. However, I had never yet seen how to do TDD with websites. You helped me overcome that hurdle.

> Unlike cathedrals, application requirements change almost continuously to meet ever evolving requirements and changes in technology, business and culture.

Your points are valid, although I still maintain that you can tell the difference between a UI with conceptual integrity (built like a cathedral) and one without it, Amazon being a great example.

> Comments in *application* code are generally a code *smell*

I agree. There's nothing worse than a 150 line function with no docstrings and a bunch of comments.

Those aren't the type of comments I use a lot of. I use a lot of API comments. Basically, I add an API comment (aka a docstring) for almost every module, class, and method to give the reader an idea of what they're looking at. That doesn't make sense so much in Rails, but I think it does in other settings.

For instance, I remember looking at a codebase that had three files named autocomplete. I had no clue why there were three, and if all three were used. It turned out that one was the webservice entry point, one was for talking to the database-based autocomplete service, and the other was for talking to the Lucene-based autocomplete service. A one line comment at the top of each file would have made my life much easier.

Anyway, thanks for all your feedback. I'm going to give it a shot and see how it turns out ;)

kellyfelkins said...

> Your points are valid, although I still maintain that you can tell the
> difference between a UI with conceptual integrity (built like a
> cathedral) and one without it,
Amazon being a great example.

I think Amazon makes the case for agile UI development. The amazon site continuously evolves. If it appears to have conceptual integrity at the moment it is because people have worked hard to create or maintain conceptual integrity in the site's current revision. The amazon UI has often been inconsistent and sometimes even broken. The company is truly amazing, highly innovative, and the site is always moving as their business model moves.

Shannon -jj Behrens said...

I think you misunderstood me. The Amazon UI is horrible. It makes me feel uncomfortable and overwhelmed every time I use it ;)

Adam Wolff said...

JJ - I'm glad to see more thinking on this (and anyone straining against the agile juggernaut.)

One thing that I think is funny about TDD is that it seems like it's actually a replacement for just plain thinking for a lot of programmers. My canonical example of this is the whole bowling fiasco. It never fails to crack me up that the classic TDD example is a complete and utter mess -- thousands of lines of code that could be easily replaced by a single function that takes a list of numbers and yields an output. But as the story shows, it's easy and fun to just jump in without even bothering to draw a damn picture on the whiteboard.

But I think it's also telling, because the bowling example is undertaken in the absence of any real requirements. When you're handed a problem like this (akin to "provide a web-service that does X") you end up carefully modeling the domain of the real world (separate objects for throws and frames and players) because you don't really know what your code has to do. I think TDD makes developers feel good. They feel more productive than they do just starting at the blank screen. After all, they're writing all this code! (never mind that most of that code tests the underlying library.)

Of course TDD, and even more-so, test isolation has its place. I actually think that most of the value from TDD comes from the architecture that it necessitates: the components are loosely coupled enough that they can be tested separately.

Shannon -jj Behrens said...

Thanks for your comment, Adam. It's nice to get an update on your thoughts on the matter.