Friday, June 06, 2014

A Blog Post on Excel, 4x4s, Moonshots, and General Purpose Software

Using Excel is like driving a 4x4. Writing custom software is like building an interstate freeway.

With a modest amount of training, you can do amazing things with a 4x4. You can go to places where there are no roads. If you don't like where you are, you can just as easily drive somewhere new. The downside is that when you're driving off-road, you generally have to drive pretty slowly, and you have to bring your own gas.

You do have to be a specialist to build an interstate freeway. In fact, it requires an amazing amount of work from a large number of people. Yet, once it's built, a 100,000 people a day can fly by at 70mph. It offers them tremendous convenience; they don't even need to bring their own gas! On the other hand, a freeway is only useful if it goes in the direction you need to go.

Hence, Excel offers tremendous flexibility, whereas custom software offers tremendous convenience. Excel is easy to get started with, but grows unwieldy as you scale the size of your models and teams. Custom software is fairly difficult to get started with but is easier to scale in terms of size and usage.

I have to interact with Excel a lot, and yet I've spent most of my career writing custom software. What amazes me about Excel (or, rather, spreadsheets in general) is that it's so amazingly general purpose. Think of how useful spreadsheets are to businesses all over the world. There are few pieces of software that are as general purpose or as widely useful as spreadsheets are.

To some degree, it seems to me that Silicon Valley is drunk with the power of being able to write custom software. Pretty much my whole career has been spent writing it. But what about writing new general purpose software like Excel? As far as I can tell, not many people get to work on general purpose software, and even fewer people get to create new types of general purpose software.

The first commercial spreadsheet was VisiCalc. What an amazing invention! Although VisiCalc was used by far fewer people than, say, Excel, it's notable because it represented a new "type" of general purpose software; it changed the world.

Now, think about all the other types of general purpose software. Most of it was invented a long time ago. Word processors, operating systems, web servers, databases, ERP systems, etc. were all first created decades ago. That's ancient history as far as computing goes! (Although, to be fair, all of these things are incredibly recent in terms of the history of civilization or the history of mankind.)

So, where are the new types of general purpose software? It's actually kind of hard to recognize them because they haven't been around long enough or copied enough times to be recognized as a "type" of software. However, I think that we're far enough along to recognize Facebook is an instance of the type "social network." Similarly, Twitter is a "microblog" (among other things). Both of these things have had a fairly worldwide impact.

Facebook and Twitter also illustrate another thing. You could say that both of them are really just custom software, whereas the truly general pieces of software that they're built on are web browsers, web servers, databases, programming languages, and operating systems. On the other hand, as I said above, they're also instances of new types of software: social networks and microblogs. Innovation is like that. An instance of one type of thing can become its own new type of thing. Innovation is always pushing toward greater specialization.

Now consider programming languages. We've seen a lot of evolutionary improvement, but a lot of the revolutionary ideas first happened decades ago. Think of subroutines, OOP, exceptions, processes, threads, coroutines, types, recursion, closures, compilers, interpreters, virtual machines, etc. These concepts have been around for decades. Sure, we have new programming languages that each have a different philosophy on how these things should work, but most of the really big ideas happened decades ago. I thought software transactional memory might be a counterexample since it seems fairly recent, but even that started about two decades ago.

Imagine, if you were to invent a fundamentally new replacement for subroutines, create a new way to package data and/or code together, or create a new form of concurrency, what would it look like?

Of course the flip side is that it's really hard to get people to understand and accept fundamentally new ideas. It took at least a decade or two before OOP was generally considered a useful technique, and that's still not universally accepted ;) The same is happening now for functional programming. And who knows where we'll go with concurrency in the future!

Truly new ideas just don't come along that often. It takes a while for a new instance of a thing to be recognized as a new type of thing. There aren't that many "moon shots" as Google likes to put it. And when they do come along, they often take a really long time before they gain widespread acceptance. Furthermore, the first mover often isn't the one who reaps the greatest rewards. Excel wasn't the first spreadsheet. Google wasn't the first search engine. Facebook wasn't the first social network.

I'm not really saying anything new. Kevin Kelly did a much better job explaining all of this in his wonderful book, What Technology Wants, but I'll leave you with a quote from another excellent book, The Myths of Innovation, Don't worry about people stealing your ideas. If they're any good, you're going to have to cram them down people's throats!

2 comments:

Anonymous said...

Hi,

I've followed your blog for several years now, but this is my first comment. I rarely comment anywhere, but this one is too compelling to leave unanswered. There's too much here to treat properly in a single comment, so I'll focus on just a couple of things.

Why are there no more spreadsheet-type general-purpose apps? Marshall McLuhan remarked that the content of a new medium is always an older medium. Print 'contains' manuscript, which in turn contains speech. By that reasoning, word processing contains (we might say 'virtualizes') typewriter typing, powerpoint contains the old acetate overhead slides, and so forth. Excel contains the green ledger books my dad used as an accountant. The pre-PC, pre-Internet world provides only a limited number of candidates for transfer from the physical world into software.

As for programming, think of all the fundamental concepts that appeared in the 1950s, for example, functional programming, COBOL, (anyone remember APL?), context-free grammars, and regular expressions. All of these happened when computers of any type were accessible to a tiny group of people, so these ideas could not be 'big' for a long time. So it may be with our time; among the crowd of ideas all around us, we cannot know which few will be influential decades hence.

My own candidate for most significant development of the past two decades in software is the free and open-source movement. When I started in computing, in the 1980s, software patents were already being talked about. In the debate it was presumed by all sides that, without some form of reward, software would not be developed or even maintained. It's hard to imagine now, but it seemed self-evident at the time.

But the FOSS movement has shown that not only will people donate their valuable time and effort, but that they often beat commercial vendors in quality, responsiveness, documentation, and every other measure. I would suggest that this is miraculous, and suggests that software is somehow different from other areas of human endeavor; its practitioners so love the process of creation that they are called to do it for pleasure and personal expression. I can think of no precedents for such a phenomenon.

So perhaps it's not specific technical ideas, but rather the cultural context made possible by the decades of accumulated technical context.

Shannon Behrens said...

> I've followed your blog for several years now, but this is my first comment.

Thanks.

> Marshall McLuhan remarked that the content of a new medium is always an older medium.

I think it's a good point, but I'm not sure how it answers the question.

> So it may be with our time; among the crowd of ideas all around us, we cannot know which few will be influential decades hence.

Agreed.

> My own candidate for most significant development of the past two decades in software is the free and open-source movement.

Agreed.

> But the FOSS movement has shown that not only will people donate their valuable time and effort, but that they often beat commercial vendors in quality, responsiveness, documentation, and every other measure.

Actually, I think most programmers who work on open source software do so as a part of their paid work.

Furthermore, I've unfortunately come to the conclusion that FOSS and quality are completely orthogonal. Sometimes the FOSS is better. Sometimes it's worse.

> I can think of no precedents for such a phenomenon.

Art and academic papers are like that. I think blogging is becoming that way.