Rant: UNIX vs. the Web

For all its strengths, developing for the Web has become a gigantic pain in the rear, especially when compared to the Unix style of development.

A few months ago, I joined a new startup. Almost all of it is backend processing that doesn't even use a database until almost all the work is done. Our only Web application is for a Web services API. Since I was mostly writing the code from scratch, my boss and I agreed that taking a Unix approach was best. Hence, we have a bunch of simple, standalone tools. Writing such tools is so refreshing. You know exactly what you need to do. They're only a few hundred lines long. You build a nice command line interface using optparse, you write some tests using nose, etc. It's all very straightforward and linear. Wanna know how to do a UNIX-style mashup? You use a pipe.

Recently, I went to work on a Web application again, and I realized just how much of a giant pain in the rear it is. Here are some of the things you need to think about. You need to be fluent in Python (or PHP, Ruby, etc.), HTML, CSS, SQL, and JavaScript. You have to have a Web server, a database, and possibly other auxiliary items like a load balancer. You have to think of the front end and the back end. Usually, you'll use a Web framework, a templating engine, and possibly an ORM. They're all probably young projects, so you'll have to be on the mailing list for each. Despite all the time you'll invest in these projects, you probably won't be using them still in five years.

Besides implementation concerns, the Web is simply complex these days. Have you read all the papers on session fixation attacks, cross site scripting vulnerabilities, SQL injection attacks, and cross site forgery attacks? Do you remember the HTTP response codes for SEE_OTHER and TEMPORARY_REDIRECT? Do you know when you should use each? I see tons of books about various Web frameworks and libraries, but where is there a really good book on how to be good at plain old Web development?

How do you deal with logins? OpenID hasn't really taken off yet, and not everyone can depend on Facebook for authentication. That means you'll need account services. What if the user forgets his password? Did you know that if you URL encode something that's been base64 encoded and then send it in an email, it might not make it through Hotmail in all cases? However, if you make the link too long, users will get confused when their email client breaks it into two lines.

The browser is now one of the most complicated pieces of software on a standard desktop. Everyone knows the DOM is a mess. How do you respond to events? There are at least three ways, and they're all painful for various reasons unless you're using a JavaScript framework. XHR, which is now a staple in the Web world, started life as a Microsoft hack. innerHTML is another such hack. Yet as useful and convenient as it is, it hasn't been blessed by the standards bodies. Seriously, who the heck thought of createNode, setAttribute, etc. in order to inject some HTML? It's very Java-ish and not very JavaScript-ish at all.

By the way, concerning standards bodies--you know, the ones who spent so much time creating XHTML?--I'll remind you that the Mozilla Web Author FAQ still says that "Serving valid HTML 4.01 as text/html ensures the widest browser and search engine support." I.e., use HTML not XHTML. The Webkit guys say pretty much the same thing.

But the fun doesn't end there. Need to do a Web request against a foreign domain and JavaScript won't let you? There's a workaround for that too. You just use a script tag and JSONP.

The Web is a strange place where the HACKS become the standard by which you get stuff done.

Once you get past all those difficulties, you're still stuck with something that's still not as snappy as a desktop app and a lot more difficult to code than an old school (i.e. no JavaScript) Web app. Essentially, the tough thing about Web apps is that they're large and effectively connectionless. Unless you're using something like Seaside, you have to handle each new Web request completely from scratch.

Hello, who are you? Oh, you have a cookie? Let me see if I know anything about you. Oh, the memcache server says it has a session for you. Let me talk to the database to see if he can tell me more. Ok, here's a form. Get back to me when you're ready.

Hello, who are you? Oh, you have a cookie? Let me see if I know anything about you...

And let's not forget that you're simultaneously carrying on about a hundred such conversations at any given time. You have all the drawbacks of multithreaded coding except, you really can't count on anything being in memory because you're spread across several servers. It's the worst of both worlds.

The nice thing about UNIX tools is that once you get them working, you don't need to think much about them anymore. When was the last time you worried much about cut, uniq, or sort? On the other hand, with a Web app, plan on rewriting it five years from now. Oh, and it'll be even harder and more messed up by then.

Of course, all my complaints just don't matter because the Web has too many good properties. It's vendor and OS neutral. You can run millions of different applications, and the only thing you need to download is a browser. (Oh wait, you already have one? An ancient version of IE? No worries, we can support that too!)

Yet again, we are reminded that worse is better. Apparently, much, much worse is also much, much better.

Python: Google App Engine: Cookie Users Beware

By default, Google App Engine Web applications runs on yourapp.appspot.com. That means that some other app, e.g. badguyz.appspot.com, can set a cookie for appspot.com, and your app will get that cookie from the user's Web browser on subsequent requests to your site.

This isn't some remarkable new exploit or anything. It's just something to keep in mind when running on subdomains like this. If you're worried about security, you should use your own domain name and cryptographically sign your cookies (here's some example source code).

Python: Debugging Google App Engine Apps Locally

Python has a wonderful interactive interpreter (i.e. shell). However, sometimes you need more setup before you can start coding. Previously, I wrote Python: Coding in the Debugger for Beginners.

Google App Engine works a bit like CGI in that output to STDOUT goes to the browser. This breaks my normal "import pdb; pdb.set_trace()" trick. However, it's not hard to put STDOUT, etc. back so that you can use pdb in the way you normally would:

for attr in ('stdin', 'stdout', 'stderr'):
    setattr(sys, attr, getattr(sys, '__%s__' % attr))
import pdb
pdb.set_trace()

The Bipolar Lisp Programmer

Have you ever wondered how it could be that Lisp is so powerful, and yet C is so much more successful and ubiquitous? How is it that so many brilliant coders know Lisp, and yet we so rarely hear from any of them other than Paul Graham? This is a great article that tries to explain it: the bipolar Lisp programmer.

Joel on Software: Never Rewrite from Scratch

I was thinking of Joel on Software's famous post Things You Should Never Do, Part I where he says, "[Netscape] did it by making the single worst strategic mistake that any software company can make: They decided to rewrite the code from scratch."

Since Joel is from Microsoft, I was pondering what would have happened if the Microsoft NT developers had taken that advice and based NT on DOS. Perhaps it's illustrative to compare the quality of Windows ME vs. Windows 2000 and XP.

Linux: Resetting GNOME Settings

My mother-in-law uses Linux, but she's far from the most savvy computer user in the world. It's tough when you start using Linux in your seventies ;) Anyway, somehow she occasionally deletes her GNOME panel. That's a problem because it prevents her from starting any applications. After much practice, I've narrowed down the list of steps that are the easiest to dictate over the phone:

1. Restart the computer.
2. Don't log in. Rather, hit Cntl-Alt-F1 in order to get to a text-based login screen.
3. Now log in.
4. Type "rm -rf .gnome* .gconf*".
5. Now, hit Cntl-Alt-Delete.
6. Let it reboot and then log in normally. Things should be fixed.

Note that this will wipe all of your GNOME settings. However, since it's easy to type, and she doesn't have any settings anyway, it's perfect. The reason I do this delicate dance of rebooting is to prevent gconfd from doing things behind my back.

User Experience: mint.com

mint.com ("free personal finance software, money management, budget planner and tools, online financial planning") is both amazingly intelligent and has an incredible user interface. All I can say is "Wow!"

UNIX: "join"

My buddy Pat Tufts showed me that Unix has a command called "join" that lets you do relational database style joins with text files. Let me show you how it works.

Let's suppose I have a bunch of purchase orders in a TSV (i.e. tab separated values) file called orders.tsv. The fields are order_id, item_id:

1 3
1 2
1 1
2 1
2 2
3 3

Next, I have all my items in another TSV file called items.tsv with fields item_id, item_name:

1 Hammer
2 Screw driver
3 Wrench

I want to print out the orders with item names instead of item ids.

The first thing I need to do to use join is pick the join field. Clearly, I'm going to join on the item_id field. The next thing is to make sure both files are sorted by item_id. This is a strange requirement if you're used to RDBMSs, but it makes sense if you think of how it's implemented. Clearly, items.tsv is already sorted by the item_id. However, I'll need to sort orders.tsv by item_id:

sort -k2n orders.tsv > orders-sorted.tsv

Hence, I sort orders.tsv by its second field, treating the values as numbers. I end up with:

1 1
2 1
1 2
2 2
1 3
3 3

Remember, the fields are order_id and item_id. Now, I can join the two files.

join -1 2 -2 1 -t '  ' -o 1.1,2.2 orders-sorted.tsv items.tsv | sort -k1n

I join the first file (orders-sorted.tsv) on its second field (item_id) with the second file (items.tsv) on its first field (item_id). My field delimiter is tab (to type a literal tab into a shell command, use cntl-v and then tab). I output the fields 1.1 (order_id) and 2.2 (item_name). Because the output comes out sorted by item_id instead of order_id, I need to sort it again by the first field (order_id). I end up with:

1 Hammer
1 Screw driver
1 Wrench
2 Hammer
2 Screw driver
3 Wrench

Hence, to fulfill order 1, I need a hammer, a screw driver, and a wrench.

Clearly, this isn't as easy and convenient as using a relational database. However, it's neat to see that it exists. My buddy Pat warned me though that dealing with all the required sorting and implicit field numbers can drive you crazy. Furthermore, it's really easy to end up with garbage.

Still, it's fun to see what's possible using plain text files and a shell. I've always heard that the "old timers" were able to implement relational databases using sed, awk, and sh, but it seems so much more plausible now that I've seen join in action.