JJinuxLand

YouTube: Vibop by NewBlue

2012-01-27T15:03:00.001-08:00

I'm proud to announce that we launched a new YouTube.com/create partner, Vibop:

Vibop makes your videos shine with just a few clicks. Add an animated intro, a vintage filter, a cartoon look, a silent movie theme, and dozens of other effects. Brighten dark images and fix a shaky camera. Fun, fast, and easy, Vibop will take your memories to the next level!

Dart at BayPiggies

2012-01-27T14:43:00.000-08:00

I gave a talk on Dart last night at BayPiggies. We had a great turn out, and the crowd was super interactive! Never underestimate a Python programmer's ability to question every aspect of a new programming language, especially one that uses curly braces and semicolons ;)

If you're interested, here are the slides.

Walking Skeletons and TODO Outlines

2012-01-24T14:21:00.000-08:00

These days, applications are so complicated and contain so many layers that it's difficult to know where to start. Should you work bottom up or top down? How much should you work on one layer before starting to work on the next layer? How can you ensure that the layers work properly together? Building a walking skeleton and managing a TODO document in outline format are two techniques that work well together to conquer complex problems, even ones involving multiple layers. Best of all, you don’t have to worry about trying to get things right the first time or getting lost along the way.

A Walking Skeleton is a tiny implementation of the system that performs a small end-to-end function. It need not use the final architecture, but it should link together the main architectural components. The architecture and the functionality can then evolve in parallel. -- Alistair Cockburn

Building a walking skeleton is a great way to handle the complexity of dealing with multiple layers. Start with the simplest possible feature, and implement it top down. Ideally, the feature should force you to work your way all the way down the stack. The goal is to make sure the layers work together.

As you’re building the walking skeleton, you may think of many things that you need to add, test, or in general worry about. It's helpful to maintain a TODO document in outline format so that you can organize and plan your attack, especially when you’re working with multiple layers at the same time. Eventually, each TODO item can be transferred into a test, a piece of code, an issue in the bug tracking system, or perhaps just an email to someone else.

Once you’ve built a walking skeleton, should you go back to developing one layer at a time? For most applications where the cost of change is low, probably not. Actively building one layer at a time is frequently very inefficient. A more efficient approach is to focus on one feature at a time. Sketch out the feature using a set of TODOs and build it top-down, managing the TODOs as you go. If you focus on one feature at a time instead of one layer at a time, you won’t end up building a lot of code in different layers that never actually gets used. The time saved by only building what you need and only building it when you have all the information you need more than compensates for the refactoring time.

Thanks go to Chris Lopez for his help with this post.

Books: Out of their Minds: The Lives and Discoveries of 15 Great Computer Scientists

2012-01-16T18:10:00.000-08:00

I just finished reading Out of their Minds: The Lives and Discoveries of 15 Great Computer Scientists. In short, I liked it. I wouldn't say I liked it as much as, say, Coders at Work, however, I'm glad I read it. What I liked most about it was that it contains biographies from really early computer science pioneers such as Ada Lovelace, John von Neumann, John Backus, and John McCarthy. I know computer history after 1970 really well, but this book contains a lot of stuff from before 1970.

I jotted down a few interesting tidbits while I was reading the book. However, since I read the Kindle version of the book, I only have percentages, not page numbers. Anyway, I hope you're as entertained by some of these as I was.

John Backus, who lead the team that created Fortran at IBM, flunked out of college [2%]. He had a metal plate installed in his head [3%]. He disliked calculus but liked algebra [3%] (just like me!). These days, Backus is a proponent of functional programming [7%].

John von Neumann, who helped establish the fundamentals of computer architecture, thought that creating a programming language (i.e. Fortran) was a waste of time since programming wasn't a big problem [4%].

Ada Lovelace, who was the first programmer (not to mention, a girl!), was a gambler, an alcoholic, and a cocaine addict. She died of cancer at the age of 36. She is credited with inventing loops and subroutines [5%].

Fortran only had globals. Algol, which is considered an ancestor of C, added locals, thus permitting recursion [6%].

McCarthy, who designed Lisp, was born in 1927 to Communist party activists. He had an Irish, Catholic father and a Lithuanian, Jewish mother [8%]. McCarthy is the reason Algol had recursion [11%]. (I didn't know that C got recursion because of Lisp.)

Alan Kay, who did pioneering work on object-oriented programming and helped create Smalltalk, got thrown out of school for protesting the Jewish quota [15%].

Edsger W. Dijkstra, who did influential work on a lot of early computer science problems such as concurrency, did very well in school and wanted to turn programming into a respectable discipline [21%].

Fred Brooks, who wrote "The Mythical Man-Month", wrote this about iterative development:

In "The Mythical Man-Month" I said build one and throw it away. But that isn't what I say anymore. Now I say, build a minimal thing--get it out in the field and start getting feedback, and then add function to it incrementally. The waterfall model of specify, build, test is just plain wrong for software. The interaction with the user is crucial to developing the specification. You have to develop the specification as you build and test.

Python: Shrapnel

2011-12-30T10:35:00.001-08:00

After many years of hopeful expectation, IronPort (now part of Cisco) finally open sourced a bunch of stuff, most importantly Shrapnel, our proprietary version of stackless Python. Back when I worked at IronPort (2004-2006), I was dying for them to open source this stuff. The existing open source version of stackless Python at the time never had quite the same flavor or emphasis as Shrapnel, but these days gevent is very similar.

Anyway, congratulations to Sam Rushing, Mark Peek, etc., and thank you Cisco! By the way, there's a reference to me in the code ;)

Dart at g|egypt 2.0

2011-12-16T13:33:00.000-08:00

I gave a talk on Dart at g|egypt 2.0. All I can say is, wow!

First of all, I wasn't even on the agenda. In fact, the room I was using wasn't really even labelled on the map--it took me several minutes to find it. It was a small room. 2 minutes before my talk was supposed to start, I only had 3 people in the audience. However, 10 minutes later, every seat was taken, and people were standing at the back and along the sides. The room was packed!

Since Dart is so new, we haven't spoken about it at very many places. I told the Egyptians that I was giving them a chance to become the best Dart programmers in the world because they were seeing the talk before most of the rest of the world had seen it. I almost got a standing ovation ;)

People were very excited about Dart and asked a ton of excellent questions. Even after my talk, I had to stand around for 2.5 hours answering more questions.

The comments on Google+ were very supportive:

Hady Allam said, "your DART session is one of the best sessions i have ever attended. Thank You +Shannon Behrens and hope to see you again in Egypt."

Abdurrahman Alraies said, "The dart session was the best surprise of the day. Thank you very much +Shannon Behrens and +Sebastian Trzcinski-Clément It says much about how important is Egypt to Google."

Saied Attala said, "really i have opened the comment box for 30 minutes and can't write every thing for you. i loved your Dart sessions and your sense of humor. i enjoyed talking with you and having pics with you."

Those comments were very touching to me. To all the Egyptians out there who went to my Dart talk, thank you for making me feel so special! I hope I get to come back to Egypt again!

After the trip, I accidentally ran into Lars Bak, the creator of Dart, in Zurich. That was very exciting for me!

If you're interested, here are the slides.

Thanks also to Saied Attala and Ayman Reda Bedair for the pictures and Seth Ladd for most of the slides.

YouTube for Your Business at g|egypt 2.0

2011-12-12T06:20:00.000-08:00

I gave a talk called YouTube for Your Business at g|egypt 2.0.

The talk went very well. There were about 200 people in the audience (and almost 900 in total at the event), which is a great turnout for a YouTube API talk. People asked me questions for about an hour afterwards. I even created a video for the talk:

I was really worried that I would lose my voice during the talk. It's very crackly right now.

Since this is the first time I've visited the middle east, I'm suffering a little from culture shock. I forgot to remove a beer joke from one of my slides. It took me a few seconds to realize why people weren't laughing. What's really funny is that I don't even drink!

I was very amazed by a few things. There were a lot of women at the event. There are far more female programmers in Egypt then there are in the US, as far as I can tell. The next thing that surprised me was that almost everyone here uses Windows--even on the server side. That's certainly not the case in the San Francisco Bay Area. At all the companies I've worked at recently, people tended to use Apple laptops and Linux on the servers. The last thing that surprised me was how many people wanted to have their picture taken with me. For about two hours, I felt like a total rock star ;)

I have a talk on Dart tomorrow and a talk on Python the next day. Let's hope my voice holds up ;)

Thanks go to Mohamed Naguib for taking the picture.

Computer Science: The Travelling Salesman Problem

2011-11-29T15:16:00.000-08:00

I was thinking about the Travelling Salesman problem this morning. I came up with an algorithm that permits a few nice optimizations. My guess is that Knuth probably already came up with this algorithm, formally analyzed it, and then came up with 15 others that were much better. Nonetheless, I figured I'd jot it down.

Warning: This is a rough draft of an algorithm. You shouldn't bother reading this if you want real rigor. I'm just writing it down to get it off my chest.

Rather than picking the first segment of the journey and working my way to the end, I want to pick a middle segment of the journey and work my way outwards recursively.

Given a list of distances (or times, or costs, or whatever) between cities, sort the list from shortest distance to longest distance regardless of which cities are involved. Now, loop over this list and use each element of the list as a middle segment of your journey. This gives us a list of "first steps" (or rather, first middle steps). Looping over the list from shortest distance to longest distance is an important optimization because it increases the likelihood that we'll find an optimal path early, allowing us to skip over a lot of potential paths later.

Also sort the list of distances so that for each city, we have a list of other cities in ascending distance order. By sorting all of the city pairs in order of (first city, distance), you can use one sort for all of the pairs.

Now, here comes the recursion. We have a bunch of middle parts of the journey. Use recursion to extend the journey either by adding to the beginning of the journey or by adding to the end of the journey. Keep recursing until you have a complete path or a partial path that is already longer than the best path seen so far. Now, we can either extend the journey at the beginning or at the end. Recursively try extending the journey by either adding to the beginning or the end. However, do it in order so that you try adding the shorter distances first. There's an implicit merge sort going on in this algorithm. This, again, is an optimization to allow you to skip work later.

While we were recursing, we had a function that took two things, a middle chunk of the journey and a set of cities not yet visited. Apply memoization so that anytime we get the same parameters, we return the same answer (by using a cache, of course). This is an important optimization.

Last of all, using the above algorithm, we'll quickly come up with the first complete path that has a decently minimal distance. Keep track of this as the best complete path seen so far. Anytime we are recursing and we come up with a partial path that is longer than the best complete path seen so far, we can stop recursing, give up, and try something else. This is an important optimization to save work "at the leaves of the tree".

I can't even wrap my head around the big O of this algorithm. I suspect it's one of those cases where you use words like "amortized cost".

This algorithm does have a weakness. If every distance between cities is exactly the same, it'll try every possibility. Similarly, if every distance between cities is exactly the same except one pair of cities which has a longer distance, it'll still try every possibility. I'm not sure of the degree to which the memoization fixes this problem. It'd be good to extend the algorithm somehow to recognize this situation and short circuit, i.e. automatically throw out paths of duplicate length.

Math: Factoring Numbers

2011-11-29T14:46:00.001-08:00

I was thinking this morning about factoring numbers. I wonder if it might sometimes be helpful to use real numbers to gain an interesting perspective in order to solve certain problems involving integer numbers (i.e. number theory problems). For instance, I was thinking about factoring large numbers.

For every natural number, C, (that isn't equal to 0 or 1), there are an infinite number of pairs of positive, real numbers A and B for which A * B = C. For instance, 6 = 1.0 * 6.0 = 2.4 * 2.5 = 2.0 * 3.0 = ... I wonder if playing around with pairs of real numbers like (2.4, 2.5) can lead you to pairs of integer numbers like (2, 3).

Imagine all the pairs (A, B) for which A * B = C. Let's create a way to graph all such pairs in a funny sort of way. Let's pick a bunch of A's going from 0 to C. For each different A, we can calculate B via C / A. Let's consider the parts of A and B that are to the right of the decimal point to see if one pair, (A, B) can lead us to another pair (A', B') which are integers (i.e. have only zeros to the right of the decimal point). In fact, let's see if we can come up with a numerical analysis approach where we use estimations to hunt down (A', B').

To do this, let's create a funny sort of three dimensional graph. Here's the pseudo code (assuming we're trying to factor some number, c):

for step in range (1, LARGE_NUM_OF_STEPS + 1):
    a = (step / LARGE_NUM_OF_STEPS) * c
    b = c / a
    x = a - int(a)  # x and y will always be in the range [0, 1).
    y = b - int(b)
    z = floor(a)
    draw_point(x, y, z)

If you use a very large number for LARGE_NUM_OF_STEPS, you'll create a funny looking 3D graph. Any place where x = 0 and y = 0, you'll have a pair of integers (a, b) that multiply to equal c.

Naturally, this is an extremely expensive way to factor numbers. However, I'll bet you'd learn a lot about factoring numbers by looking at this graph. In fact, I'm guessing that looking at this graph will lead you to a numerical-analysis-style approximation algorithm for honing in on valid integer pairs (a, b) where a * b = c.

Humans are fairly good at visualizing things in 3D, but it'd be really cool to extend this graph to four dimensions (perhaps using time as the fourth dimension). The fourth dimension would be used for various values of C, from 0 to infinity. I really think that looking at such a graph would help with fast number factoring.

Updated: Fixed a couple of errors pointed out by BMeph.

Computer Science: NP-complete Problems are Really NP-complete

2011-11-25T12:56:00.000-08:00

First of all, let me apologize for my sloppy terminology. I've always been a better software engineer than a computer scientist.

This is how Wikipedia describes NP-complete problems:

In computational complexity theory, the complexity class NP-complete (abbreviated NP-C or NPC) is a class of decision problems. A decision problem L is NP-complete if it is in the set of NP problems so that any given solution to the decision problem can be verified in polynomial-time, and also in the set of NP-hard problems so that any NP problem can be converted into L by a transformation of the inputs in polynomial-time.
Although any given solution to such a problem can be verified quickly, there is no known efficient way to locate a solution in the first place; indeed, the most notable characteristic of NP-complete problems is that no fast solution to them is known. That is, the time required to solve the problem using any currently known algorithm increases very quickly as the size of the problem grows. As a result, the time required to solve even moderately large versions of many of these problems easily reaches into the billions or trillions of years, using any amount of computing power available today. As a consequence, determining whether or not it is possible to solve these problems quickly, called the P versus NP problem, is one of the principal unsolved problems in computer science today.

While a method for computing the solutions to NP-complete problems using a reasonable amount of time remains undiscovered, computer scientists and programmers still frequently encounter NP-complete problems. NP-complete problems are often addressed by using approximation algorithms.

Note that it says, "As a consequence, determining whether or not it is possible to solve these problems quickly, called the P versus NP problem, is one of the principal unsolved problems in computer science today." I haven't seen the proof for it, but I've also heard that if you could prove that one NP-complete problem does not have a polynomial-time solution to it, then that would prove that none of them have a polynomial-time solution to them. The theory is a little over my head, but I'm going to take a shot.

I'd like to propose a problem that is NP-complete and show that it does not have a polynomial-time solution to it. Imagine I pick an n-tuple of random 32-bit, unsigned integers. Your job is guess what the n-tuple is. Hence, where n is 2, I might pick (376, 792), and your job is to guess that tuple. If you guess a certain tuple, I can tell you whether you're right or not. Hence, I can verify a solution in polynomial-time (it takes me O(n) number of int comparisons to verify a solution). However, to try to solve such a problem, you either have to get really lucky or you have to brute force it. If you find a method other than brute force trying every possible solution, then obviously there's something wrong with my random number generator. To use brute force to guess the solution requires O(A^n) for some constant A. Since it's "to the n", it's not polynomial-time (3nA^3 is polynomial-time since the exponent is fixed; A^n is not polynomial-time since the exponent varies with the size of n).

Now I know that some of you reading this understand the computer science a lot better than I do. Have I actually shown (at least informally) that NP-complete problems are really NP-complete? If not, can you, in lay man's terms explain what I'm misunderstanding? Thanks!

Tetris in Dart

2011-11-18T18:02:00.000-08:00

For my first Dart program, I decided to implement Tetris. I was inspired by Alexei Kourbatov at javascripter.net which contains an implementation of Tetris he wrote way back in 1999!

You can play my version here. The source code is here. It's part of a treasure hunt I'm building, but I'll say more about that later.

Overall, I like Dart. The Dart version of my code is about the same length as the JavaScript version of my code (the Dart version is slightly longer because I switched from using innerHTML to using the DOM). The combination of the optional static typing and Dart Editor helped me avoid many, many bugs. I'm still a Python guy at heart, but my experience with Dart was a very pleasant one.

I jotted down some lessons I learned. Warning: This is my first Dart program, and I haven't even read the tutorial yet, so I could be missing the boat on some of these points!

Don't try to play with DOM elements before the DOM is loaded. The easiest way to achieve this is to import your Dart-compiled JavaScript at the bottom of the page.

Dart has generics. This is a little unfamiliar for a scripting guy like me.

I had a hard time finding methods like JavaScript's Math.abs and Math.floor. They're now members of num.

This is how you add to the DOM:

HTMLDocument doc = window.document;
HTMLParagraphElement p = doc.createElement('p');
p.textContent = message;
doc.body.appendChild(p);

Using innerHTML works too, but I suspect it's frowned upon.

DOM manipulation isn't as easy as using innerHTML. It's not quite as horrible as I feared, but it's still a pain in the butt.

Sometimes you need to use a temporary variable to help out the type system. If you write:

doc.getElementById("s-" + i + "-" + j).src = 'images/s0.png';

it'll say "'src' is not a member of Element." However, you can easily fix this by writing:

HTMLImageElement img = doc.getElementById("s-" + i + "-" + j);
img.src = 'images/s0.png';

Translating from JavaScript to well-typed Dart is a linearly time-consuming, but relatively straightforward progress.

The intelligent code completion in Dart Editor was really helpful since I don't really know the language or its APIs. I can edit text a lot faster in Vim, but code completion helps me know what to write.

Trying to figure out the proper way to handle keyboard events in Dart is hard because of the DOM. I'm looking at developer.mozilla.org/en/DOM/KeyboardEvent, and it looks like there are two ways to do things, the deprecated way, and the way that isn't implemented yet.

Command-g is the usual Apple shortcut to find the next occurrence of something, but it doesn't work in Dart Editor.

They're working on a new DOM API. I've decided not to use it until they say it's stable.

The way main() works in the Sunflower Example doesn't match the way main() works in sample applications generated by Dart Editor. I'm not sure why.

Eclipse vs. Intellij for Android, PlayN, Google App Engine, and Python Development

2011-11-02T18:37:00.000-07:00

I can't figure out whether I should use Eclipse or IntelliJ.

Eclipse is good in that it's open source and free. Furthermore, Google has released several plugins for it such as the ADT (Android Developer Tools) plugin and the Google Plugin for Eclipse. However, Eclipse generally leaves me feeling confused, overwhelmed, and out of control. I spent two days reading a bunch of tutorials, but I still feel like I can't do what I want. I installed Aptana Studio 3 (which includes PyDev) in order to play around with my Python Google App Engine project. However, I couldn't figure out how to do two things: 1) update the PYTHONPATH for my project to include a few project-specific directories 2) use two space indents in Python code just for this project (which is the standard at Google).

On the other hand, there's IntelliJ. I've never used IntelliJ for Java development, but I've used PyCharm and RubyMine for Python and Ruby development. The downside is that it's fairly expensive. The upside is that it's really good. It doesn't leave me feeling confused, overwhelmed, and out of control. In general, I'm able to get it to do what I want. Furthermore, the IDEAVim keybindings are pretty good (not as good as the JVi plugin for NetBeans, but still pretty good).

I'm hoping to start with a toy project that uses the PlayN framework (e.g. Java), Android, etc. It'd be nice if I could use the same IDE for Java, Python, web stuff, etc. I don't really know all the various Google APIs such as Google App Engine for Java and Android, so an IDE that can guide me along would be helpful. It seems like the only solution is to use Eclipse for the Google stuff and IntelliJ for Python, Ruby, and web stuff. I haven't purchased a license yet, but I did win a license for RubyMine which might be applicable.

I just read Android Development: Eclipse vs. IntelliJ IDEA which said a) IntelliJ is way better b) all of the Android tools are still accessible outside of the IDE anyway. I also noticed that IntelliJ has come out with its own Android plugin, and it's even open source. I'm leaning toward IntelliJ, but I hate being the only one around using a certain tool. Any advice you guys have would be welcome.

(By the way, I'm sure someone is going to come along and plug Vim, Emacs, or TextMate. I'm personally a Vim diehard. However, I've come to appreciate the benefit of using Vim keybindings within a larger IDE. YMMV.)

YouTube: Magisto

2011-09-22T15:09:00.001-07:00

It's tough work being a developer advocate at YouTube. I often have to do things like blog, talk tech with other programmers, and invite people to lunch. It even requires a higher standard of personal hygiene. For instance, the other day, I had to buy a new Google T-shirt because I had a large toothpaste stain on the shirt I was wearing, and I had a business lunch with a potential partner.

Well after months of effort, my work has finally come to fruition. TechCrunch just announced that Magisto is now part of YouTube.com/create.

Hmm, probably a good time to go to the gym. All those business lunches are starting to show ;)

Humor: Millions of Microprocessors

2011-08-30T09:21:00.001-07:00

Today I ran "fortune", and I got:

I went to my first computer conference at the New York Hilton about 20 years ago. When somebody there predicted the market for microprocessors would eventually be in the millions, someone else said, "Where are they all going to go? It's not like you need a computer in every doorknob!"

Years later, I went back to the same hotel. I noticed the room keys had been replaced by electronic cards you slide into slots in the doors.

There was a computer in every doorknob.
-- Danny Hillis

Aside from the fact that that's an awesome quote, I'm pretty excited about this because Dannis Hillis was one of the founders at a company I worked for (Metaweb).

Linux: Vim Freezing

2011-08-23T11:53:00.000-07:00

I had a problem with Vim freezing, and I finally figured out why. If you log into a Linux machine with X11 forwarding turned on, but without an X server, vim hangs when you try to start it. So do various other commands.

I recently switched from an Ubuntu laptop to a MacBook Pro running OS X. In my .ssh/config, I had:

Host my-server-name

 ForwardX11 yes

When I sshed in from my Linux box, things were fine. However, when I sshed in from my Mac, things were broken. There are three workarounds:

Don't enabled X11 forwarding. X11 forwarding is enabled for ssh by either the -Y or -X flags or by the ForwardX11 configuration variable.
If you do enable X11 forwarding, make sure you start an X11 server first and then log in with a fresh xterm or terminal window so that the DISPLAY environmental variable is properly set.
Pass -X to vim so that it doesn't try to connect to the X server. Vim will not be able to alter the window title or the clipboard. It was surprising to me that vim will try to connect to the X server even if you're not using gvim. This approach won't fix other applications that may rely on X in unknown ways.

CSS: Fading Clipped Text

2011-08-11T18:35:00.000-07:00

CSS has a feature, "text-overflow: ellipsis;", that works really well when you want to clip a piece of text and replace it with "...". However, it only works for a single line of text. It doesn't work when you have a paragraph of text, and you want to clip all the text below a certain point.

This is a problem I've struggled with a couple times. I've always wanted to figure out how to have the text fade out at the bottom of the box. Before now, I've aways just clipped the text at the bottom of the box; that looks ugly since most of the time, the text is vertically clipped in the middle of a sentence.

During a recent HTLM5 Hackathon at Google, I learned about gradients and masks. After about 45 minutes worth of futzing around, I finally achieved the effect I was looking for. Here's the code. It basically puts a mask over the text with a gradient that makes the text fade out.

<!DOCTYPE html>



<html>

  <head>

    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

    <title>Fading Text</title>

    <style type="text/css">

      .box {

        list-style: none;

        padding: 5px;

        background-color: #F5F5F5;

        border: 1px solid #DCDCDC;

      }



      .video-list {

        padding-left: 0;

      }



      .video-list li.box {

        width: 400px;

        height: 125px;

        margin-bottom: 5px;

        overflow: hidden;

        position: relative;

      }



      .video-title {

        font-weight: bold;

      }



      .video-duration {

        padding-bottom: 1em;

        font-style: italic;

        font-size: 75%;

      }



      .video-description {

        font-size: 75%;

      }



      .video-description-fade {

        position: absolute;

        bottom: 0;

        left: 5px;

        height: 2em;

        background-color: #F5F5F5;

        -webkit-mask-image: -webkit-gradient(

        linear, left top, left bottom,

        from(rgba(0,0,0,0)), to(rgba(0,0,0,1)));

      }



      .video-description,

      .video-description-fade {

        width: 272px;

      }

    </style>

  </head>

  <body>

    <ul class="video-list">

      <li class="box playlist-item">

        <div class="video-title">Google I/O 2011: HTML5 Showcase for Web Developers: The Wow and the How</div>

        <div class="video-duration">60:23</div>

        <div class="video-description">

          Eric Bidelman, Arne Roomann-Kurrik



          We'll share the strengths and extents of HTML5, showing magnificent

          demos of bleeding-edge features in Google Chrome. Digging into

          high-fidelity graphics, performance, and system integration, we'll

          break each demo down on the big screen to show how it was

          constructed. Then we'll show you how to use Chrome to its full

          potential in your own projects.

        </div>

        <div class="video-description-fade"></div>

      </li>



      <li class="box playlist-item">

        <div class="video-title">Chicken Monkey Duck</div>

        <div class="video-duration">1:20</div>

        <div class="video-description">

          Get it on iTunes:

          http://itunes.apple.com/us/album/the-very-last-songs-i-will/id379861243



          LYRICS



          Monkey chicken chicken,

          Monkey Chicken, duck duck,

          Chicken monkey monkey, Chicken Monkey,

          chicken chicken monkey duck.

          Monkey duck,

          Chicken duck,

          Monkey monkey duck duck,

          Chicken Monkey, chicken chicken monkey,

          "Chicken Monkey Duck."



          Chicken chicken monkey duck,

          Chicken Monkey, duck duck,

          Chicken chicken monkey,

          Chicken monkey,

          Chicken duck.

          Chicken duck duck,

          Chicken monkey monkey duck,

          "Chicken Monkey Duck?"

          Chicken duck.

          Monkey duck duck:

          Chicken chicken, monkey,

          Chicken, monkey monkey, Chicken Monkey.

          Chicken chicken monkey?

          Chicken, monkey monkey,

          "Chicken Monkey Duck."



          Chicken chicken, monkey,

          "Chicken Monkey Duck."

          Chicken chicken, duck.

          Chicken Monkey, monkey,

          Chicken Monkey—chicken duck.



          Duck, Chicken Monkey

          Chicken chicken monkey, chicken.

          Duck duck,

          Chicken chicken, duck.

          Chicken, monkey monkey...

          Chicken!



          Duck duck, Chicken Monkey,

          Chicken chicken, monkey.

          Chicken Chicken (monkey monkey),

          Chicken monkey,

          "Chicken Monkey Duck."



          Chicken chicken chicken,

          Monkey monkey,

          Chicken Monkey:

          Duck duck duck duck duck duck duck duck,

          "Goose."



          The Album:

          iTunes — http://itunes.apple.com/us/album/the-very-last-songs-i-will/id379861243



          Bandcamp — http://music.mikephirman.com

        </div>

        <div class="video-description-fade"></div>

      </li>

    </ul>

  </body>

</html>

Python: Newbie Nugget: Lambda Expressions

2011-08-11T16:41:00.001-07:00

I gave a talk for the SF Python Meetup on lambda expressions:

(If you can't see the slides above, you can see them here.)

Linux: VNC Stopped Working

2011-08-03T11:30:00.001-07:00

I have been using VNC to connect from my Ubuntu box to an OS X box. On the Ubuntu side, I was using vinagre, which is the standard "Remote Desktop Viewer" application. On the Mac side, I was using standard desktop sharing.

Things were working for months, but they stopped working for me the other day. I could connect just fine, but I could only see a black screen. Apparently, this is a known issue.

Switching to Remmina (sudo apt-get install remmina) fixed the problem.

Go: A Second Impression

2011-07-23T15:08:00.000-07:00

My initial impressions of Go have always been somewhat negative because I was comparing it to things like Scala, Haskell, and Python. However, having read a good part of the tutorial, I've changed my opinion. I really like Go!

Go doesn't try to be an extension of C to make it more like Smalltalk--that's Objective C. Go doesn't try to be the end-all-be-all of object-oriented languages derived from C--that's Scala.

Since it neither maintains backwards compatibility with C nor adds a ton of features, I originally had a hard time getting excited about Go. Now I see that Go is a modern language that tries to follow what I think of as C's philosophy. It's simple, elegant, small, and native.

There's an old saying:

Do not seek to follow in the footsteps of the wise men of old. Seek what they sought.

- Matsuo Munefusa (”Basho”)

I think that perfectly describes Go.

Humor: Proving Programs

2011-07-16T08:08:00.000-07:00

I've always been weary of programming proofs.

For instance, I can mathematically prove that 4195835 / 3145727 > 1.3338. However, I know of certain Pentium processors that would disagree with me. If I try to prove that the following bit of C code prints out "Hello World":

if (4195835.0 / 3145727.0 > 1.3338)
    printf("Hello World\n");
else
    system("rm -rf /");

I might be a bit surprised when it deletes my hard drive instead ;)

1 bunny + 1 bunny = 2 bunnies, right? Well it depends on their sexes. It's possible that in a given time period, 1 bunny + 1 bunny might equal 5 bunnies. As I joked in a previous blog post, "All models are wrong. Some models are useful."

I really think this same thing applies to proving programs. Donald Knuth famously said, "Beware of bugs in the above code; I have only proved it correct, not tried it."

Books: Masterminds of Programming

2011-07-16T06:52:00.001-07:00

I just finished reading Masterminds of Programming: Conversations with the Creators of Major Programming Languages

Masterminds of Programming features exclusive interviews with the creators of several historic and highly influential programming languages. In this unique collection, you'll learn about the processes that led to specific design decisions, including the goals they had in mind, the trade-offs they had to make, and how their experiences have left an impact on programming today.

In short, I really enjoyed it. Here's an extremely abbreviated and opinionated summary:

Adin D. Falkoff (APL) made programming as mathematical as possible.

Thomas E. Kurtz (BASIC) was generally a nice guy who wanted to bring programming to the masses.

Charles H. Moore (FORTH) frustrated the heck out of me. He stated that operating systems are the software industry's biggest con job. I disagree. Operating systems protect me to some degree from bad and malicious code. They also let me run multiple programs at the same time and allow me to keep running even when one of the programs crashes. He also said that a piece of code written in any other programming language will be 10 times as large (in number of lines of code) as the same code written in Forth. I'd like to see him try that trick with Python!

Robin Milner (ML) was completely fascinated with programming models and proving the correctness of code. That reminds me of the quote, "All models are wrong. Some models are useful."

Donald D. Chamberlin (SQL) showed me some of the history of SQL. I didn't know IBM research was such an interesting place.

Alfred Aho, Peter Weinberger, and Brian Kernighan (AWK) were as good as I expected.

Charles Geschke and John Warnock (PostScript) talked about Adobe and the history of PostScript. I just don't like that Charles guy, and I don't like Adobe. However, they're smart guys.

Bjarne Stroustrup (C++) was as frustrating as I expected.

Bertrand Meyer (Eiffel) was really interesting. He wrote a book in French that has had a profound impact on French programmers. If he had translated that book into English, it's likely he'd be as famous as, say, Richard Stevens (the author of "UNIX Network Programming").

Brad Cox and Tom Love (Objective-C) showed me that Objective-C's goal was to enhance C in the smallest way possible to make it a bit more like Smalltalk.

Larry Wall (Perl) was awesome, as usual.

Simon Peyton Jones, Paul Hudak, Philip Wadler, and John Hughes (Haskell) were fascinating, as usual.

Guido van Rossum (Python) was practical and interesting, as usual.

Luiz Henrique de Figueiredo and Roberto Ierusalimschy (Lua) were okay. (I'm a Python guy, so it's a bit hard for me to get excited about Lua.)

James Gosling (Java) appears to suffer from premature optimization.

Grady Booch, Ivar Jacobson, and James Rumbaugh (UML) left me even less interested in learning UML.

Anders Hejlsberg (Turbo Pascal, Delphi, C#) was awesome. I knew I liked this guy from previous books, but this interview made me like him even more.

Overall, I think this book was a bit drier than, say, Coders at Work: Reflections on the Craft of Programming, so you should read that one first. However, if you're a guy like me who loves programming languages, this book is a must read.

Books: Python 3 Web Development Beginner’s Guide

2011-07-15T23:52:00.000-07:00

Packt Publishing asked me to review Python 3 Web Development Beginner's Guide. I'll have to admit, it's a bit of an odd duck. A better (albeit overly verbose) title might have been "An Introduction to Rich Internet Application Development Using jQuery UI, a Very Modern Version of Python, a Relatively Old Python Web Application Framework Named CherryPy, and an Ancient Version of HTML Written by a Guy Who Uses Windows".

The first tipoff that this book was a bit strange was that the author uses Windows and some combination of Firefox and IE. It seems like most web developers use OS X (or occasionally Linux), and they prefer Chrome over IE.

The next tipoff was the use of jQuery UI. jQuery UI is a very modern technology which is often used to build rich internet applications. RIAs really aren't the sort of thing that I would expect to see in a book for beginners. What happened to the old days when beginning web applications focused on the server dynamically generating HTML? If I took the time to count the number of lines of code, I wouldn't be surprised if this book had more JavaScript than Python.

The title of this book mentions Python 3, but if you search for "Python 3" in the book, there are extremely few mentions of it. This book really isn't about Python 3 per se (as compared to Python 2); it has a lot more to do with jQuery UI.

Whereas Python 3 and jQuery UI are very modern technologies, standing in contrast is the book's use of HTML 4 and CherryPy. HTML 4 is an *ancient* version of HTML. I would expect anyone using jQuery UI to use either XHTML or HTML5. At the very least, I would have expected one of the transitional DTDs. Similarly, he uses CherryPy. Although I agree that CherryPy is solid code, it's also fairly old. It predates any of the modern Python frameworks.

This book claims to teach web development "without having to learn another web framework" [p. 1]. That's simply not true. It makes heavy use of CherryPy. The home page for CherryPy calls it an "HTTP framework" and says that it has "everything you would expect from a decent web framework." It's not as full-featured as, say, Django, but parts in the example code such as "@cherrypy.expose" [p. 36] are certainly framework features. In fact, "@cherrypy.expose" is part of CherryPy's object publishing system, which it uses as a replacement for regex-based URL routing.

Another thing that's a bit strange about this book is that the author doesn't use a client-side or a server-side templating language. In JavaScript, he tends to use string concatenation, which is weird because there is a templating plugin for jQuery. On the server, he embeds HTML directly in the Python code, which is pretty ugly (as he mentions on p. 229).

Furthermore, the code is extremely sloppy. The code does not follow Python's style guide concerning whitespace (PEP-8) (see, for example, p. 145) even though PEP-8 is extremely standard in the Python community. I don't know of anyone who puts a space before the colon in expressions such as "if not isinstance(name,str) :" [p. 146]. Nor is it even self consistent. The indentation in the JavaScript is not only non-standard and inconsistent, it's occasionally completely wrong [p. 118] (i.e. the indentation disagrees with the braces).

Aside from bad style, I'm a little concerned about various coding practices. For instance, the JavaScript at the bottom of p. 40 has variables that don't use var. This means they're effectively global. This is extremely bad practice. Fortunately, he does use var in other places in the book.

On the subject of security, there are several standard security vulnerabilities that web applications must protect against: XSS (cross-site scripting vulnerabilities), SQL injection attacks, XSRF (cross-site request forgeries), and session fixation (or session hijacking) attacks. Every book on web development should cover these.

The book mentions XSS, but I fear it's approach may not be thorough enough. It does not mention the term "SQL injection" attack, but the ORM shown in the book does look to be somewhat safe. It mentions XSRF, but says that it's out of scope. It doesn't mention "session fixation" or "session hijacking" at all. In general, I don't think the book is good enough about "escaping things" properly. For instance, on p. 293 the author creates a URL in JavaScript using values from a form, but he doesn't take care to URL encode the parameters.

Despite all of the above, I can say this about the book. The author does a good job explaining the web to beginners. Modern web applications are fairly complicated beasts. There's the client, the web server, and the database server, and they each require their own syntaxes. The author does a decent job explaining what runs where. It can be difficult for an expert web developer, such as myself, to remember that newbies might not know all these things.

In summary, will this book help you become a competent, professional web developer? Absolutely not. Is it as well written as, say, Agile Web Development with Rails. No. However, might it be a good way for a beginner to dip his toes in web development with Python and jQuery UI? Maybe.

(Disclaimers: Packt gave me a free electronic copy of this book in trade for my review. I have not read the whole thing. I did read the first 50 pages and skimmed various key sections.)

Google I/O 2011

2011-07-07T07:36:00.000-07:00

I went to Google I/O May 10-12, 2011. Note that the first day of Google I/O was my second day as a Google employee. These are my personal notes. If you're interested in any of these talks, you can find them on YouTube.

Keynote

One of the slides during the keynote showed a picture of an android eating an apple.

There were 5000 people in the room for the keynote.

There were 110 cities watching at viewer parties.

Android has come a long way: 100 million activations, 450,000 developers, 215 carriers, 310 devices, 112 countries, 400,000 activations per day, 200,000 apps in the Android market, 4.5 billion apps installed.

Android 3.1 is Honeycomb.

Android can now act as a USB host. Hence, you can now get a keyboard and a mouse for your tablet.

You can use an X-Box controller with it.

Android 3.1 will be used for Google TV.

Android market will be available on Google TV.

Ice Cream Sandwich will be the new Android release.

The hope is for one Android OS everywhere.

It will be open source.

They have Android software that recognizes where you're looking and can recognize your head movement too.

There are books on Android market.

There are movies on Android market. They're not tied to any specific device. They're just tied to your Google account.

You can "pin" a movie in order to download it to your device.

Verizon seems to have the coolest new Android stuff.

Google has a new music service in beta.

If you add music to your account, you can listen to that music from any of device.

There are Windows and Mac clients.

There's a web-based music manager.

It's completely synced on all devices.

There's an "instant mix" feature based on machine learning.

The music is kept in the cloud. There's no need for syncing.

You can cache music recently played.

You can make stuff available instantly.

The Music application is initially by invitation only.

It's free while in beta.

You can upload 20,000 songs.

You need Android 2.2 or higher for the Android app.

There's going to be an alliance to determine how quickly Android devices should be updated and how long they should be maintained.

Sprint is in the Alliance. So is Verizon. I'm a little peeved at Sprint because they won't upgrade my Samsung Moment beyond Android 2.1, and I need at least 2.2 for some of the apps I'm interested in.

Android open-accessory is a standard accessory platform. It has hardware and software components.

There was a demo of an exercise bike plugging into a phone.

The accessory development kit (ADK) is based on Arduino.

There was a demo of a full-sized (i.e. man-sized) labyrinth board controlled by tilting a tablet.

The ADK doesn't have any NDAs or fees. It's completely open.

Android @ Home is an OS for your home.

There's an Android @ Home framework. It has its own wireless protocol. It has very low cost connectivity.

He said to envision a real world Farmville ;)

He showed a demo of the lights being connected to a game of Quake.

LightingScience has prototype lightbulbs.

Project Tungsten is a hub for your home. It runs Android @ Home.

They showed a prototype where you can just touch a CD to the hub and it starts streaming music from the cloud.

Android@Home.

They gave away free tablets!!!

There wasn't any mention of GAE. It was mostly about Android.

Life of a Google API Developer

They were showing off Google API explorer, which looks pretty neat.

There's a list of APIs. Try them.

There are lots of APIs.

See code.google.com/apis/explorer.

The API explorer is much nicer than using curl.

They use OAuth2 with delegated auth.

The Google API Discovery Service is an API to serve docs about APIs.

There are generic client libraries for the Google stuff for Java, Python, PHP, .NET, Ruby, etc.

OAuth2 is complicated, but they have nice helper libraries.

They're using Mercurial on code.google.com.

Daryl Spitzer said Go is amazingly succinct.

Passwords are like underwear. Keep them secret. Change them often.

The APIs have quotas. However, they're reasonable for moderate apps.

Google is introducting some APIs that you have to pay for.

5000 tablets were given to users for free a month before the actual release of the tablets.

The talks for Google I/O are on YouTube.

Go for Web Apps

Rob Pike (of UNIX fame) gave part of the talk!

Go is fun, efficient, and open source.

Go has a web server.

It's natively compiled.

It has gdb support.

It simplifies some stuff.

It was originally intended for systems code like building web servers, but it's now more general.

It has 1st class functions.

It has low level types, like float32.

I don't understand its replacement for exceptions. [Several weeks later, I read an article on Defer, Panic, and Recover.]

goinstall is its package manager.

It has closures.

It is statically typed.

Google API clients are coming for Go.

Go now has support for GAE! The apps are compiled in the cloud.

Go is the first native language on GAE.

Go is not theoretically interesting. It's just really useful.

Go's library support is improving quickly.

It has garbage collection and full control over how memory is laid out.

Programming Well With Others

The format for of this talk was really entertaining. In fact, it was my favorite talk. They pretended to do a radio show called "The Ben and Fitz Show".

There are people who are friendly but steal too much time and attention.

Perfectionists suck energy out of a project.

"You are not your code."

Python at Google

This talk was by Wesley Chun.

io2011 is the tag to use for Google I/O this year.

Python just turned 20.

C++ is Google's primary language.

Python was used at Google before Google really existed. It was used in the original crawler.

Java came later.

Google has a Python training class.

YouTube gets 2 billion views/day.

There are 35 hours of video uploaded to YouTube every minute.

Welcome to Computer Programming For Kids (CP4K) looks like a fun book.

HTML5 Showcase

This talk was amazing.

Here's the video.

Here's some code.

HTML5 can handle binary data.

There's a file uploader that can upload multiple files or even a whole directory. You can drag and drop images to upload them.

The talk was overwhelmingly awesome!

There's 3D support in the browser.

There's WebGL 3D. It runs on the GPU.

They showed a 3D file browser tied to an in-browser command line.

They showed real-time audio processing in JavaScript.

They showed CSS that produces 3D effects declaratively.

The three parts of this talk were about files, graphing, and audio.

YouTube APIs iframe Player

Using the iframe player simplifies things.

HTML5 doesn't have fine tuned video support yet: there's a content protection problem; there's no full-sceen HD (except Webkit); there's no microphone and camera access; there's a format problem.

However: HTML5 has better accessability; there's mobile support; it has faster start times; it's more reliable.

HTML5 doesn't support all the features used by YouTube.

Custom AS3 players won't work on iOS, of course.

There's a YT object you can play with on the Chrome console.

A guy named Greg wrote the HTML5 player.

WebM is a better encoding.

The performance of HTML5 video on older hardware is not necessarily better than Flash. (That's disappoints me because I know Flash can be a hog. I know Quicktime ran pretty well on older Mac hardware that can't keep up with watching modern movies via Flash.)

Fireside for Developer Relations

Mike Wenton leads developer relations.

There are lots of GTUGs (Google technology users groups).

The public forums need more Google attention, especially for GAE.

Google needs to reach out to unconverted fans at business conventions.

Android's scale of growth has been mind blowing.

There are a bunch of programmers who are nuts about Google.

There were lots of developer advocates in the crowd.

Closure Talk

The speaker was the one who did did the closure book. He worked on Google calendar. He's no longer at Google.

He defined a large project to be 30k+ lines of code, 4+ people, and 6+ months.

How should you code a RIA like Gmail and Calendar? Use GWT? Write a ton of JS? Calendar, Gmail, Maps, Blogger, Docs, and Reader were all pure JavaScript.

Closure has many parts. There is the the Closure library, Closure templates, and the Closure compiler (which also does static checking). They're all independent. It even makes sense to use the Closure compiler with jQuery.

Code written using the Prototype library is not easy for people to read unless they already know Prototype.

This code always returns undefined because of ";" insertion:

function f() {
 return
  "foo";
}

Too many script includes in the head leads to slowness.

Create a special version for mobile. However, this can lead to ugly forking.

The Closure compiler can compile templates + your JavaScript + the Closure library iteself into optimized JavaScript.

The compiler is a whole program optimizing compiler.

The templating language escapes HTML by default.

The closure library can compress an amazing amount. In advanced mode, it's incredible.

YUI compressor can compress JavaScript to 27% of original size. However, the Closure compiler running in advanced mode can compile JavaScript to 0.5% of its original size.

It does things like strip functions that are never called, which is useful if you're using a library like jQuery.

Don't create a mobile version of your JavaScript library. Use 1 version, and let the Closure compiler strip stuff not needed by defining which user agent you can assume. Hence, you can compile multiple different versions of the same library.

The compiler can statically enforce type hints that you give in docstring comments.

Closure tools help manage and optimize large JavaScript apps.

Software Engineering: Code Reviews vs. Peer Programming

2011-07-06T12:53:00.000-07:00

I've been thinking lately about the benefits of code reviews vs. peer programming. In general, I think most companies that really care about code quality use either code reviews or peer programming. Various companies are famous for using either one approach or the other, which leads me to wonder which approach is better under which circumstances?

Pivotal Labs is famous for doing full-time peer programming. It's well known that they do a good job writing software. However, I wonder if full-time peer programming might be too expensive for mundane code. I also wonder if it makes sense to work as a pair when someone needs to spend a few days reading, learning, and researching.

In contrast, Google is famous for code reviewing all commits before checkin. Certainly this frees up engineers to get more work done since they can spend a high percentage of their time working in parallel on separate tasks. However, I wonder if a code reviewer really has the ability to make the same level of architectural improvements as a peer programmer. Certainly a code reviewer can catch style mistakes, but it's much harder to tell someone their entire approach is wrong (for instance, threaded code vs. asynchronous code).

Furthermore, I wonder if code reviewers in general can catch all the little assumptions that get built into code. It reminds me of a story my boss once told me. A few decades ago, he was working on satellite control software. There was a piece of code that made it through three levels of code review even thought it contained a bug. It was missing a minus sign in some equations having to do with navigation. When the satellite was launched, it started spinning because it kept "thinking" it was upside down. It wasted half of its fuel before they could get the problem under control. My boss said that for satellites, the lifespan of a project is directly connected to the amount of fuel onboard. Hence, this came to be known as the three million dollar minus sign.

If this code had been peer programmed rather than code reviewed (or in addition to code review), would the peer have spotted the problem as the equation was being worked out? Certainly this taught me a valuable lesson about code review. It's far too easy to get hung up on less critical issues that are easy to spot, like style, instead of focusing on more important issues that require more brain power to understand.

In the book Professional Software Development: Shorter Schedules, Higher Quality Products, More Successful Projects, Enhanced Careers (see my blog post), Steve McConnell said that NASA found that the single most effective way to cut down on defects was to always have a second pair of eyes present. They were talking about building the shuttle, but I think the same thing applies to software.

Despite their pervasive use of code reviews or peer programming, Google and Pivotal Labs have both had their fair share of bugs. Even NASA makes mistakes. Hence, it's easy to see that neither code reviews nor peer programming can banish all defects. Given how fallible human beings are, it seems to me that the best way to keep people from dying because of mistakes is to avoid situations where mistakes are fatal. Driving is a fairly dangerous activity, and tens of thousands of people die each year in the United States because of driving errors. However, even more than that make mistakes and yet survive because of various safety precautions.

So even though I think code reviews and/or peer programming are important factors in writing high quality software, I think it's even better to avoid situations where software defects can cause serious damage. I certainly think that the more mission critical a piece of software is, the smaller and stupider it should be.

Amazon Web Services Summit 2011: Navigate the Cloud

2011-07-05T11:35:00.000-07:00

On June 21, 2011 I went to an Amazon Web Services conference in San Francisco. These are my notes.

In general, the conference was informative, but a bit subdued. It was about a quarter of the size of Google IO, and they weren't giving away anything huge for free. Furthermore, it was only one day, and the size of the crowd was much smaller. However, that makes sense since they have multiple of these conferences per year. Nonetheless, it was interesting, and I learned some stuff.

Keynote

The part of the keynote was given by Dr. Werner Vogels, the CTO.

AWS doesn't lock you into an OS or language.

There are 339 billion objects in S3, and S3 handles 200,000+ storage transactions per second.

On a daily basis, they add as much capacity as they had in total in
2000.

Spot pricing allows you to spin up an instance when the price is right.

AWS CloudFormation deploys a stack of software.

AWS Elastic Beanstalk is easy to begin and impossible to outgrow.

CloudFormation and Elastic Beanstalk don't cost anything extra.

They now have Simple Email Service.

They're building features by working from the customer backwards. They start by writing the press release, then the FAQ, next the documentation, and then finally the code.

They mentioned that Amazon RDS (Relational Database Service) is not as scalable as S3.

AWS is now a PCI level 1 service provider.

amazon.com (i.e. the retail business) is moving to AWS.

They mentioned Amazon VPC (Virtual Private Cloud).

Autodesk

There have been more than 2 million downloads of Sketchbook Mobile.

Homestyler is the fast, easy, free way to design your dream home.

Autodesk uses EC2, S3, EBS, and CloudFront.

Homestyler can now do photo realistic renderings of your home.

They mentioned project Nitrous. It provides a rich, in-browser experience. It's like Google Docs but for your 3D modeling documents.

They're doing stuff we're they're rendering in the cloud and then streaming to an iPad.

TellApart

They handle customer data in real-time (i.e. with 100ms latencies) on AWS.

They have a primarily ex-Google team.

They provide customized ads based on user behavior.

They have some serious latency requirements. They had to hit 120ms end-to-end. They worked closely with AWS to meet their needs.

Their ad performance is incredible. They have a 7.5% click through rate.

Adobe

This part of the keynote was given by Mitch Nelson, Director, Managed Systems.

LiveCycle is a managed service.

They talked about the Flash Media Server.

Adobe Connect is a managed service. It's an online meeting product. It's based on Flash.

Cloud Computing at NASA

This part of the keynote was about the JPL (Jet Propulsion Laboratory).

They wanted to augment availability in multiple geographic regions.

Some things can be safer in the cloud.

They're using the VPC (Virtual Private Cloud). They use IPSec.

They've saved a lot of money by adopting cloud computing.

Buying infrastructure way ahead of time is expensive for NASA. Cloud computing saves NASA a lot of money by allowing them to scale to meet their needs.

He talked a lot about the Mars Rover.

They spin up a bunch of instances to handle bursty traffic, processing data from the Mars Rover, which sends data once a day (I think).

The speaker said that the "missions" are his "customers".

These projects are using cloud computing: Mars Rovers, Deep Space Network, Lunar mapping
product, and airborn missions. None of them look like they involve astronauts.

SmugMug

SmugMug is profitable. They have no debt. They're a top 250 website. It's a private company. They had $10M+ in yearly revenues in '07.

They provide unlimited storage for storing photos and unlimited bandwidth for uploading and download photos. They're motto is "More, better, faster" (in comparison to other photo sites).

They can handle photos up to 48 megapixels in size.

They can handle 1920x1080p video.

They use a hybrid of AWS and their own datacenter. They have 5 datacenters. They're moving completely to AWS. They have many petabytes of data stored in S3.

Amazon really listens to their customers, trying to solve their needs.

One time, SmugMug's RubberBand project tried to spin up 1920 cores in a single API call. Amazon spun them all up as requested. SmugMug renamed that project SkyNet.

SmugMug lets customers tie their Amazon and SmugMug accounts together so that SmugMug can pass on the cost of storing raw photos and videos onto their customers (since they're so expensive to store).

SmugMug is designed for breakage, so there was minimal impact during the "Amazonpocalypse". They made use of multiple availability zones.

EBS is just like a hard disk. If you want local redundancy, use RAID. Want GEO redundancy? Replicate. EBS can/does fail just like HDDs. EBS does solve lots of problems, but use it in the appropriate way.

Customer Panel

Use Akami for dynamic generation of content, and use CloudFront for more static content.

Dealing with Akami is a pain in the butt.

The customers on the panel made it through the Amazonpocolypse fairly well. However, they had to architect for it.

The elasticity of Amazon's services is what saves the most money. Customers only pay for resources when they use them. The customers on the panel were pretty good about spinning up and spinning down instances. Also, the customers said that they can't build their own data centers fast enough to meet their own needs.

TellApart uses Spot instances to run Hadoop jobs. That means they only spin up instances when the price is low enough.

Without AWS, a lot of the customers would never have been able to attempt certain projects.

Akami hasn't broken for SmugMug. HAProxy also hasn't broken for them.

AutoDesk is using Scala.

TellApart uses GAE (Google App Engine) for its own dashboards.

The NASA guy uses CloudFormation to setup templates for their clusters.

Amazon often builds infrastructure that their customers have already built so that all the other new customers don't have to rebuild that same infrastructure.

RDS has very unpredictable latency. None of the panel customers were using it.

Security and Compliance Overview

This talk was given by Matt Tavis, Principal Solutions Architect.

Here's a link to the AWS Security Center.

Security on AWS is based on a shared responsibility model.

AWS is responsible for facilities, physical security, network infrastructure, etc.

The customer is responsible for the OS, application, security groups, firewalls, network configuration, and accounts.

AWS has complex features for who has access to interact with which AWS APIs if you have multiple people at the same company.

AWS favors replication over backup.

I wonder if you can mess around with EC2 or EBS and try to read the raw disk to see if there is stuff left over from a previous user.

By default, there's a mandatory inbound firewall, which defaults to deny.

VPC is a networking feature.

High Availability in the Cloud: Architectural Best Practices

This talk was by Josh Fraser, the VP of business development at RightScale.

RightScale is the world's #1 cloud management system.

RightScale serves 40,000 users and 2.5 million servers.

RightScale is a SaaS.

RightScale is the layer between the app and the cloud provider.

You need to design for failure.

Backup and replication need to be taken seriously.

A cloud is a physical datacenter entity behind an API endpoint.

He says that Amazon has 5 clouds. AWS is a cloud provider.

RightScale has ServerTemplates (which act like recipes). They don't like cloud images (because they're too fixed).

They have an integrated approach that puts together all the parts needed to build a single server or set of servers.

ServerTemplates are a server's DNA.

DR is disaster recovery. HA is high availability.

Implementing HA best practices is always about balancing cost, complexity, and risk.

You need to automate your infrastructure.

Always place at least one of each component (load balancers, app servers, databases) in at least two AZs (autonomous zones?).

You need alerting and monitoring.

Use stateless apps.

It's critical to be able to programmatically switch DNS.

Use HAProxy, Zeus, etc. for load balancing.

Scale up and down conservatively. Don't bring up 1000 instances because of 1 minute's demand.

Snapshot your EBS volumes.

Consider a NoSQL solution.

They use Cassandra in multiple regions, and it works well for them.

EBS snapshots can't cross regions.

There is no one size fits all solution.

The most common approaches they see involve multi-AZ configurations with a solid DR plan.

Amazon.com's Journey to the Cloud

Retail is everything at Amazon that isn't AWS. Retail is a customer of AWS.

This talk summarizes the history of Amazon retail from 1995 to 2011. I think the switch to AWS was in 2011.

They initially ran on a single box. They housed the server in their main corporate offices. There was a "water event" that caused them to move to a datacenter.

In 2001, they started switching from Digital Tru64 to Linux.

In 2004, they had 50 million lines of C++ code.

AWS started in 2006. S3 came first, and then came EC2. AWS actually wasn't built to solve problems for the retail side. It was always built to be general purpose, and the retail side was responsible for figuring out how to use it.

IMDb is an Amazon owned subsidiary. It's very separate of the rest of Amazon.

Amazon has strict runtime latency and scale requirements. If your widget can't meet them, your widget won't get shown.

Retail uses VPC. VPC makes AWS look just like your own datacenter.

November 10, 2010 is when they turned off the last frontend server not using AWS. All traffic for amazon.com is now served from AWS.

They've had billions of orders over the lifetime of amazon.com.

Amazon has taken really old orders out of the database and moved them into S3.

They use Oracle.

Be sure to consider compliance issues.

When moving to the cloud, start with simple applications (or simple parts of larger applications).

Iterate toward your desired end-state.

The cloud can't cover up sloppy engineering.

Introducing YouTube.com/create

2011-07-05T11:22:00.000-07:00

I just wrote my first blog post for the YouTube API blog: Introducing YouTube.com/create:

YouTube.com/create is a platform for third-party applications that enable users to create videos. The idea is simple. The third-party application runs in an HTML iframe on YouTube. The user creates a video with the application, and then the application uploads the video to YouTube for the user to watch and share.

To read more, check out the story on the YouTube API blog.

Books: Python 3 Web Development Beginner’s Guide

2011-06-29T11:37:00.000-07:00

I've been asked by Packt to review a new book called Python 3 Web Development Beginner’s Guide. Here's the overview:

Build your own Python web applications from scratch
Follow the examples to create a number of different Python-based web applications, including a task list, book database, and wiki application
Have the freedom to make your site your own without having to learn another framework

I looked through the table of contents, and I scanned the first 30 page. It looks like a fun book! In fact, it's exactly the sort of book I was hoping someone would write.

Python: Increasing the Timeouts for urlfetch in Google App Engine

2011-06-22T09:57:00.000-07:00

Google App Engine provides a function, google.appengine.api.urlfetch.fetch, for fetching URLs. I do believe all the other HTTP client libraries are monkey patched to make use of that function, which is written to take advantage of various Google infrastructure. The fetch function has a default timeout of 5 seconds. You can set a higher timeout by passing a deadline parameter, but the maximum is 10 seconds. Unfortunately, passing a deadline keyword parameter is often difficult if it's a third-party library that is making the call to fetch, for instance if you're using the GData client library.

I looked for a way to set the deadline parameter in a more global way, but I couldn't fine one by mere inspection of the code. I came up with the following HACK in order to work around this problem:

# HACK: Monkeypatch google.appengine.api.urlfetch.fetch to increase the
# deadline.  This is used by the various client libraries.
def _fetch(*args, **kargs):
  from google.appengine.api.urlfetch import _orig_fetch  # Import late.
  kargs["deadline"] = 10
  return _orig_fetch(*args, **kargs)
_fetch.this_is_the_wrapper = True

# HACK: Because of the way the dev app server works when reloading code,
# things are a little tricky here.
from google.appengine.api import urlfetch
if not hasattr(urlfetch.fetch, "this_is_the_wrapper"):
  urlfetch._orig_fetch = urlfetch.fetch
  urlfetch.fetch = _fetch
else:
  assert hasattr(urlfetch, "_orig_fetch")

Google I/O 2011

2011-06-09T12:45:00.000-07:00

I went to Google I/O a few weeks ago. Although I took notes, rather than blogging all my notes, I thought I'd just link to my favorite two talks:

Programming Well with Others: Social Skills for Geeks

Are languages, compilers, debuggers, and algorithms all you need to be a successful software engineer? In a perfect world, those who produce the best code should be the most successful. Unfortunately, we live in a world of imperfect people, and collaborating with others is at least as important as having great technical skills.

HTML5 Showcase for Web Developers: The Wow and the How

We'll share the strengths and extents of HTML5, showing magnificent demos of bleeding-edge features in Google Chrome. Digging into high-fidelity graphics, performance, and system integration, we'll break each demo down on the big screen to show how it was constructed. Then we'll show you how to use Chrome to its full potential in your own projects.

Python: Getting a Fair Flip out of an Unfair Coin

2011-06-03T10:06:00.001-07:00

If you have an unfair coin (i.e. one that favors heads or tails), how do generate a fair flip (i.e. one that doesn't favor heads or tails)? My buddy Hy Carrinski and I came up with the following algorithm:

"""Get a fair flip out of an unfair coin."""

from collections import defaultdict
from random import random

FLIP_RATIO = 0.7


def flip_unfair_coin():
    """Flip an unfair coin.  Return a boolean."""
    return random() > FLIP_RATIO


def create_fair_flip():
    """Generate a fair flip.  Return a boolean."""
    while True:
        flip = flip_unfair_coin()
        if flip != flip_unfair_coin():
            return flip


# Demonstrate that it works.

if __name__ == '__main__':
    results = defaultdict(int)
    for i in xrange(1000000):
        results[create_fair_flip()] += 1
    percentage = (results[False] / float(results[False] + results[True])) * 100
    print "Percentage heads:", percentage

Personal: I Work at YouTube!

2011-05-28T20:55:00.001-07:00

I got a job at Google working as a developer advocate for YouTube APIs. It's nice because it's a bit more of a social position than I usually have, but it still involves some work on open source code. Unfortunately, my desk is in Mountain View, so my commute is three hours a day; I probably won't be able to keep that up indefinitely. Anyway, I'm still very excited :)

I started three weeks ago, and I must admit that Google is about twice as awesome as I was even hoping for :)

Personal: Name Dropping

2011-05-17T17:32:00.000-07:00

Ken Thompson (initial author of Unix) was awarded the Japan Prize today at Google. I just got to meet and talk to him, Vint Cerf (one of the "fathers of the Internet"), and Rob Pike (legendary Unix hacker, famous author, and co-author of Go). Since I shook each of their hands, I've decided to never wash my hand again ;)

Call Me Crazy: Calling Conventions

2011-04-19T14:59:00.000-07:00

It's been said that the most significant contribution to computer science was the invention of the subroutine. Every major programming language has its own variation on how subroutines work. For instance, there are procedures, functions, methods, coroutines, etc. When implementing a compiled language or a virtual machine, you must decide how to implement subroutines at the assembly language level. For instance, you must decide how to pass arguments, what goes on the stack, what goes on the heap, what goes in registers, what goes in statically allocated memory, etc. These conventions are called calling conventions. You might not think of calling conventions very often, but they're an interesting topic.

First of all, what does a subroutine provide over a simple goto? A subroutine provides a way to pass arguments, execute a chunk of code, and return to where you were before you called the subroutine. You can implement each of these yourself in assembly, but standardizing calling conventions in a language makes the process more stable, repeatable, and less prone to error.

The way you implement calling conventions in a language like C is at least conceptually something like the following. You push all your arguments onto the stack; you push your current location in the code (i.e. the IP register) onto the stack; and then you jump to the subroutine. The subroutine executes its code, puts its return value in a register, and then jumps to the return-to address you put on the stack. The caller is responsible for popping the arguments off the stack (this is necessary to support varargs). Often, the compiler may optimize the code by putting values into registers rather than on the stack.

What's the difference between subroutines, procedures, functions, and methods? I think of subroutines as a low-level, catch-all term. In Pascal as well as some other languages, a procedure is expected to have side effects but not return a value, whereas a function is not supposed to have side effects but is supposed to return a value. Of course, not having side effects is a convention, not something enforced by the compiler. A method is a function call on an object. Typically, the object is implicitly passed as the first argument, although that may not be explicit in the syntax for the language.

Coroutines are a twist on subroutines. When you call a subroutine, it'll eventually return to the caller. If A calls B, then eventually B will return to A. Coroutines don't have to behave in this manner--i.e. they don't have to return to the caller. A may call B which may call C which may call A in a circular manner, without maintaining a stack of return-to addresses.

Recursion is a technique in which a function can call itself. Some languages like Lisp provide recursion as a low-level operation, and all looping constructs are built on top of recursion. There are some algorithms that are more elegantly implemented using recursion than looping such as the Ackermann function.

Recursion implies that you have some sort of stack (whether or not you use a C-level stack). That's because if a function takes one argument and recurses five times, you need a places for five arguments. Languages like COBOL put arguments in statically allocated memory instead of on a stack. Hence, COBOL lacks recursion.

Tail call optimization is an interesting technique that reduces the use of the stack. If function A calls function B, and then B's last step is to call function C, rather than having a return-to address for A and a return-to address for B, you can just have C return directly to A since there's nothing else in function B that needs to be done.

This works really well when a function calls itself as its last step. If F calls itself as its last step, rather than having a large stack of return-to addresses for F, you can optimize it away. In this way, what would normally be a recursive algorithm can be translated into simple interation. This is a critical technique in languages like Lisp that rely on recursion for looping; however, it's not present in languages like Python. (Guido van Rossum has argued that it's harder to debug bugs if parts of the stack have been optimized away. It's possible to end up in a function with no way of knowing how you got there.)

Stacklessness is a technique whereby the language's stack does not rely on an assembly or C-level stack, but rather exists on the heap. Because of the way C stacks require contiguous space, it's difficult to allocate space for 1000s of them at the same time. In a stackless interpreter, because stack frames are allocated dynamically from the heap, you can allocate a greater number of them stacks. This technique is used by stackless interpreters such as Lua as well as the greenlets library in Python in order to implement something like lightweight multithreading.

Some languages provide more flexibility than others in the way arguments are passed. C has the ability to pass a fixed number of arguments, but there's also a facility to pass an arbitrary number of arguments call varargs. Some languages like Perl and Ruby allow you to pass additional keyword arguments in the form of a hash to the function (e.g. "f(1, 2, :arg => 1)"). Some languages like Common Lisp and Python have advanced support for keyword arguments allowing you to call a method that looks like "def f(a, b, c=1, d=2, e=3)" like "f('a', 'b', e=3, d=3)". Notice that you can leave out or rearrange the order of keyword arguments. Python allows you to pass a variable number of positional arguments as well as a variable number of keyword arguments and it will even raise an exception if you pass a keyword argument that isn't expected. Common Lisp was one of the first languages to have extremely sophisticated parameter passing, including sophisticated support for keyword arguments.

Currying is a technique whereby if a function accepts two parameters, A and B, then what you really have is a function that takes an argument A and then returns another function that takes an argument B. In this way, all functions can be reduced to a list of functions that accept 0 or 1 arguments. When a language supports currying, you can pass a value for A and then later pass a value for B. Haskell supports currying.

Partial application is a related technique that lets you pass an arbitrary number of arguments, perhaps in a random order if keyword arguments are supported, and then actually call the function later. If you have a function that takes one parameter, then you can partially apply it to one argument, and you are left with a function that takes no parameters that you can call at a later time. Partial application is more general than currying. It's supported in Python. However, in languages like Python, Haskell, Lisp, etc., it's always possible to create a new function that takes no parameters that simply executes another function passing a particular value to that function. Hence, it's trivial to implement partial application manually.

Lazy evaluation is a technique whereby the arguments to a function are not evaluated until they are used within the function. If you call f(1/0) in Python, you will always get a DivisionByZero exception. However, in languages such as Haskell, if you call f (1/0), as long as f never evaluates its argument, you'll won't get an error. Languages that do not have lazy evaluation are said to be "strict". Hence, Haskell is not strict.

It is an error prone situation to have a language that is not strict (i.e. it supports lazy evaluation) but also supports side effects. For instance, consider writing the expression f(print(1), print(2) in a language that wasn't strict but permitted side effects. Depending on whether or not f evaluated its parameters, 1 and 2 may or may not be printed, and you couldn't be sure which order they would be printed in. Hence, functions in Haskell are both lazy and side-effect free.

Inlining is an optimization technique that helps avoid the overhead of a function call. If a function A calls a very small function B, an optimizing compiler might take a copy of the compiled code for B and put it "inline" in the compiled code for A. This reduces the function call overhead but results in larger binary sizes.

In some situations, some compilers can aggressively inline all functions. Alternatively, the compiler may be able to put all the code for all the functions side by side and simply jump between the different parts of the code without using the stack. Some programmers use this technique manually in order to write small, but highly efficient programs. In general, the way you think calling conventions work in a language may be a simplified, idealistic way of how they actually work. In the case of inlining, the function call completely disappears.

Continuation passing style is a technique whereby all function calls can be transformed in a way that makes tail call optimization possible. Consider the following function:

def sum(n):
    if n == 0:
        return 0
    else:
        return n + sum(n - 1)

This function cannot natively take advantage of tail call optimization because after it recurses, the return value must be added to n. It's easy to change this to a function that takes advantage of tail call optimization manually:

def sum(n, _accum=0):
    if n == 0:
        return _accum
    else:
        return sum(n - 1, _accum + n)

In this case, an accumulator is passed explicitly. (Remember, however, that even though I'm using Python syntax, Python does not support tail call optimization.)

Rather than passing an accumulator that contains a value, you can pass an accumulator that contains what other code needs to execute after the recursive call. Here is an example of that:

def sum(n, _callback=lambda x: x):
    if n == 0:
        return _callback(0)
    else:
        return sum(n - 1, lambda x: _callback(x + n))

By the way, if you find that code difficult to understand, don't feel bad! It took me more than half an hour to write it, despite the fact that I had seen the technique before! That goes to show you that switching to continuation passing style isn't easy. However, it can be accomplished via a mechanical (i.e. "just follow the steps") process, and there are languages that can do it for you automatically. Scala has a compiler flag that turns on continuation passing style so that the JVM behaves differently (conceptually it uses a lot of anonymous functions that exist on the heap rather than return-to addresses that exist on the stack).

Returning to C, you might think that the calling conventions of C were decided upon years ago, and the only variations would have to do with compiler optimizations. Surprisingly, that is not the case. For instance, AMD helped define the new calling conventions for System V on AMD64. In fact, the calling convention are neither C nor AMD64 dependent. It's entirely defined by the operating systems application binary interface standard. Hence, the calling conventions for AMD64 systems are slightly different under Windows (specifically in relation to what registers are used to pass variables). See here and here for more information.

So if you're thinking about implementing a new language, and you wonder what's in a subroutine, you make the call!

Perl: Text-based Address Books

2011-04-19T14:41:00.000-07:00

Over the years, I've stored my addresses in a variety of formats. I used to have a Palm Pilot. These days, I have an Android phone, and Google keeps track of my addresses. However, I've always had a second copy of my addresses in a text file. The format looks something like this:

A & R Stock to Performance:
  Address: 2849 Willow Pass Rd. #A, Concord, CA  94519
  Work Phone: (925) 689-1846

There are many advantages to using a text-based format. For instance, since I'm an expert Vim users, I can search and edit the file extremely quickly. Best of all, the format works on every operating system and never goes obsolete. The only downside is that I have to input the address twice: once for Google and once for my own notes.

Long ago, I wrote a Perl script to convert my notes file into a Palm Pilot database. Even though I don't have a Palm Pilot anymore, I keep the script around. I can easily alter it to output the addresses in different formats.

If you like the format I used above, here's the source code so that you can hack it to do whatever you want:

#!/usr/bin/perl -w

#
# Author: Shannon -jj Behrens
# Email: jjinux@gmail.com
#
# This program converts the addresses stored in my notes file into a Palm
# database file (a pdb), which I can then import into my Handspring Visor.  
# See usage for more details.
#
# Global variables: 
# $_0 - This is $0, but without the path.
# $pdb:: - This is a handle to the Palm address database being edited.
#

# %phone_mappings - Mappings from things such as "Cell" to integers.  These 
#   values were deduced from the @Palm::Address::phoneLabels array as well
#   as usage in my notes.txt file.
%phone_mappings:: = (
    Work => 0,
    Home => 1,
    Fax => 2,
    Other => 3,
    Email => 4,
    Cell => 7
);

# $MAX_PHONES - The maximum amount of "Phone" type fields.
$MAX_PHONES:: = 8;

use strict;
use Palm::PDB;
use Palm::StdAppInfo;
use Palm::Address;


# Output usage information to the user and exit with value 64 (see man 
# sysexits on a FreeBSD machine).
#
# By the way, here's how I use this program:
#
# mkdir palm
# pilot-xfer -b palm
# ./notes_to_palm ~/notes.txt palm/AddressDB.pdb
# pilot-xfer -r palm
# rm -r palm
sub usage {
    print "usage: $_0:: notes.txt AddressDB.pdb\n";
    exit 64;
}


# There's an ugly "bug" that causes Perl 5.6 to complain about unamed
# categories.  Hence, I have to do a little bit of initializing or else I get a
# whole bunch of uninitialized warnings.  
sub init_category_names {
    for (my $i=0; $i < Palm::StdAppInfo::numCategories; $i++) {
        if (!defined($pdb::->{appinfo}{categories}[$i]{name})) {
            $pdb::->{appinfo}{categories}[$i]{name} = "";
        }
    }
    $pdb::->{appinfo}{dirtyFields} = 1;
}


# Return a new record.  I have to do a little bit of initializing or else I get
# a whole bunch of uninitialized warnings.  I wonder why the libraries don't do
# this automatically since they initialize everything to undef anyway.  Also,
# I'll insert a phoneIndex variable into the $record so that I can use the
# phone slots on a first come first serve basis.
sub new_record {
    my $record = $pdb::->new_Record;
    foreach my $field (qw( name firstName company phone1 phone2 phone3
                           phone4 phone5 phone6 phone7 phone8 address
                           city state zipCode country title custom1
                           custom2 custom3 custom4 note )) {
        $record->{fields}{$field} = "";
    }
    $record->{phoneLabel}{reserved} = 0;
    $record->{fields}{phoneIndex} = 1;
    return $record;
}


# Process a name line, such as "Shannon -jj Behrens (author)".
sub handle_name {
    my ($fullname) = @_;

    # Check for note.
    if ($fullname =~ /^(.*)\((.*)\)/) {
        $fullname = $1;
        $record::->{fields}{note} = $2;
    }

    # Assume last word is last name, everything else is first name.
    my @name = split / /, $fullname;
    $record::->{fields}{name} = pop @name;
    $record::->{fields}{firstName} = join " ", @name;
}


# Process a phone type (e.g. Cell) and a value (e.g. "(925) 209-6439").  Please
# read p5-palm's Address.pm module to see the interesting way Palm phone
# numbers work.  When possible, use the email address as the primary "Phone"
# field.
sub handle_phonelike_field {
    my ($type, $value) = @_;

    my $phone_index = $record::->{fields}{phoneIndex};
    return if ($phone_index > $MAX_PHONES::);
    my $phone_field = "phone$phone_index";
    $record::->{fields}{$phone_field} = $value;
    $record::->{phoneLabel}{$phone_field} = $phone_mappings::{$type};
    $record::->{phoneLabel}{display} = $phone_index - 1
        if ($type eq "Email");
    $record::->{fields}{phoneIndex}++;
}


# Process an address line, such as "125 Gilger Ave., Martinez, CA  94553".
sub handle_address {
    my ($fulladdress) = @_;

    my @address = split /, /, $fulladdress;
    $record::->{fields}{address} = $address[0];
    $record::->{fields}{city} = $address[1];
    my @state_zip = split /  /, $address[2];
    $record::->{fields}{state} = $state_zip[0];
    $record::->{fields}{zipCode} = $state_zip[1];
}


# Handle one line, $_, from the notes.txt file.  Remember, there must be at 
# least one blank line after every address record in the notes.txt file in 
# order to flush it to disk.
#
# Creates globals:  $section::, $record::.
sub handle_line {
    # Ignore anything not in the Contacts section.
    if ($_ =~ /-{5,} ([^\-]+) -{5,}/) {
        $section:: = $1;
        return;
    }
    return if (!defined($section::) or
               $section:: ne "Contacts");

    # A blank line is a signal to flush the current record, if any.  Otherwise,
    # it can just be ignored.
    if ($_ =~ /^[ \t]*$/) {
        if (defined($record::)) {
            $pdb::->append_Record($record::);
            $record:: = undef;
        }
        return ;
    }

    # If the line doesn't start with a space, this is a new record's name.
    # Start a new record, and then handle the name.
    if ($_ =~ /^([^ ].*):/) {
        my $fullname = $1;
        $record:: = new_record;
        handle_name $fullname;
        return;
    }

    # Check for phone numbers and email addresses.  
    foreach my $type (keys %phone_mappings::) {
        if ((($type eq "Email") and ($_ =~ /  Email: (.*)/)) or 
            (($type ne "Email") and ($_ =~ /  $type Phone: (.*)/))) {
            handle_phonelike_field $type, $1;
            return;
        }
    }

    # Check for address.
    if ($_ =~ /  Address: (.*)/) {
        handle_address $1;
        return;
    }

    # Check for company.
    if ($_ =~ /  Company: (.*)/) {
        $record::->{fields}{company} = $1;
        return;
    }

    # Anything else can be appended to the additional notes section.
    s/^\s+//;
    $record::->{fields}{note} .= $_;
    print STDERR "Additional notes: $_";
}


my @pieces = split /\//, $0;
$_0:: = $pieces[$#pieces];
usage if (scalar(@ARGV) != 2);
my $notes = shift;
my $addresses = shift;
open(NOTES, $notes) || die "$_0::: $notes: $!\n";
$pdb:: = new Palm::Address;
init_category_names;
handle_line while (<NOTES>);
$pdb::->Write($addresses);
close NOTES;

Of Neuroscience and Jazz

2011-04-19T12:07:00.000-07:00

I am neither a neuroscientist nor a very accomplished musician, but I'd like to talk about the intersection of neuroscience and music. I have a theory that there is a neuroscience basis for why it takes a mature musical palate to enjoy jazz.

First, let me say a little something about neuroscience (based on the limited understanding I've gained by watching a bunch of talks). One of the things your brain is particularly good at is recognizing patterns and predicting patterns. At the lowest level, if two nerves are close to each other, and they both fire, it's counted as a pattern--i.e. those two things are connected. Similarly, if a nerve fires and then a short while later it fires again, that's a pattern as well. Hence, if both of my fingers feel something, there's a pattern, or if I feel a tapping on a single finger, that's a pattern as well.

However, the brain is not limited to low level patterns. Rather, it can respond to a hierarchy of patterns. Paraphrasing Pavlov, if a dog hears a can opener and then smells food and then gets fed, we all know that the dog will soon recognize the pattern and start salivating as soon as he merely hears the can opener. My point is that a sophisticated pattern is composed of lower level patterns recursively, all the way down to the level of individual neurons firing.

Next, let me talk about sine waves and chords. Let's start with a single note. A single note might look something like y = sin(x):This wave is very simple, and somewhat boring.

(By the way, I'm using wolframalpha.com to create these graphs. WolframAlpha is really interesting. It's a computational knowledge engine, and graphing things is just one of a ton of things it can do.)

Now, let's look at a chord consisting of a single note as well as the note that is an octave above it. Here's y = sin(x) + sin(2 * x):This curve is somewhat more sophisticated. However, you can still recognize the pattern by the time x = 10.

Now, let's look at what I think is a fifth, which is still a nice sounding chord. Here's y = sin(x) + sin(1.5 * x):This pattern is slightly more sophisticated, but it's still pleasingly recognizable.

Now, let me show you something that is not pleasing to the ear. This curve represents what might happen if you hit two keys that are right next to each other on the keyboard. Here's y = sin(x) + sin(1.1 * x):This curve pulses in an ugly way. You have to get all the way out to x = 120 or so to even see the pattern. In a manner of speaking, the pattern is only there by "brute force".

So what's my point? Simple patterns are easy to recognize, and they can be recognized in less time (i.e. for smaller variations of x, which I probably should have called t).

Here's something that I think is more of a jazz chord. Here's y = sin(x) + sin(1.25 * x) + sin(1.5 * x) + sin(2 * x):This curve is really interesting. You can recognize what's going on by the time you get to x = 50, but the "texture" of the curve is a lot more interesting. Whenever I play this sort of chord, it sounds deep, rich, and interesting. If the first chord reminds me of "Mary had a Little Lamb", this last chord reminds me of the forbidden love between Lancelot and Guinevere and the pain it caused King Aurthur.

So what's my point? You have to go higher up the pattern recognition pyramid in order to recognize more sophisticated patterns. A more sophisticated musical palate is able to recognize patterns that a simpler musical palate may not recognize. That's why jazz requires a sophisticated palate--it uses chords that require more effort to recognize.

Of course, there are many dimensions to music, and individual chords are just one dimension. There are also chord progressions and beats. The same sort of thing applies to these other dimensions.

Here's the circle of fifths (thanks Wikipedia!):Pick any three chords in a row on the circle of fiths, such as C, G, and D, and you have the basis for a simple, feel-good song. If you just bounce around between the different chords on a guitar using different strumming patterns, you'll immediately recognize a song you already know or create a new song that sounds pleasing. Throw in a few 7th chords and a few minor chords, and the song starts taking on additional richness because the pattern becomes less simple.

If you try to use more than three chords from the circle of fifths, your brain might be left thinking, "Wait a minute. What key am I supposed to be in?" It may be more difficult to recognize the pattern. However, this is exactly the sort of thing that happens in jazz.

The same thing applies to beats. Here's a simple walz, "Um pa pa. Um pa pa." Here's a typical rap beat, "Bum chbum bum chbumbum ch bum bum ch." More sophisticated beats require more sophisticated pattern recognition.

Every song is composed of some pattern repetition and some pattern violation. Simple, pop songs that are enjoyable by youthful palates involve a lot of pattern repetition and less pattern violation. Sophisticated music palates enjoy songs with less pattern repetition and more pattern variation. Furthermore sophisticated music palates get bored of songs more quickly. They recognize the patterns quickly, and they're ready to move on.

Furthermore, popular songs are played more often on the radio so that your brain has a better chance to become more familiar with the patterns. A listener will usually enjoy the song more after listening to it 10 times than if he's only listened to it once. If a song has so much pattern repetition that you enjoy it the first time, you'll often grow bored of it very quickly. Conversely, it's very difficult to enjoy sophisticated classical pieces the first time you hear them. All the popular classical pieces have been driven into our heads since we were kids.

Why is this? I think this can be explained by neuroscience as well. The brain works hard to find patterns, and when it does, the simple recognition of a pattern is somewhat calming. It's says something like, "Hmm, I've seen that pattern before. Coooooool." However, if a pattern is repeated too often, the brain starts to filter it out; it becomes boring. It says, "Nothing new here. Pay attention to something else." Pattern violations catch your attention. You brain says, "Hey, wait a second. I didn't expect that! You should pay attention because something new is happening!" Hence, composers must always straddle the line between calming pattern repetitions and exciting pattern violations. How far you go in either direction dictates how sophisticated a musical palate will be required to enjoy the song, and thus, who will enjoy it.

Ok, so that's my theory of neuroscience and music. I don't have any MRIs to back it up, nor do I have the educational background to claim real expertise on the subject. Nonetheless, I think there's some truth to it.

Gaming: How Do Characters Know What to Say?

2011-04-19T11:34:00.000-07:00

My wife and I like to play games like Paper Mario together. Paper Mario is a long game with a lot of dialog. At any point in the game, you can talk to any character, and that character will say something "sensible". For instance, they'll ask you to help them out, or they'll thank you if you've already helped them out. I've always wondered how that's coded. Similarly, I've always wondered how many ways I could think up to code it.

The simplest approach is to use a complicated set of possibly nested if/else statements. For instance, if Mario has this item, then say this. Otherwise, if he has beaten this level, say that. Certainly that's a valid approach, and it doesn't even matter if it's slow. Since people read so slowly, trying to optimize how quickly you can come up with what text to show next is absolutely the last thing you would ever need to optimize in a video game.

At the opposite extreme, this problem could be solved with a rules engines. There are libraries that let you wrangle control of complex if/else hierarchies. (The first time I heard about rules engines was when I was interviewing at a consulting company where they were building a welfare system for various counties in California. Apparently, the rules surrounding welfare systems are so complex that a rules engine is a necessity.)

You can also use a mix of if/else statements and switch statements. For instance, the main game is a mostly linear progression from one state to the next, which is easily solved by a switch statement. If you are in this state, say this. If you're in that state, say that. However, sometimes Mario goes on side quests. In cases like that, a particular character might need to have a separate switch statement to know what to say for that side quest.

Similar to using a switch statement, you can use a non-sparse array per character where each state is an index into that array, and each spot in that array has a pointer to a message. Since the array contains every state, many of the states will point to the same message.

You can create a mapping from state to message using a sparse tree. To find out what to say, traverse the tree to find the state that is closest to, but still less than the current state. A sparse tree has a benefit over a non-sparse array in that the tree is only as large as the number of messages the character might say.

Rather than having an array of states for each character, you could also have a single array of states for the entire game. When you save the player's position in the game, you simply save his index into the array. The first spot in the array has a mapping of what every character would say at the beginning of the game. Each new spot in the array has a list of updates for all the characters who would say something different for that new state. This works like a database transaction log. Since there are only a few hundred or a few thousand characters, you can easily store in memory what each character would say at the current state in the game.

I'm going to take a guess and say that they probably just used a simple set of nested if/else statements, but it's kind of fun to think of other approaches.

Ruby: My Take on Pivotal Labs, Part II

2011-04-18T19:34:00.000-07:00

As I mentioned in my previous post, I have tremendous respect for how Pivotal Labs builds software. In this blog post, I want to cover why practices used at Pivotal Labs may not always be appropriate at other companies. The core of my argument is that Pivotal Labs is a consultancy; hence, their priorities are not always the same as the priorities for a startup building its own software.

First of all, let me talk about full-time pair programming. In the book Professional Software Development, Steve McConnell states that NASA discovered that the single most effective way to reduce defects (in manufacturing, etc.) is to always have a second pair of eyes present (i.e. to work in pairs). However, NASA must go to extreme lengths to avoid defects because lives are on the line. That's rarely the case with most startups. Most defects are merely embarrassing. In many cases, code review may be more efficient than full-time pair programming. In some cases involving purely aesthetic matters, it may sometimes be acceptable to even skip the full code reviews and settle for aesthetic reviews instead.

Full-time pair programming is also a great way to train newbies. It's definitely in Pivotal Lab's interest to have all of its employees extremely well trained and interchangeable. However, it's really the customer that's paying for this training, and it's certainly expensive. In a startup building its own code, this expense may not be warranted. Certainly, there's huge value in having overlap between employees so that no one employee who gets hit by a bus can take out the entire company. However, some specialization may be appropriate. When I worked at IronPort, we had a low-level, FreeBSD kernel guy. He spent a lot of time tweaking the kernel in a way that only a FreeBSD kernel guy can. We also had people who specialized in JavaScript. Expecting every person to be capable of pair programming on FreeBSD kernel development as well as ajax hacking is simply asking for too much, and it's too expensive in terms of programmer time. Sometimes a little specialization can be helpful.

Full-time pair programming may also help a particular task get done faster. However, if you have a limited number of engineers at a small startup, certainly the total time to completion for all programming tasks will go down if all programming must be done in pairs. Pivotal Labs has the advantage that they can scale up the number of people on a project if a job calls for it, and they make more money when they do so. It's the opposite at a small startup. The number of programmers at a small startup is often somewhat limited and fixed, and there are economic advantages to getting more done with fewer people. If you only have two programmers, and one of them is a Ruby expert and the other an ActionScript expert, it might be best to just let them do their thing individually so that at least two things are getting done at the same time.

Don't get me wrong. I'm not saying that pair programming isn't extremely valuable. I'm just saying that full-time pair programming is extremely expensive, and sometimes the cost isn't justifiable for smaller startups that aren't consultancies.

The next thing I'd like to cover is tests. Pivotal Labs writes a lot of tests. It makes sense for them because they get paid to write them, and they never want to get caught with their pants down. If Pivotal Labs has a choice of implementing two features with a 30% chance of having a defect or of implementing one feature with a 2% chance of having a defect, they're only going to implement one feature. Since they don't get paid by the feature, they get paid the same either way.

The economics just aren't the same at a startup building its own software. Don't get me wrong--I really appreciate the value of testing, but I also try to keep in mind the cost. My approach to building software at a small startup is to consider the return on investment when writing tests. Some tests provide a very high return on investment. They catch a lot of potential bugs and require very little work. I'm thinking of high-level Cucumber and Webrat tests. Some tests require a lot of work, but don't catch very many actual bugs. I'm thinking of view tests. A view test neither tests that a view looks right, nor does it test that its interface with the controller is correct. My point is that it may be in Pivotal Lab's best interest to implement view tests, but if I'm working at a small startup that isn't a consultancy, it probably isn't in my interest to implement view tests.

So what's my point in bringing all of this up? As much as I appreciate how Pivotal Labs does things, I understand that those practices may not always be appropriate for a small startup building its own software. It reminds me of that old essay, The Rise of "Worse is Better". As much as I hate to admit it, sometimes worse is better.

Ruby: My Take on Pivotal Labs, Part I

2011-04-18T16:22:00.001-07:00

Pivotal Labs is one of my favorite companies. I have a tremendous amount of respect for how they develop software. They put the "extreme" in extreme programming, and more importantly, they get stuff done. However, there are some things that Pivotal Labs tend to do that I disagree with.

I have such high regard for Pivotal Labs that I specifically try to find startups that started at Pivotal Labs when I'm looking for a job. Since these companies often make you do a three hour pair programming session on their code, I've seen the code of multiple companies that started at Pivotal Labs. Hence, although I don't know if Pivotal Labs has an official opinion on these topics, I've seen these things at enough companies that I feel it's worth commenting on.

First of all, many of the companies that I interviewed at didn't use database-level foreign key constraints and database constraints in general. I know that it's trendy in the Rails world to try to enforce your constraints at the application level. (I'll admit that in some cases involving sharding, you can't use foreign key constraints. However, very few Rails applications are built with sharding.) Unfortunately, the application is simply incapable of truly enforcing certain constraints such as uniqueness constraints and foreign key constraints. Only the database can apply these constraints because they interact with ACID and transactions.

Secondly, as far as I can tell, several of the companies I interviewed at are susceptible to mass assignment vulnerabilities because they don't properly lock things down with attr_accessible such that things are not accessible by default. Perhaps this problem is going away in Rails 3, I don't know. However, many of the companies I talked to didn't want me to explain to them why their code was vulnerable.

Thirdly, a lot of projects at Pivotal Labs tend to use view tests. I think that unit testing views is a total waste of time. Apparently, so does Sarah Mei at Pivotal Labs.

Rather than unit testing the model, view, and controller separately, I prefer to rely on Cucumber and Webrat more heavily. I still write some model tests using RSpec, but I never duplicate things that are already tested by Cucumber and Webrat. I explained my approach and my reasoning in another blog post. Unfortunately, not everyone agrees with me, and this is a hot topic right now.

Last of all, I've seen several Pivotal Labs code bases that use Rails 2.3.X, but haven't switched to using the rails_xss plugin. The rails_xss plugin can really help avoid XSS vulnerabilities. It allows you to remove most of the h() calls in your templates since it escapes things by default. I think this behavior is the norm in Rails 3, so this complaint will soon go away for new code bases.

There's an old saying that if two people agree on everything, than only one of them is doing all the thinking. Hence, it should come as no surprise that I disagree with a few things that I've seen in various Pivotal Labs projects. Hopefully no one at Pivotal Labs will think worse of me for pointing those things out.

Math: Sierpinski's Triangle is a Variation of Pascal's Triangle

2011-04-18T15:58:00.000-07:00

Here's a picture of Pascal's triangle from Wikipedia. It's animated just in case you don't remember how Pascal's triangle is created:

Here's a picture of Sierpinski triangle, also from Wikipedia:

You can see in the top image that to calculate a spot in Pascal's triangle, you just add the two above spots. To get Sierpinski's triangle instead of Pascal's triangle, you just xor the above two spots. Cute trick, eh?

I learned this trick while reading Concepts, Techniques, and Models of Computer Programming, which is a fantastic book, by the way. Here's my Oz code for printing out Pascal's triangle:

% This is a generic version of Pascal's triangle that let's you specify the
% operation instead of just using "+".

declare GenericPascal OpList ShiftLeft ShiftRight
fun {GenericPascal Op N}
   if N==1 then [1]
   else L in
      L={GenericPascal Op N-1}
      {OpList Op {ShiftLeft L} {ShiftRight L}}
   end
end

fun {OpList Op L1 L2}
   case L1 of H1|T1 then
      case L2 of H2|T2 then
  {Op H1 H2}|{OpList Op T1 T2}
      end
   else nil end
end

fun {ShiftLeft L}
   case L of H|T then
      H|{ShiftLeft T}
   else [0] end
end

fun {ShiftRight L} 0|L end

declare
fun {FastPascal N} {GenericPascal Number.'+' N} end

for I in 1..10 do {Browse {GenericPascal Number.'+' I}} end

JavaScript: Jasmine

2011-04-18T15:39:00.000-07:00

I went to a talk the other day on Jasmine:

Jasmine is a behavior-driven development framework for testing your JavaScript code. It does not depend on any other JavaScript frameworks. It does not require a DOM. And it has a clean, obvious syntax so that you can easily write tests.

describe("Jasmine", function() {
  it("makes testing JavaScript awesome!", function() {
    expect(yourCode).toBeLotsBetter();
  });
});

Jasmine started life at Pivotal Labs and is like RSpec for JavaScript. Thanks to the flexibility of JavaScript, it has a powerful mocking / stubbing framework. Jasmine is not a replacement for Selenium. It's good for testing JavaScript functions that calculate things, but it doesn't try to make testing the DOM any easier. Just as unit tests written using RSpec are not a replacement for integration tests written using Cucumber and Webrat, similarly unit tests written using Jasmine are not a replacement for integration tests written using Selenium. Nonetheless, it looks useful.

Ruby: nil is a Billion Dollar Mistake

2011-04-18T13:22:00.000-07:00

The fact is, I really like Ruby. However, there are some ways in which it uses nil that I really disagree with. For instance, in Ruby, if you try to look up something in a hash that doesn't exist, you get a nil. Similarly, if you try to reference an @attribute that hasn't been set yet, you'll get a nil.

That reminds me of this article, Null References: The Billion Dollar Mistake:

Tony Hoare introduced Null references in ALGOL W back in 1965 “simply because it was so easy to implement”, says Mr. Hoare. He talks about that decision considering it “my billion-dollar mistake”.

Compounding this problem is Ruby's current lack of real keyword arguments (although, I know they're coming). Hence, if you pass a keyword argument like f(:foo => 1), and then try to use the keyword argument in the function like options[:foooo], the misspelling will result in a nil as if the argument hadn't been passed. This masks a real problem. All of these have resulted in real bugs in my code and lots of frustration.

It doesn't have to be this way. In Python, if you try to look up something in a dict that doesn't exist, you get a KeyError exception. If you try to use an attribute that doesn't exist, you get an AttributeError exception. If you misspell a keyword argument, you get a TypeError exception. Exceptions are nice because they catch the problem right away instead of allowing it to fester. In Ruby, if I get a nil that I'm not expecting, I might not find out that there's a problem until much, much later in a different part of the code when I try to call a method on the nil thinking it's a real object.

Since I'm on the subject of hashes, I think I should mention that Haskell and Scala take a different approach. If you look up something in Scala that may not exist, it returns an Option instance. An Option instance may or may not contain a real value. (Similarly, Haskell uses the Maybe monad.) You have to do work to get the value out of the Option instance, and the type system will catch you if you just blindly assume that there's always something in the Option instance. This is one case where the ML type system can help you avoid a whole class of bugs.

Python: Adding New Methods to an Instance

2011-04-18T13:08:00.000-07:00

It's easy to dynamically add new methods to a class in Python, but all my attempts to add new methods directly to an instance had always involved hacks. When I discovered that Ruby had syntax to add methods directly to an instance, my curiosity in the subject was rekindled. Fortunately, someone pointed me to this blog post that shows how.

Why should you care? Most of the time, you shouldn't. You might want to use this trick if you were trying to do something like JavaScript's prototypal-based inheritance system. In my case, I think I was trying to do some funky monkey patching where I only wanted to monkey patch a particular instance instead of the class as a whole. Anyway, even if I can't come up with a good use case, I'm still glad I know how to do it ;)

Linux: Fish Hanging

2011-04-18T12:56:00.000-07:00

fish is a "user friendly command line shell for UNIX-like operating systems such as Linux." I've been using fish for about a year, and I really like it. Unfortunately, until recently, I had a problem that I couldn't log into virtual consoles in Linux. If I hit Cntl-Alt-F1 and tried to log in, fish would just hang. This was mentioned on the mailing list a long time ago. I'm pleased to say that I finally solved the problem.

I had the following code in my ~/.config/fish/config.fish:

type fortune > /dev/null
and begin
 fortune
 echo
end

Basically, this code says, "If fortune exists, run it. Otherwise, don't complain." I figured out through a process of elimination that was the culprit. I replaced it with:

if status --is-interactive
 fortune
 echo
end

This code will result in an error if fortune doesn't exist, but it won't die. It turns out to be fine for me since I only run fish on my laptop, and I usually install fortune at the same time I install fish.

JavaScript: Perfectly Encapsulated May Mean Perfectly Untestable

2011-04-18T12:22:00.000-07:00

It's no secret that JavaScript is like Lisp in that you can accomplish amazing things using a huge number of small, nested functions. In fact, you can write things like:

(function () {
    function pickNose() {
    }

    function fart() {
    }

    pickNose();
    fart();
})();

In this code, an anonymous function is defined and immediately called. pickNose() and fart() are two internal functions that are used by the outer function, but they are not available to the outside world.

It's amazing what you can get done using nested closures like this, but there's a cost. How do you write tests for pickNose() and fart()? Certainly, you can write a test for the outer function as a whole, but there's no way to test those inner functions in a standalone way without doing some refactoring. In a certain sense, the code is like a script in that you can test the thing as a whole, but you can't test the parts in a standalone way.

What's the solution? I'm sure there are many. You could have the outer function take a parameter such that when the correct value is passed, the code could flip over to testing mode and test itself. Another approach is to use Douglas Crockford's module pattern. The outer function could return an object that has references to the inner functions. That way you can call them externally and test them. However, that may not be an option if you are really paranoid about the outside world getting references to those functions.

In my opinion, getting overly paranoid about people calling your inner functions isn't very fruitful. JavaScript doesn't have much to help you keep modules away from each other. As soon as you have hostile JavaScript on the page, it's sort of game over. All of the JavaScript operates with the same permissions--it's not like you can sandbox a particular module.

Furthermore, it's difficult if not impossible to prevent people from just grabbing your JavaScript source code and hacking it to do whatever they want (especially if they can rely on the help of an external server). To state the obvious, JavaScript doesn't have very good internal security boundaries, so your server must always be distrustful of the JavaScript that it's talking to. Of course, that's the way the web has worked for as long as I've been coding.

Python: Using Louie with Twisted

2011-04-18T11:53:00.000-07:00

Louie "provides Python programmers with a straightforward way to dispatch signals between objects in a wide variety of contexts. It is based on PyDispatcher, which in turn was based on a highly-rated recipe in the Python Cookbook." Louie is like an event system used in a GUI toolkit. Similarly, you can think of Louie as an internal pubsub system. Twisted is a framework for building asynchronous network servers. Using Louie in your Twisted applications can make your applications a little less "twisted".

When you code a Twisted application, you often put a lot of your logic in custom Factory and Protocol classes. However, if you have one application that has to talk to, say, three different types of servers, and you have a custom Protocol and Factory class for each server (perhaps each server speaks a different network protocol), it can be confusing to have your application logic broken up all over the place. Louie can help with that.

When you are implementing a Protocol class, when you receive a new Twisted event, you can generate a Louie event. In that way, you can have a single application-specific class that responds to events from a wide variety of Twisted Protocol classes. When I used this approach on my Twisted application, the code went from being very "scatter brained" to something more linear. I had a bunch of small functions that each responded to Louie events, and all the functions were in order in the same class. It certainly helped me wrangle control of the complexity of the system.

It may seem strange to translate Twisted events into Louie events, but considering the fact that Louie has support specifically for Twisted and considering the fact that I learned this trick from Drew Pertulla, a Twisted user, I know I'm not the only one using this trick.

I have a couple more tips. First of all, Louie tries really hard to pass only the arguments that your function is looking for. Hence, if your function doesn't accept a sender argument, it won't pass a sender argument. This is really helpful, but it can bite you. The magic tends to break down if your function is wrapped by a decorator or if your function is a closure (i.e. a nested function). In these cases, Louie can't figure out exactly which arguments to pass, and stuff can break. However, it's usually easy to work around situations like this once you figure out what's going on.

I have one other mildly off-topic trick for working with Twisted. Queues and Twisted work really well together. If you have one piece of Twisted code that simply reads from a queue and then does something, then other parts of the Twisted code can put things in the queue in a synchronous manner. It sounds simple, but this one trick really helped me out a few different times.

Last of all, I wanted to mention gevent. Twisted is certainly a mature library with support for a range of protocols, but if you're working with green field code, you might want to take a peek at gevent instead. Aside from having excellent performance, it also lets you write code in a synchronous manner, which is helpful for mere mortals like myself. If you're thinking about using Twisted vs. gevent, check out this talk that the Meebo guys gave at PyCon.

JavaScript: JS.IO is Solid, Hookbox Might be the Bee's Knees

2011-04-18T11:18:00.000-07:00

Recently, I needed to add realtime (i.e. comet, websockets, flash sockets, etc.) support to an application. JS.IO is a JavaScript library that provides a long polling system using the Comet Session Protocol (CPS). It doesn't try to do what Socket.IO does, i.e. abstract all the various transport mechanisms such as websockets and flash sockets. Rather, it provides long polling and leaves the switch to websockets or flash sockets to the user. There are server-side libraries to integrate with JS.IO using Twisted, Eventlet, Erlang, etc.

My experiences with JS.IO were very positive. Although the documentation was sorely lacking, Michael Carter (the author) was extremely helpful when I was getting started. Furthermore, my testing with crossbrowsertesting.com proved that JS.IO was very reliable across an incredible range of browsers. It was the most reliable comet library I tried. By the way, I was doing this cross-domain.

These days, Michael Carter has started a new project called Hookbox:

Hookbox is a Comet server and message queue that tightly integrates with your existing web application via web hooks and a REST interface.

I blogged about Michael's talk at PyCon. Although Hookbox doesn't perfectly solve all my problems (it doesn't yet having clustering support), I'm extremely hopeful that this will turn out to be a great project!

JavaScript: Socket.IO Didn't Meet Needs

2011-04-18T10:56:00.000-07:00

Recently, I needed to add realtime (i.e. comet, websockets, flash sockets, etc.) support to an application. Socket.IO is a library built on top of Node.JS that "aims to make realtime apps possible in every browser and mobile device, blurring the differences between the different transport mechanisms." Since Socket.IO did exactly what I needed, I was hoping it would solve my problems easily and that I wouldn't have to implement what Socket.IO did myself.

Unfortunately, things didn't work out so well. I had to do things in a cross-domain manner. Although the browser support list for Socket.IO is very good, that didn't match up with my actual experience. I built a simple application that tried to send and receive a message using Socket.IO, and then it reported on which transport was used. Unfortunately, many of the browsers that I wanted to support such as IE 6 and 7 and Opera just didn't work, even though they were supposed to. Here are some of my results.

Furthermore, if you watch the mailing list, a lot of questions just get dropped on the floor. I'm sure this is because the authors of Socket.IO are completely overwhelmed. Hopefully as Socket.IO matures and the list of expert users grows, this will improve.

By the way, you might wonder how I tested so many browsers. I paid for an account on crossbrowsertesting.com. It was worth every penny! First, I would use it to try to take a snapshot of my test page on all the different browsers. Then, I would use the web-based VNC system to log into the systems and view the page manually in order to see what was going wrong. This was very helpful to determine exactly which browsers did and didn't work.

Anyway, as I said, I hope Socket.IO gets better because it solves a real need. Perhaps it already has, since I was doing this testing several months ago. However, if you need to use Socket.IO, I heartily recommend you make use of crossbrowsertesting.com to make sure that the browsers you need to support actually work.

In my next post, I'll cover JS.IO.

ZeroMQ is Amazing!

2011-04-18T10:41:00.000-07:00

Recently, I had to improve the performance of mesh networking in a mesh of, say, 10 nodes. The original code used a simple RPC system built using JSON on top of netstring. Every message to every node involved a new connection. On a cluster of 10 nodes, I was getting 30 messages per second.

I enhanced the code by using persistent connections. I also switched from RPC (i.e. using a whole roundtrip that blocks the whole connection) to message passing (i.e. passing a message doesn't necessarily result in a response and doesn't tie up the socket). This improved the performance to 300 messages per second.

Next, my buddy encouraged me to try out ZeroMQ. Man was I amazed! I hit something like 1800 messages per second on a cluster of 10 nodes! I can only imagine what ZeroMQ was doing in order to hit this number. Perhaps it was batching messages more intelligently (from my experience, that's an amazingly effective technique).

I ran the same test on a range of cluster sizes, from 2 to 10. The performance graph had a sawtooth shape. The shape was consistent between runs. I'm inferring that ZeroMQ doesn't try to have one node send the message to all the other nodes. Rather, the message filters through the cluster in a tree-like manner.

Anyway, I'm sorry I can't share the graphs or the code, but let me just say that I was very impressed with ZeroMQ!

Linux: ^\

2011-04-18T10:36:00.001-07:00

Am I the only one who didn't know that you could use ^\ (i.e. control backslash) to kill a process when ^c doesn't work? Usually, I have to use ^z to background the process, and then type kill -9 %1. I think ^\ makes the process dump core, but since dumping core seems to be turned off by default, it works out well. Here's an example of my killing a process under fish (my shell):

fish: Job 1, “nosetests” terminated by signal SIGQUIT (Quit request from job control with core dump (^\))

Thanks to Jeff Lindsay for the tip.

Videos: IBM Centennial Film: 100 X 100 - A century of achievements that have changed the world

2011-04-16T15:34:00.000-07:00

IBM Centennial Film: 100 X 100 - A century of achievements that have changed the world

The film features one hundred people, who each present the IBM achievement recorded in the year they were born. The film chronology flows from the oldest person to the youngest, offering a whirlwind history of the company and culminating with its prospects for the future.

I found the film to be very moving.

Python: I'm Looking for a Python Instructor

2011-04-13T15:34:00.000-07:00

I'm looking for a talented, friendly Python programmer to do corporate training. I'm helping out a company called Marakana. They do corporate training and have multiple gigs lined up for Python. Unfortunately, their normal Python instructor is getting a little bit busy right now, and so am I. If you're interested in giving a four day training session on Python, please send email to me at jjinux at gmail dot com. My buddy Robert Zuber has already prepared the course materials, and the pay is pretty decent. They're also looking for people to teach Ruby on Rails, HTML5, JavaScript, and Android, but I figured most of my readers are Python programmers.

I'm sure this is obvious, but you must be comfortable with public speaking. You get bonus points if you've given a talk at PyCon or a local users group. You get double bonus points if you're well known in the Python community, especially if I know you ;)

Update: The response was pretty overwhelming, so I think I can stop looking. Thanks, guys.

PyCon: Closing Lightning Talks

2011-04-06T13:11:00.000-07:00

PyCon will be held in Montreal in 2014 and 2015.

Twiggy is a new Pythonic logger. It has a totally new design (i.e. it's not like log4j). It's the first really new logging design in 15 years. It uses lots of chaining method calls like jQuery. It makes parsing logs easier. It has a modern config system. It has better traceback printing. It has an asynchronous logger.

Askbot is a Stack Overflow clone in Python.

The Python Miro Community has Python videos. They're rolling out universal subtitle support.

Minuteman is a tool to replace your build.sh. It acts as a workspace and project manager. It is like zc.buildout. It's also like Maven and Gentoo. It's a "metabuild system." It has no docs, no tests, and no users.

Hold Old is My Kid? is a website that helps you figure out your kid's age in days, months, and years.

flufl.enum is an enum library written by Barry Warsaw.

MOE_write is a library for dealing with Python2 vs. Python3. It's from Google. It allows you to maintain 2 or 3 branches (?). It runs 2to3 and 3to2 to patch back and forth between branches. It's a scary hack.

Microsoft stopped funding IronPython. It's still going forward in the open source community. There's a project called IronPython Tools for Visual Studio. It has intellisense that can do crazy, static type analysis. It can figure out the type of a variable even through function calls.

A guy wrote Adventure in Python3. He ported it from the original Fortran code. He even did a weird thing where you can play it within the Python shell. What's weirder is that he wanted people to be able to do things like write "north" in the Python shell, without needing to add the parenthesis to make it a function call. Hence, he overwrote the __repr__ method so that calling "north" would automatically call the function. Cute hack.

Python has a ton of mock testing libraries. Almost everyone should make use of mocks, at least to mock out services only available over the Internet or to test your exceptions handling. There's a new library called "Fudge" that was inspired by "Mocha".

Side note: my buddy Kyle wrote a mocking library called ditto at IronPort/Cisco which has various useful features.

PyCon: Hidden Treasures in the Standard Library

2011-04-06T12:59:00.000-07:00

Hidden Treasures in the Standard Library

The talk was by Doug Hellmann. He writes Python Module of the Week. By the way, that is the most popular Python blog according to Google Reader, at least the last time I checked. Doug is publishing the series of blog posts as a book. Oh, and he's a nice guy :)

Side note: I met a guy who worked at a company that provided mapping software. The software was used by Google maps. He worked on the routing algorithms. He said the whole system consisted of 200 million lines of code. (I'm not sure how that's possible.) However, he also said that they distribute an SDK that contains .NET, the JRE, and Python within it.

Use the csv module with your own "dialects" to handle data that has fields with characters between each field.

SQLite3 has been added to the standard library. You can create custom column types in Python.

You can create signed, serialized data using the hmac library.

You can serialize data using the built in JSON support. Look at json.dumps(default=...).

You can override sys.excepthook with your own function.

You can create complex logging handlers. For instance, you can send non-verbose messages to the console, or you can send verbose messages to a log file. You can even combine these two things. For instance, you can configure it so that it logs tracebacks, only to file, and only on error.

PyCon: Greasing the Wheels of Exploration with Python

2011-04-05T17:13:00.000-07:00

Greasing the Wheels of Exploration with Python

Here's the talk summary:

The control of the Mars Exploration Rovers (MER) requires a complex set of coordinated activites by a team. Early in the MER mission the author automated in Python much of the task of one of the operation positions, the Payload Uplink Lead, for 7 of the 9 cameras on each rover. This talk describes the MER rovers, the operation tasks and that implemented system.

They used gigapan images.

They use virtual reality to visualize what's going on.

Dust was a serious problem for the rovers.

There's lots of Python ~~on the rovers~~ used to control the rovers.

The speaker's background is in machine learning and robotics.

The rovers have been running for 6-7 years. They find 1-2 bugs a year. Bugs are usually fixed in a matter of hours.

They uses Ames Vision Workbench, Nebula, and OpenStack. All three of these are open source.

The speaker was from Ames Research Center, NASA.

Side note: unfortunately, I missed the first five minutes.

PyCon: Fun with Python's Newer Tools

2011-04-05T17:03:00.000-07:00

Fun with Python's Newer Tools

collections.namedtuple works just like normal tuples, but it lets you assign a name to each field. It's fast, and there is no additional per-tuple space cost.

There's a lot of cool stuff in the collections package.

Named tuples can be used for "instance prototypes" by using "other_tuple._replace(field=5)". Don't be discouraged by the "_"--they had to do that so as to avoid a namespace conflict.

You can subclass a named tuple. It's based on __slots__.

In Python 3.2, there is a functools.lru_cache. It's a decorator. It accepts a maxsize argument.

Side note: I missed the first five minutes of the talk.

PyCon: An Open success for the cloud: OpenStack

2011-04-05T16:59:00.001-07:00

Side note: I missed part of the talk because I had to leave early to go to mass.

OpenStack turns a pile of hardware into a cloud.

Swift is their object storage system.

Nova is their system for provisioning virtual machines.

Glance is their system for image storage.

Burrow is their distributed message system.

Dash is a Django user interface for managing virtual machines.

NASA uses OpenStack.

Rackspace is going to use it. They were acquired by Rackspace.

PyCon: Disqus: Serving 400 million people with Python

2011-04-05T16:38:00.000-07:00

Disqus is a commenting system for blogs.

It's the largest Django application.

They get 500 million visitors a month.

It's used by CNN, IGN, MTV, etc.

Every engineer at the company is also a product manager.

They have a flat company structure.

They're experiencing exponential traffic growth.

They have 100 servers.

They rent hardware.

Their Python code is CPU bound.

They're using Apache + mod_wsgi.

Their background tasks are IO bound. They use Celery + gevent for background tasks. This saves memory over using separate processes for each background job.

They use Graphite for monitoring. They use Etsy's statsd proxy for Graphite.

"Measure anything and everything."

They had to fork and monkey patch Django in order to scale.

They deploy to production 3-7 times a day.

They use Hudson for continuous integration. They have a large test suite that takes 30 minutes to run. They love Hudson, but they said it's too Java oriented.

They do not suffer from not invented here syndrome (NIH).

They have rolling deploy with fast rollbacks.

They use the unittest module and nose.

They use coverage.py and Pyflakes.

They love pep8.py and Pyflakes.

They learned Django and then learned Python.

Python package management is a mess.

They're a good Python shop, and they have a good engineering culture.

They have open sourced a bunch of stuff.

PyCon: Going Full Python - Threadless

2011-04-05T16:26:00.000-07:00

The keynote was by a guy from Threadless.

Threadless sells clever, artistic T-shirts.

The company has lots of culture. It's very hip.

The designs are community driven. The community votes on them.

They don't actually print the shirts themselves.

Threadless is in Chicago. Rahm Emanuel visited. After he visited, he Twittered, "Seriously, this city used to build things. Now we're just assholes with novelty t-shirts. I'm with motherfucking stupid."

They switched from PHP to Django. They're still fighting a lot of technical debt.

Threadless is an "art community".

The speaker was hilarious.

PyCon: Lightning Talks

2011-04-05T16:08:00.000-07:00

Read the Docs is a site that hosts documentation. The speaker said "shit" a lot.

Chef or Puppet--choose one.

Haystack is a project for doing full-text search.

"Emacs pinky" is a real problem.

DjangoZoom enables turnkey deployment for Django.

PyCon did a donation drive for Japan. People could donate by texting to 90999. It wasn't actually very successful, but the Python Software Foundation pitched in to help out.

JavaScript is like English--it's a real mess, but it works.

In ECMAScript 5 (ES5), you can start your script with '"use strict";'.

Firefox (because of the ES Harmony project) has added all sorts of weird additions to JavaScript. Many of them were taken from Python.

flufl.i18n is a high level API for i18n. It's higher level than gettext. It makes it easy to handle multiple languages at the same time. It was written by Barry Warsaw at Canonical.

PyWO is the "Python Window Organizer". It works on top of your existing window manager. It allows you to move and resize windows in useful ways. It support tiling. However, it's less controlling / automatic than a normal tiling window manager.

Grace Law (a Python recruiter) said that you should be learning and hacking in your spare time. It helps with interviews to talk about something you're excited about in your coding life. Speed is also important. You should be quick.

Side note: I saw lots of Ubuntu users and lots of Macs. Windows users were more unusual this year.

PyCon: Supercomputer and Cluster Application Performance Analysis using Python

2011-04-05T16:00:00.000-07:00

Supercomputer and Cluster Application Performance Analysis using Python

The speaker was from Sandia labs.

The goal of his project was measure and track the performance of super computers.

His software used Python, matplotlib, MySQL, etc. It was a Tkinter application.

The interface was definitely created by an engineer ;) It was somewhat overwhelming.

There was some UI in the application to manage database schemas.

It's weird to see Tk applications when you're so accustomed to "dot-comish" web apps.

It seems painful to me to have to recreate an admin to manage database schemas when good ones already exist.

I think the speaker was running Windows 2000.

There was some UI in the application for creating queries to query the database.

The application could show graphs plotted by matplotlib.

Kiviat charts are hard to code, but they're very useful for plotting multivariable data.

His projects were called Pylot, Co-Pylot, and eCo-Pylot. He plans on open sourcing them, but that type of thing takes a while at a big lab.

His project, Mantevo, is already open source.

Since the goal of the project is to measure and track the performance of super computers, I asked if they had come up with any interesting results yet. They haven't yet because they're just getting started.

PyCon: HTTP in Python: which library for what task?

2011-04-05T15:50:00.001-07:00

HTTP in Python: which library for what task?

The talk was by a Mercurial guy. He works on Google Code.

When you think of HTTP, what you're probably thinking of is RFC 2616, i.e. HTTP/1.0. HTTP/1.1 has features that you're probably not remembering to handle correctly. This resulted in a bug that took him a very long time to diagnose.

HTTP/1.1 allows pipelining, which means you send all the requests right away and then wait for the responses to come in serialized over the same socket.

Using chunked encoding lets you keep the connection open between requests even if you don't know how much data will be streamed. Did you know that you can specify additional headers between chunks?

Did you know the "100 Continue" response gets sent before you finish sending the body?

httplib (used by urllib2) is very minimal. It doesn't even do SSL certificate validation! It doesn't support keepalive. It doesn't have unit tests.

httplib2 is just a wrapper around httplib that adds some stuff.

There is PycURL. It's based on libcurl, which is the gold standard for HTTP libraries. However, it's not very Pythonic and it has a steep learning curve.

twisted.web.http only supports HTTP/1.0 [or perhaps it doesn't support very much of HTTP/1.1].

The author is working on a new library. It uses select and is thus non-blocking. It has "100 Continue" support. It has lots of unit tests. However, it doesn't support pipelining.

Using httplib (via urllib2) is okay if your needs are simple.

PycURL is awesome if you can tolerate the steep learning curve.

You can get the author's new library from http://code.google.com/p/py-nonblocking-http/.

PyCon: Advanced Network Architectures With ZeroMQ

2011-04-05T12:57:00.000-07:00

Advanced Network Architectures With ZeroMQ

This was a talk on ZeroMQ by Zed Shaw. He and his talk were a lot more sedate than I expected. He's also older than I thought he was.

He uses Emacs, screen, and Ubuntu.

ZeroMQ servers should not be exposed to the naked internet. If you send it invalid protocol data, assertions will kill the process.

The talk was very fast, but most of it was like the ZeroMQ guide.

PyCon: Exhibition of Atrocity

2011-04-05T12:52:00.000-07:00

Exhibition of Atrocity

Here's the summary:

Believe it or not, but you can write pretty horrendously awful code even in a language as elegant as Python. Over the years, I've committed my share of sins; now it's time to come clean. Step right up for a tour of twisted, evil, and downright wrong code, and learn some strategies to avoid writing criminally bad code--if you dare!

This was a fun talk full of fun examples.

The speaker showed code he had worked on over the course of 11 years.

They used Hungarian notation at one point!

snake-guice is a simple, lightweight Python dependency injection framework based on google-guice.

PyCon: Using Coroutines to Create Efficient, High-Concurrency Web Applications

2011-04-05T12:43:00.000-07:00

Using Coroutines to Create Efficient, High-Concurrency Web Applications

Here's the summary:

Creating high-concurrency python web applications is inherently difficult for a variety of reasons. In this talk, I'll discuss the various iterations of application server paradigms we've used at meebo, the advantages/disadvantages of each approach, and why we've settled on a coroutine-based WSGI setup to handle our high-concurrency web applications going forward.

They started with CGI. Then they switched to mod_wsgi. Then they switched to Twisted. Finally, they switched to gevent and Gunicorn. They said that this was the best of both worlds.

Guido said, "I hate callback based programming."

The speaker showed a great chart which showed the strengths and weaknesses of the various approaches.

Gunicorn is a lightweight WSGI server. It has worker processes. It supports gevent.

mod_wsgi is fast. However, if you use Gunicorn with multiple processes, it'll beat mod_wsgi.

Use Greins to manage Gunicorn.

gevent-profiler is useful.

I Code Sooooo Slowly!

2011-04-05T11:56:00.000-07:00

One thing I've learned over and over is that a programmer's skill with his preferred editor is no indication of his skill as a programmer. One of the best programmers I know stuck with Pico for years! Certainly many of the programmers mentioned in "Coders at Work" use Emacs without even trying to learn it well.

If you've been reading my blog for a while, you know that I'm obsessed with productivity, especially when it comes to editors. There's a reason why. I'm slow...really slow!

When I was at PyCon, I participated in a coding challenge. The grand prize was a MacBook Air, and there were only 8 participants. I figured I stood a good chance at winning the MacBook Air which I needed since I was leaving Twilio.

The challenge used SingPath:

SingPath is the most FUN way to practice software languages! SingPath provides a platform to those that want to test their programming skills in a competitive and fun environment.

(By the way, did you notice that it's spelled Singpath in the title and SingPath in the introductory paragraph?)

Anyway, the challenge consisted of 10 questions that you have to write Python code to solve. I finished 6th out of 8! I was the slowest programmer among the professional programmers. It gets worse ;) The guy who won the competition was a Russian guy (I think). He finished all 10 problems in 10 minutes. I only finished 8 problems after 30 minutes. I would have finished the others, but I ran out of time.

It was definitely humbling to realize just how slow I am. What's ironic, though, is that I'm really fast at manipulating text in an editor. (Sometimes I use Vim and sometimes I use NetBeans with the jVi plugin so that it feels like Vim.) I can make text fly, but that doesn't translate into coding faster.

Fortunately, I have other assets as a programmer. For instance, my code is well known for being extremely clean, and I rarely have very many bugs. Nonetheless, it's frustrating that I'm so slow, and I really wish there was something I could do to improve my speed.

Having spent so much time futzing with my editor, I think it's clear that that isn't the problem. I think the problem is that my mind is constantly in "proof mode". I double check everything, code review my own code multiple times, think about all the edge cases, etc. Trying to code fast just doesn't work for me. When I did the programming competition, I was coding as fast as I possibly could. If I had taken my time like normal, I probably would have taken 2-3 times as long (but the code would have had really nice documentation and tests too).

Does anyone have any realistic advice for how I can increase my speed without sacrificing quality?

Python: Python IDEs Panel

2011-04-05T11:47:00.000-07:00

Python IDEs Panel

Side note: There were surprisingly few people at this talk. It seems like most Python programmers get started with either Vim or Emacs and then don't change. It's ironic that I'm so obsessed with programmer productivity given that I'm such a slow coder ;)

The panel consisted of representatives who worked on Python Tools for Visual Studio, PyCharm, Komodo IDE, Wing IDE, and a Python mode for Emacs (pythonmode.el). There was no one present to champion PyDev or NetBeans.

Michael Foord prefers Wing IDE.

Python Tools for Visual Studio has debugging support for high performance computing (HPC). It supports MPI. It can debug a program that uses multiple processes. It supports both IronPython and cPython. You can use iPython within Visual Studio to control a cluster of machines. You can write Python code to analyze the variables in the individual frames of a stack.

PyCharm makes test driven development (TDD) fast! The speaker was using PyCharm to test drive the development of a class which he was creating quickly in the test file. PyCharm can automatically create the scaffolding for a class as you use the class in your test. It can create the scaffolding for methods, add imports, add constructors, etc. all automatically as you try to use those things in your tests. It has helpful coding suggestions and refactoring support. It has code snippets. It's crazy how much it can guess what to create automatically. It has great Django support. It has Django-specific code completion. The demo for PyCharm was flat out amazing!

Komodo Edit is free and open source. However, Komodo IDE is commercial. It adds support for debugging, etc. It opens very quickly. It's good at working with multiple languages at the same time. It has an HTML preview feature that can show you an HTML page within Komodo (presumably because it's written in XUL). It has a regular expression debugger. There are about 80 extensions. In 2011, InfoWorld rated Komodo IDE as the best Python IDE.

Wing IDE is commercial. It supports multiple keyboard personalities. It has a debug shell. The debugger and intellisense work well together.

The champion for Emacs was Barry Warsaw. He showed how to integrate Pyflakes. The demo wasn't particularly inspiring.

Most of the IDEs have been around for about 10 years.

When I brought up the fact that so many successful programmers use Emacs, the general consensus was that the people who succeed using Emacs have been doing the same sort of thing for a really long time. They have it entirely in their head. They don't really need an IDE, and an IDE isn't really helpful for their situation.

PyCon: The Data Structures of Python

2011-04-05T11:33:00.000-07:00

The Data Structures of Python

Use types idiomatically.

Sometimes you don't have a choice.

Be efficient when it doesn't cost you anything.

Think about set vs. frozenset, mutable vs. immutable.

There is now a collections.OrderedDict class in Python 2.7 and 3.1.

Sometimes you need your data structure to address more than one concern. Use combinations of things from the standard library.

collections.deque is a linked list. It's pop(0) and insert(0, item) operations are O(1), whereas those operations are slower with normal lists. In Python 2.7, there's a maxlen parameter for the deque class (which I assume turns it into a circular queue).

You can use the array type to efficiently represent an array of ints, etc.

heapq is also interesting.

collections.abc contains abstract base classes.

"Don't subclass dict ever!" He said this is true of other containers as well. He said there are too many edge cases. You should instead subclass things like collections.Mapping instead.

There's an ordered set class on the Python Cookbook site. (Presumably, it combines a set with a list.)

Don't do more than necessary. ABCs (abstract base classes) can help.

You can use a frozenset as a dict key, and you can use frozensets as members in other sets.

Tuples are more efficient than lists.

Side note: htraf.htsql.org is an insanely good WSGI / ReST interface for databases.

PyCon: Ten Years of Twisted

2011-04-05T11:20:00.000-07:00

Ten Years of Twisted

The talk was by Glyph Lefkowitz, the original author of Twisted.

He mentioned Medusa and Sam Rushing.

Twisted started because Glyph wanted to add email and HTTP handling to his video game, Twisted Reality. He started by using the "select" function call.

Twisted is good because it unifies the protocols.

Glyph is only the 3rd most prolific Twisted committer.

When you write tests for Twisted code, you can have both the client and server in the same process.

Twisted is event-driven and asynchronous.

Before switching to Python, Glyph wrote Twisted Reality in Java using threads.

Twisted is powerful and flexible.

Twisted is switching from the term "framework" to "engine".

"Reactor included."

Use "twistd --help" to see which servers are available for free with Twisted.

Glyph said, "I hope we have another 10 years of Twisted."

Side note: Gevent was really popular this year. During his keynote (which I missed because I overslept, unfortunately), Guido said that he rejected the "everything is a callback approach". However, he also said he didn't like approaches using Greenlets because you couldn't be exactly sure where the context switches were happening. I know he also doesn't like threads. He seems to prefer explicit yields for asynchronous IO. As far as I can tell, many Python programmers aren't following his suggestions because Gevent seemed really, really popular this year. It's certainly my favorite approach for asynchronous network servers.

PyCon: Lightning Talks

2011-04-04T18:00:00.000-07:00

Qtile is a tiling window manager written in Python. It's an alternative to Awesome.

In Tunesia, they were using flame throwers against protesters.

In a "do-ocracy", you don't govern, you just do.

PyParsing and SPARK are good parsing libraries.

Resolver is a Python-powered spreadsheet that works on Windows.

Dirigible is the same thing as Resolver, but it runs as a web application.

PyCon: Hookbox: All Python web-frameworks, now real-time. Batteries Included

2011-04-04T17:52:00.000-07:00

Hookbox: All Python web-frameworks, now real-time. Batteries Included

Hookbox is a new server written by Michael Carter (of JS.IO and Orbited fame) for doing comet. It is a simple comet server that uses web hooks to talk to your normal web server. Your normal web server can take care of all the business logic, and Hookbox can take care of the comet.

Side note: SF.net uses MongoDB and has ugly denormalization problems.

Hookbox replaces what you would have had to do with RabbitMQ + Orbited.

Michael called Hookbox a "web-enabled message queue".

It's based on Eventlet.

It uses the comet session protocol (CSP).

It takes care of pub/sub, history, presence, moderation, etc.

The documentation is not very complete.

It doesn't support clustering.

Hookbox is really good at delegating business logic to your application.

PyCon: Porting to Python 3

2011-04-04T17:39:00.000-07:00

Porting to Python 3

py3ksupport.appspot.com has a list of the top 50 Python projects and which of them support Python3. As of the talk, 34% of the top 50 Python projects supported Python3. I just checked, and it's up to 54%.

There are multiple strategies to porting to Python3:

Only support Python3.
Use separate trees for Python2 and Python3.
Include both versions in a single download and set package_dir in setup.py
Implement "continuous conversion" using 2to3. This approach is recommended for libraries. Distribute can help.
Use a single codebase with no conversion. This requires loads of compatibility hacks. It's fun, but it's ugly. Check out the "six" project if that's what you want to do.

Try 2to3 first. If in doubt, use distribute.

Libraries should port as soon as possible.

In order to prepare, use Python 2.7 with the -3 flag. Fix all the deprecation warnings.

Use separate variables for string vs. binary data.

Add "b" and "t" to file flags.

For sort, switch from "cmp" to "key".

Reminder: __foo__ is often pronounced "dunder foo" (aka "double under foo").

Increase your test coverage. This significantly helps!

Use "2to3 -w ." to port an entire directory.

In setup.py for distribute, use use_2to3=True.

urllib2.urlparse is not right. Use the urlparse module directly.

Bytes vs. unicode is the hard part.

Using == to compare bytes and str doesn't work. b"a"[0] does not equal "a"[0]. Hence, the comparison will return False in a way that is unexpected and silent.

Trying to support Python 2.5 and Python3 at the same time is REALLY hard. It's much easier to support Python 2.7 and Python 3.0 at the same time.

Seeking in a UTF-8 text file is slow because it involves decoding, not simple indexing.

See http://docs.python.org/py3k/howto/pyporting.html.

There's a book called "Porting to Python 3".

PyCon: Status of Unicode in Python 3

2011-04-02T14:36:00.000-07:00

Status of Unicode in Python 3

The talk was by Victor Stinner. I went to dinner with him and a few other people. He was a nice, French guy.

The encoding for source code defaults to UTF-8 in Python 3.

Surrogate escapes are a new feature in Python 3.2. They let you deal with stuff that can't be decoded as UTF-8. For instance, you can decode a filename string to a unicode object without losing data even if the decoding isn't clean.

There are still issues to work on.

Victor had bootstrap issues implementing all this stuff.

It took a lot of hard work to improve all this stuff.

Check out Programming with Unicode, which is a book that Victor wrote.

Victor has event more Unicode fixes in store for Python 3.3.

Side note: I had an idea. It'd be cool to create a tool that shows you a call tree for your application. In the call tree, it can show you where all the encodes and decodes are done. This would help you know where to do encodes and decodes. This would really help when porting from Python2 to Python3. Figuring out where to do encodes and decodes is a lot more subtle than you might think. It doesn't always make sense to do it at the very edges of your application.

PyCon: Opening the Flask

2011-04-02T14:23:00.000-07:00

Opening the Flask

The talk was by Armin Ronacher, one of the authors of Flask.

Flask started as an elaborate April Fool's joke.

The guys who did Flask did a lot of cool stuff before doing Flask, such as Jinja2.

Flask was originally Jinja2, Werkzeug, and some glue code cleverly embedded in a single file download.

Marketing beats quality.

Originally, they didn't do any testing or any code review. That's changed.

They wanted to restart with good docs and good tests. They documentation and tests are pretty good these days.

The name "Flask" is a play on another framework, "Bottle".

Flask is still based on Werkzeug and Jinja2.

Flask can use Blinker as a signalling system (?).

Flask is about 800 lines of code, 1500 lines of tests, and 200 A4-sized pages of documentation.

There are lots of extensions.

Flask uses decorator-based routes, but there are other options.

Flask supports URL routing as well as URL generation.

Jinja2 uses template inheritance.

They use context locals to make writing apps simper. Armin suggests that you might as well embrace context locals--you'll end up needing them anyway.

Armin is very against import-time side effects.

Flask is in favor of explicit application setup. For Flask, this has occasionally led to circular imports that they had to find workarounds for.

WSGI middleware works with Flask.

Flask has several advantages over Bottle.

You can override various aspects of how Flask works. For instance, you can use the "@app.before_request" and "@app.after_request" decorators.

They make use of lots of extensions.

Flask has a goal of keeping the core very small.

You can use other templating engines if you want, but Jinja2 will always be a dependency just in case any extensions need it.

Documentation matters.

Communication matters. It's very easy to submit issues to Flask. You don't even need to create an account.

It's important to keep committing to projects otherwise the project looks dead.

Consistency matters.

It's important that even the documentation be beautiful.

Flask uses Sphinx for documentation.

Armin didn't have a strong rule of thumb for when you should use Flask vs. Pyramid. Neither the Flask guys nor the Pyramid guys are overly religious about their frameworks.

The audience turnout was about the same for both the Flask and Pyramid talks.

The Flask guys are thinking about redoing the code for reverse routing.

PyCon: State of Pylons/TurboGears 2/repoze.bfg

2011-04-02T13:36:00.000-07:00

State of Pylons/TurboGears 2/repoze.bfg

There are about 2000 people on the Pylons mailing list.

Ben Bangert said that Pylons relies too heavily on subclassing. When people subclass and override stuff in a framework's parent class, it makes it difficult to alter the parent class without breaking people's code.

Side note: The new Pyramid T-shirt is beautifully done, but evil aliens (or anything else for that matter) are a big turn off for me.

Pylons is big, but Ben said that too much of Pylons is dependent on him.

Pylons is merging with repoze.bfg. It's going to be called "The Pylons Project", but it's based on the code in repoze.bfg.

Chris McDonough wrote repoze.bfg. It's a great, but relatively unknown framework. Pylons has better name recognition.

TurboGears 2 is built on Pylons.

The new framework is called Pyramid and is part of "The Pylons Project".

TurboGears 2 and Pylons are going to be maintained together so as not to strand a bunch of legacy projects.

Side note: There were a lot of big beards at this PyCon. Big beards have always been popular among hackers, but it's even more popular right now. I wonder if this has anything to do with the Giants pitching staff and with the Giants winning the World Series.

TurboGears will experiment and innovate on top of Pyramid in the same way it used to experiment and innovate on top of Pylons.

repoze.bfg has a catchphrase: "Plumbing Zope 2 into the WSGI pipeline." It actually doesn't take any code from Zope, but it does borrow some ideas.

There are about 200 people on the repoze.bfg mailing list. It has 100% statement coverage. 100% of the code is documented--if a feature isn't documented, then it doesn't exist. There are 80 committers to the repoze.bfg project!

Pyramid is just repoze.bfg renamed.

There is a Paster template for Pyramid--multiple actually.

These are the features in repoze.bfg: it maps URLs to code; it has an authentication framework; it supports I18N; it supports single file applications as well as larger projects; it makes use of PasteDeploy; it has unit, functional, and integration testing; it uses WSGI; it has great docs; it supports multiple templating engines; it supports Google App Engine; it has plugins; it can serve static files; it has sessions; it has cross site request forgery (CSRF) protection; it supports events; it has good exceptions handling; it supports WSGI middleware; it's extremely fast; it has 100% statement coverage.

One reason that it is so fast is that a lot of work is done at startup so that less work needs to be done at runtime.

It is not a full-stack framework. It has no persistence layer.

It currently has 16 dependencies, but single file applications are still possible.

It is "unopinionated".

Chris (the repoze.bfg guy) said, "I'm a Django fan. I love Django." Obviously, he doesn't think Django is appropriate for all situations.

Pyramid != Zope. It's not Pylons. It's not MVC. Chris hates that term.

Flask may be more appropriate for small projects. Pyramid may be a better fit for larger applications.

Pyramid exposes plug points all over the place.

Pyramid supports row-level security.

There's a Pyramid OpenID library.

They use Lettuce together with a web driver. It can work without Selenium.

WebTest from Ian Bicking looks interesting.

The next version of Pyramid will target Python3.

Paste is in a state of flux, but it's too important to be abandoned.

PyCon: Python - The Secret Sauce in the Open Cloud

2011-04-01T11:35:00.000-07:00

Python - The Secret Sauce in the Open Cloud

This talk was given by Jason Huggins, the creator of Selenium.

The talk was about OpenStack which is open source, cloud computing software.

It used to be that OpenStack was very difficult to get started with. Things are getting much simpler.

Cloud computing is still on top of a huge hype wave.

OpenStack allows you to run your own cloud.

Side note: there were a lot more women this year.

You can use whichever hypervisor you want: KVM, QEMU, Xen, VirtualBox, VMware, etc.

OpenStack glues together a bunch of third-party parts. It has an API. It uses a database. It has a messaging protocol. It has a lot of glue code.

He said that his most important slide was a link to http://ansolabs.com/deploy.

Install VirtualBox.

Install Vagrant. It's an API on top of VirtualBox.

Get chef recipes from the OpenStack cookbooks.

Using OpenStack has become substantially easier during the last four months.

What out for disk I/O. Use copy-on-write disk images.

Make the VM as small as possible.

When asked about Apple VMs, he said, "Apple is like Oracle: Oracle doesn't scale financially."

PyCon: Distributed Tasks with Celery

2011-04-01T11:23:00.000-07:00

Distributed Tasks with Celery

Celery is an asynchronous job queue based on distributed message passing.

ShootQ looks like a useful site for professional photographers.

Celery takes care of scheduling. It also handles errors with retries.

It works with Web hooks too, although normally you can stick to just plain Python.

It can control nodes via remote control by broadcasting messages.

It can run on top of Reddis, etc. as well.

Use CELERY_ALWAYS_EAGER when unit testing.

RabbitMQ does NOT like running out of memory! Really bad things happen.

RabbitMQ doesn't really have replication.

Side note: Convore is really popular. I had never even heard of it, and it seemed like about half the people present were using it. It's a web-based chat system. It's a mix of Twitter and IRC. It's very threaded.

Celery doesn't yet have priority queues.

PyCon: Keynote

2011-04-01T11:11:00.000-07:00

Steve Holden: Python Software Foundation Chairman

The attendance in 2009 was really down; the PSF lost a lot of money. 2010 was better. 2011 has the biggest attendance ever. There were 1400+ registrations.

Dr. Hilary Mason from Bit.ly

List comprehensions are the most popular feature in Python according to her data.

The Kinect For Xbox 360 from Microsoft is awesome.

She recommended the book "Collective Intelligence".

Side note: Tavis Rudd has a totally awesome speech recognition setup. He has created his own spoken language / grammar with words customized to doing everything, especially controlling Emacs. Listening to him speak, it sounds like a foreign language, but he's able to control his machine very quickly. He had to develop this because he was suffering from RSI very badly. He has a Mac with two virtual machines. One runs Windows with Dragon Speak. The other runs Linux. He logs into Linux from Windows. He has high-level words that allow him to operate very quickly.

PyCon: Cooking with Python 3

2011-04-01T10:45:00.000-07:00

Cooking with Python 3

Side note: I met a guy from Lockheed Martin who was working on the Joint Strike Fighter using C++. He said that "malloc" and "new" were not allowed; everything had to be statically allocated.

Python3 now uses "except ValueError as e:" instead of "except ValueError, e:".

There are a bunch of really subtle changes in Python3, such as the fact that some_exception[0] no longer works.

Python3 chains exception messages if an exception occurs while another exception is being handled.

If you write "[x * x for x in nums]; print x", x won't be in scope. This is a bit surprising.

The exception object does not escape the except block. It won't be defined outside the block. This is really surprising in situations like:

e = True
try:
    ...
except Exception as e:
    ....
print e

It'll say that e isn't defined when you try to print it. It was flagged as a bug in the bug tracker, but then closed as invalid.

Exec has been locked down a lot in Python3. For instance, this no longer works:

def foo():
    exec("a = 23")
    print(a)
foo()

execfile is gone.

reload has moved to the imp module.

There is now a new set literal notation, "fruit = {'apple', 'banana'}", although you can't use it to create an empty set for obvious reasons.

There are now set comprehensions: "upper_fruits = {f.upper() for f in fruits}".

There are now dict comprehensions: "d = {name.upper(): value for name, value in stock.items()}".

"some_dict.keys()" gives you a "view" on the dict, not a list.

"zip" now produces an iterator, not a list. This sort of thing is very common in Python3.

You can now write: "a, *middle_items, b = [1, 2, 3, 4]".

The "*" in the following forces all the subsequent arguments to be keyword arguments (meaning you must specify the keyword when passing them): "def foo(x, *, y)".

Function annotations are new. They look like "def add(x: int, y: int) -> int:...". By default, they don't actually do anything. However, libraries can create whatever meaning they want for them. The thing after the colon can be any Python expression. This adds an __annotations__ member to the function object.

You could easily build a function decorator that uses the annotations to do optional static type checking.

If multiple libraries use annotations for different things, you could end up with conflicting uses of function annotations.

Use StringIO for strings and BytesIO for bytes.

This is a byte object: b"byte".

argparse is a new argument parser descended from optparse.

The division operator no longer truncates. It gives you a float. If you want integer division, use "1 // 3".

Use "2to3 -w ." to port an entire directory to Python3.

"unicode" and b"bytes" are completely separate.

2to3 is not a magic bullet. It's actually pretty stupid.

The bytes vs. unicode thing is by far the hardest part of porting.

We tried porting redis-py to Python3. I now think that the unicode vs. bytes problem is going to making porting extremely difficult, and it's going to lead to a lot of bugs.

The following is False 'b"H"[0] == "H"'. That's because it's actually equal to 72. This is going to lead to a lot of subtle bugs during porting.

There is a unicodedata module to look up unicode character metadata.

Books: Coders at Work

2011-03-31T16:43:00.001-07:00

I just finished reading Coders at Work:

Based on nearly eighty hours of conversations with fifteen all-time great programmers and computer scientists, the Q&A interviews in Coders at Work provide a multifaceted view into how great programmers learn to program, how they practice their craft, and what they think about the future of programming.

In short, I really enjoyed this book. It was relaxing, stimulating, humbling, and enabling all at the same time.

One thing that really stood out was that programmers used to be able to understand systems extremely deeply because they could fit the whole thing in their heads. In contrast, it simply boggles my mind to think of how many lines of code are involved in showing Google Maps to me (think of my laptop, all the routers, Google's servers, the JavaScript, the server-side code, etc.). If you're up for some light technical reading, this book is highly recommended.

The Zen of Testing

2011-03-25T11:44:00.000-07:00

Tests that use a style you disagree with are better than no tests at all.

Tests that catch more errors are better than tests that catch fewer errors.

Getting more done with less effort is better than getting less done with more effort.

Tests that are readable are better than tests that are unreadable.

Tests that are easy to update are better than tests that are hard to update.

Tests that fail when appropriate are better than tests that fail when inappropriate.

Tests that test the stuff you care about are better than tests that test stuff you don't care about.

Tests that take little effort and catch a lot are better than tests that take a lot of effort and catch little.

Python: Behavioral Driven Development for Telephone Applications Using Lettuce, Flask, and Twilio

2011-03-23T23:07:00.000-07:00

I'm going to be giving a talk tomorrow night at BayPiggies called "Behavioral Driven Development for Telephone Applications Using Lettuce, Flask, and Twilio". Here's the description:

Behavioral driven development is a style of programming popular in the Ruby world using tools such as Cucumber and Webrat. In this talk, I'll show how the same tricks can be used in Python too, using a library called Lettuce. I'll also show off Flask which is a new micro web framework. Last of all, I'll cover Twilio which is an API that makes it easy to build telephone-based applications using web technologies.

I'm going to be showing code samples from my demo application PyTeladventure. If you can't make it to the talk, you might way to have a peek at the code.

Hardware: Fixed a Dell Inspiron 1545

2011-03-21T11:46:00.000-07:00

My buddy Adam Ulvi loaned me a laptop since I didn't have one. (My last day at Twilio was last Thursday.) He had a Dell Inspiron 1545 lying around, and he was nice enough to mail it to me. The only catch was that there was some breakage around the display near the hinge. Apparently, this is extremely common, so I decided to try to fix it.

I bought a replacement part from Parts-People.com. The part I bought was the display cover, and it came with replacement hinges. It cost about $50. I went to Dell's site to find the service manual. It was actually a fairly involved repair, but I managed to get it done. My only regret was that I didn't buy a replacement part for the piece of plastic that frames the front side of the screen. I didn't notice that it was broken too, so I had to make due with some carefully applied electrical tape. Fortunate, it doesn't look as ghetto as it sounds.

Anyway, the repair went really well. The laptop looks pretty much brand new, with the exception of one small piece of electrical tape that blends in pretty well anyway. The new lid made a big difference. I've learned two things from this experience: 1) Dell's build quality sucks (which is something that my buddies have been telling me for a while now). 2) Dell's repair manuals are incredibly good. Following the instructions was a piece of cake, even it did take me two hours.

Linux: lubuntu

2011-03-15T15:34:00.000-07:00

I decided to give lubuntu a try:

lubuntu is a faster, more lightweight and energy saving variant of Ubuntu using LXDE, the Lightweight X11 Desktop Environment.

If you already have Ubuntu installed, trying lubuntu is really easy; just run "sudo apt-get install lubuntu-desktop".

In summary, it's very pretty, super fast, and crazy small. In fact, its memory usage was almost laughable considering I was running it on a 4GB MacBook Pro. I think my total memory usage was something in the 200-300MB range.

The downside is that there are a lot of things that I've grown accustomed to in Ubuntu that I can't live without. Ubuntu has a GUI to swap the capslock key with the control key, and it has a GUI to tweak my touchpad and power management settings. There are probably ways to configure these things by hand under lubuntu, but I've grown mildly impatient in my old age ;)

The biggest challenge for me was that lubuntu doesn't know about encrypted home directories. I have an encrypted home directory, and Ubuntu knows that it has to run ecryptfs-mount-private when I log in; in fact, it doesn't even need to ask me for my password again since it does it as part of the login process. When I ran lubuntu, I had to log in, run ecryptfs-mount-private (typing in my password again), log out, and then log back in.

I have a couple more tips. To access the OpenBox menus, middle click on the desktop. If you decide to install lubuntu, don't tell it to switch to lxdm. Otherwise, if you remove lubuntu, you'll end up with a borked gdm setup. To fix it, you'll need to remove and reinstall gdm.

In summary, I really like lubuntu. It makes my machine feel lightning fast, faster than any other computer I've ever owned--at least until I fire up NetBeans ;) I'm not sure if they'll fix the things I mentioned, but if those things don't affect you, I think lubuntu is worth a try.

Linux: Awesome is Awesome!

2011-03-13T20:18:00.000-07:00

I decided to give Awesome a try. Awesome is a tiling window manager like xmonad. I've been using it for about a month, and I like it a lot.

It integrates with GNOME much better than dwm. Most users of dwm don't use GNOME, but I do. Its default settings are a lot nicer than xmonad's. When I use xmonad, I spend all my time futzing with my .xmonad.hs, but I haven't had to tweak Awesome at all the whole time I've used it. For instance, Awesome comes with a ton of layouts built in.

Getting gvim to work perfectly under xmonad required some tweaking, and I couldn't figure out how to get it to work perfectly under dwm. I know that this is gvim's fault because it doesn't follow the exact size prescribed by the window manager, so the bottom part of the screen gets messed up. I'm sure most dwm users don't use gvim--they probably run vim in a terminal. It worked perfectly under Awesome.

Getting NetBeans to work well took even more work, and even now it sometimes freezes in predictable (and avoidable) ways. I had to "sudo apt-get install suckless-tools", and then I added "xterm -e wmname LG3D" to my startup applications (i.e. my GNOME session). Finally, I switched to JDK 1.7.0 and edited /usr/local/netbeans-6.9.1/etc/netbeans.conf so that "netbeans_jdkhome="/usr/local/jdk-1.7.0".

Since Awesome and GNOME each have their own panels, I did some tweaking to make them get along better. I have Awesome's panel on the top, and one GNOME panel on the bottom. I got rid of things from the GNOME panel that were already provided by the Awesome panel. I'm pretty pleased with the result, and I didn't even have to futz with any configuration files to do it.

I really like the keybindings in Awesome. Since I had already played with xmonad and dwm, I didn't really have to re-learn the keybindings for Awesome since they're so similar. I just had to realize that all the keys were based on the command key (i.e. Mod4) instead of the control key.

I especially like having a different set of tags (i.e. virtual desktops) per monitor. That's exactly the paradigm I like the most. I use a big monitor for most things, and I use a lot of different tags. I use my laptop screen just for instant messaging applications, and I only use a single tag.

Plugging in or unplugging my monitor is a pain, especially since I have to tweak the Nvidia settings every time. I know it's a bit "ghetto", but my solution was to restart X every time I want to plug in or unplug my monitor. I think this actually takes less time than shuffling my windows around and futzing with my Nvidia settings.

I have found one Awesome bug. When I first login, my cursor is a clock, and it doesn't go away. If I play with the menus, it does. It's not a big deal, so I don't care.

The default layout is floating, and each tag has its own layout. This default works well for me. I only have to press Command-space to switch to tiling.

One of these days, I might give Qtile a try since it's written in Python and my buddy really likes it. However, in the meantime, I'm pretty happy with Awesome.

Personal: Should I Leave LinkedIn?

2011-03-13T20:13:00.001-07:00

I got email from LinkedIn:

Give your productivity a boost - for free!
Exclusive offer: Try LinkedIn Premium free for 1 month*

Dear Shannon -jj,

Find and manage high quality contacts with LinkedIn Premium. Get started today and get 1 month free!.

LinkedIn Premium helps you be great at what you do, with tools such as:

• InMail: Contact anyone on LinkedIn without an introduction - response guaranteed!**
• Expanded Profile Views: See expanded profiles of everyone on LinkedIn, even people outside of your network
• Who's Viewed My Profile: Get the inside scoop on how you're viewed professionally by accessing the complete list

You'll also get a lot more including up to 700 profiles per search.

Your exclusive offer expires soon so get started today!

This means that LinkedIn is going to be making money off of spammy recruiters at my expense. That is really a violation of my trust. Am I wrong? I'm seriously thinking about deleting my account.

Humor: Donald Knuth Jokes

2011-03-03T08:09:00.000-08:00

At the age of two, Donald Knuth implemented Conway's Game of Life on Babbage's difference engine which was pretty impressive since Conway hadn't been born yet...neither had Babbage.

The next version of TeX will be self hosting...and will run on bare metal.

Skynet is actually an out of control TeX macro.

Knuth can store all integers between 0 and 127 in 4 bits.

Knuth invented binary.

Chuck Norris is actually an android that Donald Knuth created. He was funded by DARPA.

Knuth once wrote, "Email is a wonderful thing for people whose role in life is to be on top of things. But not for me; my role is to be on the bottom of things." He succeeded.

Knuth started to dabble in botany in his backyard. He gave it up, though, when all his trees ended up with squareroots.

Knuth is the last highlander.

Knuth created the Antikythera device when he was a young lad vacationing in Greece.

When Knuth wants to speed up an algorithm, he doesn't code in assembly--he codes in binary.

Donald Knuth enjoys coding in Python, but he prefers to write the .pyc files directly.

One time Knuth decided to crosscompile his algorithms from binary to quaternary. In the process, he invented DNA.

Knuth's brain is the most powerful source of energy known to man. Unfortunately, harnessing it would destroy mankind.

Knuth recently created a new sorting algorithm. Its performance is O(1).

Knuth invented literate programming so that TeX could be compiled by less capable compilers. Knuth's own compiler compiles the English.

If you can think of more, please leave them in the comments below :)

Personal: Looking for a New Job

2011-02-28T21:30:00.000-08:00

I'm looking for a new job. I really like my current company, but it's not a very good fit for me for various reasons. Anyway, here's my resume. Here's what I'm looking for:

I'd like to find something in San Francisco or the East Bay
I'd like to go somewhere where the code is clean and high quality
As much as I enjoy startups, I'd really like to find something calm and relaxing
I prefer to code in Python or Ruby, but I also really enjoy weird programming languages like Scala, Haskell, and Erlang

Bless: A New Twist on Concurrent Programming

2011-02-25T22:51:00.000-08:00

(Disclaimer: I'm just a guy who likes learning new programming languages. I'm not an academic language expert.)

I've been reading Coders at Work lately and really enjoying it. Fran Allen is a woman who won the Turing Award for her work on optimizing compilers. She said something interesting. She said that we've been in a rut since C came along because the pointers in C prevent the compiler from doing a lot of interesting optimizations. She said that programming languages before C were actually more optimizable. She also said something about using offsets rather than pointers so that the compiler could easily move memory around.

It reminded me of Dreaming in Code. Alan Kay said that when it comes to software, we're still building pyramids because we haven't yet discovered the programming version of the arch. Pyramids are heavy, whereas arches can support an amazing amount of weight while not having as much weight themselves.

I remember years ago that Alan Kay's statements made me think of actors in Erlang. Fran Allen's statements also made me think of actors in that an actor could be pointed to by a pointer, whereas all the state in that actor could be represented by offsets of that pointer. This is how you can isolate actors from each other within the same UNIX process. (Of course, you have to make sure the language doesn't provide the equivalent of pointer arithmetic with indexes. That's easy. References are indexes, and you can't set them arbitrarily.)

One thing is clear in "Coders at Work". Everyone is trying to figure out better ways to do concurrency. Currently, the three competing models are threads, actors, and STM (software transactional memory). One lesson that I learned when I read Why Science Fiction So Often Fails to Predict the Future is that life isn't homogeneous. That post pointed out that every form of human transportation is still in use somewhere in the world. Did I walk to work, drive, or take the train? The answer is a little bit of all three. Similarly, I've read that the CISC vs. RISC debate has fizzled because these days chips are CISC on the outside and RISC on the inside. Hence, I've been thinking of ways to use threads, actors, and STM together.

Bless is a new "twist" on concurrent programming. I haven't written it yet--I'm just thinking out loud to get feedback from fellow language enthusiasts. Bless is a little bit like Erlang at a high-level and a little like Python at the mid-level. I haven't decided on the low-level parts yet.

First, let's start with Erlang-style processes. I like to think of them as "green processes" because they form their own "address space", but they're implemented by the VM. However, to avoid confusion, I'm going to call them actors. An actor in Bless is an entity that is implemented by the VM. It has its own VM-level stack and its own VM-level heap. The VM-level stack must not use the C stack. At the C level, an actor can be referenced by a pointer (i.e. a location in memory), but all data within the actor is referenced by offsets to that pointer. Hence, the actor can move around in memory without disturbing the contents.

Each actor has its own green process. Actors can be scheduled onto any number of operating system threads. Erlang is like this.

Each actor can receive asynchronous messages using a queue. I think there are queue implementations that don't require locks; however, I was also thinking of implementing the queue using STM.

All data is local to an actor. The code for an actor looks sort of like a Python module. It has lexically scoped functions that have local variables. Variables global to the actor are kind of like instance variables in object oriented languages. If a module contains the code for an actor, you can run that same code on multiple green processes. For instance, you might use a new actor for every web server connection. Perhaps it makes sense to use the term "module" for the source code for an actor and "actor" for when that module is run on a green process (remember, I say "process" not in the UNIX sense, but rather because each actor has its own data).

Data is not immutable like In Erlang. However, messages between actors are passed by deeping copying the data. I'm thinking of the messages as being JSON-like messages. This is something my buddy Warren DeLano was a real fan of. Immutable data made a lot of sense when Erlang was created because you can implement message passing via passing a pointer to immutable data. However, mutable data is more familiar to most programmers, and if you have 100+ CPUs, you can afford to copy some data around. In a perfect implementation, messages would be contiguous blocks of data, and they could be memcpy'd between green processes. One thing I learned from the Sam Rushing is that you'd be amazed at how fast memcpy is, especially if you have 100+ CPUs to play with.

I know that a lot of functional people are going to be upset with my use of mutable data. That's okay. Most of these ideas still apply if you want to go purely functional. That's cool too.

Actors can block on IO, like in Erlang. (I don't buy Node.JS's argument that everything should be a callback. A little brilliance at the VM level can make everyone else's life easier. Think of how Java frees programmers from the pains of manual memory management.) However, like Erlang, you can use asynchronous IO under the covers.

Bless will need a scheduler in order to schedule actors onto the real kernel threads underneath. This sort of thing is done in Erlang and Scala, so there's nothing particularly mysterious about this. Note that if you have 100+ CPUs, the time slices can be longer which cuts down on the amount of work you need to do for scheduling.

I want Pythonic syntax. I want indentation-based syntax. I know some Python haters will reject Bless because of this, but they'd probably find other reasons to reject Bless anyway. Besides, Python isn't the only language to use indentation. Another one of my favorite languages, Haskell, does it too.

I want blocks. Blocks in Ruby are wonderful. I think the best way to represent a block in Python is "function_call(args) do: ...". The "do:" behaves the same way "if ...:" does. Blocks should follow the same scoping rules that if statements do. I'm not opposed to anonymous functions such as "def (args):".

I want keyword arguments. Python, Common Lisp, and Smalltalk all got this right.

I want variable declarations. I would prefer the keyword "var", but it should behave sort of like "my" in Perl. Variable declarations can help the system catch you if you misspell things, and they also help when you want to set a variable in an outer scope.

There is no "self" in method calls. It's just like functions in a module. However, you can of course "instantiate" the module (i.e. run the actor) multiple times in separate green processes. There's no need for "@" like in Ruby or "self." in Python because it's all lexically scoped.

If I decide to do static typing, I want it to look like ActionScript3, "var a: type". If I decide to do an ML-style type system, the ":type" can be optional.

Although I really like Python syntax, I think I'd be okay with a slightly-lower level language like Java. I.e., I'm not against using a compiler, and I am interested in resolving symbols at compile time. If you think about levels of dynamism, Python, Java, and Haskell all have valid, interesting approaches. I really like the middle ground that ActionScript3 provides. At this point, I care more about the approach to concurrency in Bless than which level of dynamism to use. Certainly, having both a compiler and an interpreter (like Common Lisp, Haskell, and Ocaml) is very appealing.

Compared to how crazy you can get in Python, I'd be okay with sacrificing a little bit of that dynamism for a little more safety, although I'm not a fan of babysitting the compiler like in Java, and I am still a fan of either duck typing or ML-style typing.

By the way, there are other forms of concurrency such as data parallelism, data-flow programming (aka promises and futures), etc. To some degree, I think they are orthogonal--i.e. you can do them in Bless as well, but that's not my focus. It's best to avoid scope creep and second system syndrome.

I think there are a few things I haven't straightened out in my head. I haven't thought about how to reuse functions between actors. Perhaps mixins make sense. I also haven't thought about inheritance.

Notice that if you do it right, you can safely run multiple programs in the same UNIX process using separate actors. However, to do this, you'll need to control how actors are able to obtain references to each other. This would be a nice feature, but it can easily be thrown out if it makes the implementation too complex--there's a reason we have UNIX processes. It may make sense to think of the actors within a UNIX process as all being "friendly", whereas separate UNIX processes are allowed to be somewhat less "friendly" with each other.

That's it. It's basically Erlang-style concurrency with Python syntax and actors with mutable "process-local" state (note, that I know that Erlang has mutable process-local state too). Easy peasy. Anyone know how to implement this thing? ;)

By the way, if I ever did manage to implement it, my next step would be to implement an operating system as I described here. Basically, start with a microkernel and then code all the things like the filesystem in userland using Bless.

Python: SSL Hell

2011-01-25T18:07:00.001-08:00

I was having a hard time getting SSL to work with gevent on Python 2.6. It turns out I had two problems.

The first resulted in this error message:

SSLError: [Errno 336265218] _ssl.c:337: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib

It turned out to be a permissions issue. I ran "cat" on the file, and it turned out that I didn't have access to it:

cat: /etc/mycompany/certs/httpd/mycompany-wildcard.key: Permission denied

I ran the command with sudo, and the problem went away.

The second error was related to using urllib2 under gevent:

URLError: <urlopen error [Errno 2] _ssl.c:490: The operation did not complete (read)>
<Greenlet at 0x2add8d0: start_publisher> failed with URLError
...
SSLError: [Errno 8] _ssl.c:490: EOF occurred in violation of protocol
<Greenlet at 0x2add958: <bound method WSGIServer.wrap_socket_and_handle of <WSGIServer at 0x2b48750 fileno=3 address=127.0.0.1:34848>>(<socket at 0x2b48a10 fileno=5 sock=127.0.0.1:34848, ('127.0.0.1', 37858))> failed with SSLError

This problem was because I was using gevent to monkeypatch the socket module, but I wasn't using it to monkeypatch the ssl module. Once I monkeypatched the ssl module, everything worked.

I had a heck of a time writing nosetests that would fire up a server using gevent and connect to it over SSL using urllib2. However, those nosetests proved very valuable in helping me figure out when and where SSL was breaking for me.

Here's what one of those nose tests looked like:

# Unfortunately, this monkey patching is not isolated to just this module.

from gevent import monkey
monkey.patch_all(thread=False)  # Nose uses threads.

import urllib2

import gevent

from myproj import server

TEST_INTERFACE = "127.0.0.1"
TEST_PORT = 34848
URL = "https://%s:%s" % (TEST_INTERFACE, TEST_PORT)


def test_server():

    test_successful_box = [False]

    def start_server():
        server.main(interface=TEST_INTERFACE, port=TEST_PORT)

    def start_publisher():
        response = urllib2.urlopen(URL)
        assert response.msg == "OK"
        test_successful_box[0] = True
        gevent.killall(greenlets)

    greenlets = [gevent.spawn(start_server), gevent.spawn(start_publisher)]
    gevent.joinall(greenlets)
    assert test_successful_box[0]

Python: use_twisted Decorator

2011-01-07T15:09:00.001-08:00

This is a decorator that you can put on a Python function that will temporarily fire up Twisted's reactor and run a function under Twisted. This is useful if most of your program doesn't use Twisted, but you have a function that must use Twisted.

Here's an example of using it:

@use_twisted
@defer.inlineCallbacks
def do_something_twisted():
    value = yield do_something_else_twisted()
    other_value = yield do_more_stuff()
    defer.returnValue(value + other_value)

Here's the decorator:

def use_twisted(twisted_function):

    """This is a decorator to run a function under Twisted.

    Temporarily fire up the reactor and run the function under Twisted.
    It should return a deferred, of course.

    Unfortunately, there's a bug in Twisted that only allows you to
    start and stop the reactor once.  See
    http://jjinux.blogspot.com/2011/01/python-usetwisted-decorator.html.
    Hence, this decorator will prevent you from calling it twice.  So
    sorry :(

    """

    from twisted.python.failure import Failure

    captured_value = []  # Think of this as a box.

    def wrapper(*args, **kargs):
        reactor.callLater(0, call_twisted_function, twisted_function, args,
                          kargs)
        reactor.run()
        assert captured_value
        value = captured_value[0]
        if isinstance(value, (Exception, Failure)):
            raise value
        else:
            return value

    def call_twisted_function(twisted_function, args, kargs):
        deferred = twisted_function(*args, **kargs)
        deferred.addBoth(capture_and_stop)

    def capture_and_stop(value):
        captured_value.append(value)
        reactor.callLater(0, reactor.stop)
        return value

    if globals().get("_reactor_run", False):
        raise AssertionError("The use_twisted decorator can only be used once")
    else:
        globals()["_reactor_run"] = True

    return wrapper

Updated: Added code to prevent the decorator from being called twice.

Linux: pssh

2011-01-06T17:54:00.000-08:00

Have you ever needed to run a bunch of shell commands over ssh on a bunch of servers? I know there are probably a ton of tools out there to do this, but when I asked my operations buddy Geoff which he preferred, he told me to try out pssh (aka parallel-ssh). I tried it, and I was pleased to discover it was easy to setup and easy to use.

It's a Python package. Make sure you have setuptools installed (on Ubuntu, use "sudo apt-get install python-setuptools"). Then run "sudo easy_install pssh". It creates the following binaries in /usr/local/bin: prsync, pssh, pnuke, pslurp, pscp, and pssh-askpass.

It's best if you install your ssh key on each system. I have a shell script called ssh-installkey to do that:

#!/bin/sh
#
# Install my ssh key on a remote system.

[ -n "$1" ] || {
 echo "usage: ssh-installkey username@host" >&2
 return 1
}
ssh $1 "mkdir -p -m 700 .ssh"
ssh $1 "cat >> ~/.ssh/authorized_keys2" < ~/.ssh/id_dsa.pub
ssh $1 "chmod 600 ~/.ssh/authorized_keys2"

Unfortunately, you'll have to run this script manually for each of the servers, which involves typing in your password a bunch of times. However, once you have your ssh key installed, your life will be much more pleasant.

To use pssh, you should create a hosts file with the hosts that you want to control. It's a simple file with one host per line. If you need to specify a username, you can use the format username@host, just like ssh.

Now, you can try out pssh, "pssh -h hosts.txt -i ls". The "-i" tells pssh to output the results from each server "inline" (which looks nice). If you don't care about the output of the command (for instance, if you're compiling something), you can just leave out the -i.

There are a couple of gotchas to be aware of. First of all, each command starts with a fresh login. That means using "cd" in one command doesn't help at all for the next command. I tend to use commands like "cd dir && do_something" when running pssh. Secondly, if your command takes a long time to run, pass "-t -1" to turn off timeouts.

Lastly, you'll need to do some more work if you want to use sudo. By default, sudo won't run if you don't have a tty, which you won't if you're using pssh. To fix this, you'll have to manually log into each server and edit /etc/sudoers. Comment out the line that says "Defaults requiretty". Once you do that, you'll be able to use sudo with pssh.

I was able to use pssh to control a cluster of 10 EC2 instances in order to install ZeroMQ. (In real life, I'd add ZeroMQ to the AMI so that it was already installed on each server, but using pssh helped me get something up quickly so that I could experiment.)

Linux: Unity

2011-01-06T17:48:00.000-08:00

I tried out Unity which is going to be the user interface for Ubuntu 11.04. It's built on top of GNOME, and it's currently used in Ubuntu's Netbook edition. Overall, I liked it. It's definitely one step closer to OS X. For instance, menus are shown at the top of the screen rather than at the top of each window. Unfortunately, I encountered several bugs when trying to use an external monitor, so clearly Canonical's programmers have their work cut out for them to hit Ubuntu's 11.04 release (due in April, of course).

Linux: CrunchBang Linux 10 on a MacBook Pro

2011-01-03T16:34:00.001-08:00

I tried CrunchBang Linux 10 on a MacBook Pro. Previously, I had a lot of trouble dual booting with OS X, so I did the same thing I did for Ubuntu--I told it to use the entire disk. This turned out to be a big mistake.

I put GRUB on the MBR since I wasn't dual booting. I also set up an encrypted LVM. The system wouldn't boot. I just got a flashing folder with a "?" icon. I think this is a known problem with Debian right now.

I also wanted to try MEPIS. It's based on Debian as well. It even has a utility that you run from within OS X that sets stuff up for dual booting. Unfortunately, since I wiped OS X, that wasn't available.

I was doing all this on my company's spare Macbook Pro (while my Macbook Pro was in the shop). Unfortunately, the DVD for my Macbook Pro wouldn't boot on the other Macbook Pro (which was just a few months newer). Hence, I couldn't reinstall OS X.

My solution was to install Ubuntu. I gave it the whole disk, and everything worked out okay. I was a little bummed since I wanted to try a new distro (either CrunchBang or MEPIS), but it's hard to argue with a working system ;)

Apple: Ouch!

2010-12-21T16:05:00.001-08:00

I had my laptop wrapped in a blanket in my backpack when I dropped it on concrete at BART yesterday. It landed on the corner with quite a lot of force and bent a few metal pieces. Everything seems to work, with the exception of the ethernet port. Thankfully, I rarely use that. I'm hoping this is covered by the warranty, although it seems unlikely. The bent aluminum reminds me of body damage on a car.

Vim: PeepCode Videos

2010-12-15T22:31:00.000-08:00

There are commercial videos on PeepCode that show how to use Vim. They're fantastic! The production quality is very high, so they're worth paying for. I bought both of them, and I learned quite a lot. Even the first one covered a ton of advanced stuff, so I highly recommend it.

If you're just getting started with Vim, I suggest you:

Read this blog post from Yehuda Katz. Basically, he suggests you use the GUI to get started and iteratively gain more experience with Vim. I think that's good advice.
Next, I suggest you fire up Vim and take the internal tutorial.
Last of all, watch Smash into Vim part I.

If you're an advanced Vim user, watch both videos and tell me what your favorite part was ;)

JavaScript: IE and DTDs

2010-12-10T15:01:00.001-08:00

I was having a hard time getting Internet Explorer 8 (IE8) working with Socket.IO's "htmlfile" transport (or any other transport for that matter). I was getting weird permissions issues. I finally fixed the problem by adding a DTD to the top of my HTML file, "<!doctype html>". I was just being lazy not having a DTD, but I was really surprised when that fixed my problem. Of course, I'm still having a hard time getting IE7, IE6, and Opera working cross-domain (technically, I'm just using a different port, but eventually it'll truly be cross-domain).

Ruby: Why I Prefer Cucumber

2010-12-08T11:50:00.000-08:00

I've been struggling to articulate why, in most cases, I prefer high-level integration tests with Cucumber and Webrat (or Capybara) over low-level model, view, and controller tests with RSpec. I think I finally have an example that conveys what I'm trying to say. Consider the following Cucumber test:

Scenario: create a new film unsuccessfully
  Given I am logged in as "admin"
  And I am on the admin films page

  When I follow "New film"
  And I press "Create"
  Then I should see "There were problems with the following fields:"
  And I should see "Name can't be blank"
  And I should see "Url name can't be blank"
  And I should see "Sort name can't be blank"
  And I should see "URL doesn't look like a valid RTMPE URL"
  But I should not see "Trailer URL doesn't look like a valid RTMPE URL"
  And I should not see "Scene URL doesn't look like a valid RTMPE URL"

This test is short, readable, and easy to write. It doesn't test every possible validation failure, and it's not the only test I have. (In fact, I have some RSpec model tests that test the more esoteric URL validation rules.) However, it does test the model, view, and controller's handling of validation failures, and it even tests that they integrate with each other.

Can you imagine trying to write the same tests by separately testing the model, view, and controller using RSpec? Now, imagine trying to use a separate test for each assertion. That's a lot of code for something so trivial--this ain't rocket science, guys! Finally, remember that when you test the things separately, there's nothing preventing the code from crashing when you put all the pieces together. (For instance, what happens if the controller and view each pass their RSpec tests, but they disagree on the spelling of one of the assigns?)

Is there benefit to testing things separately--absolutely. Is it worth it in this case--absolutely not. I think it's important to remember that at a certain level, our job is to implement features that work. Tests are a means to an end--they help us keep the code working. They don't really have any intrinsic value for the stakeholder. They only have the secondary value of keeping the code working when it is extended.

Just as there is engineering value in implementing features using less code (as long as it remains readable), there is also engineering value in implementing features using less testing code (as long as the tests continue to serve their purpose of preventing regressions).

My point is that Cucumber lets you test more using less effort.

Personal: Wife in Labor

2010-11-26T10:00:00.000-08:00

Wife in labor. Built Twilio app to time contractions. Call it to try it out: (866) 948-3615.

Software Engineering: Coping When You Must Repeat Yourself

2010-11-19T11:02:00.000-08:00

These days, most software engineers are familiar with the acronym "DRY" which stands for "don't repeat yourself". In general, it's good advice. In a perfect world, no code would ever be duplicated and every piece of "truth" would only exist in one place. However, the real world isn't quite so perfect, and things are far less DRY than you might imagine. The question is, how do you cope?

First let me show you some reasons why you can't always keep it DRY:

Often, the same truth is duplicated in the code, the tests, and the docs. Duplicating the same piece of truth in the code and in the tests helps each verify the other. Generally, the code is more general than the tests (the tests verify examples of running the more general code), but the duplication is there. When you update one (for instance to change the API), you'll need to update the other. This is a case where not keeping it DRY pays off--if you have to update the tests, that's a reminder that you'll also have to update all the other consumers of your API.

Similarly, the API docs often duplicate the truth that is built in to the code. That's because it's helpful to explain the truth to the computer in one way (using very precise code) and explain the truth to the reader in another way (using friendly, high-level English). Every truthful comment duplicates what the code already says, but not every piece of code is easily and quickly readable by human readers--this is especially true in, say, assembly.

Another area where truth is duplicated is in APIs. The function defines a name and an API. The caller uses it. They must agree on these things or the code won't work. If the caller decides to use a different name or a different API, the code will break. Essentially, programmers have decided that it's better to duplicate the name and the API rather than duplicate the contents of the function. This points to a useful trick--sometimes a small amount of duplication saves a large amount of duplication. You'll also see this sometimes in comments when they say "see also..."

Another source of duplication concerns public vs. private. For instance, in C, the same API is duplicated in the .h file and the .c file. Sometimes, the same piece of code must be duplicated in different projects. For instance, one operating system might need to define the same C types as another operating system because there's no easy way for them to share the same header files.

At a higher level, one time I had to add the same function I wrote to two projects. One project was proprietary company code. The other was open source (I had permission, of course). For technical reasons, it was impractical for the company code to import or subclass the open source code, so I was stuck just duplicating it.

Often, you'll need to duplicate the same piece of truth in multiple languages. For instance, think of how many HTTP client libraries there are in all the different programming languages. It doesn't matter how good an HTTP client library is if it's not easily accessible from the programming language I'm currently coding in. Sometimes there will be multiple HTTP client libraries for the same language because they're implemented differently (for instance, syncronously vs. asyncronously).

I mentioned tests before. Often tests duplicate some setup or teardown or perhaps the same pattern of interacting with a function. Refactoring is sometimes appropriate, but not always. It is commonly held that this is one area where keeping it DRY is less important than keeping it simple and isolated. A perfectly DRY collection of unittests that is difficult to comprehend and difficult to debug when something fails is less helpful than a set of simple, isolated unittests that contain a small amount of duplication. If the duplication causes multiple tests to fail, you'll know to keep fixing the tests until they all pass.

The question remains, how do you cope when you can't keep it DRY?

Greppability is very important. (By grep, I mean any tool that can search for a string or regular expression. I don't necessarily mean the UNIX tool "grep".) In highly dynamic languages like Ruby (that have great facilities for metaprogramming, but no static typing or interfaces) and highly factored frameworks like Rails (that use lots of files and levels of indirection), even a brilliant IDE can fail in comparison to a simple "grep tour". If you refactor a class in Ruby, how will you remember to refactor all the mocks of that class? You might have a user of your class that has a mock of your class that still makes use of your old API. The tests might be passing even though the code will assuredly crash. If you use grep, you can update all the callers of your class as well as all the mocks of the class. Grep can also help you find instances of a string in non-code documentation, and it even occasionally works with binary files. My point is, don't underestimate the utility of grep. Rather, you should aim for greppability. A function named "f" is not greppable, but a function named "calculate_apr" is. (By the way, naming all your loop variables "iterator" does not improve greppability, it just wastes time.)

Another way of coping when things aren't DRY is to have cross referencing comments. If you know that you must duplicate the same piece of truth in 5 places, add a comment next to each of those 5 places that refers to the other 5 places. Don't be afraid to duplicate the comment exactly. Your comment can say something like, "If you change this, don't forget to update..."

Another thing that helps mitigate duplication is proximity. Docstrings belong in the code because if a programmer updates one, he'll be more likely to update the other (although even proximity can't always help lazy programmers). If all the API documentation is in a separate file, that file will go stale very quickly.

Parallelization also helps. For instance, this code has a small amount of duplication:

 some_a = 1
 some_a.invoke_method()
 register(some_a)
 call_something_unrelated()
 some_b = 2
 some_b.invoke_method()
 register(some_b)

Sometimes you can factor out this duplication. However, in less dynamic languages like C, it may not always be easy to do so. However, parallelization can really help:

 some_a = 1
 some_b = 2
 some_a.invoke_method()
 some_b.invoke_method()
 register(some_a)
 register(some_b)
 call_something_unrelated()

Another old trick for coping with duplication is to have one source generate the other. Generating API documentation using javadoc is a good example of this. Sometimes you can use a program to generate code for multiple programming languages. There's another example of "generation" that I sometimes use in Python. I use string interpolation when creating docstrings. For instance, if there's a piece of documentation that should be duplicated in multiple places, string interpolation makes it possible so that I only have to write that piece of documentation once.

Another source of duplication has to deal with the plethora of tools programmers must use. There is the source code itself, a revision control system, a bug tracker, and a wiki. Often times, the same piece of truth needs to be duplicated in all of these places. This is one place where Trac really shines. Once you properly configure Trac, you can reference the bug number in each of your commits. Trac's commit hook will take that commit and add it as a comment in the original bug with a reference to the source code in Trac's source code viewer. Hence, Trac (which is a bug tracking system, a wiki, and a source code viewer) and the revision control system work together to reduce duplication.

It's unfortunate that life isn't always as DRY as you'd like it to be. However, keeping a few tricks in mind can really help mitigate the problems caused by having to duplicate a piece of truth in more than once place. If you have other tricks, feel free to leave them in a comment below.

JavaScript: Naughty Socket.IO Example

2010-11-18T11:01:00.000-08:00

File this under the "things you probably shouldn't do, but are fun anyways" category. Socket.IO is a library for Node.JS that provides Comet using a plethora of different approaches (WebSocket, Flash socket, AJAX long polling, etc.). I hacked the Socket.IO chat demo so that it reads HTML from my terminal and just dumps it to the browser. Hence, I can control people's browsers from my terminal. Insecure? Yeah. Fun? Oh yeah!

Anyway, here's how I hacked the server.js file in Socket.IO's chat demo:

io.on('connection', function (client) {

 // Read from /dev/tty and send it to the browser.
 var stream = fs.createReadStream('/dev/tty', {encoding: 'ascii'});

 stream.on('error', function (exception) {
   client.send({announcement: 'Exception: ' + exception});
 });

 stream.on('data', function (data) {
   client.send({html: data});
 });
 ...

And here's how I hacked chat.html:

function message(obj) {
 var el = document.createElement('p');
 if ('html' in obj) el.innerHTML = obj.html;
 ...

Here's what it looks like in my terminal:

sudo ./server.js
18 Nov 10:14:04 - socket.io ready - accepting connections
18 Nov 10:14:06 - Initializing client with transport "websocket"
18 Nov 10:14:06 - Client 5832344139926136 connected

<i>I'm typing this in to control the page.</i>
<script>alert('Oh baby!');</script> This doesn't work with innerHTML, thankfully.
<ul><li>Node.JS</li><li>Socket.IO</li></ul>