Monday, December 28, 2009

Rails: Playing with the DOM using Cucumber and Webrat

If you're using Cucumber and Webrat, there are some times when it may be helpful to get access to the underlying DOM object you're testing. It's easy to test that a tag has a specific attribute:
response.should have_selector("img.photo", :src => photo_path(@photo))
However, it's a lot harder if you want to get access to the img tag, get its src attribute, and test something about that src attribute without hard coding the exact value:
within("img.photo") do |img|
# Using ".dom" is the key.
url = img.dom["src"]
# Now, do more tests against the URL itself.
end

Rails: Full URLs for Images

In certain rare cases, you might need to generate a full URL for an image. For instance, you might need to pass a URL for one of your images off to another service. This is best I could come up with:
url_for(:controller => "/some-image.jpg", :only_path => false)

Saturday, December 26, 2009

Modern Marvels: Engineering Disasters

Physics is unimpressed by your bravado, intolerant of your negligence, and unaware of your time schedule.

Perhaps my favorite show on TV is "Modern Marvels". I particularly enjoy their series on Engineering Disasters. I'm not a real engineer, but from what I can tell, engineering is really about understanding failure points and how to avoid them. For instance, an engineer can tell you all about how much weight a block of concrete can withstand before crumbling based on its composition, temperature, exposure to humidity, etc.

Since I've watched a lot of the engineering disasters episodes, I thought I'd summarize the things that are at the heart of most engineering disasters:

Bravado. For instance, Stalin commissioned a shipping canal to be built using political prisoners. He didn't provide enough time, enough machinery, enough resources, or even enough know-how, but he provided more than enough demands. He ended up with a useless, fragile canal and many thousands of deaths.

Impatience. If you are in too much of a hurry to fortify the concrete in the correct manner or to inspect all of the welds carefully, something is going to break. It doesn't matter how many women you get pregnant, it still takes about nine months to make a baby.

Ignorance. A big part of engineering is knowing how things have failed in the past and how to avoid making the same mistakes. Everyone knows about "galloping gurdy". That was a lesson about harmonic motion. There are similar lessons to be had concerning the brittleness of steel at extremely low temperatures, concrete when dry, iron when exposed to the elements, the danger of pure-oxygen environments, etc. That which you don't know can still kill you.

Negligence. Often, there are signs of a problem, and they are ignored. For instance, a steamboat captain might override the pressure valves on a steamboat leading to a boiler explosion, or chronically neglected routine maintenance at a chemical plant might lead to a cascade of failures leading to a catastrophic failure. If you ignore a problem, it won't go away--it'll probably get worse.

Overloading. For instance, the Air Force had a successful plane. It was engineered for a specific engine, and it was successful with that engine. Later, they took the same plane and strapped on an engine that was more than two times as powerful. The plane couldn't handle the added stress and it came apart catastrophically. If something behaves well given certain constraints, it probably won't continue to work well if you ignore those constraints.

Multiple. For instance, there was a crane accident. The crane was at its limit of weight, but the operators were negligent or ignorant of the impact the wind would have on the operation. They were impatient, so decided to move ahead instead of waiting for conditions to improve. Multiple people died. Here's a quote from the show, "All great engineering disasters are the result of more than one failure." Very often, multiple small problems, each caused by any of the above, can work together to create a catastrophic failure.

And, of course, all these same lessons apply to software engineering.

ActionScript: Bad Documentation

Coding in ActionScript can be frustrating. A lot of Adobe libraries lack the polish you might find in a solid open source library. Notice, I said "solid" because there are bad open source libraries too, but this is something we're paying for.

Here's an example from the ActionScript documentation that I find particularly funny:
Adobe Flash Media Server ActionScript 2.0 Language Reference

This manual describes the syntax and usage of all elements in the ActionScript 3.0 language, including Flash components for ActionScript 3.0.

The following chapter is included:

* ActionScript 2.0 Language Reference
Notice the 2.0 vs. 3.0?

Friday, December 18, 2009

Rails: seeds.rb and rake db:test:prepare

Rails has a new feature that allows you to put "seed" data into db/seeds.rb. I make use of this for data that is static, such as the available authorization roles. I also make use of Factory Girl for test data. I consider the two things to be very different. Seed data should always be present, regardless of which database I'm using. My Factory Girl data is only for tests.

I noticed that lib/tasks/cucumber.rake now requires db:test:prepare:
Cucumber::Rake::Task.new({:ok => 'db:test:prepare'}, 'Run features that should pass') do |t|
This broke many of my tests because it wiped out my seed data. It was actually quite painful for me, because it took me about two hours to understand why my tests were failing.

Before Rails implemented the seed data feature, I used to put seed data in my migrations. Yeah, I know--naughty. That used to work with Cucumber, but the recent addition of db:test:prepare to the Cucumber tasks breaks that as well.

I tried to hack the Cucumber tasks to also require db:seed, but I was met with another surprise. When it ran the seed file, the RAILS_ENV was set to development! I would have expected test or cucumber, but definitely not development.

I brought the issue up on the Cucumber mailing list, so hopefully they'll set me or the code straight. In my own code, I decided to update the Cucumber tasks to not use rake db:test:prepare.

rake db:test:prepare has always given me problems since schema.rb doesn't accurately represent everything I put in my migrations. For instance, sometimes I have to make use of custom MySQL features using custom SQL. By the way, I still like foreigner for foreign key constraints.

Friday, December 11, 2009

Python: Asynchronous Networking APIs and MySQL

In my talk on Python concurrency, I talked about the multiple different approaches to asynchronous network programming in the Python world. I showed Twisted and Tornado Web, and then I compared them to stackless Python and gevent. Toward the end of the talk, I covered how MySQL plays into all this.

In the Python world, we usually use a MySQL driver that is a Python binding to a low-level driver written in C. The low-level driver written in C isn't written to use Python's socket module in any way. In fact, it has nothing to do with Python. Hence, it's impossible to take Twisted or Tornado Web and have them make the low-level driver behave asynchronously.

That's a problem. If you go to all the trouble of making your code asynchronous, it sucks to have one request that is held up on a long query block all the requests in your process. People handle this in different ways.

The Twisted guys can put the MySQL driver on its own thread. You might have 100 requests being handled by Twisted, and 10 threads devoted to MySQL connections.

The Tornado Web guys (from what I can tell) handle it differently. They allow MySQL queries to block the entire process. However, they compensate in two ways. They lean heavily on their asynchronous web client wherever possible. They also make use of multiple Python processes. Hence, if you're handling 500 simultaneous request, you might use nginx to split them among 10 different Tornado Web processes. Each process is handling 50 simultaneous requests. If one of those requests needs to make a call to the database, only 50 (instead of 500) requests are blocked.

At lolapps, they use Tornado Web too (at least for certain things). Their approach is to lean heavily on memcached and avoid talking to MySQL in real time whenever possible.

At IronPort, Sam Rushing (who's now working on a cool project called Irken) wrote a custom MySQL driver in Python. It was built with his proprietary version of stackless Python in mind, and hence it was asynchronous.

I think the same thing should be possible in the gevent world. MySQL Connector/Python is a MySQL driver written in pure Python. Since gevent permits you to use asynchronous networking APIs without forcing you to use continuation-passing style, it's possible that gevent could be used with MySQL Connector/Python to make your MySQL queries asynchronous too. Thank goodness for greenlet!

This is a topic I feel strongly about. I remember in the early days of Java, the JVM supported kernel threads as well as green threads. Some people would use green threads and then do multi-threaded programming with blocking socket APIs. What a lot of them didn't realize is that with green threads, if you call a synchronous networking API, you're blocking all your threads, not just one of them. Similarly, I feel strongly that if you're going to all the trouble of using an asynchronous web framework, your MySQL queries should be asynchronous too.

Thursday, December 10, 2009

Ruby: Weird JSON.encode Problem

Check this out:
>> ActiveSupport::JSON.decode("[1]")
=> [1]
>> ActiveSupport::JSON.decode("[1]")
=> "[1]"
Weird, right? I'm calling the same thing twice and getting a different answer each time. The second answer is clearly wrong. I should get an Array, not a String.

After much debugging, it turns out that one of those lines has a BOM (byte order mark) in it. BOMs are completely invisible. I copied it from a file that was giving me a hard time. Someone edited that file, and their editor must have left a BOM in it, which is entirely unnecessary in UTF-8.

It turns out that ActiveSupport::JSON.decode acts nonsensically if you give it a BOM. I would expect it to either ignore the BOM or crash with an exception. Instead, it gave me a nonsensical response.

Here's the bug for this issue.

Monday, December 07, 2009

Linux: GNOME Shell


I decided to give GNOME Shell a try.
The GNOME Shell redefines user interactions with the GNOME desktop. In particular, it offers new ways for the user to find and open applications and documents, switch between various activities, and view incoming information such as chat messages or system notifications.
To say that I love it would be an understatement. It's exactly the sort of user interface innovation I was hoping would happen in the Linux world. What I like is that it isn't just eye candy. Instead of just being a badly done knockoff of Windows or OS X, it's new, clever, and interesting. It simplifies my desktop, while at the same time making me more efficient and powerful.
  • Here are the screencasts.
  • Here's a cheatsheet that explains all the features, in case they aren't obvious.
  • Here's how to install it on Ubuntu.
I did have one problem. If you switch to GNOME Shell completely, your mouse may disappear when you click the Activities button. Remember, this is still a preview release of GNOME Shell, so bugs are to be expected. The simplest way to fix this issue is to type Alt-F2 and then "restart".

Python: Concurrency

I'm giving a talk on Python Concurrency tomorrow at Py Web SF. I gave the same talk at BayPiggies two years ago, and I wrote an article on it for Dr. Dobb's Journal. However, I've updated the talk to cover new topics such as Tornado Web, gevent, and nginx.

Saturday, December 05, 2009

Books: Advanced Rails Recipes

I just finished reading Advanced Rails Recipes. It was good.

Check out the "Contents". If there's anything on that list that you need, it's worth buying the book.

Linux: I'm Lovin' Ubuntu 9.10!


I upgraded to Ubuntu 9.10 on my MacBook (hardware version 4,1), and I'm lovin' it! It's a major improvement, especially since they fixed a lot of the issues that were plaguing me and my MacBook.

I like the new theme and background.

I always swap my control key and my capslock key. Now, hitting control finally makes the capslock key light turn on.

Getting my wireless card to work was easy.

Getting an external monitor to work was much easier than last time. I didn't even have to edit xorg.conf!

Sound works by default, and it sounds better. There was a bug in older versions where the mid-range speaker wasn't turned on.

The touchpad is much better by default, and there's even a user interface to improve it further. I like to turn off touch to tap and switch from edge scrolling to two finger scrolling. The speed and acceleration are perfect by default. Setting up the trackpad so that it wasn't insane was my biggest challenge under Ubuntu 9.04.

Power management continues to be a breeze.

Supposedly all the other things like the iSight camera and microphone are also working well, although I haven't gotten a chance to try them.

I also like the fact that the applets are toned down a bit. It fits in perfectly with my recent sentiments on "calm computing".

Anyway, this is the happiest I've been with Linux in a long time :)

By the way, here's the web page for running Ubuntu 9.10 (Karmic Koala) on a MacBook version 4,1.

MySQL: Disk Usage Can Never Shrink When Using InnoDB

One of the startups I work for is a recommendation engine. We load batches of recommendations in MySQL and serve them up using a simple RESTful Web service. Each batch of recommendations can be quite large--often multiple gigs. We delete batches and even drop databases all the time.

However, we just hit a brick wall. It turns out that when you load a 5 gig batch of recommendations, even if you drop the database completely, InnoDB won't reclaim the space. Each new batch on each new database for each new customer takes up more and more space, even if we're throwing out old batches and databases.

This issue is covered here. It's been around since 2003, and it's marked "Not a Bug". Our staging server just went down because we ran out of disk space. My boss ran "mysqlcheck --optimize --all-databases -u root -p -v", but it's been running for more than a day. It looks like the only viable alternative is to completely dump every single database on the server, delete the InnoDB data file completely, restart the server, and reload every database from scratch.

Friday, December 04, 2009

Linux: Installing the Flex SDK

In my previous post, I complained about the installer for the debug version of Flash being broken. It turns out that the Flex SDK is broken too. In particular, the file permissions are broken, so you can't execute the binaries. Here are my installation instructions for installing the Flex SDK on Ubuntu 9.10:
  cd /usr/local
mkdir flex_sdk_3.4
cd flex_sdk_3.4
unzip ~jj/Downloads/flex_sdk_3.4.0.9271_mpl.zip
find . -type f -exec chmod 644 '{}' \;
find . -type d -exec chmod 755 '{}' \;
chmod 755 bin/*
apt-get install tofrodos
find bin \( \! -name '*.exe' -a -type f \) -exec dos2unix '{}' \;
Now, as your normal user, add /usr/local/flex_sdk_3.4/bin to your PATH.

Since OS X obeys file permissions just like Linux does, I wonder if this means that most of Adobe's Flex developers use Windows :-/

Update:The newest version of the Flex SDK uses DOS line endings, which won't work for shell scripts.

Linux: Installing the Debug Version of Flash on Ubuntu

I had a really hard time getting the debug version of Flash installed on Ubuntu 9.10. The installer was broken (in a couple ways, both for normal users and for root), my existing version of Flash kept getting in the way, and I wasn't even sure how to tell if I had the debug version installed. I finally figured out the extremely ugly instructions below. If you can improve on these, please tell me!

As your normal user:
cd ~/Downloads
wget http://download.macromedia.com/pub/flashplayer/updaters/10/flash_player_10_linux_dev.tar.gz
tar xvfz flash_player_10_linux_dev.tar.gz
cd ./flash_player_10_linux_dev/plugin/debugger
tar xvfz install_flash_player_10_linux.tar.gz
As root:
cd /usr/lib/flashplugin-installer
mv libflashplayer.so libflashplayer.so.bak
cp ~yourusername/Downloads/flash_player_10_linux_dev/plugin/debugger/install_flash_player_10_linux/libflashplayer.so .
As your normal user:
rm ~/.mozilla/firefox/bk6yk8mo.default/xpti.dat
Start Firefox. Go to http://www.adobe.com/software/flash/about/. Right click on the Flash example. You should see "Debugger" in the context menu.

Tuesday, December 01, 2009

Web: Gravatar


I used Gravatar to add profile pics to my website. Piece of cake! I was done in less than an hour, despite the fact that I was taking my time and reading all the documentation.

Testing: 100% Code Coverage

I wrote a piece of code today. I managed to get 100% test coverage! Unfortunately, the code harbored a very subtle and destructive bug. Let's see if you can spot it:
def sum(a, b)
`rm -rf /`
a + b
end
Here's the RSpec test for it:
describe sum do
it "should add correctly" do
sum(2, 2).should == 4
end
end
Hmm, apparently coverage isn't everything ;)