Wednesday, July 29, 2009

Python: Planet Python Update

If you read my blog through Planet Python, you'll now only see my blog posts which have been explicitly tagged with Python. If you want to see all the rest of the things I blog about, such as JavaScript, Rails, Oz, Haskell, Erlang, Linux, etc., you'll need to subscribe directly to my blog.

Sorry for the inconvenience. I had to make this change because a lot of Python programmers didn't like seeing Rails-related posts showing up on Planet Python.

Thanks for reading!

Rails: Configuring Admins, Checkboxes, and attributes= Vulnerabilites

I'm using acl_system2 together with authlogic to provide ACLs for my Rails app.

I wanted to provide a list of checkboxes so that an admin can configure the list of roles assigned to a user. Since Rails really prefers to work with one model per form, and the list roles are separate from the user record itself, it took me a while to figure out how to do this. Thankfully, I found this post which showed me how to do it. The code looks like:
<% restrict_to "admin" do %>
<p>
<%= form.label :role_ids, "Roles" %>
<% Role.all.sort.each do |role| %>
<% field_id = "role_#{role.id}" %>
<br />
<%= check_box_tag "user[role_ids][]", role.id, @user.role_ids.include?(role.id), {:id => field_id} %>
<%= label_tag field_id, h(role.title) %>
<% end %>
</p>
<% end %>
Without any changes to my controller, it just worked. However, that worried me.

I suddenly realized that if you give a user access to edit a record from ModelA and ModelA has_and_belongs_to_many ModelB, then a user can hack his form to pick which records from ModelB he has. I.e., my app was totally insecure--and, probably, so was everyone else's! I started freaking out ;)

Fortunately, the guys on the SF Ruby mailing list showed me this blog post. Apparently, I was right. It's a huge security vulnerability that even the third edition of "Agile Web Development with Rails" doesn't cover very well.

After analyzing all the options, I created config/initializers/disable_mass_assignment.rb with:
# Force every model to make use of attr_accessible.
# See: http://railspikes.com/2008/9/22/is-your-rails-application-safe-from-mass-assignment
ActiveRecord::Base.send(:attr_accessible, nil)
Then, I added calls to attr_accessible in my models like:
attr_accessible :username, :email, :first_name, :last_name, ..., :password, :password_confirmation
I ran my tests, and I ended up with this ugly error message:
Mysql::Error: Column 'session_id' cannot be null: INSERT INTO `sessions` (`data`, `created_at`, `updated_at`, `session_id`) VALUES('BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo\nSGFzaHsABjoKQHVzZWR7AA==\n', '2009-07-29 19:08:46', '2009-07-29 19:08:46', NULL) (ActiveRecord::StatementInvalid)
It took me a while to figure it out, but finally I stumbled across this blog post. I added the following to the bottom of config/initializers/disable_mass_assignment.rb:
ActiveRecord::Base.send(:attr_accessible, :session_id)
That fixed it.

I had to fix a few remaining issues, usually by tweaking the attr_accessible calls in my models. Finally, my code that would allow an admin to create another admin broke, because attr_accessible wasn't allowing access to role_ids. That was a good sign ;) I tweaked the controller like:
def update
restrict_to "admin" do
@user.role_ids = params[:user][:role_ids] || []
end
if @user.update_attributes(params[:user])
...
I even wrote the following Cucumber test to verify that everything worked:
Scenario: a user cannot hack his form to trick Rails into making him an admin
Given I am logged in as admin
When I follow "My Account"
And I follow "Edit"
And I check "admin"
And I check "subscriber"
And someone else strips me of my admin role
And I press "Update"
Then I should not see "Roles"
And I should not see "admin, subscriber"
Voila! All is happy in Rails land again!

Wednesday, July 22, 2009

Rails: Forcing a Controller to have a Comment

I'm using acl_system2 for authorization. As a general rule, I think apps should deny access to everything, and then open up permissions where appropriate. However, acl_system2 makes it hard to restrict permissions in ApplicationController and open them up in each subclass.

That means I have to remember to control access in each controller. acl_system2 will allow access unless you tell it not to. That could lead to accidents. Hence, I started with the following in ApplicationController:
# === Access controls
#
# Each controller is responsible for enforcing access controls properly. It
# should either have some variation of::
#
# before_filter :require_user
# access_control :DEFAULT => 'admin'
#
# Or at the very least::
#
# # No access control is required:
# # before_filter :require_user
# # access_control :DEFAULT => 'admin'
#
# See http://github.com/ezmobius/acl_system2/tree/master for more details.
Of course, that's just a comment which no one will ever read anyway. Hence, I wrote a RSpec test to enforce it:
context "Controllers" do
controllers = Dir[File.expand_path(File.dirname(__FILE__) + "/../app/controllers/*.rb")]
it "should not be empty" do
controllers.should_not be_empty
end
controllers.each do |f|
contents = IO.read(f)

context f do
it %{should contain "before_filter :require_user" at least in a comment (see application_controller.rb)} do
contents.should =~ /before_filter :require_user/
end
it %{should contain "access_control :DEFAULT => 'admin'" at least in a comment (see application_controller.rb)} do
contents.should =~ /access_control :DEFAULT => 'admin'/
end
end
end
end
Conceptually, what's going on is that I have an interface that I want child classes to follow. Part of that interface is that you must at least provide a comment about why you don't have to require access control. Funky!

Rails: Tests vs. Docs

I've been accused before of relying too heavily on documentation and not heavily enough on tests. I wrote the following in my README_FOR_APP:
==== Unnecessary Files

You probably should not need to create files in any of these directories, and
you should not commit any autogenerated files to these directories. Each of
these directories has a README explaining why and what we're using instead::

* app/views/layouts
* test/fixtures
* test/functional
* test/integration
* test/unit
* test/unit/helpers
(I'm using RSpec, factory_girl, etc., so a lot of those directories are unnecessary. I only have one layout.)

I thought to myself, how would a TDD person solve this problem with tests instead of documentation? Hence, I created spec/hieararchy_spec.rb:
require File.expand_path(File.dirname(__FILE__) + '/spec_helper')

context "the file hierarchy in this Rails app" do
directories = {
"app/views/layouts" => ["main.html.erb"],
"test/fixtures" => [],
"test/functional" => [],
"test/integration" => [],
"test/unit" => ["helpers"],
"test/unit/helpers" => []
}
directories.each do |directory, expected|
it "should have no unexpected files in #{directory} (see the README)" do
Dir["#{File.dirname(__FILE__)}/../#{directory}/*"].each do |f|
(expected + ["README"]).should include(File.basename(f))
end
end
end
end
Now, I can get rid of the documentation. The test itself will tell the user what he did wrong and which README to look at.

Tuesday, July 21, 2009

Rails: authlogic and acl_system2 under Rails 2.3.2

Updated: Documented my migration.

I got authlogic and acl_system2 to work together under Rails 2.3.2. authlogic provides authentication. acl_system2 provide authorization, i.e. ACLs. authlogic is very up-to-date, but acl_system2 is a bit dated. That's okay, though, because it's not the sort of thing that should need to change much.

Let me cover some of the stumbling blocks I encountered after I followed the authlogic tutorial and the acl_system2 documentation.

acl_system2 is not available as a gem. Hence, you need to install it via:
script/plugin install git://github.com/ezmobius/acl_system2.git
If you followed the authlogic tutorial, you'll end up with the ApplicationController#current_user method being private. To work with acl_system2, it should instead be protected. Otherwise, you'll end up with this:
You have a nil object when you didn't expect it!
The error occurred while evaluating nil.roles (NoMethodError)
.../vendor/plugins/acl_system2/lib/caboose/role_handler.rb:15:in `check'
.../vendor/plugins/acl_system2/lib/caboose/logic_parser.rb:43:in `process'
.../vendor/plugins/acl_system2/lib/caboose/access_control.rb:101:in `allowed?'
.../vendor/plugins/acl_system2/lib/caboose/access_control.rb:28:in `access_control'
Here's what my migration looks like:
class CreateRoles < ActiveRecord::Migration
def self.up
create_table :roles do |t|
t.string :title, :null => false

t.timestamps
end

add_index :roles, :title, :unique => true

create_table :roles_users, :id => false do |t|
t.integer :role_id, :null => false, :options => "CONSTRAINT fk_role_id_roles REFERENCES roles(id)"
t.integer :user_id, :null => false, :options => "CONSTRAINT fk_user_id_users REFERENCES users(id)"
end

Role.delete_all
Role.create :title => "admin"
end

def self.down
Role.delete_all
drop_table :roles_users
drop_table :roles
end
end
Each of my controllers has something like:
before_filter :require_user
access_control :DEFAULT => 'admin'
I decided to add the following to ApplicationController as protected methods:
def permission_denied
render :text => "Forbidden", :status => "403 Forbidden"
end

def permission_granted
end
That way the HTTP status gets set for "Permission denied".

In order to test the above using Cucumber and Webrat, I added a feature step like:
And the HTTP status should be "403 Forbidden"
Then, I added a step definition:
Then /^the HTTP status should be "([^\"]*)"$/ do |status|
response.status.should == status
end
So far, I'm pleased :)

Saturday, July 18, 2009

This Isn't a Pure Python Blog

I've been getting some flack about my Ruby-related posts showing up on Planet Python. On the one hand, I have never intended this to be a pure-Python blog. On the other hand, since most of my readers come from Planet Python, I surely don't want to upset them.

I work at two startups. One uses Python with Pylons. The other uses Ruby on Rails. This puts me squarely in both camps. In general, I try to code in and blog about as many technologies as I have time for. Hence, you'll often see posts about weirder things like Haskell, Erlang, Oz, etc.

I can understand how it would be frustrating for some people to read about Rails on Planet Python. In fact, I got kicked off of Planet Haskell because I spent too much time blogging about Python! Now, I could probably wax philosophic about the increasingly insular nature of programmers in our industry, but given how many times I've made fun of Java, it probably wouldn't do me any good ;)

If you have any advice, I'd love to hear it. I'd hate to create separate blogs, because all of these topics together are part of who I am and what I do. One of my heroes is Isaac Asimov, and that guy would write about anything! I see that there is a way to view just my Python posts: http://jjinux.blogspot.com/search/label/python. However, I don't see an atom feed for that. I've asked the guys on Planet Python for advice concerning this issue.

In the meantime:

If you have read my blog in the past, and you enjoy it, and you don't mind my occasional posts on weird things like Ruby, Oz, Haskell, etc., please consider subscribing to my blog directly: http://jjinux.blogspot.com/

Oh, and as usual, thanks for reading!

Friday, July 17, 2009

Rails: RSpec, Cucumber, Authlogic, and factory_girl

After a day of work and a week of reading The RSpec Book, I got RSpec, Cucumber, Authlogic, and factory_girl to play nicely with each other. I can now test user registration, logging in, and logging out.

I can't possibly cover everything from scratch, but I'll cover some of the trickier integration points.

Setup Rails, RSpec, Cucumber, and Authlogic per their own documentation. I ended up with the following in both config/environments/test.rb and config/environments/cucumber.rb:
# There is some duplication between test.rb and cucumber.rb.
config.gem "cucumber", :lib => false, :version => ">=0.3.11"
config.gem "webrat", :lib => false, :version => ">=0.4.4"
config.gem "rspec", :lib => false, :version => ">=1.2.6"
config.gem "rspec-rails", :lib => 'spec/rails', :version => ">=1.2.6"
config.gem "thoughtbot-factory_girl", :lib => "factory_girl", :version => ">=1.2.2", :source => "http://gems.github.com"
I created a factory in test/factories/user.rb:
Factory.define :user do |u|
u.username 'john'
u.email 'john@example.com'
u.password 'funkypass'
u.password_confirmation 'funkypass'
u.first_name 'John'
u.last_name 'Smith'
u.billing_address "1 Cool St.\r\nBerkeley"
u.billing_state 'CA'
u.billing_zipcode '94519'
u.other_zipcode '28311'
u.phone '(925) 555-1212'
end
I created a Cucumber feature in features/user_sessions.feature:
Feature: User Sessions

So that I can blah, blah, blah
As a registered user
I want to log in and log out

Scenario: log in
Given I am a registered user
And I am on the homepage
When I login
Then I should see "Login successful!"
And I should see "Logout"

Scenario: log out
Given I am logged in
And I am on the homepage
When I follow "Logout"
Then I should see "Logout successful!"
And I should see "Register"
And I should see "Log In
To implement this, I created step definitions in features/step_definitions/user_sessions_steps.rb
def user
@user ||= Factory :user
end

def login
user
visit path_to("the homepage")
response.should contain("Log In")
click_link "Log in"
fill_in "Username", :with => "john"
fill_in "Password", :with => "funkypass"
click_button "Login"
response.should contain("Login successful!")
response.should contain("Logout")
end

Given /^I am a registered user$/ do
user
end

When /^I login$/ do
login
end

Given /^I am logged in$/ do
login
end
That's it.

Authlogic's testing documentation talks about stuff like:
require "authlogic/test_case"
...
include Authlogic::TestCase
...
setup :activate_authlogic
UserSession.create(users(:whomever)) # logs a user in
In theory, you should be able to log a user in using some code--without going through the login form. I worked on this for several hours, but I wasn't able to get it to work. I googled quite a bit, but I wasn't able to find anyone else who could get this to work (i.e. integrate RSpec, Cucumber, Authlogic, and factory_girl). It seems that everyone else came to the same conclusion I did--just force the tests to use the login form.

Monday, July 13, 2009

Software Engineering: A Book on $foo

Rant warning:

I'm reading a new book. Let me summarize:
$foo is awesome. It will help you get your projects done on time and on budget. Traditional software projects fail because they don't use $foo. The people who do manage to deliver software on time and on budget only do so because they are heroic programmers. Their process is actually fighting against them. They should use $foo instead. Traditional software projects fail because they deliver the wrong features too late. This is because they use the waterfall approach to software design instead of using $foo. If you use $foo, you'll deliver your software on time and on budget, you'll have fewer bugs, and you'll have fun doing it!
Like, gag me with a spoon! Books been coming up with new approaches, making the same promises, and criticizing the same waterfall model since the '70s!

Ok, here's a fun idea. Come up with a design book that doesn't criticize the waterfall model, but criticizes some other $foo technique instead, or come up with a book from within the last decade that actually thinks the waterfall model is still a good idea.

Saturday, July 11, 2009

IDE: NetBeans after Six Months

I've been using NetBeans for six months. I thought I'd whip out a quick post to tell you how it's gone. My earlier post is here.

I found a bunch more things I like. I finally figured out how to open files quickly. In the Projects window on the top left, click on the name of the project. Now, typing one letter at a time, you can get autocomplete. It took me a little while to figure out how it works, but now I can open files more quickly than I can visually parse things.

I also figured out that if I maximize the editor, and I shrink the size of the whole window, I can get the editor to fit in smaller places without taking up the whole desktop. That's helpful when I'm copying code I'm reading from a PDF to NetBeans even though I only have a 13" MacBook screen. (I like to manually copy code samples when I'm reading a book in PDF form because it helps me to learn the code better. I'm stuck with a tiny screen because I do a lot of my work at Starbucks.)

I still love jVi. It's not perfect, but it's pretty good.

I love the way NetBeans automatically fills in closing ", ], }, end, etc. for Ruby. This really speeds me up.

I don't like the fact that jVi can't wrap my comments and docstrings using gq} as well as Vim can.

I like having NetBeans launch my Ruby web server. It's one less window I don't need to worry about.

However, I still prefer to execute most things like rake tasks and ruby script/generate in the shell.

The coding hints aren't perfect. They're sometimes wrong, but that's not the end of the world. They're still a net positive.

I love using the find window when doing grep hunts. Since the window is open, it's easy for me to go down the list carefully making changes where necessary.

I'm happy with the plugin system. I didn't need to read any documentation, yet I've installed plugins for PHP, Ruby on Rails, jVi, etc.

In general, NetBeans is way easier to get up to speed with than Eclipse, which is the thing I like most about it.

Thursday, July 09, 2009

Agile Programming: I'm Stuck in the Middle

I found out yesterday, that at least according to Xmarks, my blog is far more popular than even I thought. However, before you think my head has gotten too big, let me tell you about two companies I interviewed with recently that turned me down. Twitter turned me down a couple years ago, and Aardvark turned me down a couple weeks ago (which is really a shame because I think Aardvark is awesome!).

Both of them said pretty much the same thing. They said I was too methodical and relied too heavily on documentation. More importantly, I'm sure, was that I didn't yet embrace test-driven development, TDD.

In TDD, you write your tests first, and then you write your code. However, ever since college, I've written my docstrings first, and then I write the code. I generally write the tests after I've written some code, and then I grow the two together. Although, I don't practice TDD, I am test infected, which means I feel uncomfortable about code that I've written unless I've also written tests.

I guess I've always been influenced by Knuth's literate programming. Apparently, someone forgot to tell me that literate programming is no longer cool.

My buddy Dan Aronson introduced me to a great programmer at Pivotal named Kelly Felkins. Pivotal is famous for being one of the most agile shops around. They practice TDD, they peer program pretty much all the time, etc., and they write good software. When I explained to Kelly my style of "docstring-driven development", he said I'd never make it at Pivotal. The other engineers would just delete all my docstrings. I think the idea is that in TDD, comments are to be distrusted, whereas tests are the true way of documenting code.

In real life, comments usually should be distrusted, but that's not the case with my comments. That's because in my style of docstring-driven development, I refactor the comments before I refactor the code. In the Python world, there's even doctest which lets you embed your tests right in your docstrings, which ensures that they are more likely to be current.

Until yesterday, I had never considered there to be a contradiction between literate programming and test-driven development. I just figured most engineers were too lazy to write both tests and documentation. As we all know, many engineers are too lazy to write either tests or documentation ;) Nonetheless, Kelly's arguments for TDD were pretty convincing. For instance, one of his best arguments was that TDD leads to much better test coverage.

It's interesting to note that not everyone likes agile or extreme programming (extreme programming is a specialized form of agile programming). For instance, Donald Knuth, the godfather of computer programming, said:
Let me just say that almost everything I’ve ever heard associated with the term "extreme programming" sounds like exactly the wrong way to go...with one exception. The exception is the idea of working in teams and reading each other’s code. That idea is crucial, and it might even mask out all the terrible aspects of extreme programming that alarm me.
Adam Wolff, who worked for several years on OpenLaszlo wrote a particularly insightful anti-agile piece:
Without even looking at the telltale URL, you can see these apps from a mile away. They still have the smell of the original auto-generated scaffolding, but more importantly, it's pretty clear that they were developed piecemeal...If you step back a bit, it's pretty easy to see the whole Web 2.0 thing as giant circlejerk of technology creators making it easier for mediocre developers to ship the next marginally useful application, while they struggle to implement a user interface that has the quality of a typical '80s PowerBuilder app...My second point is potentially more important: piecemeal approaches yield piecemeal solutions...In fact, many of the agile best practices reinforce short-sightedness and lock-in, even as they claim to emphasize nimbleness...The whole point of design is to incorporate the big picture in the little details, but when you're pursuing a "let's try it and see how it goes" approach, there is no big picture; there's just a bunch of stories.
If you've ever used an OpenLaszlo app, you know that he has a point. An OpenLaszlo app generally does a better job conveying the big picture. In general, OpenLaszlo apps put Ajax-based Web 2.0 apps to shame.

Perhaps user interfaces are one thing that shouldn't be designed in an agile way. One of the best user experience people I know, Stephanie Hornung, really drove this point home for me. Consider Amazon.com. It leaves me feeling overwhelmed and confused because there are too many piecemeal features vying for my attention. It's like an experiment in agile programming gone wrong. Stephanie argued that good user experience people aren't agile. They have to understand and plan for the big picture. I think this is one thing that Apple is particularly good at.

At the risk of using an overused metaphor, I don't think you can use agile techniques to build Gothic architecture:


(from Wikipedia)

However, most of us don't live in Gothic cathedrals. My own house is 100 years old, and has had at least five addons. It's a total mess, but unlike most Gothic cathedrals, it has central heating and air ;)

And just to be fair, even though Pivotal is an agile shop, their tool Pivotal Tracker doesn't feel like a piecemeal solution. It's easy to use, and the big picture is obvious.

The question remains, are agile programming and literate programming at odds with each other? If you're writing a Rails application, documenting the big picture is pointless. There's too much magic (aka convention) going on for in-line documentation to be helpful. The fact of the matter is, if you're building a Rails app, you need to read either a book or a tutorial ahead of time. Once you've done that, there's not much point in writing documentation that covers what you've already read.

However, not everything is a Rails app. For instance, Structure and Interpretation of Computer Programs, SICP, talks about the challenges of writing a memory manager for Lisp. Either you get it right, or you're completely screwed. Imagine trying to write a memory manager from scratch on a new hardware architecture. If you mess it up, your entire object graph is going to be toast. In such cases, I think it does make sense to embrace literate programming. I also think that working it out on paper ahead of time and heavy code review are necessary. Of course, just like Gothic cathedrals, not many people have to write memory managers for Lisp these days. Banging out Rails apps for Web 2.0 is far more the norm.

While I'm on the subject of SICP, perhaps it's telling that M.I.T. has switched from Scheme to Python for its first computer science course that students must face. Gerry Sussman said, "Nowadays, a real engineer is given a big software library, with a 300-page manual that’s full of errors...The engineer must learn to perform scientific experiments to find out how the software and hardware actually work, at least enough to accomplish the job at hand." Sussman pointed out that we may not like it this way (”because we’re old fogies”), but that’s the way it is, and M.I.T. has to take that into account. When you put it that way, TDD makes all the sense in the world.

Futhermore, Kelly pointed out that if you only write as much code as need to make the tests pass, you avoid overengineering a bunch of code that may not match the real world.

Where does that leave me? I'm going to use TDD for my Rails app, but I still maintain that a truly brilliant user interface must be developed like a Gothic cathedral. If you're interested in doing TDD for building a Web app in Python, I suggest some mix of Twill, Nose, and Selenium. If you're using TDD for building a Web app in Rails, Kelly suggested some mix of Cucumber, RSpec, and Selenium.

Happy Hacking!

Wednesday, July 08, 2009

Rails: Internationalization

I'm reading through Agile Web Development with Rails. I'm surprised at how naive the I18n support in Rails appears to be.

It seems to get number formatting right, but it gets a lot of other things wrong. For instance, as far as I can tell reading the book, it assumes all languages have the same pluralization forms as English. It forces you to create YML-based "constants" for all your localization strings. Hence, your templates contain things like I18n.t('checkout.submit') instead of, say, _("Submit your order"). I have no clue what happens if you're missing a constant in your YML. I also don't think it understands how to degrade gracefully. For instance, if I ask for some weird sub-dialect, I bet it doesn't know how to fall back to the main language. Furthermore, the YML files painfully duplicate strings for translation. I haven't seen anything for dealing with the Accept-language header, but that could just be the book's fault.

I may be misunderstanding some things, but I think the gettext library used in C, Python, etc. is much smarter and more convenient.

Please feel free to tell me how wrong I am. However, if you do, you must also explain how the heck Russian speakers can possibly cope with their crazy pluralization rules in real time ;)

Friday, July 03, 2009

Rails: Validating URLs

Here's a quick-and-dirty way to validate URLs in your model. Updated: Only allow certain schemes.
require 'uri'

class Film < ActiveRecord::Base

VALID_URI_SCHEMES = ['http', 'https']

validates_presence_of :url
validate :url_must_be_valid

protected

def url_must_be_valid
parsed = URI.parse(url)
if !VALID_URI_SCHEMES.member?(parsed.scheme)
raise URI::InvalidURIError
end
rescue URI::InvalidURIError => e
errors.add(:url, 'is not a valid URL')
end
end