Wednesday, May 23, 2012

Async: To Be or Not To Be

Just because I have to use a callback-oriented style on the client doesn't mean I want to use a callback-oriented style on the server. Now, before anyone gets all upset and tells me that I don't know the difference between async and a kitchen sink, let me explain :)

The client is necessarily an event-oriented place. If I don't know which button the user is going to press, it makes a lot of sense to use a different callback for each button. The server is different. If I'm waiting for the result of a database query before I can continue processing a request, it sure is convenient to just block and wait.

My key point is that it's important to separate what style you want to code with and what performance and scalability characteristics you want. You shouldn't necessarily pick a callback-oriented style just because you want the performance and scalability characteristics of asynchronous networking APIs.

My favorite two examples are gevent and Erlang, but Go is similar. When you code using gevent or Erlang, your code looks like synchronous, blocking code. However, below the covers, they use asynchronous networking APIs. Now, before anyone tells me that it's impossible, buggy, or that it'll never work, let me point out that these tricks have been in production for decades at Ericsson, Yahoo Groups, and IronPort Cisco.

Furthermore, I should point out that asynchronous networking APIs aren't a perfect fit for every problem. For instance, if your goal is to send 10 gigabytes of information to another server, it turns out that synchronous networking APIs will actually outperform asynchronous networking APIs. The reason asynchronous networking APIs are so popular is because they can handle a larger number of clients than synchronous networking APIs can and because they use less memory than a large number of threads, which each have to have their own stack. gevent and Erlang can handle a large number of clients, don't use up much memory, and don't require a real OS-level stack per client.

So what's my problem with the callback-oriented style? I find it a lot harder to read. I've coded projects in Twisted, Node.js, etc., and I prefer the gevent approach. You get roughly the same performance and scalability characteristics, but with much easier to read code. Of course, what's readable to me may not be readable to other people. I've met people who are perfectly happy using Twisted Web 1 and don't think that callback-oriented code poses any real challenge.

If you're interested in hearing more about my thoughts on async and concurrency, check out my other blog posts, which include a link to my Dr. Dobb's Journal article on Python concurrency.

Tuesday, May 22, 2012

HTML5 Dev Conf

I went to HTML5 Dev Conf yesterday. The following is a loose collection of my notes.

First of all, the place was packed! Even the second-tier talks didn't have any standing room left over.

The Backbone.js talk was especially packed. Not only could I not get into the room, I couldn't even get anywhere near the door. In general, Backbone.js was an extremely hot topic. However, most people admitted that they hadn't used it.

Douglas Crockford's Keynote

JSON has no version number. "Everything with a 1.0 gets a 1.1 and a 2.0, and it's all crap until you get to 3.0, so JSON has no version number."

He talked about the first browser, Mosaic. Some company patented an <embed> tag to embed content in a rectangle in the browser. He said that the company was a patent troll, and they went after everyone trying to make money. They figured their patent covered JavaScript, image tags, etc. Apple settled. M$ fought them for years. Yahoo finally led the defense and won.

Crockford was using an Alienware laptop.

Crockford is a calming speaker, but it's easy to overlook the fact that he has strong, unusual opinions.

Crockford cancelled his Facebook account because of their privacy policy.

Crockford is against software patents (and possibly against all patents?).

Yahoo sued Facebook because of patent infringement. That's why he left Yahoo. He works at PayPal now.

Crockford spent some time bashing Yahoo. However, he was also hopeful that the company could be turned around.

He said Yahoo had a globally scalable NoSQL database in 1995.

"Yahoo is profitable and has always been profitable."

Yahoo's business is to do "Anything [on the web] for a dollar."

Yahoo is organized into properties. It's 100 companies that share a cafeteria.

They "misuse" agility.

Crockford kept answering his cellphone during his talk. At first I thought it was a joke, but I'm pretty sure it wasn't. It wasn't even someone important.

He doesn't have a degree in CS. He has a degree in broadcasting.

He thinks that Yahoo should:

  • Optimize for customer value.
  • Create a culture of excellence ("Yahoo doesn't have that.").
  • Fork Node.JS and create Ynode.
  • Desilo the company.
  • Focus on improving and integrating Yahoo Groups (it's basically 12 year old software).
  • Fix the research group.
  • Bring back Jerry Yang as chief Yahoo (instead of CEO).

He ended the talk by saying, "Don't forget your semicolons!"

He called himself "Doug Crockford".

He said, "I love CoffeeScript," but also expressed certain reservations about it.

Behind the Scenes of "Cut the Rope"

This talk was given by Giorgio Sardo who does Windows evangelism at Microsoft Corp.

He showed PacMan on an really large board (80,000 user generated boards all connected).

"IE9 Hardware Accelerated HTML5."

"Cut the Rope" was originally a native iOS application. They rewrote it in HTML5.

It was 40,000 lines of code. They translated it directly to JavaScript. They went from OpenGL to Canvas.

They're using setInterval for animation. They have one image with all the different animation states.

The rope was the hardest part. It's made of 20,000 triangles. There's another 10,000 triangles for the shadow. It's easier on HTML5 because they can use bezier lines.

He used a JavaScript profiler to fix his performance problems. It saved the project.

He's using Microsoft Visual Studio (of course).

He's using jQuery.

They worked on the project for 12+ weeks.

The original game developers are in Russia.

Windows 8 makes it easy to take a web app and make it a desktop app.

There's a Windows 8 store.

Benefits of CoffeeScript

For some reason, the speaker didn't show up. I offered to give an introduction to Dart. Someone else offered to give an introduction to CoffeeScript. After doing our respective short introductions, we held a panel-like discussion in which we answered questions from the audience.

CoffeeScript has an emphasis on conciseness, and it is inspired by Python and Ruby. Dart has an emphasis on building extremely large, client-side applications, and it is inspired by JavaScript and Java.

I had always heard that CoffeeScript had a weird scoping issue. If you have a local variable, and someone introduces a module-level variable of the same name, your local variable will start referring to the module-level variable instead of shadowing it. The CoffeeScript panelist said that he worked around this problem by using a different version of CoffeeScript that has ":=" syntax for setting non-local variables.

CoffeeScript places an emphasis on translating to JavaScript in very predictable ways. Dart places an emphasis on avoiding "puzzlers" (such as those present in the "Java Puzzlers" book).

CoffeeScript generates JavaScript that it is easy to debug. Dart can be debugged natively using a version of Chromium that has Dart support built in.

CoffeeScript takes JavaScript and makes it "even more dynamic". Dart takes JavaScript and adds optional static typing in order to facilitate building larger projects.

Overall, the panel discussion was friendly and enlightening.

Cross Platform Mobile and Web

This was a talk from a guy at LinkedIn on how they handle developing for the web and multiple mobile platforms.

Unfortunately, I missed the first 15 minutes.

Their iOS application is a mix of native and web code. It's 20% native code and 80% web code.

They use native code for all infinite lists because it's too hard to do infinite lists using web code.

They decided PhoneGap was not a good fit for them. One of the reasons was that they wanted to decide whether to use native code or web code on a case-by-case basis.

On Android, they wrote more native code than they did on iOS. About 80% of the code was native. However, they're slowly transitioning more toward HTML5.

For testing, they're using many projects, including: Vows, Robotium, Selenium, FoneMonkey, GHUnit, Hudson, PhantonJS, and some internal stuff.

They ship everything twice a week.

The apps and server deploy independently.

They're using Node.js. They upgrade it frequently.

A single Node.js instance can achieve 150qps and only requires 50mb.

LinkedIn uses Java, Scala, Python, Ruby, Node.js, etc.

IntelliJ can step through Node.js code thanks to the WebKit debugger.

They don't have a good debugging solution for mobile. They rely on logging.

They use MVC on the client side.

A "screen" is a "div to div move."

A "page" requires a "whole new browser page load."

A single page can have multiple screens. Each screen has a controller and a model.

They want to reduce the number of frameworks they use and increase the number of libraries they use. A framework calls you, whereas you call a library.

They use Backbone.js. It:

  • Provides routing from screen to screen.
  • Controls screen lifecycle.
  • Controls view updating based on model changes.
  • Has model construct for validation.

They're very tied to Backbone.js.

They use several libraries including: Zepto (to manipulate the DOM), iScroll (for inertial scrolling), and Underscore (which provides collections helpers, binding constructs, and templating).

They use SASS.

They embrace raw JavaScript instead of using CoffeeScript, etc.

They use Closure to minify and optimize their JavaScript. It also compiles their templates.

A bundle is a list of screens, templates, etc.

They cache bundles, images (?), etc. into local storage. They use MD5 hashes for files, and they only have to download diffs of files when they make changes. They said that mobile browser caches were very fickle, so they prefer to use offline storage to manage their own cache.

They use Jasmine as well as Selenium.

They use Capistrano.

Responsive Web Design: Conquering the Web, One Screen at a Time

This talk was mostly geared toward designers.

Responsive Web Design emphasizes web designs that work well on a variety of web and mobile devices. It came out of the "fluid layout" movement. However, these designs use CSS and other tricks to adapt themselves to look good everywhere, even on mobile.

Break points define when you should switch to a different layout. Try resizing your window drastically to see if there are break points that cause the application you're using to switch to a different layout.

Layouts are the thing that change the most. However, you might want to change the type (i.e. font) as well.

On mobile, you should probably use 14px Arial for your font.

You might want to change which image you show based on the screen size. The way to do this hasn't completely been resolved. There's some new "picture" element they're playing with in the standards bodies (the W3C?).

"Starting with the desktop may be an increasingly backwards way of thinking about a Web product." -- Luke Wroblewski

He said half of your users are mobile.

Opera is the most popular mobile browser world wide (I assume this refers to cellphones that aren't running Android or iOS).

He said QA is very important. You have to test on many different devices.

Focus on your content!!!

He claimed that Flash doesn't work reliably on Android.

mediaqueri.es has a lot of examples of sites that use responsive web design.

There is a fold, but ignore it. Focus more on horizontal sizes, not vertical.

These are the screen widths you should consider: 320px (important), 480px (sideways iPhone), 640px, 768px (important; older iPads), 960px (important), 1184px, and 1400px.

Sweat the small stuff such as type (i.e. font), navigation, and links.

44px is a good rule of thumb for a nice navigational link.

Design in tiles. Tiles should be flexible. They can be swapped.

If you want to learn more, search for "media queries".

See also developer.mozilla.org/en/CSS/Media_queries.

Media queries let you specify different CSS for different situations.

Examples:

@media screen and (max-device-width: 810px) {
  div {
    background: blue;
  }
}

<link rel="stylesheet" media="screen and (color)" href="foo.css">

There are many media features you can use such as width, device-width, orientation, color, etc.

See also: responsive.is

"Working on the web...is defined by its transcience, often refined or replaced within a year or two." -- Ethan Marcotte

He uses SASS. He doesn't like LESS because it's based on JavaScript.

Twitter has a framework called Bootstrap. He doesn't use it.

Adding a Server-side Backend to Your JS-powered App

This was a pitch for StackMob.

StackMob, in my own words, is a platform as a service offering for pure client-side applications written in JavaScript. They provide the server-side infrastructure so that you don't have to.

StackMob uses Backbone.js.

He said that when you use the Facebook API and the Twitter API, it's easy to forget that those companies have massive backends.

StackMob's goal is to create a server-side backend for your JavaScript-powered application.

They provide many backend services such as user auth, a datastore, security, GeoSpatial APIs, push support, static file hosting, social integration, custom server-side code, and analytics.

They're a "mobile backend platform".

They are "giving everyone access to 'big company' backend power."

They are "lowering barrier of entry."

The backend is accessible via a JavaScript API built on top of a RESTful API.

It's based on OAuth2, Ajax, and Backbone.js models.

They've integrated persistence to StackMob into their Backbone.js-based API.

They have a nice looking dashboard.

It looks like their datastore is schemaless.

They use a mix of MySQL and MongoDB.

You can fetch an object as well as all of its related objects, up to depth of 3. I'm worried about the security implications of retrieving things you shouldn't have access to.

I'm also worried that someone can hack your JavaScript to do things that your app wouldn't want you to do. I don't think StackMob has any way to differentiate trustworthy JavaScript from non-trustworthy JavaScript.

They're using OAuth2.

Public/Private keys identify your app's requests.

They're going to add access controls "very soon". This would allow you to share access to certain objects.

I really don't think they have a viable, comprehensive security model yet.

They're a Java/Scala shop.

They have a way for you to run your Java on their servers.

They don't have an approach to server-side validation in the SDK yet.

If you use a password field, they will explicitly not give it back to you when you load the user object.

They deal with cross-domain issues by hosting your static files.

They support custom domains.

They're integrated with GitHub. I.e. they can update your code when you push to GitHub.

They give you separate database environments for development and production.

They encrypt your data before storing it in the database.

It's free for limited use. They charge for higher volume use.

They have support for using S3.

He used reveal.js for his slides.

0-60: Animating the Web

This talk was given by Jeremy Kahn who works for YouTube. (I've never met him.)

Animation is used to draw a connection between two separate things.

"Flash is not the future of the web."

CSS3 has powerful APIs for smooth animations on the web.

Ideally, we want 60 frames per second. Flash's default frame rate is 12fps.

Tweening is a technique where you move from a start position to a stop position over a period of time. The goal is to let the computer do the hard work of making the animation between the start position and stop position work. Tweening is a popular technique in Flash. It's also supported by jQuery's animate function.

Keyframing is a little more complex than tweening. Keyframing lets you have any number of points in between the start and the stop.

Here are examples of the two techniques using jQuery:

// Tweening
$ell
  .css({ left: -700 })
  .animate({ left: 0 }, 2500);

// Keyframing
$ell
  .keyframe(0, { left: -700 })
  .keyframe(2500, { left: 0 });

Do not use setInterval for animation. If your callback takes too long, all sorts of bad things will happen.

It's better to use setTimeout.

It's even better to use requestAnimationFrame. It tries to call itself 60 times a second.

When you think about requestAnimationFrame, take into consideration the time delta, and make sure you have a terminating condition.

You can use MVC informally to break up your animation code into separate functions.

You can do animation using only CSS3 (i.e. without using JavaScript).

There is CSS syntax for keyframes (it uses @).

CSS has support for easing.

Ceaser is a tool for visually generating bezier curves for CSS easing.

Use different easing functions for X and Y to get interesting curves in your animations.

JavaScript animation:
Pros: flexible, runs anywhere
Cons: Everything lives in the JavaScript thread

CSS animation:
Pros: Better performance / smoother (it uses your computer's graphics card)
Cons: Limited easing formula choices, not supported everywhere

The difference in performance is huge on mobile devices.

Minimize reflows.

One approach is to use "position: absolute".

A better approach is to use the transform property.

Animating <canvas> requires different techniques.

Avoid clearing the canvas on every animation frame. This is the number one performance killer.

There are a couple techniques that help with <canvas> animations--pre-rendering and using multiple canvases (one for the background and one for each shape).

CSS transforms can use sub-pixel rendering.

When you scale stuff, CSS doesn't smooth out stuff once you get above 1.0. Hence, always scale up to 1.0. Try not to exceed 1.0.

Here is a list of CSS tools: Ceaser, SASS mixins

Here is a list of tools for JavaScript tweening: jQuery, Zepto, Shifty (which he wrote)

Here is a list of keyframing tools: Rekapi (which he wrote; it's a model and controller for keyframe animations)

Here is a list of graphical tools: Sencha Animator, Adobe Edge, Stylie (which he wrote)

HTML5 is still trying to catch up with Flash when it comes to authoring tools.

Should you use JavaScript or CSS for animations? He recommended using JavaScript to control CSS animations.

HTML5 is still trying to catch up with Flash's authoring environment.

He used Keynote for his slides. Keynote has some cool transitions.

Hype is an HTML5 authoring tool.

You should separate your state logic and your drawing logic.

Mobile Web Debugging and Performance

The speaker works at Oracle. He uses PhoneGap.

He wants to replicate the experience that Developer Tools provide on Chrome, but in a mobile context.

There's a JavaScript debugger on iOS. It shows errors. It's very limited.

The same is true in the debugger for Android. It's a "little better than iOS."

WebKit supports remote debugging. Hence, you can debug an app running on the simulator.

Chrome for Android 4.0 and above has USB debugging. This lets you debug on a separate port. It looks like Developer Tools on Chrome. Search for "Chrome for Android remote debugging".

abd = Android Debug Bridge

See: code.google.com/chrome/mobile

This doesn't currently work for WebView widgets because those don't run in Chrome.

"The native WebKit browser is the default for WebView apps, and it's way behind the curve."

Opera also supports remote debugging. You need Opera on your desktop and Opera mobile on your mobile device.

Firefox Mobile also has remote debugging, but it's brand new. It was developed at a hackathon.

iWebInspector is a web debugging tool for the iOS simulator. It was written by the guy who wrote jQuery Mobile.

Remote JS is a tool from Sencha Touch. It's tool for Android applications that use WebViews. You don't have to use Sencha Touch in order to use it.

Wiener (see the Apache Cordova Project) is another tool for doing remote debugging. However, it's somewhat limited.

jsconsole.com is a JavaScript console that you can add to your app.

Socketbug is a project written using Node.js and Socket.IO. It's another tool for mobile JavaScript debugging. It supports iOS, Android, and Palm.

Adobe Shadow is a commercial tool for mobile JavaScript debugging. It's currently free. It allows you to control multiple remote devices at the same time.

blaze.io has a free mobile web performance testing tool called Mobitest.

MobilePerf is another mobile web performance testing tool.

Friday, May 18, 2012

Code Reuse has Finally Arrived...Obviously!

I got into a discussion about code reuse with my co-worker, Jarek. I remember reading books on object oriented programming from a few decades ago in which OOP promised to increase code reuse. A lot of people doubted that serious code reuse would ever happen. OOP suggested that code reuse would be possible by reusing and subclassing existing classes. I even remember hearing C++ programmers talk about purchasing libraries of useful classes.

To some degree, this did actually happen. For instance, QT is a wonderful library for C++, and it certainly makes coding in C++ a lot easier. It was even proprietary up until a few years ago. Library reuse also happened via language-specific package sites such as CPAN (for Perl), PyPI (for Python), and a range of sites for Ruby. Furthermore, most languages come with a lot more stuff built in these days. For instance, compare Python's standard library ("batteries included") with that of C.

However, compared to decades ago, there is less of an emphasis on class-level reuse these days. For instance, rather than trying to code classes that will be flexible enough to solve tomorrow's problems, agile programmers suggests you write your code in such a way that you will be able to modify it to solve tomorrow's problems (with an emphasis on tests that will prevent regressions when you modify the existing code).

These days, rather than focusing on class-level reuse, we've achieved reuse via RESTful web services (and variations thereof) as well as via open source stacks. RESTful web services are usually closed source and proprietary. However, they have a lot of benefits. For instance, they're language agnostic. It's also a lot easier to control access to a RESTful Web Service than it is to a class library. This permits good authorization controls and various monetization strategies.

As I mentioned before, another place where we see a lot of reuse is in the open source stack. Just as Taco Bell has a million things on the menu that are all made from the same 10 ingredients, there are a million startups out there all based on the same stack of open source software. We even have a word for it--LAMP. Sure, there are variations of this--PostgreSQL instead of MySQL, Nginx instead of Apache, Python and Ruby instead of PHP--however, the fact remains that a lot of startups are only possible because of the tremendous amount of code that is available for reuse.

Hence, it's clear that code reuse arrived during the last 10-20 years. It didn't come exactly as we expected, but it's definitely here.

Documentation Sucks, Video Games Rule!

One thing that's great about a lot of video games is that you don't have to read any documentation in order to start playing them. I tend to play games on the Wii with my wife, and those games tend to teach you what you need as you go along.

I think the same thing applies to user interfaces. Having to read documentation kind of sucks. Being able to learn as you go along is much nicer. A user interface that is easy to get started with is great, and one that teaches you on the fly how to become a power user is even better. For instance, I think that menus are a great way to enable users to discover new features. Menus that also show you the relevant shortcut keys are great because you become a power user by gradually memorizing frequently used commands. A user interface that does useful things automatically (assist and suggest) is the best of all (for instance, code assist).

Another thing I feel strongly about is that a piece of software doesn't have to be hard to use in order to be powerful. Compared to raw SQL, spreadsheets are very easy to use. However, they're also very powerful.

I continue to be a fan of Vim. However, when I first tried TextMate, I was amazed at how well its menus were organized. It even had a system for organizing the actions provided by plugins. This permits you to learn the advanced features (even of plugins) as you go. In fact, I was able to do stuff in TextMate that I still don't know how to do in Vim. I'm sure that Vim doesn't lack such features; it's just that it was a lot easier for me to discover how to use them in TextMate.

These days, I've been using Sublime Text 2. One thing that it can do that I really like is automatically suggest word completions as I type. The visual feedback is very discoverable. The Chrome location bar does the same thing.

Wednesday, May 16, 2012

Dart in Glasgow, Scotland

I gave a talk on Dart in Glasgow, Scotland. A big round of thanks go to my friends at Moviecom.tv and 29studios for not only setting up the talk, but also recording and editing it.

Tuesday, May 15, 2012

Unit Tests Don't Find Bugs: the Death of QA

Unit tests don't find bugs. They find regressions. This is a painful lesson I learned when I first started doing TDD (test-driven development), and it's well known among most TDD circles.

TDD's goal is to prevent programmers from introducing new bugs into working code. However, when you're writing code from scratch, your tests won't help you find all the bugs in your code. That's because you can't possibly write tests for all the ways your software will be used (or abused). When I first started doing TDD, I had really good tests, but I was too tired to do much exploratory QA. However, my boss wasn't, and I was very embarrassed to find that my software had lots of bugs. Simply put, he used my software in ways that I hadn't intended.

I've seen a lot of companies that don't bother writing any tests or doing any QA. They just let their users find all the bugs. Needless to say, I've never had respect for those companies.

However, it's growing more and more popular to destaff the QA department and just require engineers to write lots and lots of tests. Often, these are in the form of unit tests. Even though integration tests can conceivably catch more bugs, they take much longer to run. Hence, even integration tests are often deprioritized.

What I'm discovering is that a lot of projects have both lots of unit tests and lots of bugs. These are bugs that could have been found manually by a QA engineer, but it seems that manual QA testing (i.e. exploratory testing) has gone out of vogue.

I used to think that code that was well-documented, well-styled, well-tested, and code reviewed would rarely have bugs. Sadly, I no longer believe that to be the case. I think we need to go back to the days when we had decently-sized QA departments, perhaps in addition to all the other things we do.

To tweak what Knuth said, "Beware of the above code. I have only tested that it works. I haven't actually tried it."