Two weeks ago Bruce and I released JSGameBench version 0.1. Today marks the release of version 0.2, a much faster and cleaner version. We continue to learn both from tightening the code and from the strong HTML5 community. Version 0.2 reinforces our belief in HTML5 as a strong, horizontal platform for games and highly interactive applications across the web.

Benchmarking

In order to talk about browser performance, we needed to standardize. We now have two machines that will be our testing machines:

• For OS X: a MacBook Pro laptop, currently OS X 10.6.6, 4GB of RAM, 2.66 GHz Intel Core i7, and NVIDIA GT 330M with 512MB of RAM.
• For Windows: a Lenovo T410s laptop, currently Windows 7 Enterprise, 4GB of RAM, 2.53 GHz Intel Core i5, and NVIDIA NVS 3100M with 512MB of RAM.

Both of these laptops are significantly less powerful than the Mac Pro the original tests were run on. In addition, the 3100M offers approximately half the performance of the 330M.

More is Better

Thanks to help from the HTML5 community, JSGameBench has roughly doubled performance in the most widely used browsers in only two weeks. This a very promising development for the future of HTML5 games and interactive apps. The two most significant changes are clamping positions to integer values and using the transform/transform3d style property.

Also, despite the far less powerful graphics cards and slower CPU in the Lenovo 410s, Win 7’s hardware accelerated browsers -- Firefox 4, Internet Explorer 9, and Chrome 11 -- continue to dominate, achieving scores above 1,000. Our new leader, Firefox 4, is well ahead of the pack.

Some Thank Yous

First, thanks to Bruno Garcia, a developer at Three Rings, who submitted a pull request clamping position values to integers. An obvious improvement for software render pipelines, but one we hadn’t thought to try. For axis-aligned sprites, clamping to integer values was faster in every browser except for Chrome 11 on Windows. A huge win and the majority of the performance gains. Thanks, Bruno!

Second, thanks to Zynga’s Laurent Desegur, Paul Bakaus and Rocco Di Leo. At the HTML5 tech talk, we had a chance to talk about HTML5 performance and some of their ideas. Paul also blogs extensively about HTML5 tricks.

Desktop Recommendations

For the desktop browsers, the following techniques generated the highest frame rates:

• Snapping positions to integer values rather than using floating point values. JavaScript supports several different ways to snap, such as “parseInt” and “| 0”. We would expect using the or operator to be faster than a function call, but have not yet tested this. Note that snapping could generate visible artifacts, particularly at very low resolutions.
• Using the DIV tag with background image instead of IMG tag. The difference is particularly noticeable when using individual sprites instead of sprite sheets, as changing the IMG’s src property is very slow.
• Many individual sprites are faster than sprite sheets in nearly every case. The exceptions were the fully hardware accelerated browsers on Windows, where sprite sheets offer a 5-10% advantage. Unless you are only targeting those browsers, we recommend staying with individual sprites. For browsers with the CANVAS tag, we expect an optimum balance of download and draw performance is to download sprite sheets but then use CANVAS to cut them up.
• Update DOM elements rather than using innerHTML to load the parent. For several browsers, Firefox 4, Chrome 10 and 11, and IE9, innerHTML was faster, particularly for rotated sprites. However, for those browsers CANVAS tag performance was far superior to DOM manipulation, so update and switch to CANVAS when possible.
• For axis-aligned images with integer position values, the CANVAS tag was superior across all desktop browsers. This had not been the case with floating point values -- and indicates software blitting in their render pipelines -- but if you are snapping and drawing axis aligned images, use CANVAS when you are able to.
• For browsers that support 3D transforms -- we are currently testing on Chrome and Safari -- using translate3d(x,y,z) is equal to or faster than translate(x,y).
• On desktops, using CSS transitions for motion or CSS keyframes for animations were slower than simply using JavaScript for these tasks. Worse, they often generated noisy framerates, so they are not a good solution for games in desktop browsers.

In order to reduce JSGameBench’s testing time, we have chosen to use these recommendations -- positions snapped to integer values, DIV tags with background images, updating DOM elements, 3D transforms when possible, no CSS transitions or keyframes -- as our default test. One further tweak was to settling on the transform property “rotate” since “rotate“ versus “matrix“ generated no difference. We believe that HTML5 game developers and browser developers will gain the most if we optimize the smallest number of render paths, rather than constantly testing every possibility.

Mobile Differences

We thought the most interesting part of the HTML5 tech talk was how different our performance test results were. Much of this week was spent exploring where those differences might have come from, beyond normal testing variance. We believe the single largest reason for differences in our results comes from the iPhone -- or more specifically, Mobile Safari running on the iOS ecosystem. While many HTML5 techniques work well on Mobile Safari, truly wringing maximum performance will require more special case code than any other browser.

We have primarily been testing desktop browsers so far, but what follows are our preliminary points of data about iOS.

CSS

For Safari, both desktop and mobile, CSS transitions and animations are hardware accelerated. As we previously discussed, this acceleration is not a win on desktop, but on iOS, it generates intriguing results.

CSS transitions are pretty simple. For a given CSS property, such as position, CSS transitions provide a method to interpolate between the current value and the new value. For example, on a Webkit browser the following CSS:

#Foo {
  -webkit-transition: left 0.1s linear;
  left: 45px;
}


This will interpolate between Foo’s current left value and “45px” in 0.1 seconds using linear interpolation. (The Art of the Web has a nice discussion and demos of transition timing.) Generally, CSS transitions work for smoothly varying a CSS property.

CSS keyframe also provide a different approach, allowing you to set keyframes for a given property. Normally, those properties would still be smoothly interpolated, but as Paul Baukus discovered (and refined), by exploiting numerical precision limits in the browser, it is possible to snap between values.

For example, to use keyframes to run a 2 frame animation of the flickering drive flame on our spaceship, you would use a CSS keyframe like this:

@-webkit-keyframes 'ship' {
  from { background-position: 0px 0px; }
  49.99% { background-position: 0px 0px; }
  50% { background-position: -128px 0px; }
  to { background-position: -128px 0px; }
}


This example animates the background image position. To accomplish the same effect using a DIV element masking an IMG element, you would instead use:

@-webkit-keyframes 'ship' {
  from { -webkit-transform: translate(0px,0px,0); }
  49.99% { -webkit-transform: translate(0px,0px,0); }
  50% { -webkit-transform: translate(-128px,0px,0); }
  to { -webkit-transform: translate(-128px,0px,0); }
}


In both cases, you would associate the animation with the DIV via:

img.ship_animating {
  -webkit-animation-name: 'ship';
  -webkit-animation-timing-function: linear;
  -webkit-animation-duration: 0.25s;
  -webkit-animation-iteration-count: infinite;
}


Using HTML5 techniques form the desktop, both iOS and Android return values in the 20-30 sprites range, using a 20 fps baseline. By using the DIV with background position approach, performance on the iPhone increases by 25% or more, warranting further research.

Taking advantage of the second CSS keyframe technique -- using a DIV to mask an IMG tag -- generated similar performance increases, but began severely stuttering JS execution. We found similar results using CSS transitions to move the game objects: the JavaScript thread would execute erratically, likely as a result of the UI and render threads on iOS being higher priority.

CSS Only?

Viewing the thread starvation led us to try a further experiment. Rather than relying on JavaScript to update object positions, what performance levels are achievable if all of the sprite motion is in CSS? The results were impressive, with the iPhone 4 easily pushing over 100 moving, animating, and rotated sprites. Unfortunately, this level of performance is achieved at the expense of JavaScript execution, with the JS thread getting called at an erratic 2-5fps.

Is it possible to build a game using this approach? Almost certainly, but it is going to be a very quirky development path and iOS specific. Still intriguing, however.

Browser Improvements

As always, version 0.2 brings a list of requests to browser manufacturers:

• JavaScript control of screen rotation, including the ability to lock to landscape or portrait. With mobile, auto rotation is incredibly disruptive to interactive apps.
• Although the CSS tricks are interesting, the broadest win for HTML5 as an effective interactive platform is for browsers to expose hardware accelerate render pipelines to JavaScript directly rather than forcing developers to attempt to contort their game engines into JS/CSS hybrids.

That’s All for Now

We will continue to improve and refine JSGameBench in the coming weeks. We’ve begun to reorganize the code to better serve as a reference design for JS games and have a list of tasks ahead of us. We continue to be sure that we’ve missed good ideas and look forward to continued feedback. HTML5 continues to impress us and we are excited by what we will bring to JSGameBench next.