Google has created a new suite of benchmarks called Octane for testing a browser’s JavaScript performance.
The new benchmark suite supersedes their older V8 benchmarks by adding a number of real-world tests to the suite. The added tests adapted from on current JavaScript applications and libraries. The new benchmark is also designed to be run from tablets or mobiles.
The newly added benchmarks are:
- Pdf.js: This is a PDF rendering library by Mozilla, and it is being integrated in Firefox as the inbuilt default way to view PDF files.
- Box2DWeb: This the the JavaScript port of what was originally a C++-based 2D physics library. It is used in popular games such as Angry Birds.
- GB Emulator: This is an open source GameBoy Emulator written in JavaScript.
- Mandreel: This is a compiler that converts C / C++ and OpenGL code to web-ready code. It is used to covert the C++-based 3D Bullet Engine to JavaScript.
- CodeLoad: This benchmark tests how fast the browser can get popular JavaScript libraries such as Closure and jQuery up and running.
By including real-world applications in their tests Google aims to have a benchmark that tests the real performance of the browser rather than an artificial number based on features that are rarely used. This is similar to what Mozilla hoped to achieve with Kraken.
Benchmarks are a tricky business, since they can be as distracting as they can be useful. Consider the Acid 3 benchmark which tests certain features that have little practical use. Browsers that focussed on brining their Acid3 scores to 100/100 did so just to get a better score, not because the features being tested had any practical use. This is why Firefox didn’t score a perfect 100 for quite some time. The issue is when browser vendors focus on getting better scores in benchmarks while ignoring real-world issues.
Which is why it matters what a benchmark tests, since that is what is being used
as a measure of the performance of the browser. If the tests really do represent real-world usage, then that is what the browser is optimising itself for, otherwise all you get is a browser that is better at running that benchmark.
Even Google’s latest benchmark falls into the same trap, it measures only JavaScript performance while few applications are purely JavaScript based. Most applications will actually render content, and those operations are not tested by these benchmarks. Of course as a JavaScript benchmark, it is supposed to test the real world JavaScript performance; our point is only that this is not the only important aspect of a browser’s performance.
You can find our more about the Octane benchmark at its Google Code project page, or you can see how your browser performs on it by clicking here.
