Measuring ASM.JS performance

What is ASM.JS?

Now that mobile computers and cloud services become part of our lives, more and more developers see the potential of the web and online applications. ASM.JS, a strict subset of JavaScript, is a technology that provides a way to achieve near native speed in browsers, without the need of any plugin or extension. It is also possible to cross-compile C/C++ programs to it and running them directly in your browser.

In this post we will compare the JavaScript and ASM.JS performance in different browsers, trying out various kinds of web applications and benchmarks.

About the tests

All tests can be run in your browser, most of them are fully automatic and don't need any special setting.
Some benchmarks provide score as result, while others measure the required time to complete a test. In every case where the site didn't provide confidence intervals, the test was repeated five times and it was calculated manually. Some tests run until you quit them, for these usually the average frames per second was taken.

Test environment

The system configuration used for testing:

  • Intel Core i5-2320 3GHz
  • NVIDIA GeForce GT 440
  • 8GB RAM
  • Ubuntu 14.04 and Windows 7 Professional

The tests ran on the latest official releases and on a build of EFL MiniBrowser:

  • Firefox 31 (Linux and Windows)
  • Chrome 36 (Linux and Windows)
  • Chromium 34 (Linux only)
  • EFL MiniBrowser r171296 (Mon Jul 21)

Chromium has no official Windows build (unless you count Chrome as one), and EFL WebKit currently has build issues on Windows, so they were tested only on Linux.

General JavaScript tests

First, let's test how these browsers perform on the usual JavaScript tests. Some of these sites already contain asm.js code too, so we can get a good image about how well they can handle most of the scripted websites. The following benchmark suites were used:

The first set of tests results are scores, so higher values are better, while others show the required times to complete the test, and lower values win there.

General JavaScript test results

As you can see, in most cases Blink/WebKit based browsers win here, often with significant difference compared to Firefox.

Computation performance tests

Now let's see how well can browsers handle asm.js code. The following tests are simple handwritten scripts, they show the number of operations per second for each test case.

And the results:

Hadwritten ASM.JS test results

These tests show a surprising difference between browsers. In some of them Firefox has horrible performance, then in others it is twice as fast as the competition. It seems the results depend on too many things, so we can't clearly name a winner here.

Emscripten tests

The embenchen benchmark suite is a collection of performance testing codes (micro) and real-world examples (macro). They were converted to asm.js using Emscripten.
This set of tests measures how fast can the different browsers finish the benchmarks, so lower numbers are better.

The benchmark was run 5 times. This table shows the averages for each test, and you can also find the 95% confidence intervals in the full statistics, attached to this post.

Emscripten test results

In this set Firefox wins for almost every test. While the difference in the micro benchmarks is not too big, the macro tests (box2d, bullet, lua_binarytrees, zlib) run significantly faster on Firefox. These complete 2-3 seconds sooner than in other browsers, meaning about 20-30% performance increase in most cases.

Multimedia and real-world uses

The most well known use of asm.js is the many web applications, converted from native code using Emscripten or other softwares. Most of these also use WebGL, as this technology allows developers to port every kind of graphics softwares, including 2D and 3D games, emulators and so on. It is important to note that by using advanced graphics, the computer's video card and driver software can heavily influence the user experience - and the test results.

Multimedia test results

These tests show the real power of asm.js and what can be done by using it. The winner is Firefox here, and depending on the platform (Linux or Windows), Chrome has nice result in some cases too.

Conclusion

After comparing the results of the tests, what we can see is that Firefox performs better on asm.js specific benchmarks and tests, while Blink/WebKit based browsers usually slower. On the other hand, Chrome and Chromium leads in general JavaScript benchmarks.

While the JSC performs better on traditional JavaScript code, if we want to get full speed on the latest web applications, it might be wise to implement asm.js optimization in some form. Personally I think this should go together with improving WebGL. As the browsers slowly become gaming platforms too, asm.js and WebGL performance will be an important value for those who are in search of a better browser.

 

AttachmentSize
asmjs-statistics.zip88.15 KB

Till Schneidereit (not verified) - 09/27/2014 - 13:56

I didn't look into the other benchmarks, but note that the "Computation performance tests" are not testing what they say they do.

If you open the console in Firefox while running one of them, you see lots of warnings saying "Using //@ to indicate sourceURL pragmas is deprecated. Use //# instead". Those warnings are logged each time a script with a deprecated style of supplying original source information for compiled-to-JS code is used. The *content* of the warning isn't actually interesting here. What is interesting is that you get hundreds to thousands of them per run of each benchmark.

So what the benchmarks are really testing is performance of parsing, compiling, optimizing and then running the code. Firefox's asm.js-optimizing OdinMonkey does all optimization up-front, so the startup costs are a bit larger than in other engines (including Firefox without "use asm"). OdinMonkey does this for two reasons:
- because it needs to verify that a program adheres to the asm.js spec, so that it can rely on certain guarantees at runtime.
- to guarantee stable performance at runtime, without intermediate slowdowns from the JIT compiler. This is crucial for games, where stable frames per second are extremely important.

Now obviously, almost all or all real-world code will be compiled once, not thousands of time, making the benchmark far less interesting than it appears to be.

Additionally, OdinMonkey usually optimizes for the cases of repeated compilation by storing the compiled code in the browser cache. That usually means that repeated startups boil down to just loading the code from the cache. I guess that mechanism doesn't work with code introduced using `new Function`.

Kadir Topal (not verified) - 09/27/2014 - 16:17

Firefox 31 is 4 generations behind the state of the art for Firefox. Since v31 there have been significant improvements. You can actually see for yourself on www.arewefastyet.com

matyas.mustoha - 10/06/2014 - 14:36

Thank you for your explanation, Till. I wonder if those tests could be modidified somehow to test different parts of them (parsing time, execution, first time run, etc.).

Kadir, we used the latest stable versions available at the time of the measurement. Of course, it's possible that we will try different versions of Firefox sometime in the future too.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • No HTML tags allowed
  • Lines and paragraphs break automatically.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Fill in the blank