How big is that performance gain after all?

I have a quite fundamental problem... Let's assume that I came up with a bright new idea how to speed up the execution of JavaScript programs. Once implemented, of course, I'd like to measure it's effect and present the results to the world.

There are benchmark suites that exist exactly for this purpose. Now, if it takes 100 seconds for the unaltered JavaScriptCore to execute the whole SunSpider benchmark, and this drops to 77 seconds with my improvements applied, then what should I report?

For me, a natural way of reporting the result would be to use the execution time of the original JSC as a baseline and give the improvement in terms execution time: 1 - 77/100 = 0.23, i.e., 23% progression in runtime.

However, the scripts coming with SunSpider take another way. They would report that my version is 100/77 = 1.29x as fast as the original. (And this is mathematically sound, too.) Often, this is reported as 29% progression in performance.

So, for the same result, we have two interpretations. And the only explanation for the difference in the figures is hidden in the words standing after 'progression in'. I'm not sure that we always care about that last word. (Especially, if it's omitted, like '10% progression'.) Unfortunately, this can make the comparison of different improvements wobbly.

Moreover, from marketing point of view it does matter whether you 'sell' your work as being 23% or 29% better. Of course, if it's marketing, one should always go for the higher figure... But then there is the risk that someone will blame you for 'tweaking' math for your own good...

Well, there is no real conclusion. The question remains on the table: how big is that gain? Shall it be reported in terms of execution time, performance, or both?

Or am I simply overcomplicating things?

bkil (not verified) - 05/15/2009 - 23:10

Yes, the same issue has come to my mind some years ago, but I have given an answer myself. The solution is to always give the figure that looks the best, and keep the others for the detailed paper. Cheers!

spongyabob (not verified) - 07/07/2009 - 23:22

Here is the solution:
http://portal.acm.org/citation.cfm?id=5673
Bye!

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • No HTML tags allowed
  • Lines and paragraphs break automatically.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Fill in the blank