Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 2 down vote favorite

Since the beginning of the Javascript race -- which I would situate around Google Chrome launch in 2008 -- the improvement in the Javascript engine performances have been impressive.

The web is crowded with comparisons such as "Firefox V3.42.7 vs. Safari 3.0-prealpha2", and the winner of those comparisons changes every few months and differs on each benchmark.

But the big picture, independently of who got their new version out last, is that the average speed of current up-to-date browsers has improved a lot over the last years. Yet this long-term improvement is difficult to quantify:

  • people usually compare the last version of each browser, and not different versions of one browser

  • announced performance improvements do not generally pile up: when someone announce V3 twice as fast as V2, and later V4 twice as fast as V3, this does not mean that V3 is fourth times as fast as V1, because they usually mean "in a favorable case", and the favorable case in the V3-4 transition are not necessarily the same as in the V2-3 transition

  • benchmarks themselves evolve over time; what is referred as "the Sunspider test" today is not the same as in 2008, so we cannot compare raw scores over time.

Does anyone know of a valuable measurement of javascript engines performance improvement over the last few years?

share|improve this question
Define "valuable." In the end, it's going to come down to the specific performance considerations for your particular application. Anecdotally, Chrome seemed to be the first to make a concerted effort to improve their Javascript engine's performance, and that put pressure on everyone else to improve theirs also. So it's all good. But without a specific performance issue that you have personally profiled, I don't see how this can have a constructive answer. – Robert Harvey Aug 10 '11 at 15:43
@Robert: I don't think it's unreasonable, for example, to wish to have a concrete measurement of the performance difference between the first public version of chrome and its current version, or Firefox between 3.0 and now. It's nice to say that "everyone improved their JS performance as a result of the fierce competition", but concretely, can we quantify this global improvement? It would provide data, for example, for the kind of speedup that a given implementation change (from an AST-walking interpreter to a JIT) can provide to a dynamic language. – gasche Aug 10 '11 at 19:56
@Robert: as to "valuable", that's precisely left to the answerer appreciation. All benchmarks have their flaws, and I know there won't be any specific objective, ultimate, usecase-independent answer. But any benchmark people find reasonable to run to compare different browsers at a given time would possibly provide reasonable result to compare different browsers at different points in time. – gasche Aug 10 '11 at 19:59
I upvoted Sean's answer. – Robert Harvey Aug 10 '11 at 20:00
Performance of what though? Just because there's a spec doesn't mean there aren't a zillion ways to implement the spec. Yes, one browser does the math faster but it's still gets results of a huge DOM append on the page slower because it's rendering algorithms are crap. JS timers won't necessarily give you an accurate picture on something like that. From what I can tell, Chrome is on top, Firefox is trying out Scrum or something with the constant releases and appears to actually be reducing performance/stability and IE is finally getting its act together but still behind. – Erik Reppen Aug 12 '11 at 7:24
show 1 more comment

3 Answers

up vote 1 down vote accepted

As a side effect of keeping the benchmarks game files in CVS, anyone willing to do careful work should be able to gather together a little performance history.

For example,

1) You can find out when each new version text for V8 was committed.

2) You can find the data.csv file version just after the dates of your selected version files

3) You can "select for diffs" and diff the data.csv files for just after those dates

4) You can use page search to find all the V8 rows with status 0, and copy to a text file

5) Then you need to check that the data rows are for the same program. Firstly, that they have the same prefix (nbody,v8,1) and secondly that they reported the same program size gzip (1287).

6) So for - V8 version 1.2.5 (candidate) [console: dumb] - from Jun 9 2009 you could dig out these data rows

name,lang,id,n,size(B),cpu(s),mem(KB),status,load,elapsed(s) fasta,v8,1,25000000,791,125.240,10088,0,2% 1% 1% 100%,125.241 nbody,v8,1,50000000,1287,211.753,10088,0,0% 0% 0% 100%,211.755 spectralnorm,v8,1,5500,311,88.122,12588,0,0% 0% 0% 100%,88.120

7) And you could match those with these data rows for - V8 version 3.4.7 [console: dumb] - from Jun 28 2011

name,lang,id,n,size(B),cpu(s),mem(KB),status,load,elapsed(s) fasta,v8,1,25000000,791,30.750,5712,0,0% 0% 0% 100%,30.762 nbody,v8,1,50000000,1287,71.068,12852,0,0% 0% 0% 100%,71.118 spectralnorm,v8,1,5500,311,33.462,6580,0,0% 0% 0% 100%,33.474

8) Start with a later version (like V8 version 1.3.11.1) and you should be able to match more of the programs.

Have fun.

And for Mozilla (2008-2011)

1) You can find out when each new version text was committed.

3) You can "select for diffs" and diff the data.csv files for just after those dates (April 8 2011 is the last date to include tracemonkey)

4) You can use page search to find all the tracemonkey rows with status 0, and copy to a text file

etc

And for older Mozilla (2005-2007) on a different machine, using different programs

1) You can find out when each new version text was committed.

3) You can "select for diffs" and diff the data.csv files for just after those dates

4) You can use page search to find all the javascript rows with status 0, and copy to a text file

etc

The problem you'll see in that old data is that over time some tasks were removed and some new tasks were added, and there isn't a nice way to figure out when that happened. Probably the easiest approach would be to make a note of the earliest timestamp for each task description text, and then check the data file for a date just after that looking for a javascript program that implements the new task.

I should probably also say that you could dig out the source code for the programs, for example binarytrees.javascript

share|improve this answer
Thanks, that's definitely in the direction of what I'm looking for. One should be able to extract valuable information on the evolution of performances over a large span of v8's life. I hoped that somewhere, someone had already done the work, but apparently not. – gasche Aug 11 '11 at 9:36
One remark on these numbers: the downside is that they all apply to the same software, v8, which has seen intensive performance work, but I'm not sure as made big changes to its architecture: it was designed as a JIT from the start. So these numbers will allow to quantify the improvement over v8 lifetime, but not to quantify the performance gain when moving from a non-JIT to a JITed implementation. Of course, that's not meant to criticize your data, as I don't think the previous non-JIT software was deployed in a way suitable for the shootout. – gasche Aug 11 '11 at 9:40
Well there are also numbers for Mozilla JavaScript, but you'll need to be more careful digging through those - and sadly I never figured out how to get a useful version string out of spidermonkey. – igouy Aug 11 '11 at 15:27
up vote 2 down vote

You could always check out, build and test old versions of the browsers since some of them are open source. I bet you could find older binary versions of those who aren't.

share|improve this answer
up vote 4 down vote

The first thing that comes to mid when you ask this question is Are We Fast Yet. (Link is to the "old" page, because the current page is straight lines.

(Historical note, the original page just said "No.")

share|improve this answer
I knew of Are We Fast Yet -- sorry for not mentioning it in my question -- but unfortunately the time span is quite short. It's mostly a description of performance improvement during the development of one experimental engine landed in one Firefox version. In term of global improvement, it won't tell us much more than a Mozilla release announcement at the launch of Firefox 4. It would be interesting to see that timeline extended, both in the past and in the future -- and, why not, with separate data on Trace, Jäger and IonMonkey. – gasche Aug 10 '11 at 20:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.