Update: I got a lot of comments on my post. I am trying to answer them as they come in, so check back after you leave a comment.
Brendan Eich and Mike Shaver have posted an update on our progress on TraceMonkey. There has been a lot of buzz around Google’s new Chrome browsers and its V8 JavaScript VM. Some voices have claimed that V8 is several times faster than TraceMonkey. We did some head to head comparisons and these claims don’t match our observations.
We used Apple’s SunSpider benchmarks for our tests. Depending on the OS and machine configuration we are 1.18x to 1.28x faster than V8. Since V8 is only available for Windows, we didn’t perform any tests on MacOSX and Linux, both of which we support already. Our latest builds also work on ARM, by the way.
I am sure you can derive different results by tweaking the benchmarks or designing entirely new custom benchmarks alltogether, but since SunSpider has been used fairly intensively in the past two years to measure the evolution of JavaScript performance in Safari, Firefox, Opera, and IE, I think SunSpider is probably the most reliable cross-platform benchmarking tool at this point (which doesn’t say that its a particularly good one, its just the best we have right now.)
Talking about IE, our tests also indicate that we are about 15 times faster than IE 7, and about 5 times faster than IE 8 beta on the SunSpider aggregate scores.
If you want to give TraceMonkey a try, take a look at our nightly builds. You can enable the JIT in the about:config settings. The nightly builds are certainly not ready yet for wide-spread use, but we have improved stability significantly since our initial preview release. Firefox with TraceMonkey enabled is now my default browser, and I am writing this post with it.
Andreas Gal 
Pingback: Mozilla in Asia » Blog Archive » TraceMonkey vs. V8
You should append data from the five JS tests G. is delivering with Chrome. Matt Cuts is already throwing out blog links to charts created with the G. Chrome JS test set.
I made some tests on Dromaeo yesterday, and Chrome’s result is really impressive: 5.12x faster than Firefox 3.0.1 on the same virtual machine (WinXP on Parallels), still 3.37x faster than a nightly build with JIT enabled run on Leopard (so not inside a VM).
The post is in italian, but you can find all the links to the Dromaeo tests
http://www.pseudotecnico.org/blog/2008/09/02/veloce-e-veloce/
It does crash for me on a lot of sites (I’d say about 10% of sites using the nightlies). I’ve only enabled it in the Chrome now; although I do get odd results using some extensions. I am submitting my crash reports!
This benchmark doesnt reflect a multi tab / long term /real browsing environment doesn’t it?
What about http://dromaeo.com/ ? :)
What kind of speed improvement magnitude do you expect in a near / further future with TraceMonkey ?
How far TraceMonkey is different from V8 ?
With Sunspider, the latest Firefox nightly seems to be ~1.2-1.3x faster than Chrome on my machine. But if I try the benchmark on http://dromaeo.com/, Chrome is faster than Fx on every single test. With Fx I got 1624ms, while with Chrome 718ms. Why is there such a huge difference between the results of the two benchmarks?
Well my bench on Windows Xp (P4 2.8GHz, 512MiB) were more like:
V8: 3317.0ms
TraceMonkey: 5827.0ms
Trace monkey did better in
bitops,
V8: 94.6ms TM: 78.2ms
access:nsieve,
V8: 27.4ms TM: 20.8ms
crypto:sha1,
V8: 27.8ms TM: 19.6ms
date:format-xparb,
V8: 642.4ms TM: 244.2ms
math,
V8: 209.0ms TM: 113.6ms
regex,
V8: 547.8ms TM: 303.0ms
string:base64,
V8: 101.0ms TM: 27.0ms
string:tagcloud:
V8: 243.0ms TM:236.8ms
TraceMonkey did noticeably worse in
access,
V8: 122.2ms TM: 484.4ms
controlflow,
V8: 4.0ms TM: 142.6ms
I don’t have precise data at hand for google’s own benchmarks, but I have no doubt chrome does great on the benchmarks it was tuned for/developed with. We ran the SunSpider microbenchmarks yesterday and we beat chrome 4-5x on those. We didn’t publish those numbers because people are less familiar with the microbenchmarks then the actual SunSpider suite.
Different JITs with different compilation approaches deliver different performance results for different benchmarks. What I was trying to say with my blog post is that we are not “guaranteed many times slower than V8”, as someone suggested.
While Dromaeo contains very similar benchmarks, we don’t perform nearly as well on them as on the original SunSpider benchmarks. I have not deeply looked into this, but I will definitively do so over the next few days. Dromaeo looks more interactive than SunSpider. If it talks to the DOM during the benchmark, we currently don’t compile across such calls (we plan to for Beta2 though).
As for stability, I am sorry if Minefield crashes for you that much. Please post bugs for those failures. Feel free to CC me on the bugs directly, that way they go into my queue right away (bugzilla ID is gal at uci dot edu).
As for “how real are these benchmarks”, I think none of them really are. We definitively need better benchmarks that capture web applications, which is what people really want. Not tests of recursive function calls in different variations.
As for our future, I think it is bright. TraceMonkey exists for a little over 10 weeks now. For that we are running already quite fast for some code. We are working on tracing more code. Keep in mind that for some SunSpider tests we don’t speed up over interpretation at all because we don’t compile anything (most notably we don’t compile recursion at the moment). Once we have improved coverage for such cases, you will see pretty drastic improvements for cases we didn’t compile before (5-10x). For really computationally-heavy code (like the imagine manipulation demo schrep showed), which we already do compile, I expect that we can get at least another 2x speedup over time, probably more. But talk is cheap. Give us some time to speed it up, and I promise to blog about it :)
As for the numbers Matthew posted, we seem to beat V8 in almost every category but we come out 2000ms slower in the total? That looks weird. Maybe just a pasting error. But there is definitively some variance in the benchmark with different machines, and V8 and TM are very close in performance. I wouldn’t be surprised if in some configurations they beat us by a bit, just as we beat them by merely 20% or so in the configurations we tested with.
Pingback: Chromed at aleatory
@Andreas I just posted the SunSpider suite benchmarks where TraceMonkey did better (with the exception of the very slow marks at the bottom) to show there were some things TM did well. There were plenty of slower, unlisted results for TM as well. I had weird problems copying the urls for the results, otherwise I would have just posted links to the two test runs.
The V8 docs say they don’t have an interpreter; they compile straight to machine code at execution time. It’ll be interesting to see how that works out on real-world pages, and whether it limits the amount of low-level optimization they can do.
Andreas, is there an agreed-upon “threshold” whereby TM will be enabled by default in the nightlies?
Pingback: Chrome: First Look :: David Mandelin’s blog
Pingback: Weasel, the Geek - TraceMonkey vs. V8
It’s a great post, thanks…
But the there is a flaw in the link to the nightly builds. You are linking to the mozilla and not the firefox builds.
If anyone would like to try out the nightly firefox build the right URL is
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/
Have a nice day :)
Fixed the link, thanks Brunsgaard.
I’m also curious about #14 :)
#14: I think Mike Shaver is the better person to ask this, but I think the main show stopper at the moment is stability, and thats what we are focusing on. The goal is certainly to be as stable for standard web use as with jit off, and thats where we are trying to get to for Beta1. If you want to help, grab a nightly and file bugs :)
Pingback: Chrome讓我們期待更多了 « 小白的窩
Pingback: Chrome? - Page 2 - CodeCall Programming Forum
Is TraceMonkey optimize only for SunSpider?
As people have mentioned, on dromaeo Chrome still beats Tracemonkey. In google V8 test
http://code.google.com/apis/v8/run.html
it beats Tracemonkey as well. Am I missing something here?
I think the entire discussion over who has the best VM is nonsense. Both VMs are open source, and both VMs are new. Over time, we’ll probably see tracing making it’s way into V8, as we’ll probably see general compilation making it’s way into TraceMonkey. Both teams have done an amazing job, and I say congratulations to both.
Pingback: Google Chrome « NPFet.com
The new javascript engine seems to choke on scripts from redfin.com
It works fine if I disable the JIT.
Pingback: Wurstsalat
What’s the status about the powerpc port?
Pingback: ulu's me2DAY
Andreas, Any chance you can write a new blog post on what’s happening in TraceMonkey land in the last few weeks in particular how is tracing for recursion coming along
Pingback: SquirrelFish Extreme entre en el juego de los motores JS | aNieto2K
Pingback: Emerging browser war: Google Chrome vs. Mozilla Firefox « Sriniani’s Weblog
I don’t think it really matters whether TraceMonkey/V8/SquirrelFishExtreme are a bit faster at certain (synthetic) benchmarks against each other. What matters, and that’s good, is that we will have really, really fast JavaScript in most major browsers in the future. It’s good competition.
Of course it’s especially nice for Firefox since parts of the browser and extensions are written in JS.
Pingback: Mozilla Firefox and Google Chrome Harmony | Bits And Buzz
Pingback: A personal view at Firefox vs. Chrome | Tiago's Tech Blog
Sorry, I meant SpiderMonkey ! (not seamonkey)
What I was talking about can be found in more details here : http://news.qooxdoo.org/good-old-firefox
Thanks.
Pingback: Color Management and Firefox 3.1 at freakified
Pingback: JS++ | Maister.org.uk
The less restrictive LGPL license does not apply the restrictions to other software that merely links with the program. This is mostly used on libraries.
Pingback: Firefox 3.1 Beta 3 Candidate zum Download | Reverse Engineer Techblog