TraceMonkey vs V8

Update: I got a lot of comments on my post. I am trying to answer them as they come in, so check back after you leave a comment.

Brendan Eich and Mike Shaver have posted an update on our progress on TraceMonkey. There has been a lot of buzz around Google’s new Chrome browsers and its V8 JavaScript VM. Some voices have claimed that V8 is several times faster than TraceMonkey. We did some head to head comparisons and these claims don’t match our observations.

We used Apple’s SunSpider benchmarks for our tests. Depending on the OS and machine configuration we are 1.18x to 1.28x faster than V8. Since V8 is only available for Windows, we didn’t perform any tests on MacOSX and Linux, both of which we support already. Our latest builds also work on ARM, by the way.

I am sure you can derive different results by tweaking the benchmarks or designing entirely new custom benchmarks alltogether, but since SunSpider has been used fairly intensively in the past two years to measure the evolution of JavaScript performance in Safari, Firefox, Opera, and IE, I think SunSpider is probably the most reliable cross-platform benchmarking tool at this point (which doesn’t say that its a particularly good one, its just the best we have right now.)

Talking about IE, our tests also indicate that we are about 15 times faster than IE 7, and about 5 times faster than IE 8 beta on the SunSpider aggregate scores.

If you want to give TraceMonkey a try, take a look at our nightly builds. You can enable the JIT in the about:config settings. The nightly builds are certainly not ready yet for wide-spread use, but we have improved stability significantly since our initial preview release. Firefox with TraceMonkey enabled is now my default browser, and I am writing this post with it.

40 thoughts on “TraceMonkey vs V8

  1. Pingback: Mozilla in Asia » Blog Archive » TraceMonkey vs. V8

  2. It does crash for me on a lot of sites (I’d say about 10% of sites using the nightlies). I’ve only enabled it in the Chrome now; although I do get odd results using some extensions. I am submitting my crash reports!

  3. With Sunspider, the latest Firefox nightly seems to be ~1.2-1.3x faster than Chrome on my machine. But if I try the benchmark on, Chrome is faster than Fx on every single test. With Fx I got 1624ms, while with Chrome 718ms. Why is there such a huge difference between the results of the two benchmarks?

  4. Well my bench on Windows Xp (P4 2.8GHz, 512MiB) were more like:

    V8: 3317.0ms
    TraceMonkey: 5827.0ms

    Trace monkey did better in
    V8: 94.6ms TM: 78.2ms

    V8: 27.4ms TM: 20.8ms

    V8: 27.8ms TM: 19.6ms

    V8: 642.4ms TM: 244.2ms

    V8: 209.0ms TM: 113.6ms

    V8: 547.8ms TM: 303.0ms

    V8: 101.0ms TM: 27.0ms

    V8: 243.0ms TM:236.8ms

    TraceMonkey did noticeably worse in
    V8: 122.2ms TM: 484.4ms

    V8: 4.0ms TM: 142.6ms

  5. I don’t have precise data at hand for google’s own benchmarks, but I have no doubt chrome does great on the benchmarks it was tuned for/developed with. We ran the SunSpider microbenchmarks yesterday and we beat chrome 4-5x on those. We didn’t publish those numbers because people are less familiar with the microbenchmarks then the actual SunSpider suite.
    Different JITs with different compilation approaches deliver different performance results for different benchmarks. What I was trying to say with my blog post is that we are not “guaranteed many times slower than V8”, as someone suggested.
    While Dromaeo contains very similar benchmarks, we don’t perform nearly as well on them as on the original SunSpider benchmarks. I have not deeply looked into this, but I will definitively do so over the next few days. Dromaeo looks more interactive than SunSpider. If it talks to the DOM during the benchmark, we currently don’t compile across such calls (we plan to for Beta2 though).
    As for stability, I am sorry if Minefield crashes for you that much. Please post bugs for those failures. Feel free to CC me on the bugs directly, that way they go into my queue right away (bugzilla ID is gal at uci dot edu).
    As for “how real are these benchmarks”, I think none of them really are. We definitively need better benchmarks that capture web applications, which is what people really want. Not tests of recursive function calls in different variations.
    As for our future, I think it is bright. TraceMonkey exists for a little over 10 weeks now. For that we are running already quite fast for some code. We are working on tracing more code. Keep in mind that for some SunSpider tests we don’t speed up over interpretation at all because we don’t compile anything (most notably we don’t compile recursion at the moment). Once we have improved coverage for such cases, you will see pretty drastic improvements for cases we didn’t compile before (5-10x). For really computationally-heavy code (like the imagine manipulation demo schrep showed), which we already do compile, I expect that we can get at least another 2x speedup over time, probably more. But talk is cheap. Give us some time to speed it up, and I promise to blog about it :)
    As for the numbers Matthew posted, we seem to beat V8 in almost every category but we come out 2000ms slower in the total? That looks weird. Maybe just a pasting error. But there is definitively some variance in the benchmark with different machines, and V8 and TM are very close in performance. I wouldn’t be surprised if in some configurations they beat us by a bit, just as we beat them by merely 20% or so in the configurations we tested with.

  6. Pingback: Chromed at aleatory

  7. @Andreas I just posted the SunSpider suite benchmarks where TraceMonkey did better (with the exception of the very slow marks at the bottom) to show there were some things TM did well. There were plenty of slower, unlisted results for TM as well. I had weird problems copying the urls for the results, otherwise I would have just posted links to the two test runs.

  8. The V8 docs say they don’t have an interpreter; they compile straight to machine code at execution time. It’ll be interesting to see how that works out on real-world pages, and whether it limits the amount of low-level optimization they can do.

  9. Pingback: Chrome: First Look :: David Mandelin’s blog

  10. Pingback: Weasel, the Geek - TraceMonkey vs. V8

  11. #14: I think Mike Shaver is the better person to ask this, but I think the main show stopper at the moment is stability, and thats what we are focusing on. The goal is certainly to be as stable for standard web use as with jit off, and thats where we are trying to get to for Beta1. If you want to help, grab a nightly and file bugs :)

  12. Pingback: Chrome讓我們期待更多了 « 小白的窩

  13. Pingback: Chrome? - Page 2 - CodeCall Programming Forum

  14. I think the entire discussion over who has the best VM is nonsense. Both VMs are open source, and both VMs are new. Over time, we’ll probably see tracing making it’s way into V8, as we’ll probably see general compilation making it’s way into TraceMonkey. Both teams have done an amazing job, and I say congratulations to both.

  15. Pingback: Google Chrome «

  16. Pingback: Wurstsalat

  17. Pingback: ulu's me2DAY

  18. Andreas, Any chance you can write a new blog post on what’s happening in TraceMonkey land in the last few weeks in particular how is tracing for recursion coming along

  19. Pingback: SquirrelFish Extreme entre en el juego de los motores JS | aNieto2K

  20. Pingback: Emerging browser war: Google Chrome vs. Mozilla Firefox « Sriniani’s Weblog

  21. I don’t think it really matters whether TraceMonkey/V8/SquirrelFishExtreme are a bit faster at certain (synthetic) benchmarks against each other. What matters, and that’s good, is that we will have really, really fast JavaScript in most major browsers in the future. It’s good competition.

    Of course it’s especially nice for Firefox since parts of the browser and extensions are written in JS.

  22. Pingback: Mozilla Firefox and Google Chrome Harmony | Bits And Buzz

  23. Pingback: A personal view at Firefox vs. Chrome | Tiago's Tech Blog

  24. Pingback: Color Management and Firefox 3.1 at freakified

  25. Pingback: JS++ |

  26. Pingback: Firefox 3.1 Beta 3 Candidate zum Download | Reverse Engineer Techblog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s