Technical Leadership at Mozilla

Today, I am starting my role as Mozilla’s new Chief Technology Officer. Mozilla is an unusual organization. We are not just a software company making a product. We are also a global community of people with a shared goal to build and further the Web, the world’s largest and fastest-growing technology ecosystem. My new responsibilities at Mozilla include identifying and enabling new technology ideas from across the project, leading technical decision making, and speaking for Mozilla’s vision of the Web.

I joined Mozilla almost six years ago to work with Brendan Eich and Mike Shaver on a just-in-time compiler for JavaScript based on my dissertation research (TraceMonkey). Originally, this was meant to be a three-month project to explore trace compilation in Firefox, but we quickly realized that we could rapidly bring this new technology to market. On August 23, 2008 Mozilla turned on the TraceMonkey compiler in Firefox, only days before Google launched its then-still-secret Chrome browser, and these two events spawned the JavaScript Performance Wars between Firefox, Chrome and Safari, massively advancing the state of the art in JavaScript performance. Today, JavaScript is one of the fastest dynamic languages in the world, even scaling to demanding use cases like immersive 3D gaming.

The work on TraceMonkey was an eye-opening experience for me. Through our products that are used by hundreds of millions of users, we can bring new technology to the Web at an unprecedented pace, changing the way people use and experience the Web.

Over the past almost six years I’ve enjoyed working for Mozilla as Director of Research and later as Vice President of Mobile and Research and co-founding many of Mozilla’s technology initiatives, including Broadway.js (a video decoder in JavaScript and WebGL), PDF.js (a PDF viewer built with the Web), Shumway (a Flash player built with the Web), the rebooted native Firefox for Android, and of course Firefox OS.

For me, the open Web is a unique ecosystem because no one controls or owns it. No single browser vendor, not even Mozilla, controls the Web. We merely contribute to it. Every browser vendor can prototype new technologies for the Web. Once Mozilla led the way with Firefox, market pressures and open standards quickly forced competitors to implement successful technology as well. The result has been an unprecedented pace of innovation that has already displaced competing proprietary technology ecosystems on the desktop.

We are on the cusp of the same open Web revolution happening in mobile as well, and Mozilla’s goal is to accelerate the advance of mobile by tirelessly pushing the boundaries of what’s possible with the Web. Or, to use the language of Mozilla’s engineers and contributors:

“For Mozilla, anything that the Web can’t do, or anything that the Web is not faster and better at than native technologies, is a bug. We should file it in our Bugzilla system, so we can start writing a patch to fix it.”

Trust but Verify


It is becoming increasingly difficult to trust the privacy properties of software and services we rely on to use the Internet. Governments, companies, groups and individuals may be surveilling us without our knowledge. This is particularly troubling when such surveillance is done by governments under statutes that provide limited court oversight and almost no room for public scrutiny.

As a result of laws in the US and elsewhere, prudent users must interact with Internet services knowing that despite how much any cloud-service company wants to protect privacy, at the end of the day most big companies must comply with the law. The government can legally access user data in ways that might violate the privacy expectations of law-abiding users. Worse, the government may force service operators to enable surveillance (something that seems to have happened in the Lavabit case).

Worst of all, the government can do all of this without users ever finding out about it, due to gag orders.

Implications for Browsers

This creates a significant predicament for privacy and security on the Open Web. Every major browser today is distributed by an organization within reach of surveillance laws. As the Lavabit case suggests, the government may request that browser vendors secretly inject surveillance code into the browsers they distribute to users. We have no information that any browser vendor has ever received such a directive. However, if that were to happen, the public would likely not find out due to gag orders.

The unfortunate consequence is that software vendors — including browser vendors — must not be blindly trusted. Not because such vendors don’t want to protect user privacy. Rather, because a law might force vendors to secretly violate their own principles and do things they don’t want to do.

Why Mozilla is different

Mozilla has one critical advantage over all other browser vendors. Our products are truly open source. Internet Explorer is fully closed-source, and while the rendering engines WebKit and Blink (chromium) are open-source, the Safari and Chrome browsers that use them are not fully open-source. Both contain significant fractions of closed-source code.

Mozilla Firefox in contrast is 100% open source [1]. As Anthony Jones from our New Zealand office pointed out the other month, security researchers can use this fact to verify the executable bits contained in the browsers Mozilla is distributing, by building Firefox from source and comparing the built bits with our official distribution.

This will be the most effective on platforms where we already use open-source compilers to produce the executable, to avoid compiler-level attacks as shown in 1984 by Ken Thompson.

Call to Action

To ensure that no one can inject undetected surveillance code into Firefox, security researchers and organizations should:

  • regularly audit Mozilla source and verified builds by all effective means;
  • establish automated systems to verify official Mozilla builds from source; and
  • raise an alert if the verified bits differ from official bits.

In the best case, we will establish such a verification system at a global scale, with participants from many different geographic regions and political and strategic interests and affiliations.

Security is never “done” — it is a process, not a final rest-state. No silver bullets. All methods have limits. However, open-source auditability cleanly beats the lack of ability to audit source vs. binary.

Through international collaboration of independent entities we can give users the confidence that Firefox cannot be subverted without the world noticing, and offer a browser that verifiably meets users’ privacy expectations.

See bug 885777 to track our work on verifiable builds.

End-to-End Trust

Beyond this first step, can we use such audited browsers as trust anchors, to authenticate fully-audited open-source Internet services? This seems possible in theory. No one has built such a system to our knowledge, but we welcome precedent citations and experience reports, and encourage researchers to collaborate with us.

Brendan Eich, CTO and SVP Engineering, Mozilla
Andreas Gal, VP Mobile and R&D, Mozilla

[1] Firefox on Linux is the best case, because the C/C++ compiler, runtime libraries, and OS kernel are all free and open source software. Note that even on Linux, certain hardware-vendor-supplied system software, e.g., OpenGL drivers, may be closed source.

On management, and culling occluded layers.

My day-time job as VP Engineering for Mobile at Mozilla is to manage an engineering organization of some 130 engineers. Most of my time is spent working with people and helping them to be effective at building software. That means a lot of meetings, a lot of planning, and a lot of HR-type busy work.

My job description today is very different from when I joined Mozilla a little bit more than 5 years ago to build the first available, commercial grade JavaScript compiler at that time (TraceMonkey). After 18 months in that job I became one of the top 3 overall committers to the Mozilla project. My job was to design and build stuff, and I enjoyed that quite a bit.

As much as I miss putting my PhD in Computer Science to use these days, having such a large engineering organization behind me has its perks. The amount of stuff “I” can get done by finding so many engineers the right things to work on and removing any roadblocks they face is really amazing, and it makes up for the lost hacking time.

Plane hacks

Still, to avoid feeling like a useless paper pusher, I usually have a hacking project going on on the side. I can’t really justify spending time hacking at work. There are simply too many meetings to attend, emails to reply to, and problems to solve. However, whenever I sit on a plane without WiFi, I can’t really effectively do my VP job since that job is all about communication. Time on the plane is hacking time, and as it happens, I travel a lot–I obtained Global Services status with United 2 years in a row, almost exclusively flying Economy (that must be some sort of World Record).

When I pick a project to hack on, I tend to pick the hardest problems I come across. Its usually a problem an engineering team is stuck on, and there is no easy fix. Some of my favorite projects last year were PhoneNumber.js, a library to format international phone numbers, and predict.js, a predictive text engine for the FirefoxOS keyboard. Both projects have long graduated into production. JavaScript hacks aside, the other area I am very interested in right now its Graphics and our layers system. Layers are used in browsers to accelerate animations and scrolling using the GPU. Not a lot of people work on this particular code, and its fairly hairy and complex. The perfect playground for a closet-engineer.

Whats a layer anyway?

In Gecko, our rendering engine, we try to detect when frames (a “div” is a frame, for example) are animated, and if so, we put those frames into their own layer. Each layer can be rendered to an independent texture, forming a tree of layers representing the visible part of the document, and we use a compositor to draw those layers into the frame buffer (frame buffer is the technical term used by OpenGL, “window” is what this means on most systems in practice).

Gecko has a couple different kinds of layers. Color layers consist of a single color. The body element of a document is usually white, and we use a color layer to draw that opaque, white rectangle. Image layers can hold one single image and are a special case of content layers (internally called Thebes, for historical reasons). Content layers is where we render arbitrary content into (text, etc).

When rendering the visible part of a document we already try to skip invisible frames, but when frames are animated (are moving around), we often end up having a layer tree where multiple layers are painted on top of each other, partially hiding each other. The compositor draws these layers in Z order, so the result is correct, but we sometimes composite pixels that are guaranteed to be occluded by layers that are pasted right on top of them. On desktop this is wasteful from a power consumption perspective, but in practice usually not a big deal. On mobile, on the other hand, this can actually cause significant performance problems. Mobile systems often have unified memory (texture data and the frame buffer share memory with the CPU) with fairly low memory bandwidth. Overcompositing (drawing pixels that aren’t visible in the end) wastes precious memory bandwidth. In extreme cases this can cause the frame rate to drop below our target frame rate of 60 frames per second for animations.


Flatfish is a tablet we have ported FirefoxOS onto. It has a high resolution screen and a comparatively weak GPU. As a result, over-compositing can cause the frame rate to drop. In case of the home screen for example we were compositing a color layer (blue in the image below) that was completely hidden by a content layer (yellow star). Setting each pixel in the frame buffer to black before copying the actual content over it caused us to miss the 60 FPS target for homescreen animations.

To solve this problem, I wrote a little patch (bug 911471) for the layers system that walks the children of a container layer in reverse Z order and accumulates a region of pixels that are guaranteed to be covered by opaque pixels (some layers might be transparent, those are not added to this region). As we make our way through the list of layers, any pixel that is covered by layers we paint later (remember, we are walking in reverse Z order) we don’t have to actually composite. It would be overwritten by an opaque pixel anyway. We use this information to shrink the scissor rectangle the compositor uses to composite each layer. The scissor rectangle describes the bounds of the OpenGL draw operation we use to composite.


Occluded Color Layer

Not a perfect solution, yet

This approach is not optimal, because the scissor rectangle is just a rectangle, and the layer might be partially occluded. Such partial occlusion is properly described by the region we are accumulating (regions can consist of multiple rectangles), but when setting the scissor rectangle I have to take the bounds of the region to paint (since GL doesn’t support a scissor region). This can still cause over-composition. However, in essentially every test case I have seen this doesn’t matter. Layers tend to be occluded by exactly one other layer, not by a set of layers partially occluding the layer.

It is possible to precisely solve this problem by splitting the actual draw operation into multiple draws with different scissor rects. This might be slow, however, since it writes to the GPU pipeline multiple times. A faster approach is probably to split the draw into multiple quads and draw all of them with one GPU call. Since this is a rare case to begin with, I am not sure we will need this additional optimization. We can always add it later.

Open Engineering

Mozilla builds software very differently than any other company, because, well, we are not a company. Mozilla is an open-source project backed by a not-for-profit foundation for the public benefit. For us open-source doesn’t just mean the source is out there. We actually care about Open Engineering.

Open from the start

The Boot 2 Gecko project, which is today known as Firefox OS, started on July 25, 2011 with an email to Mozilla’s dev-platform mailing list by Chris Jones, Brendan Eich, Mike Shaver and myself. In that email we expressed our opinion that as an open-source Mozilla community we should build a web-based smartphone. As a rallying cry and gathering point, we created an empty github repository and Chris and I started filing the first bugzilla bugs for Firefox OS, and started hacking on the first lines of code for Firefox OS.

Firefox OS was completely open from day 1, including the design process as well as the implementation. As we are building Firefox OS today, you can follow along live and jump in and help. You can see the same source live as our device and carrier partners. You don’t have to work for Mozilla (many Firefox OS contributors do not) and you don’t have to ask for permission to help. Just join our IRC channels and mailing lists, and get involved in the design or the code.


Mozilla’s large and diverse base of contributors always included organizations and not just individual contributors. With Firefox OS, the direct engagement of large organizations in the design and implementation of our products has increased tremendously. Qualcomm, Telefonica, Deutsche Telekom, and many others have large product and engineering teams committed to the project, building Firefox OS alongside with us. They are able to do so because of the openness of our processes, and they are motivated to do so, because Firefox OS is the only truly open smartphone platform where everyone who is willing to help can have an equal stake in. Mozilla doesn’t own Firefox OS. We merely care for it.

Last week we wrapped up a major Firefox OS engineering milestone. Leading up to that milestone, Deutsche Telekom Innovation Labs hosted a work week for the project. More than 140 engineers from Mozilla and our key partners attended. We use these work weeks to get engineers into one room who are usually distributed all over the world. Mozilla has offices on every continent, and so do most of our partners.

Below you can see a picture of the team. In the background is our milestone dashboard, showing the total number of bugs remaining, as well as the number of bugs between the frontend team (Gaia, 26) and platform team (Gecko, 50). The frontend and platform teams raced each other for the whole week. On Friday the Gecko team was almost able to catch up, with only 8 bugs remaining by the time the Gaia team hit 0 bugs. Those 8 remaining bugs landed over the weekend after the work week in Berlin.


Why the Web is going to win Mobile

firefox-phoneOn July 25, 2011 Chris Jones and I, with help from Mike Shaver and Brendan Eich, for the first time publicly talked about our vision to build a smartphone based on the Web. The announcement didn’t come out of nowhere, and didn’t happen in isolation. Chris and I had been working together for Mozilla for a few years, but we didn’t get to spend a lot of time talking to each other since Chris doesn’t work in the same office. On a trip to Asia as part of the Firefox 4 launch campaign, we finally had the opportunity to catch up and we ended up talking a lot about what later became the guiding question and principle for Mozilla’s research group:

What can the Web NOT do? And if so, WHY NOT?

We identified a couple of use cases where the Web is intentionally not used today. We are both platform engineers at heart. We wrestle with bits and bytes and CPU and GPU cycles for a living and we understand the Web platform and the implementation thereof (in Mozilla’s case: Gecko) pretty well. For the life of us, we couldn’t figure out why in a few key areas the Web hasn’t progressed as much as elsewhere. Two particular areas we homed in on were plugins and mobile.


If you ever had to work on a rendering engine, you probably hate plugins just as much as we do. Plugins are like a thorn in the Web’s side. They are difficult to render efficiently, binary compatibility requirements make the browser-side support code super ugly and hard to maintain, plugins leak memory, plugins can cause all sorts of security issues, and they generally hold back the development of the Web’s native capabilities. A good example of this is Flash: because the Web relied for over a decade on Flash to play back video, the VIDEO element didn’t really take off until pretty recently and some essential capabilities such as adaptive streaming are still missing from the Web stack. Chris and I both worked on the plugin support code in the Firefox, and we agreed that for the better of the Web, we really would like to DELETE that code. There is a large number of plugins out there, but realistically, only a handful matter, including Flash, Acrobat Reader (PDF), and maybe Silverlight (we included Silverlight back then, but by now Microsoft has totally given up on it, so I guess we can as well drop Silverlight from the list). We wondered why Flash and Acrobat Reader are implemented as native code plugins, instead of building the same functionality with HTML5/CSS and JavaScript, and since we didn’t have an answer for that question, we decided we should just build it and see what happens. We started with PDF first, because it seemed like less code to write until we can see whether performance and quality are comparable. A year later, Firefox is about to ship with the JavaScript-based PDF reader we started, so I think it’s safe to say that we found the answer to the question: the Web is ready to render PDF. We didn’t forget about Flash either. A few month later we started the Shumway project, which is still evolving, but might at least partially replace the role of Adobe’s native Flash player in Firefox eventually.

On Canvas and Dashed Lines

A lesser known part of the PDF.js project’s history is that the Web was actually NOT ready to render PDF when we started. We are using the CANVAS element to render PDF documents. We ran across a few loose ends and missing features. Infamously, we pretty quickly discovered that the CANVAS element couldn’t render dashed lines. It seems the spec authors just forgot to include this feature. We ran into this problem 4 days into implementing PDF.js because the first document we chose to render had some dashed lines in it. As it happens, Chris and I are both committers to the Mozilla code base, so Chris whipped up a prototype implementation in Firefox and we posted a proposed spec to the W3C mailing list. Two weeks later Firefox trunk (Nightly) could display dashed lines in the CANVAS element, and pretty soon thereafter a bug was filed for WebKit to add support as well (WebKit is used by Safari and Chrome, amongst others). This approach is the third part of the research group’s mission: Find out what the web can’t do, figure out why not, and FIX IT.

Standards vs Prototypes

The Web is so powerful because it’s not governed by committees and standards bodies. Instead, standards bodies like W3C merely help the Web avoid fragmentation by giving browser vendors a forum to coordinate and collaborate on jointly developed technology. To add dashed lines support, we didn’t have to convince a standards body. Instead, we added a prefixed prototype implementation to Firefox, and within a few months hundreds of millions of Firefox users could start using this feature. This was also the reason WebKit quickly decided to implement dashed lines support: competitive pressure. Firefox could do something WebKit-based browsers couldn’t. This is how the Web evolves. Someone finds a missing feature and implements a prototype for it, and if it’s a good feature, the rest of the browser vendors have little choice than to add support as well. The W3C’s role is to help browser vendors implement the feature in a compatible fashion across the different browser engines. The result is a pace of innovation that is unparalleled in technology history. Hundreds of millions of users had access to a feature after a few months of two engineers realizing that it was missing, and the rest of the Web adopted the same feature another few months later. Compare this to traditional governance by committee approaches such as the Java Community Process and the C++ standards committee where the pace of progress is measured in decades.

Beyond the Desktop

The second area we were interested in was Mobile. How come the Web is so ubiquitous on the desktop, but Mobile content (Apps) is implemented with proprietary native technology such as Java/Dalvik (Android) and ObjectiveC/Cocoa (iOS)? As with plugins, we didn’t have a technical explanation, and as with plugins, we did the same thing to find an answer: build it.

At the time of our first post about Boot 2 Gecko, which later became Firefox OS, Mozilla already had a Mobile product: Firefox for Android. It wasn’t particularly popular because it was pretty slow and memory hungry. We proposed a couple architectural changes to Firefox for Android, in particular abandoning our established XUL frontend toolkit in favor of a native Android UI. Mozilla’s mobile team did a heroic effort to turn the prototype we created into a full product within 6 months. Today, Firefox for Android is one of the most popular third party browsers for Android, and is faster and uses less memory than the native browser on most devices. In contrast to the often crappy browsing experience with the built-in browser, Firefox for Android is actually a pretty decent way to surf the Web on mobile. Yet most people still spend most of their time in native Apps. Why?

Mobile, the neglected red-headed stepchild

If you want to understand why the Web isn’t as popular on Mobile as on Desktop, just surf the Web on a phone: The Web totally blows on Mobile! Almost none of the device capabilities of a modern smartphone were available to the Web around the time we started talking about Boot 2 Gecko: making a phone call, SMS, bluetooth, NFC, proximity sensors, streaming camera access, etc. Name a device capability, and the Web didn’t support it back then. Browser vendors neglected to evolve the Web on Mobile with the same vigor as we did on the Desktop. No wonder content authors choose powerful native APIs instead of writing applications in HTML5!

The solution is pretty trivial and obvious: fix it. Mozilla started the WebAPI campaign as one of the first steps of the Boot 2 Gecko project. We have added dozens of new APIs to HTML5 to support mobile device capabilities, and many of these APIs have already been proposed as new W3C standards in various stages of progress, ranging from prototype specs all the way to final specs implemented across multiple browsers.

Why Mozilla matters

Our existing user-base of hundreds of millions of users gives Mozilla a tremendous amount of influence over Web technology, and this user-base is what enables us to push out new APIs and technologies. Any new device capability we add, we tend to add across all implementations of our browsers, ranging from Firefox OS devices (which are not in market yet), to Firefox Android (which has a ton of users) and Firefox Desktop (which a third of Internet-connected humans use). This gives us a loud voice in standards bodies, and we use that voice to aggressively push the boundaries of what can be done with the Web. Today, HTML5 is as powerful as native APIs, and within a year we have essentially closed the gap between the Web and native APIs when it comes to device capabilities.

Mozilla’s role in helping the Web succeed on Mobile is so critical, because we are driven by very different motives as the other three large browser vendors: Google, Apple and Microsoft. Mozilla is a public benefit non-profit foundation. Our goal is to promote innovation and choice on the Internet. We have no profit motive, and no services or products to sell. We want users to use our products, because they are better than the competition’s. And we want this choice to drive other vendors to advance their products to compete with us. This keeps us honest, and constantly pushes the envelope on Web technology.

Why the Web is going to win Mobile

Today, Firefox on Android is a powerful Web Browser and Web Runtime for Mobile applications. The Firefox Marketplace on Android offers a way for content authors to package HTML5 applications in an easily accessible marketplace that users can download applications from.

In 2013, jointly with our partners we will bring the first Firefox OS devices to market. These devices are completely built with the Web, and as Chris and I suspected a year and a half ago: there is really no technical reason why native technology should prevail on Mobile. In fact, Firefox OS and HTML5 apps are actually often much more CPU and memory efficient than proprietary alternatives such as Android and iOS. We are capitalizing on this fact and we are targeting in particular emerging markets such as Brazil with the first set of Firefox OS devices. The Web will bring smartphones to parts of the world where people can’t afford the expensive high-end smartphones Google, Apple and Microsoft like to compete over.

Only one question remains at this point: Is the Web really going to take off on Mobile? I think the best way to answer this is as follows: picture yourself being an executive at a company such as Facebook and someone pitches a new product to you: Facebook for Windows–a product designed for users to access Facebook from Windows desktops. In all likelihood, you would laugh that person out of the room. Obviously the Web is the way users access content such as Facebook on Desktop. Just the thought of using a native application for this purpose seems surreal and nonsensical. Now replace the word “Windows” with “iOS”, and suddenly everything is different. Obviously there is native Facebook for iOS.

In reality, in a world where the Mobile Web is as powerful as the Desktop Web, Facebook for iOS is just as surreal and nonsensical as Facebook for Windows is for the Desktop. Mozilla has been working tirelessly to create that world, and I think we are getting pretty close. Once we are there, native Mobile apps will be all but forgotten, just as Win32 applications are headed for extinction today.

Mozilla Project vs Mozilla Corporation

We went through a lengthy thread on hardware-decoding and H.264 on dev-platform today. It was heated but mostly civil. We are all pretty passionate about open standards, so an intense debate was to be expected. One issue I keep running into whenever I take some idea to a public forum like dev-platform is my standing with the Mozilla Project and the Mozilla Corporation. I am the Director of Research for the Mozilla Corporation. I can speak with some authority about research stuff. I am also a contributor to the Mozilla Project. I own a couple modules, and I am a peer to another few. When I posted to dev-platform today about codecs, I was wearing my Mozilla Project contributor hat. Me being a director at the Mozilla Corporation buys me exactly zero standing and authority with the Mozilla Project. The Mozilla Project has a governance structure. Peers and module owners make decisions. Since we are talking about codecs here, and I am neither a peer nor a module owner, I have as much authority as any random contributor on whether we take the course I plotted in my email or not. I am absolutely convinced I am right, but I still don’t get to make the call. The decision is with the module owners. So next time you read something I write, keep in mind: I am allowed to have a lot of opinions, but I have actually very little authority over the Mozilla Project. The authority rests with a large experienced group of module owners, not any one individual, even less so someone hired for a job (like a director). Instead, module ownership is based on merit and contribution to the project. And that’s why Mozilla rocks.