Brendan's Roadmap Updates

ES4 News

Some news on language size: the proposed ECMAScript 4th edition (ES4) grammar is a bit more than twice as big as ES3's, counting several ways (concrete productions, abstract syntax tree node types):

Syntax                   ES3   ES4
------                   ---   ---
Concrete                 238   490
  Type Exprs                    50
  Classes & Interfaces          23
  Patterns                      17

Abstract                  33    77

This is not out of line given the eight years since ES3, which came less than three years after ES1. Much of ES4 has been developed and shipped in ES3-based languages: AS3 in Flash, JS1.6 and 1.7 in Firefox, Opera's extensions including array destructuring. And we're being generous with syntactic conveniences, which desugar to a smaller core language.

On the compatibility front, we have a new document on known incompatibilities between ES3 (ECMA-262 Edition 3 the spec, not real-world behavior of ES3 implementations in browsers -- note well) and ES4.

Contradiction

Chris Wilson: "... I don't think the other Microsoft guys ever intended to say 'everything about ES4 is bad'."

Pratap Lakshman (Microsoft): "We do not support or agree to the current ES4 proposal, either in whole or in part."

Chris,

You seem to be repeating falsehoods in blogs since the Proposed ECMAScript 4th Edition Language Overview was published, claiming dissenters including Microsoft were ignored by me, or "shouted down" by the majority, in the ECMAScript standardization group.

Assuming you didn't know better, and someone was misinforming you, you (along with everyone reading this letter) know better now. So I'll expect to see no more of these lies spread by you.

Everything that has been done to advance JavaScript to the proposed ECMAScript 4th Edition (ES4) in Ecma TC39-TG1 has been in the open for well over a year, first via the es4-discuss mailing list (complete archive) and exports of our group's wiki, along with public talks I've given. Then by ecmascript.org hosting the reference implementation for ES4 as proposed, along with our public trac database. And finally by the whole wiki being fully opened up and hosted live -- including Microsoft's "ES3.1" proposals.

These ES3.1 proposals came suddenly in March after years of passive participation by Microsoft, while the rest of the group worked on ES4. They have not been touched since April. They weren't published via wiki export at that time, by agreement among all parties in TG1 including Microsoft and Doug Crockford. For one thing, they were (and still are) extremely rough and incomplete, with unanswered comments.

Almost all of the non-deprecation ES3.1 proposals were lifted directly from ES4 proposals (see my comment here).

One document reaffirms an agreement made in March: any ES3.1 candidate specification would be developed as a subset of the ES4 reference implementation, so it could be tested mechanically for forward and backward compatibility. This agreement has not been kept.

In February, an early warning was given by Microsoft, and met with vigorous debate (no shouting). Readers should take the time to follow this link, because I think it gets to the heart of the dispute, which is superficially over whether ECMAScript should remain small and mostly the same as it is in Edition 3.

The pattern of general assertions about small being beautiful and sufficient for web application developers' needs, met by specific arguments listing use-cases where JS does not scale in time or space, or lacks basic data integrity, type safety, and programming in the large support, in turn met by absolutely zero specific counter-arguments -- this is a pattern we have seen over and over in TG1 this year.

The small-is-beautiful generalization alternates with don't-break-the-web, again without specifics in reply to specific demonstrations of compatibility. Oh, and sometimes fix-security-first is put forth, without sound and sufficient solutions.

At best, we have a fundamental conflict of visions and technical values between the majority and the minority.

However, the obvious conflict of interest between the standards-based web and proprietary platforms advanced by Microsoft, and the rationales for keeping the web's client-side programming language small while the proprietary platforms rapidly evolve support for large languages, does not help maintain the fiction that only clashing high-level philosophies are involved here.

Readers may not know that Ecma has no provision for "minor releases" of its standards, so any ES3.1 that was approved by TG1 would inevitably be given a whole edition number, presumably becoming the 4th Edition of ECMAScript. This is obviously contentious given all the years that the majority of TG1, sometimes even apparently including Microsoft representatives, has worked on ES4, and the developer expectations set by this long-standing effort.

A history of Microsoft's post-ES3 involvement in the ECMAScript standard group, leading up to the overt split in TG1 in March, is summarized here.

The history of ECMAScript since its beginnings in November 1996 shows that when Microsoft was behind in the market (against Netscape in 1996-1997), it moved aggressively in the standards body to evolve standards starting with ES1 through ES3. Once Microsoft dominated the market, the last edition of the standard was left to rot -- ES3 was finished in 1999 -- and even easy-to-fix standards conformance bugs in IE JScript went unfixed for eight years (so three years to go from Edition 1 to 3, then over eight to approach Edition 4). Now that the proposed 4th edition looks like a competitive threat, the world suddenly hears in detail about all those bugs, spun as differences afflicting "JavaScript" that should inform a new standard.

Sorry, but most of those JScript deviations are not candidate de facto standards -- they are just your bugs to fix. They should not stall ES4 for one second.

I'm testifying as an eyewitness, and I have other witnesses in TG1 who can confirm what I write here. Everyone in the majority of TG1 gave time to the late-breaking dissenters from Microsoft and Yahoo!. We went out of our way to read, cross-reference, and comment on the "3.1" proposals. Activity on those proposals stopped in April.

Anyone who wants to check my claims against the TG1 meeting notes can go to the meetings page on the wiki, and follow the links. The face-to-face notes starting in March are easy to scan for political content (including my dissenting notes), and I invite anyone who is interested to have a look.

The meeting notes were edited by many hands, including Microsoft folks to their evident satisfaction, at the time of the meetings. The change logs are public in the wiki. There was no ignoring of minority positions. There was no failure to listen on the part of the majority. On the other hand I did not stop all progress on ES4 until unanimous agreement was reached on how to proceed (since Ecma and ISO do not require unanimity to make standards).

There certainly was no shouting down of the dissenters -- that's a bold lie in view of the well-attended and friendly dinners sponsored by the face-to-face meeting hosts.

As the links above demonstrate, there has been no secrecy on the part of the majority of TG1, other than the secrecy requested by Microsoft and required by Ecma. Rather the opposite: Doug Crockford and Microsoft's reps would depart for private meetings, in the middle of the March, April, and especially the May TG1 face-to-face meetings.

My TG1 Convenor's Report, dated 4 May 2007, to Ecma TC39 (chaired by Jim Miller of Microsoft) is available to Ecma members at the Ecma members website. In it, I informed our parent TC39 committee that "[a] sub-group of TG1 is investigating possibly standardizing a smaller, compatible language, but this proposal has not yet been adopted." Since then, the sub-group has failed to present a coherent proposal -- JScript informative documents are not proposals -- and my next report to Ecma will have to say so plainly.

You may have been misled, but now that everyone can see almost all of the primary sources (the remaining documents are hidden behind Microsoft, Yahoo!, and Ecma firewalls), please stop repeating falsehoods about either my conduct as TG1 Convenor, or the treatment of dissent by the majority favoring ES4 in TG1.

If you have any documents to publish that shed light on Microsoft's actions and intentions in Ecma TC39-TG1, I welcome them.

/be

From the inimitable Chris Double of Mozilla's New Zealand brain trust: moving, rotating, scaling <video> in SVG (so who needs yet another non-standard plugin?). If you make a build from Chris's code, don't forget to view-source the demo.

/be

My Ajax Experience West keynote covers a lot of ground, with slant-wise truth telling the over-arching theme. Mozilla believes in fairly radical open source action, including open strategy. In that spirit, three new projects, the first already known via jresig:

ActionMonkey, the project to join forces between SpiderMonkey and Tamarin, being driven by Jason Orendorff with Edward Lee and some old-timers and newcomers lending a hand. The first stage of development integrates MMgc into SpiderMonkey as a replacement for SpiderMonkey's GC. (NB: the wannabe mascot has a day job.)

IronMonkey, the project to map IronPython and IronRuby to Tamarin, so that Tamarin becomes multi-lingual, over time delivering high performance for all languages. This effort is being spearheaded by Seo Sanghyeon (whose last name I apologize for misspelling -- or for using a different spelling of, at any rate -- in my slides). The idea is to make use of the MsPL-licensed open source that Microsoft has kindly given the world, and tend necessary changes or additional code in downstream truly-open repositories that can accept patches from non-MS employees, such as FePy.

ScreamingMonkey, the project to add browser-specific script-engine glue to Tamarin, enabling it as a <script> tag handler in other browsers, starting with IE. Mark Hammond is leading this charge. Where it will end, one can only speculate. At a minimum, if you have Tamarin installed on your system in the future, you may need only a small download to plug it into other browsers. Best case, it comes with all the relevant glue in its main distribution. We'll see.

I'm excited to announce these projects. They address various concerns voiced by bloggers of the form "what about other languages?" and "what about IE?". Comments welcome.

/be

As Dave Herman just posted at Lambda-the-Ultimate, we ECMA members who have been working on the successor to the JavaScript standard have a new www.ecmascript-lang.org website, which now hosts the "milestone 0" release of the ES4 Reference Implementation, written in Standard ML of New Jersey.

Adobe open-sources Flex under the MPL -- a good second step. Congratulations to all the people who made it happen.

My browser-war battle-heightened senses tell me you'll see more open-sourcing, and no small amount of "openness" hype, from several big players in the coming days. As we mozilla.organs learned the hard way, getting the license right and hosting the code in a public repository are not enough (no pixie dust).

Successful open source projects combine meritocratic leadership, "doing" more than "talking", and breadth through well-scoped extension mechanisms. It's not enough to do great work by oneself: each committer who has the stamina and remains engaged must spend time listening to users and developers, grooming helpers and successors, and refactoring or even redesigning to support what becomes, module by module, a platform.

Platforms are for applications to stand upon, for building higher tiers of platform, and ultimately (in my view) for enabling user innovation. Mozilla's platform has at least three tiers: Web, XUL, Gecko-embedded. Firefox has addons as a tier distinct from those.

Of course, closed-source platforms exist, but the successful ones hold up so much code that they end up de-facto standards, and often go open-source just to reduce costs born by the proprietor (there are better reasons than that, don't get me wrong).

One way of evaluating Adobe's success in open-sourcing Flex will be to look for more than just added components and widgets, and discern higher-level as well as different platform tiers. For instance, would patches to add a web-targeted compiler back end, in addition to the Flash Player SWF-targeted back end that's there now, be accepted if they were developed?

Or how about harmonizing XUL and MXML, making changes to underlying layout and widget models on both sides in order to converge the two, so that (with trivial XML vocabulary translation) MXML could be interpreted by browsers that support XUL, as well as compiled into a SWF bound for the Flash Player?

For many observers and fans, an open-source Flash Player is the holy grail. If that is coming, it will probably take a while. But there's a small and (I think) rising chance that "openness" competition among big players will speed up the trend at Adobe.

Then what? Obviously, forked and fragmented Player variants are not helpful, and the main distribution will be the contested ground. A better Flash Player via open-source development would be fine: good not great.

Blending Flash code into multiple browser rendering pipelines, with seamless text input (including focus) and output, an integrated DOM, full and fast scripting, and unified CSS processing would be even better. Add high-quality, unencumbered <video>, and web-friendly 3D (including an LOD-capable scene-graph model, rather than a bulky, eagerly created DOM), and you're close to my idea of the promised land.

Adobe is moving in the right direction. That's good enough for me today.

A small presentation I gave at SXSW asks what it means to be "Open" in the sense of open standards, open source, and a web whose major content formats are not controlled by a single vendor (I credited Nat Friedman's amusing keynote from last year's Free Software & Open SourceSymposium for inspiration of the OED historical "Open" example).

Ted Leung (see followups) blogged about this topic already, specifically in connection with Adobe. Ted refers to Anne Zelenka's Why Open is Good and How Open Could Be Good for Flash post, which is worth reading, including the comments. In one comment, John Dowdell of Adobe asks whether the essence of "Open" is the ability to fork. Anne notes the loaded connotation of "fork" but agrees that John's analysis is "not incorrect."

I would go further. Forking is an extreme point in a continuum of options that exist with open source. The option to fork must exist as a feedback mechanism, but it need not be used in order for users to gain benefits not available with closed source and proprietary standards. Forking can be the right thing, or it can be a kind of mutually-assured-destruction option that keeps everyone acting in the interest of not forking.

Prior to even approaching a fork, open standards and open source both empower user-driven innovation. This is old news to the Mozilla user community, who have been building and feeding back innovations for the life of the project, increasing over time to include Firefox add-ons and GreaseMonkey user scripts. (BTW, I am pushing to make add-on installation not require a restart in Firefox 3, and I intend to help improve and promote GreaseMonkey security in the Firefox 3 timeframe too.) Without forking, even to make private-label Firefoxes or FlashPlayers, users can innovate ahead of the vendor's ability to understand, codify, and ship the needed innovations.

Consider just the open standards that make up the major web content languages: HTML, CSS, DOM, JS. These mix in powerful ways that do not have correspondences in something like a Flash SWF. There is no DOM built inside the FlashPlayer for a SWF; there's just a display list. There's no eval in ActionScript, and ActionScript features a strict mode that implements a static type checker (with a few big loopholes for explicit dynamic typing). You can't override default methods or mutate state as freely as you can in the browser content model. Making a SWF is more like making an ASIC -- it's "hardware", as Steve Yegge argues.

This is not necessarily a bad thing; it's certainly different from the Open Web.

I assert that there is something wrong with web-like "rich" formats that aren't hyperlink-able or indexable by search-engines. You could argue that these bugs could be fixed, and Flash is wisely becoming more URI-addressable and view-source-able over time. But it still ain't the Web. It is not hand-authored, easily tweaked incrementally, copy-and-paste-able. It's hardware.

Given the stagnation of the web under IE and the failure of Java on the client, there's a place for Flash, and I'm sure Microsoft would argue that means there's a place for WPF/E (but I don't see the point: either bet on the Web, or bet against it in a transcendently different way from trying to overtake Flash). If I were VP of Engineering in a 10,000 person company, I would want the security blanket of the C-like syntax and a static type system for a well-known, big-bell-curve language like AS3, C#, or Java.

And indeed AS3, C#, and Java all are marketed to big companies with hordes of "hardware" hackers creating fixed-point UIs. Dead UIs, in other words, not enlivened by user innovation. And sorry to say (to Yahoo! and Google, who may be exceptions to the rule), the big innovations don't come from the big dumb companies.

Meanwhile, the Web is alive precisely because of the distributed extensibility of its content languages, both on the server side and in the client JS/DOM world, not to mention add-ons and GreaseMonkey.

Dare Obasanjo argues that developers crave single-vendor control because it yields interoperation and compatibility, even forced single-version support. Yet this is obviously not the case for anyone who has wasted time getting a moderately complex .ppt or .doc file working on both Mac and Windows. It's true for some Adobe and Microsoft products, but not all, so something else is going on. And HTML, CSS, DOM and JS interoperation is better over time, not worse. TCP/IP, NFS, and SMB interoperation is great by now. The assertion fails, and the question becomes: why are some single-vendor solutions more attractive to some developers? The answers are particular, not general and implied simply by the single-vendor condition.

Implicit in my writing is the assumption (conclusion, really) that browsers can adopt the necessary advanced rendering and faster virtual-machine programming language support that the "rich client platforms" boast (or promise in version 2.0). For instance, we will support an OpenGL-ES binding for canvas in the near term, in Firefox 3 if vlad and I can help it. There's no technical reason this can't be interoperably supported by other browsers in the near term.

Can the Open Web keep improving enough to stave off the Closed Web? Of course, Mozilla is here to help, and we want every browser to support the nascent standards we support. But at some point the answer does depend on Microsoft upgrading IE without forking web standards from what Firefox, Safari, and Opera agree on now and in their upcoming releases, which are not yet implemented in IE7.

This will mean incompatibility for old IE content, or more painful version checking in the IE engine, or a mix. Note how Flash 9 includes both the AS2 and AS3 (Tamarin) engines, since Adobe could not keep compatibility in one engine. Gecko has had to put up with more de-facto quirks implementation than we might prefer; we sucked it up to get market share. IE can do likewise, for a change to avoid losing more market share.

Such a brute-force versioning approach could work well for Microsoft, as it did for Adobe in Flash 9. Microsoft can most afford to put more megabytes on disk via Windows Update than can browser vendors such as Mozilla and Opera, who rely on voluntary downloads.

Anyway, I'm committed to improving the Open Web, even at the expense of Firefox market share if it comes to it (but we are still growing, and Firefox still has the second-place leverage to improve the Web quickly). If we should fail and just make a fancier browser whose nascent standards are not adopted by IE, at least we tried. The alternative is to renounce innovation and let the proprietary rich clients move the Closed Web forward. That might be convenient for some (big company) developers. It's not in the interest of the larger developer community, or of the public, in my book.

Let me know what you think.

This is not an original thought, but I write with some authority here.

I hacked Unix kernel code out of grad school at SGI, in SGI's "good old days" (1985-1992). Among other things, we took single-threaded (ST) kernel code and multi-threaded (MT) it on SGI's SMP boxes. I won a free trip to New Zealand and Australia in 1990 along with Bent Hagemark, with kernel source on magtape in our hot little hands, on account of others' bugs in adapting some single-threaded (ignoring interrupts) AT&T "Streams" code to SGI's SMP kernel, and made fixes in the field (thanks to the SGI sales guys in Brisbane, we even got two nights on the Gold Coast in compensation -- not long enough!).

You must be this tall to hack on threaded systems, and that means most programmers should run away crying. But they don't. Instead, as with most other sharp tools, the temptation is to show how big one is by picking up the nearest ST code and jamming it into a MT embedding, or tempting race-condition fate otherwise. Occasionally the results are infamous, but too often, with only virtual fingers and limbs lost, no one learns.

Threads violate abstractions six ways to Sunday. Mainly by creating race conditions, deadlock hazards, and pessimistic locking overhead. And still they don't scale up to handle the megacore teraflop future.

We can hope for better static analyses to find all races. In the real world, the code is C or C++ and there's no hope for static salvation. Sure, some languages try to put deadlocks in a syntactic cage, but that walks right into the overhead problem, in spite of heroic VM-based optimizations. Unexpected costs, even if constant or linear, can sink any abstraction. For example (still piquant to Mozilla hackers busy deCOMtaminating), virtual method calls cost; they should be avoided where you're writing hardware. The same goes for locks: not all abstractions must be MT-safe; some must be ST and fast.

So my default answer to questions such as the one I got at last May's Ajax Experience, "When will you add threads to JavaScript?" is: "over your dead body!"

There are better ways. Clueful hackers keep rediscovering Erlang. Then there is STM. One retro stylist I know points to an old language-based solution, Hermes.

A requirement for JS3 (along with hygienic macros) is to do something along these more implicit lines of concurrency support. In all the fast yet maintainable MT systems I've built or worked on, the key idea (which Will Clinger stated clearly to me over lunch last fall) is to separate the mutable unshared data from the immutable shared data. Do that well, with language and VM support, and threads become what they should be: not an abstraction violator from hell, but a scaling device that can be composed with existing abstractions.

So here's a promise about threads, one that I will keep or else buy someone a trip to New Zealand and Australia (or should I say, a trip here for those over there): JS3 will be ready for the multicore desktop workload.

Does this mean we won't be hand-coding MT Gecko? I think Graydon said it best: "I'd rather eat glass." Most high-level concurrent Mozilla programming will be in JS3 on the back of an evolved Tamarin.

For Mozilla 2, I proposed that we use Oink, or really Elsa, to build a tool that can help automate deCOMtamination, switching to C++ exceptions instead of nsresults, and similar tasks beyond the reach of sed or perl. The idea is to specify pattern and replacement abstract syntax trees (ASTs), and have the tool match the pattern against sub-tree instances, producing a version of the source with the replacement AST pretty-printed instead of the matching sub-tree.

Graydon came up with the insight of using source coordinates, which the Oink framework tracks accurately, to tell which lines in a given primary source (.cpp file, not .i file that results from the C pre-processor) correspond to a particular AST. The tool would read the source, counting lines until the range to delete is reached, then copy each such line matching the pattern to replace with a "- " prepended. Then write lines pretty-printed from the replacement AST with "+ " prepended. Add a bit more boilerplate and context lines, and you have a patch.

Graydon didn't just make this breakthrough, he prototyped it, with a hardwired pattern to replace struct foo pointers with struct bar pointers in a test file:

$ cat test.cc
struct foo {};
struct holdsAFoo
{
  char x;
  static const foo * const y;
  int *z;
};

$ ./staticprint -fs-print-rewrite test.cc
--- test.cc
+++ test.cc
@@ -5,1 +5,1 @@
-  static const foo * const y;
+static struct bar const * const y;

$ ./staticprint -fs-print-rewrite test.cc | patch -p0
patching file test.cc

$ cat test.cc
struct foo {};
struct holdsAFoo
{
  char x;
static struct bar const * const y;
  int *z;
};

All this wants is an Oink tool of its own (instead of a hack to staticprint), a little more indentation fidelity (bonus points for respecting the source file's Emacs and vim modelines), C++ purity (no struct before bar needed), patch context and hunk coalescing, and generalized AST pattern/replacement options, and we'll be able to make major systematic changes by generating patches. Neat!

/be

I just committed the initial revision of mozilla/js/tamarin, the open-source ActionScript Virtual Machine contributed by Adobe. But I had nothing to do with this fine work. It's the product of some talented folks at Adobe (originally at Macromedia; at least one of whom has moved on and is doing a startup).

Tamarin, the open-source AVM, is the JIT-oriented JavaScript VM that I mentioned in my last post. We have already developed, via a two-day marathon hacking session, a set of proof-of-concept patches to integrate it with SpiderMonkey. The resulting Franken-VM was compiling and linking when I last laid hands on it, but not quite limping. This exercise produced many insights to guide more durable integration. And it was great fun interacting with the new open-source contributors from Adobe, whom I would like to welcome to Mozilla:

Jeff Dyer, self-hosting compiler architect Steven Johnson, Tamarin developer Tom Reilly, garbage collector developer Rick Reitmaier, JIT developer Dan Smith, Tamarin module owner Edwin Smith, Tamarin creator and VM architect Erik Tierney, Tamarin developer My apologies to those whose names I haven't yet learned. I've had the pleasure of getting to know most of the above-named folks through the ECMA TG1 (JavaScript standards) group. They are good people and skillful hackers, and I'm grateful that they are adding their strength to the Mozilla project.

I should point out that, although Adobe did not release open source for its existing, free AS3 compiler, it did contribute a self-hosting (AS3-in-AS3) compiler prototype written by Jeff Dyer. We intend to develop this jointly into an ES4/JS2 self-hosting compiler, implementing the full and final ECMA-262 Edition 4 spec when it is done. In the mean time, if you want to generate ABC for Tamarin, you have open source options.

I'll be on IRC in a little over nine hours to chat about this great news and answer questions, and then I'm off to Web 2.0. Later this week I will blog again about some early progress on another Mozilla 2 front.

/be

Update: Frank Hecker of the Mozilla Foundation clearly and concisely answers many basic questions about What This All Means, questions to which I tend to give short shrift in this too-technical blog of mine. Recommended.

Mozilla has gone from open source whipping boy in 1999 to open source poster child since 2004, due in large part to the success of Firefox. For that we can thank some amount of luck with our "timing the market" -- the browser market that no one knew existed, until it was suddenly clear in the summer of 2004 that Microsoft had dropped the ball and Firefox was picking it up.

But we didn't just get lucky. Putting the regular user first with Firefox's default UI and interaction design, building on real-world web standards instead of start-from-zero fantasy web standards, persevering against bug adversity over time, and calculated (but secondary -- the apps came first) work on "the platform", especially to aid web developers and power users via add-ons, were all necessary to our success.

In particular, Firefox couldn't have happened without Mozilla 1.0, a release milestone about which I wrote a requirements manifesto in early 2001. Mozilla 1.0 came out in June of 2002, and finally won back reputation lost due to Netscape 6 and the big rewrite. It set the stage for Firefox, Thunderbird, and other XUL applications.

For Mozilla 1.0, the architectural die was cast in late 1998 and 1999, so the process to reach 1.0, with its stable set of APIs on which most of a browser could be built, consisted mainly of the hard work of finishing. Since then, much of the code base has been revised in some way, but always incrementally. No "big bangs."

The current CVS trunk, which will become the Mozilla 1.9 stable branch some time in the first half of next year, contains significant rendering rearchitecture, along with lots of other important work I won't go into right now.

The graphics work and the reflow refactoring are perhaps the most aggressive changes conceivable in the Mozilla milestone process. That process, more or less still the same as described in the "mozilla 1.0 manifesto", depends crucially on community QA and patch contributions, which must converge on a product alpha/beta/release cycle to have enough real-world testing to claim meaningful code coverage. This cycle continues in 1.9, on which Firefox 3 will be based.

So Mozilla is a large, mature, fairly conservatively maintained open source code base. What else is new?

Lots: the web as a system of interoperating and open standards is under renewed assault. Browser competition is heating up, which is great except where it portends innovations locked into proprietary stacks.

The mobile device space is growing dramatically, yet with power storage and dissipation limits temporarily repealing Moore's Law. Worse, carriers and handset makers control all software most users get, leaving little room for the kind of choice that led to Firefox. Let's save the mobile thought for another time.

What about desktop systems? In four years, desktops will have polycore CPUs, with (still specialized but increasingly useful) teraflops in the GPU. Multimedia and 3D content will become more tractable by mere mortals without expensive, complicated tools, or there will be a lot of wasted power and bandwidth. Never mind 3D -- just better search, better text rendering, usable video, and all kinds of user-oriented optimization tasks should soak up those hardware threads' cycles. Whatever happens, it seems a shame to leave all that capacity to games, other proprietary software, and closed content formats.

So what should we do in Mozilla-land about this?

First, we should do Mozilla 2, targeting 2008, "after" Mozilla 1.9 -- but we should start working on it very soon, next month. We should not follow 1.9 with 1.10 and run headlong into the law of diminishing returns.

Second, after Mozilla 2 is well under way, we will worry about those super-duper CPUs and GPUs. I have some thoughts, but I'll save them for another post.

Mozilla 2 means among other things a chance to break frozen API compatibility, which removes constraints on the current architecture and allows us to eliminate old APIs and their implementations, renew and improve the APIs and code we want to keep, and realize significant runtime and code size wins. For instance, we can get rid of RDF, which seems to be the main source of "Mozilla ugliness" humorously decried by Steve Yegge.

For Mozilla 2, we will have a JIT-oriented JavaScript VM (details soon) that supports the forthcoming ECMAScript Edition 4 ("JS2") language. Among the desirable characteristics of this VM will be a conservative, incremental garbage collector (GC). If it makes sense, we can use this GC module to manage DOM object memory instead of using XPCOM reference counting. We can use its conservative scanning code to assist in cycle collection. And we can JIT calls directly into DOM glue code entry points (provided no JS mutation has overridden a method property value), bypassing the powerful but relatively slow typelib-based dispatching machinery of XPConnect.

This will kick Ajax performance in Firefox up a notch or three.

This is just a start.

For Mozilla 2, because we can break API compatibility where it makes sense to do so, we can and will provide better APIs, "on the outside" of Gecko -- and remove XPCOM API boilerplate "on the inside". We can translate old C++ portability veneer into standard C++ where doing so does not cost us portability, performance, or correctness. We can even switch to C++ exceptions if doing so wins in both code size and runtime performance in a fair contest.

So instead of code like this:

PRBool
nsXULDocument::OnDocumentParserError()
{
  // don't report errors that are from overlays
  if (mCurrentPrototype && mMasterPrototype != mCurrentPrototype) {
    nsCOMPtr<nsIURI> uri;
    nsresult rv = mCurrentPrototype->GetURI(getter_AddRefs(uri));
    if (NS_SUCCEEDED(rv)) {
      PRBool isChrome = IsChromeURI(uri);
      if (isChrome) {
        nsCOMPtr

you'll see code like this:

bool
XULDocument::OnDocumentParserError()
{
  // don't report errors that are from overlays
  if (mCurrentPrototype && mMasterPrototype != mCurrentPrototype) {
    IURI *uri = mCurrentPrototype->GetURI();
    if (IsChromeURI(uri)) {
      GetObserverService()->NotifyObservers(uri, "xul-overlay-parsererror");
    }
    return false;
  }
  return true;
}

(I've taken the liberty of supposing that we can lose the ns prefix from interfaces in Mozilla 2's C++ bindings, using a proper C++ namespace [not shown] instead.)

I should add that we will not break API compatibility gratuitously; some of our APIs are fine, thank you. "No big rewrites" in the sense that Joel writes about (throw it all away and start over). And much implementation code will be kept, but transformed.

Now, we can't hope to achieve these code transformations by hand. For one thing, much of our code is not exception-safe. This is where Oink comes in. It would take an army of OCD-savants typing at 120wpm a long time to convert all of Mozilla's hand-crafted allocate/free and lock/unlock code patterns to RAII, but with help from Oink's front end Elsa, we can automate the task of rewriting the source. And (with some flow-sensitive work on the Oink side) we should be able to check that we've converted every last exception-unsafe case to use RAII.

Oink, or really its friend Cqual, is also good for more involved static code analysis than finding patterns to rewrite and ensuring exception safety. With the right qualifiers, we can enforce higher level safety properties such as "no format string comes from the network", or "chrome (UI) code must sanitize content data that flows into it".

I think we should extend the Oink framework to include JS2 in the analysis, for total data flow. With the type system for JS2, we'll finally have type soundness on which to build higher-level model checkers.

Building this Oink-based tooling won't be done in a day, and we should be careful not to overinvest if simpler techniques suffice. But from what Roc, Graydon and I have seen, Oink looks quite promising.

So Mozilla 2 is not just about simplifying APIs, removing old code and XPCOM overhead, and making the source code more approachable. It's also about material improvements to program security, which is inherently weak in all browsers implemented in languages such as C and C++. Security requires defense at every level of abstraction, from high-level JS that enforces confidentiality properties, down to buffer manipulations that should be provably memory-safe.

There is no silver bullet. Virtual machines are, as Michael Franz and others point out, a great way to reduce the size of one's TCB and track information flow, supporting richer security models and policies -- safe "mashups in the browser".

We will optimize the JS2 VM aggressively in the Mozilla 2 timeframe. But we can't switch to "managed C++" (neither can Microsoft, notice) for any near term competitive browser, nor is JS2 the right language for the low-level systems programming that lies on critical rendering and interaction paths in any browser.

We will combine approaches, moving as much "middleware" C++ as we can, when it's not on any critical path and it uses only safe pointers, into JS2. Again Oink/Elsa can help to automate this translation. I envision a checker that first scores all code against a series of costs of translating from C++ to JS2, identifying the low-hanging fruit statically. Profiling results for common user-level tasks and page load tests should be used to veto any low-cost judgments that might translate C++ that's otherwise ripe for JS2, but that actually ends up dominating a critical path.

Whether we invest in C++ to JS2 automation, do it by hand, or use a hybrid of Oink-based and by-hand techniques, I don't want to decide right now. But as with deCOMtamination, we have an opportunity to work smarter by investing in tools instead of spending programmer talent in repetitive and mostly menial labor on a very large scale.

The C++ that we can't move to JS2 can still be made more secure by combining the other key Mozilla 2 levers: conservative GC (which removes free memory read hazards), Oink-based checkers (with appropriate new qualifiers annotating our source), and dynamic 24-by-7 valgrind plus gcov tinderboxes running automated tests.

Finally, we will run at reduced privilege when we can, and otherwise use OS-enforced security mechanisms that are sound and well-documented. We should put most plugins out of process while we're at it.

I haven't really begun to talk about further graphics work (3D canvas) and security models (those safe browser-based mashups). Nor have I begun to discuss Firefox 4, the likely version to be based on Mozilla 2, except to say that we will keep an unbranded Firefox version building at all times as Mozilla 2 is developed. Mainly I am focusing on the Mozilla platform, on which Firefox, its add-ons, and other apps all stand or fall.

So the goals for Mozilla 2 are:

Clean up our APIs to be fewer, better, and "on the outside." Simplify the Mozilla codebase to make it smaller, faster, and easier to approach and maintain. Take advantage of standard language features and fast paths instead of XPCOM and ad hoc code. Optimization including JIT compilation for JS2 with very fast DOM access and low memory costs. Tool-time and runtime enforcement of important safety properties.

Oh, and isn't it time that we get off of CVS? The best way to do that without throwing 1.9 into an uproar is to develop Mozilla 2 using a new Version Control System (VCS) that can merge with CVS (since we will want to track changes to files not being revamped at first, or at all; and we'll probably find bugs whose fixes should flow back into 1.9). The problem with VCSes is that there are too many to choose from now. Nevertheless, looking for mostly green columns in that chart should help us make a quick decision. We don't need "the best" or the "newest", but we do need better merging, branching, and renaming support.

Last point: much of what I wrote here, much of my work in Mozilla, is focused on the platform, yet I noted above that we always put apps such as Firefox first, and do not claim to be "a platform" for everyone. In spite of this, people are building apps such as Songbird on top of XULRunner.

So what are we, platform or app? The answer is "both, in a virtuous cycle". Because we serve users first and most broadly, but also developers at several layers -- Web, XUL, C++ -- we have both undersold and under-invested in the C++ layer. In the long run, neglecting the C++ codebase puts the app at risk. So with Mozilla 2, we're going to balance the books.

This is enough for now; detailed roadmap and wiki work will follow. I'm intent on moving the Mozilla codebase to a true next level: cleaner, leaner, safer, with better APIs and C++ bindings, and very fast page-load and DOM performance. While this is easier said than done, it is palpably within our reach for 2008. Let's do it.

The talk I gave last Friday at XTech 2006 in Amsterdam, a revised and extended version of the talk I gave at the Ajax Experience the previous week in San Francisco, is here.

Graydon has been busy. The XPCOM garbage cycle collector lives. Huzzah!

One (half-way) down, two to go....

/be

Mark Hammond's work to support Python in XUL is nearly done. The DOM_AGNOSTIC2_BRANCH should land in the next few weeks. Already I see many on the PyXPCOM list testing Mark's fine work, chomping at the bit to use Python in XULRunner.

This brings to mind a hot topic in my recent hacking: infusing JS with Pythonic generators and iterators, including array comprehensions. Brief taste:

js> function count(n) {
    for (var i = 0; i < n; i++)
        yield i;
}
js> g = count(10)
[object Generator]
js> g.next()
0
js> g.next()
1
js> two_to_nine = [i for i in g]
2,3,4,5,6,7,8,9
js> squares_to_20 = [i * i for i in count(20)]
0,1,4,9,16,25,36,49,64,81,100,121,144,169,196,225,256,289,324,361

Why borrow from Python for iteration, generators, and comprehensions? I typed this into a wiki recently:

Given the years of development in Python and similarities to ECMAScript in application domains and programmer communities, we would rather follow than lead. By standing on Python’s shoulders we reuse developer knowledge as well as design and implementation experience. The trick then becomes not borrowing too much from Python, just enough to gain the essential benefits: structured value-generating continuations and a general iteration protocol.

The specific context was iterators and generators, not general Python imitation. There is no point in trying to turn JS into Python, but I see little to gain from differing gratuitously where the two languages are already quite similar (arrays/lists, for/in loops).