CesiumJS Migrates to ES6 Modules

When CesiumJS 1.63 is released on November 1st, it will contain one of the largest architectural refactors in Cesium’s history. We’ve migrated from Asynchronous Module Definitions (AMD) to ES6 JavaScript modules. This allows both Cesium maintainers and end-users to take advantage of modern JavaScript tooling in addition to making CesiumJS apps load faster by shrinking the library’s overall bundle size.

This post details everything you need to know before upgrading and dives into the architectural challenges it took to upgrade over 300,000 lines of code that make up CesiumJS. Here are the highlights:

If you use the combined build of Cesium.js, such as including it through a script tag, no changes are required.
If you use webpack, Rollup, or other modern build systems that support ES6, your configuration will most likely work as-is. You may have to tweak your configuration to treat Cesium as ES6 modules instead of AMD. Rather than include individual modules in your application, it’s recommended that you use Source/Cesium.js via named imports.
If you exclusively use AMD modules and your tooling does not support ES6 modules, you’ll need to switch to requiring in the combined Cesium.js file as either a UMD module or through a global script tag. You can also take the leap and update your own application to ES6.
If you are using Cesium through Node.js, no code changes are needed. Cesium now sets the "type": "module" and "module": "Source/Cesium.js" options in package.json to take advantage of ES6 module support in newer versions of Node.js. This should simplify Cesium usage through Node for developers who have already made the jump to ES6 on the server. For older versions of node that do not support ES6, we use an adapter library, esm, for loading CesiumJS modules a commonjs environment.
The file size of Cesium’s Web Workers decreased from 8384KB (2624KB gzipped) in Cesium 1.62 down to 863KB (225KB gzipped) in Cesium 1.63. This is an over 10x reduction in size and makes initialization faster, especially on low-end devices and slower network connections.
For tooling that support it, Cesium applications can now take advantage of tree-shaking to reduce overall application size and avoid including parts of Cesium you aren’t using.
For tooling that uses Node.js resolution for front-end packages, such as Rollup’s rollup-plugin-node-resolve or webpack’s resolve.mainFields option, you can now import Cesium directly without having to specify a path to the root module:
import { Viewer } from 'cesium';
var viewer = new Viewer('cesiumContainer');
Even though we’ve moved to ES6, the combined Cesium.js file still supports browsers that lack ES6 support, such as IE 11. As long as your build system can target IE 11, Cesium ES6 modules will continue to work without requiring additional transpilation.
We’ve also updated the cesium-webpack-example to illustrate some of the above features.

Diving Deeper

While we’re extremely happy with how the ES6 transition turned out and the minimal churn it should cause our users, getting there was a harrowing journey. The concern wasn’t the actual migration of the source modules to ES6, but all of the infrastructure, tooling, and packaging that surrounds it.

Pre-ES6 Cesium was ~326,000 lines of JS code, half of which was unit tests. All of it was AMD modules. This doesn’t include our demo application, Sandcastle, which contains over 200 live coding demos. Cesium also makes extensive use of Web Workers and about two dozen small third party libraries. A subset of Cesium is also supported in Node.js. If any one of these things could not be made to work with ES6, the effort would fail.

The worst part is that no amount of preparation would guarantee I wouldn’t hit an unexpected showstopper at any point in the process. I just had to dive in and hope for the best. The below account is by no means exhaustive, but provides what I feel are the more interesting parts of the migration.

Converting to ES6

Figuring out the best way to convert Cesium’s ~1300 JavaScript files from AMD to ES6 was the first step of my journey. I looked into off-the-shelf solutions, such as amd-to-es6, but unfortunately they didn’t produce usable output in all cases. I wanted to avoid reformatting the entire codebase up front and do as little as possible to the code outside of the ES6 migration. I also wanted to avoid hand-editing files because it was likely I’d discover an issue after the initial conversion was done and have to go back and reconvert the entire codebase again.

Thankfully the well-formatted nature of Cesium made writing a custom script straightforward. We actually already had a script that formatted our AMD modules and sorted our require statements, so rather than start from scratch I was able to adapt it for ES6 translation.

Translating to ES6 had 4 parts:

Replace the AMD define call with ES6 imports.
Remove use strict; directives, since ES6 modules are always strict.
Remove any early returns since they are disallowed in ES6 modules.
Update the return statement at the end of the module to use export.

Third-Party code

A lot of Cesium’s third-party code already had a hand-written wrapper for AMD. It was easier to ignore this folder in my script and manually update them to ES6 modules. In some cases, I had to de-UMD-ify modules because it caused problems when importing the same code in both Node.js and the browser. I basically had to update the wrapper of every third-party library to fix these problems.

One of the common questions we get about Cesium is why we don’t just use third-party code directly from npm packages. We would love to, but the lack of standardized modules makes it almost impossible. It requires too much manual customization to ensure third-party code works everywhere that Cesium works. Once ES6 modules are everywhere, I’m hopeful that will finally change, but we’re not there yet.

The entire conversion process took about a day. I knew the AMD modules had a lot of boilerplate compared to ES6, but I was still surprised at just how much. Switching to ES6 shaved off over 17,000 lines of source code. From ~326,000 down to ~309,000.

Baby Steps

At this point in the process, all of our source code and unit tests were in ES6 module format. I couldn’t run the unit tests or any of our build scripts, but in theory I could load up our simple Cesium Viewer reference application in the browser:

Cesium ES6

While there’s no globe, most of CesiumJS loaded as ES6 modules on the first try. Lack of Web Workers prevented the Earth from displaying.

I had never been more excited about a mostly blank screenshot in my life. All of the widgets and other core infrastructure code loaded and executed without any issue. The reason the Earth didn’t appear is because Web Workers were broken, which was my biggest concern before I started. It also made my next goal crystal clear.

Web Workers

Cesium’s Web Workers infrastructure is one area I assumed would require a major rewrite. Cesium Workers relied heavily on dynamic importing of AMD modules and I wasn’t sure if what we were doing would translate cleanly to ES6. My assumption was that I would rewrite it to load ES6-based workers during development, possibly refactoring away the dynamic imports, and run IIFE-compiled versions of the same code in production builds. This would allow for quick iteration by avoiding what was traditionally a costly build step.

My plan immediately fell apart. As it turns out, while the official Web Worker specification allows for loading of ES6 modules, no browser actually supports modules as Workers. There would be no way for me to load our ES6 code in unbuilt form. I would have to re-architect our entire worker infrastructure before I could even attempt to get the Earth to show up. I was devastated so I decided to move on to something else, Cesium’s build system.

Even though I hadn’t used it yet, I had my eye on Rollup for quite some time and wanted to replace the requirejs optimizer, our previous build tool of choice, with it. While doing some initial prototyping, I realized that Rollup could actually produce AMD modules as output. Could it be that easy? Could I add a simple build step to compile our ES6 source code back into AMD and use the existing infrastructure as-is? Rollup turned out to be a joy to work with, and in less than 30 minutes, I was ready to load Cesium again:

Cesium ES6

You’ve probably seen this default Cesium home view a million times, but never via ES6 modules.

In the end almost nothing had to change about Cesium’s Worker infrastructure, which was especially nice since I thought it would be a major barrier to ES6. This is both a testament to Rollup as well as the original forethought that went into Cesium’s Worker infrastructure to begin with. A big thank you to Scott Hunter who did Cesium’s original Worker implementation.

The only significant change was that Workers now always require a build step. Thankfully, a similar process already exists for Cesium’s GLSL Shaders and rollup is fast enough that it has minimal impact on how we develop.

The unexpected win here was that Rollup’s automatically shared code detection provided a huge 10x reduction in Worker size, as I stated in my opening highlights.

Building and Packaging

Having gotten up-to-speed with Rollup, updating Cesium’s build system to generate combined and minified ES5 Cesium.js file seemed like the next logical step. This ended up being mostly trivial, replacing calls to requirejs.optimize in our build system with calls to rollup. Overall build time is significantly faster. Our CI process decreased from ~17 minutes down to ~11. And generating a full release package decreased from 112 seconds down to 69. This was a huge unexpected benefit for maintainers.

Cesium makes heavy use of pragma comments to remove debug code. This provides greater than a 5% performance improvement for production builds. We wrote and published a small Rollup plugin, rollup-plugin-strip-pragma, for removing these from our own release builds to go along with a similar webpack plug-in we previously published. While it’s an optional step, if you are using Rollup or webpack with Cesium, we recommend you use them in production builds.

The only functional change to the actual build output is that the combined Cesium.js is now a UMD module. It should work anywhere that hasn’t made the jump to ES6 yet and still works out of the box on legacy browsers, such as IE11.

Unit Tests

It was relatively easy to update our Karma configuration to load ES6 unit tests. This allowed for running unbuilt Cesium in ES6 compatible browsers. However, Cesium historically has the capability to run unit tests against the combined and minified version as well, this catches edge cases and other unexpected behavior that isn’t detectable with the unbuilt version, such as minification-only issues. Using the built version would also be required for testing Cesium in legacy browsers, such as IE11.

I still had Rollup on my mind from the packaging refactor I had just finished so I immediately realized we could “build” the specs, just like we build Cesium, which would turn them back into a single IIFE file that could be loaded by any browser. The only problem was that building the specs would also build Cesium directly into the specs. I needed them to use Cesium as a global external instead because the goal is to test the generated bundle.

Rollup has some built-in support for externals, but it focuses on npm modules and doesn’t work with relative paths. Thankfully, I found rollup-plugin-external-globals which did exactly what I needed. As a concrete example, here is one of Cesium’s unit tests:

import { Cartesian3 } from '../../Source/Cesium.js';

it('construct with default values', function() {
    var cartesian = new Cartesian3();
    expect(cartesian.x).toEqual(0.0);
    expect(cartesian.y).toEqual(0.0);
    expect(cartesian.z).toEqual(0.0);
});

Usingrollup-plugin-external-globals, this gets transpiled into the below:

it('construct with default values', function() {
    var cartesian = new Cesium.Cartesian3();
    expect(cartesian.x).toEqual(0.0);
    expect(cartesian.y).toEqual(0.0);
    expect(cartesian.z).toEqual(0.0);
});

Now that Cesium is being treated as a global object, I can simply include the built version in a script tag as part of our test initialization and be confident that our minified code works as expected.

For coverage, we needed to use karma-coverage-istanbul-instrumenter for native ES6 support, but it was a drop-in replacement. Thankfully, our coverage numbers weren’t heavily impacted by the removal of AMD boilerplate and Cesium unit tests still cover over 93% of the codebase.

Like Web Workers, I expected running the unit tests against combined Cesium.js to be a major problem, but the above strategy allowed me to replace the guts of our unit testing infrastructure without actually changing anything from a maintainer’s perspective. JavaScript tooling has definitely come a long way since we started Cesium.

Sandcastle

The final piece of the puzzle was Sandcastle, our live coding prototyping and demo application. It is one of the oldest Cesium applications in existence and its Dojo 1.x based interface is long overdue for a rewrite. Since such an undertaking was outside the scope of our ES6 module migration, I had to determine what the minimum requirements were for declaring victory regarding ES6. I came up with two goals:

Sandcastle is used by the Cesium maintainers for prototyping, debugging, and one-off testing. To keep iteration fast, we need Sandcastle to support using the unbuilt ES6 modules directly.
Sandcastle needs to continue to work on legacy browsers that do not support ES6, but this can require using the Cesium.js UMD modules if needed.

Researching how to incorporate ES6 code into legacy applications, I stumbled onto the nomodule attribute. This allows you to specify both ES6 and non-ES6 versions of a script and both legacy and modern browsers will only load the version they actually support. With this mechanism in place, we can load the built Cesium.js file in browsers like IE 11, but use Cesium ES6 modules directly when developing Cesium features. The HTML looks something like this:

<script type="text/javascript" src="../../../Build/CesiumUnminified/Cesium.js" nomodule></script>
<script type="module" src="../load-cesium-es6.js"></script>

On legacy browsers, scripts with type="module" are ignored because the specification states that any unknown script type should cause the element to be ignored. This means in IE11 only the first script element is processed and the Cesium global is immediately available to Sandcastle demos.

On newer browsers that support ES6 modules, you would expect both script elements to be processed, but nomodule exists to solve this problem. It tells browsers to ignore the first script tag and therefore only the second script element is processed. nomodule is part of the ES6 Module specification, so the above pattern guarantees only one of the two script elements is ever processed.

Cesium Source/Cesium.js is itself an ES6 module that doesn’t affect global state, load-cesium-es6.js is a small shim which loads Cesium as a global variable and then executes the startup function that exists in all demos.

Cesium has over 200 Sandcastle examples, so I had to write another script to update the boilerplate to use the new mechanism but it was mostly an exercise in find-and-replace. All previously shared Sandcastle demos which store data in a GitHub gist or encoded URL will continue to work as well.

The bottom line is that a 9 year old, dojo-based ES5 application like Sandcastle was able to transition to using ES6 Cesium without having to be completely rewritten.

What’s next?

JavaScript continues to be a fast-paced ecosystem with ever-changing tooling options. In general, our approach has always been to go slow and use battle-tested systems that are known to work well, rather than what has the newest or coolest feature-set. That being said, there are a couple of things I would love to incorporate into Cesium that are now only possible because of the move to ES6 modules and Rollup.

Source Maps

If you use through Cesium ES6 modules, Source Maps are already supported, but switching to Rollup means we can finally generate Source Maps for built versions of Cesium. This makes debugging problems in production easier for many of our users. Unfortunately, simply turning the option on in Rollup lead to some errors that we need to address before this can happen. However, I expect Source Maps to be available as early as CesiumJS 1.64.

Use modern JavaScript through Babel

Since Chrome, Firefox, and Safari already support some of the latest JavaScript features, we should be able to keep our minimal build during development policy in place and start to leverage newer features such as async/await/Promise/generators/etc… We would still need to run the final output through Babel as part of the build process, but Rollup already has good support for this step. This would make our code a lot nicer with almost no overhead during the development process. However, its impact on our runtime performance, code size, and global scope needs to be further evaluated before we can officially adopt it in Cesium.

CesiumJS 1.63 with ES6 support ships on November 1st, feel free to reach out on our forum if you encounter any issues updating your application or have any additional questions about our migration to ES6.

Finally, thank you to the community who helped beta test the ES6 migration or otherwise provided feedback and encouragement. I learned a lot throughout the process and I don’t think it would have turned out as well as it did without your help.