The State of Cesium ion Architecture: The Infrastructure behind the Platform for 3D Geospatial

Since its inception, Cesium ion has served the 3D data and visualization requirements of millions of users. Cesium ion enables users to upload heterogenous 3D data and converts it into web streamable formats through computationally intensive jobs known as tiling. It then allows them to seamlessly stream this content in real time into a multitude of viewing engines while efficiently managing access to it. Currently ion manages over 40 million requests daily while storing over 100TB of data, with this number always growing. The architecture underpinning this data intensive application has been carefully crafted and bettered over the years. 

To build up to ion’s architecture, let’s take a step back and look at a non-exhaustive list of requirements we are trying to achieve: 

1. Uploading: As a platform that champions open interoperability, allow users to efficiently upload diverse source 3D data, including content from third parties like Sketchfab, from around the globe.
2. Tiling and Pipelines: Enable users to convert their uploaded source data into a web-streamable format and enable them to archive, clip, and export this data.
3. Hosting and Streaming: Facilitate hosting and streaming of globally curated 3D content, such as Cesium World Terrain, Cesium World Bathymetry, and Cesium OSM Buildings. Additionally, host user-uploaded tiled data and stream it while enabling users to manage authorization.
4. Functional API: Provide users with the ability to programmatically leverage Cesium ion’s capabilities listed above via an API.
5. Frontend interface: Ensure Cesium ion’s functionalities are accessible to non-technical users through an intuitive interface.
6. Security: With users entrusting us with their data, it is crucial that security is front and center in our considerations. This involves not only implementing security best practices in our application code but also extending these practices to our infrastructure.

The following section tries to provide a high level overview of the architecture representing the inner workings of the publicly exposed sides of ion.

1. API Server

Cesium ion’s API server lies at the heart of the application. The server manages user and asset metadata such as tiling progress, asset type, and user usage, while also facilitating source data uploads, executing tiling jobs and authorizing access to tilesets. It utilizes temporary credentials to allow users to upload directly to S3, providing highly available and quick uploads all around the world.

The API server itself is a Node.js based application that follows REST principles. It is containerized with docker and deployed through AWS Elastic Container Service (ECS) behind an Application Load Balancer. It is backed by a PostgreSQL database instance from AWS RDS for metadata storage. 

Since our user’s applications are often products that are derivatives of ion, resilience and availability are essential qualities. The above architecture ensures that we are providing a highly available API to our users through health checks, autoscaling, and load distribution.

2. Tiling

Tiling, as stated above, is the process of converting user uploaded source data into web compatible formats like 3D Tiles, KML and GLTF. Since ion accepts a wide variety of formats, each of these conversions has its own distinct hardware requirements while tiling. The tiling pipeline also handles archiving, clipping and exporting user data. Due to the requirement for variable hardware per job, we leverage AWS Batch to perform these functions.

AWS Batch allows us to customize hardware requirements for each job and runs them in environments blocked off from each other. This ensures that user data remains safe and isn’t tampered with by another running job. AWS Batch also places jobs into a queue until they can be executed in a compute environment, allowing ion’s tiling process to be extremely scalable by leveraging the asynchronous nature of Batch’s queue.

3. Asset Hosting

At present, Cesium ion's storage accommodates approximately 3 trillion distinct objects. Managing such a large amount of files on a file system can lead to considerable complexity. Hence, we utilize S3 for our storage solution, allowing assets to be served as a static collection of files.

We employ CloudFront, the CDN provided by AWS, which caches requests at edge locations close to the user and utilizes temporary JSON Web Tokens to validate user requests directed towards our assets. These authorization checks are handled by Lambda@Edge functions for optimized response times. Hosting our assets in AWS S3 and distributing them via CloudFront offers high scalability and extremely low latency globally.

To further diminish response times for locations distant from our primary location (Virginia us-east-1), we leverage AWS S3's replication functionality. This allows us to replicate assets to additional locations. Presently, every asset generated by tiling jobs is replicated to Tokyo (ap-northeast-1), with plans for additional regions in the future. Similar to authorization, lambda@edge functions are used here as well to route requests based on geographical proximity. 

4. Front end

The Cesium ion SPA (Single Page Application) is a statically built web application. Similar to our other statically served assets, it resides on S3 and is distributed through CloudFront. It offers a user friendly interface that sits on top of the complex architecture beneath. The UI leverages the same endpoints provided by the API server for its services and functionalities. 

As the above description suggests, all of ion’s code is containerized. It allows us to reuse the code in other container-based solutions. We have already done this using Kubernetes, which will be covered in a future blog.

To top this all off, Cesium ion employs automated deployments to accelerate development and reduce time spent on manual deploys. Our CI automatically builds and deploys containers for application code, while also updating any necessary infrastructure seamlessly. This approach allows us to deploy multiple times a day and efficiently deliver new features and bug fixes to our users.

The complexity of ion's system lies in its multifunctional nature, serving as a crucial part of the Cesium ecosystem. So, the next time you initiate an upload on ion for tiling your asset, or access 3D content through the CesiumJS client or a game engine plugin, remember that there is a sophisticated architecture behind the seemingly simple user interface.

Ready to experience the platform first hand? Sign up or sign in to your account.