Geopipe's ContextSnap Streams Entire Cities with Cesium and 3D Tiles
This is a guest post by Christopher Mitchell of Geopipe about their tool ContextSnap, which allows users to explore millions of buildings, all streamed in the browser with Cesium and 3D Tiles.
Geopipe’s New York City loaded in Unity with its SDK (left) and Geopipe ContextSnap, powered by Cesium with 3D Tiles (right).
At Geopipe we’ve built a tool called ContextSnap that can stream entire cities into your browser using Cesium and 3D Tiles. With ContextSnap you can navigate a 3D copy of a city within your browser and click on buildings and other objects to see semantic information: height, number of floors, materials, address, and more. If you need to take 3D models out into your own workflow, you can select your desired area and immediately download a 3D mesh model in your favorite file format.
Why are we building these models? From gaming to training autonomous vehicles, from architecture to defense, perfect virtual copies of the real world are increasingly vital. Unfortunately, the real world is complex, dense, and nuanced, and with today’s hand-built 3D models of cities it can take a team of professionals months to reproduce even a single neighborhood in realistic detail. While several well-known companies have automated the process, the resulting models, usually made with a technique called photogrammetry, simply don’t have the material or object information necessary for renderings or interaction.
But we realized that computers could be taught to understand the world without human-powered labeling, automatically building rich, detailed 3D models that contain information about what’s in the world, not just how it looks. Geopipe uses deep learning and other algorithmic techniques to understand cities and other areas from raw, unlabeled sensor data. These algorithms can determine every detail of the world - from where buildings are, what they’re made of, and where their windows and doors are, to the locations and properties of roads and their lanes, sidewalks, trees, and water.
It takes our pipeline less than a day to process an entire city, from raw data to complete labeled 3D models.
Geopipe uses algorithms to build perfect 3D copies of real cities, and Geopipe ContextSnap is your tool to instantly download exactly the model you need.
Once these models have been generated users need a way to explore and use them. Many of our customers want to view the resulting cities, navigate the 3D world, click on objects to see semantic information, and occassionally overlay their own information on top of those models. That’s where Cesium and 3D Tiles come in.
Geopipe reconstructs hundreds of thousands (often millions) of buildings and other objects in each city. At the finest level of detail, every pane of every window is reproduced, and no computer on the market today could load an entire city at that level of detail at the same time, especially in a browser. On the other hand, it must be possible to view our models from any perspective within the city. With 3D Tiles, we can severely cull the geometry that needs to be loaded to show the entire city at any given time, with the camera anywhere within the city.
We automatically generate our 3D models at up to 13 different levels of detail and geographic scale, facilitated by knowing the difference between (for example) buildings, trees, roads, and sidewalks, and downsampling each type of object intelligently. For medium levels of detail, we simplify models (use fewer polygons for trees, for example, or replace windows cut into the sides of buildings with facade features painted via texture). At our lowest detail models covering the broadest geographic areas, we even cull small buildings and foliage: these models will be loaded far from the camera, so the simplifications won’t be obvious.
Since a 3D Tiles tileset is a hierarchical arrangement of models covering an area at different levels of detail, the only remaining missing piece is the hierarchy of tileset.json metadata files. We feed our models into Cesium via a server that dynamically generates the 3D Tiles tileset from our models and their metadata. Because we use batched 3D models, the underlying metadata describing individual objects is retained in models Cesium displays, so Cesium’s API can be used to display information about those objects when a user clicks on them. The results is Geopipe ContextSnap.
Inspecting 3D buildings in New York City via Geopipe ContextSnap before downloading an FBX file.
Other customers want to go further and download models that they can pull into their existing toolchain for editing, rendering, and interaction. ContextSnap is still part of this workflow. Customers can preview a version of the models they’ll be getting from Geopipe (we limit ContextSnap to all but our two highest levels of detail, due to the memory and CPU constraints of WebGL), then drag a box around the area they want, choose the level of detail and file format appropriate to their use case, and download the model immediately.
Geopipe for Games SDK demonstrated in Unity and other games and tools, powered by concepts similar to 3D Tiles.
In fact, 3D Tiles are sufficiently expressive that Geopipe is using similar concepts in its Geopipe for Games SDK, a plugin for the Unity game engine and an underlying RESTful API that allow customers to directly stream Geopipe’s worlds into games, VR experiences, simulations, and other software. Resources like CPU and GPU memory are still dear, so presenting as immersive a world with as little geometry loaded as possible is important, while still showing the perspective from anywhere in a city. Once again the solution is loading immersive, high detail models close to the player or viewer and broad, lower detail models far from the player. Thus miles of a city and distant oceans, islands, and mountains can be visible at the same time while the framerate is kept at acceptable levels even on unexceptional hardware.
Whether you need 3D models for your own game, simulation, or AEC project, you need to stream models of the real world into your own application, or you need to analyze data about the objects in a city at scale, Geopipe can help. And with Cesium and 3D Tiles, customers can inspect and explore a city containing millions of objects in their browser.