Cesium ion

Getting started

This step-by-step guide will help you configure and deploy Cesium ion on Kubernetes. While we expect you to already be experienced with running Kubernetes workloads, this guide is written to take you from zero to Cesium ion with minimal prior knowledge. If you’re updating an existing installation, see Upgrading from a previous release.

What’s included

  • startHere.html - Documentation (this file)

  • cesium-ion/ - Helm chart

  • images/cesium-ion.tar - Container image for the Cesium ion API

  • images/cesium-ion-asset-server.tar - Container image for the Cesium ion asset server

  • images/cesium-ion-frontend.tar - Container image for the Cesium ion user interface

  • images/cesium-ion-tiling.tar - Container image for the Cesium ion 3D tiling pipeline

  • images/cesium-ion-job-watcher.tar - Container image for the job watcher to detect abnormally terminated jobs

  • images/postgresql.tar - Container image for PostgreSQL

  • scripts/ - Scripts that allow you to run the tiling pipeline from the command line.

  • restApi.html - REST API reference

  • sampleData/ - Sample data used throughout the documentation

  • thirdParty/ - Additional files used by the documentation

System requirements

Any x86-64 compatible system capable of running Kubernetes can also run Cesium ion. Requirements for production workloads will depend directly on your use case and Cesium ion performance will scale with additional CPU, RAM, and storage performance. For local development, we recommend a computer with the following minimum properties:

  • An x86-64 compatible processor

  • 8 or more CPU cores (see Configuring Resources)

  • 32GB of RAM or greater.

  • At least a 32GB of volume storage

Additionally, to follow this guide you will need sudo, admin, or similarly elevated permissions as well as a tool for working with container registries, such as Docker or podman.

Kubernetes has a diverse set of configuration options. The first time you install Cesium ion we recommend following this guide closely to avoid introducing uncertainty in the setup process. Once you are comfortable with the Cesium ion Helm chart and configuration, you can further customize it to your specific needs.

Installing microk8s

This guide uses microk8s, a lightweight and easy to configure Kubernetes implementation meant for local development. If you have an existing Kubernetes cluster you would like to use, you will need to update the supplied commands to those available with your Kubernetes installation. Skip to the Importing Images section if you are not installing microk8s.

  • Linux

  • Windows

Run the below command to install microk8s using snap.

sudo snap install microk8s --classic

Update permissions and configuration

Once installed, run the below command to add yourself to the microk8s user group

sudo usermod -a -G microk8s $USER

You also need to create a .kube directory in your home folder to store microk8s configuration. Run the following commands:

mkdir ~/.kube
sudo chown -f -R $USER ~/.kube
sudo microk8s config > ~/.kube/config

Since we made changes to your group settings, you must log out and log back in before continuing.

We recommend using WSL2 to set up a microk8s environment. Use the official provided WSL2 microk8s installation instructions from: https://microk8s.io/docs/install-wsl2

Update permissions and configuration

Once installed, run the below command to add yourself to the microk8s user group

sudo usermod -a -G microk8s $USER

You also need to create a .kube directory in your home folder to store microk8s configuration. Run the following commands:

mkdir ~/.kube
sudo chown -f -R $USER ~/.kube
sudo microk8s config > ~/.kube/config

Since we made changes to your group settings, you must close out of WSL2 and run it again for the changes to take place.

When working with WSL2, you can easily access files on your root drive but you must use Unix style paths. For example, if your root drive is C:\ you can access files at \mnt\c

Verify installation

Verify the installation by running the following command:

microk8s status --wait-ready

This should produce output similar to the following. The important part is "microk8s is running". If you receive an error, review the Installing microk8s section.

microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
    ingress              # (core) Ingress controller for external access
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    registry             # (core) Private image registry exposed on localhost:32000
  disabled:
    cert-manager         # (core) Cloud native certificate management
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    gpu                  # (core) Automatic enablement of Nvidia CUDA
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Load balancer for your Kubernetes cluster
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorization
    storage              # (core) Alias to hostpath-storage add-on, deprecated

Verify required features

The default Cesium ion configuration requires dns, helm3, ingress and registry to be enabled. If they are not shown as enabled in the output from the previous command, run the following commands:

microk8s enable dns
microk8s enable helm3
microk8s enable ingress
microk8s enable registry
microk8s config > $HOME/.kube/config

Microk8s installs kubectl and helm. You can run them as microk8s kubectl and microk8s helm to administer the cluster.

Importing images

Run the below commands to import the images into the microk8s registry add-on. The registry is created at localhost:32000. Importing these images may take a few minutes for each step. If you are using podman or other Docker alternative, be sure to update the commands for your tooling.

All commands throughout this documentation are assumed to be executed from the top-level directory where you unpacked the zip, i.e. the directory containing startHere.html. 
docker image load --input images/cesium-ion-asset-server.tar
docker tag cesiumgs/cesium-ion-asset-server:1.3.0 localhost:32000/cesium-ion-asset-server:1.3.0
docker push localhost:32000/cesium-ion-asset-server:1.3.0

docker image load --input images/cesium-ion-tiling.tar
docker tag cesiumgs/cesium-ion-tiling:1.3.0 localhost:32000/cesium-ion-tiling:1.3.0
docker push localhost:32000/cesium-ion-tiling:1.3.0

docker image load --input images/cesium-ion.tar
docker tag cesiumgs/cesium-ion:1.3.0 localhost:32000/cesium-ion:1.3.0
docker push localhost:32000/cesium-ion:1.3.0

docker image load --input images/cesium-ion-frontend.tar
docker tag cesiumgs/cesium-ion-frontend:1.3.0 localhost:32000/cesium-ion-frontend:1.3.0
docker push localhost:32000/cesium-ion-frontend:1.3.0

docker image load --input images/cesium-ion-job-watcher.tar
docker tag cesiumgs/cesium-ion-job-watcher:1.3.0 localhost:32000/cesium-ion-job-watcher:1.3.0
docker push localhost:32000/cesium-ion-job-watcher:1.3.0

docker image load --input images/postgresql.tar
docker tag bitnami/postgresql:15.4.0-debian-11-r0 localhost:32000/postgresql:15.4.0-debian-11-r0
docker push localhost:32000/postgresql:15.4.0-debian-11-r0

Troubleshooting importing images

Command "docker" not found

If you are running under WSL2, you will need a docker client installed inside of the container, not the host Windows machine. If needed, run the below commands inside of the WSL2 container to install and configure docker.

sudo apt update
sudo apt install docker.io
sudo usermod -a -G docker $USER
exec su -l $USER
Insecure registries

On some versions of Docker, you may receive an error regarding insecure registries. If this happens to you, configure the Docker daemon to allow this action, create or edit /etc/docker/daemon.json to include the following setting:

{
  "insecure-registries" : ["localhost:32000"]
}

You also need restart docker using the below command for the changes to take effect:

sudo systemctl restart docker

License configuration

Cesium ion requires a license, which is configured at the top of cesium-ion/values.yaml. By default it will be an empty string:

license: ""

Install your license by performing the following steps:

  1. Download your license file from https://cesium.com/downloads

  2. Open the license file in a text editor and copy the entire contents into the clipboard

  3. Open cesium-ion/values.yaml in a text editor.

  4. Paste the contents into the license string at top of the file

Volume configuration

The default Cesium ion configuration stores all stateful data across five volumes:

  • cesium-ion-sources - User uploaded source data

  • cesium-ion-assets - Processed source data served as Cesium ion assets

  • cesium-ion-stories - Images and other media uploaded as part of Cesium Stories

  • cesium-ion-postgresql - PostgreSQL database containing accounts, Cesium Stories, and asset metadata.

  • cesium-ion-archives - Processed full asset archives and processed clip and ship output

While Kubernetes has a myriad of storage options, we will use local persistent volumes for ease of setup on a single machine. Follow the below steps to configure them:

When editing cesium-ion/values.yaml on Windows, file paths should be from inside the WSL2 shell. For example, paths to your C:\ drive need to start with /mnt/c/. Additionally, the postgresql volume must live inside of the WSL2 VM, for example at /home/$USER/postgres, and can not reside under /mnt/c.

  1. Run microk8s kubectl get nodes and copy the name of the node. A node in Kubernetes refers to the hardware where the pod is run. In this case your machine will be the only node.

  2. In cesium-ion/values.yaml find the localPersistentVolumes section

  3. For node, replace # REQUIRED: Name returned by "kubectl get nodes" with the node name from step 1.

  4. Create a directory for your assets.

  5. Under assets, replace # REQUIRED: Path to output data on your filesystem. with the absolute path to the directory you created in step 4.

  6. Create a directory for your source data. This must be different from previous directories.

  7. Under sources, replace # REQUIRED: Path to input data on your filesystem. with the absolute path to the directory you created in step 6.

  8. Create a directory for your stories images. This must be different from previous directories.

  9. Under stories, replace # REQUIRED: Path to stories images on your filesystem. with the absolute path to the directory you created in step 8.

  10. Create a directory for your archvies. This must be different from previous directories.

  11. Under archives, replace # REQUIRED: Path to archives data on your filesystem. with the absolute path to the directory you created in step 10.

  12. Create a directory for the PostgreSQL database. This must be different from previous directories.

  13. Under postgresql, replace # REQUIRED: Path to postgres data on your filesystem. with the absolute path to the directory you created in step 12.

When you are done the localPersistentVolumes section should contain all the information you need for your install.

localPersistentVolumes:
  enabled: true
  node: [RESULT OF "get nodes" FROM STEP 1]
  assets:
    enabled: true
    path: [DIRECTORY FROM STEP 4]
    capacity: 32Gi
  sources:
    enabled: true
    path: [DIRECTORY FROM STEP 6]
    capacity: 32Gi
  stories:
    enabled: true
    path: [DIRECTORY FROM STEP 8]
    capacity: 32Gi
  archives:
    enabled: true
    path: [DIRECTORY FROM STEP 10]
    capacity: 32Gi
  postgresql:
    enabled: true
    path: [DIRECTORY FROM STEP 12]
    capacity: 32Gi

PostgreSQL

Except in advanced use cases, Cesium ion requires a PostgreSQL database. To make initial configuration easier, the Cesium ion Helm chart includes a subchart packaged by Bitnami with a preconfigured user and clear text password. This provides basic configuration for local development, but is not configured for production use.

Once you are comfortable with configuring and installing Cesium ion, you have two options:

  1. Connect Cesium ion to your own, externally managed database. See External PostgreSQL configuration in the "Additional Configuration" section.

  2. Configure the bundled subchart for production use by referring to its official page on ArtifactHub.

We will continue with the default configuration for this tutorial, but remember that the default configuration is meant for getting up and running with minimal effort. Properly configuring a PostgreSQL database for production use is your responsibility and outside the scope of this document.

Configuring Resources

By default Cesium ion will require 6.5 CPU cores to run the servers and a job monitoring POD. Additionally if you are tiling your own data, each job will use 2 cores.

If you want to run a development setup with fewer CPU cores, you need to update the resources subsection under assetServer, apiServer and frontendServer to collectively be less than your desired number of cores.

For running a production setup, it is recommended to increase tilingJob resources to 4 CPU cores. If available, assetServer and apiServer resources would benefit from being set to 4 CPU cores as well.

Running the install command

Your cesium-ion/values.yaml file should now have a complete and valid configuration. Install the chart into a new cesium-ion namespace by running the following command:

microk8s helm install cesium-ion cesium-ion/ --wait --values cesium-ion/values.yaml --namespace cesium-ion --create-namespace

This process takes about a minute. There is no requirement to use a specific namespace and we are simply following best practices. Once installation is complete, you should see output similar to the below.

NAME: cesium-ion
LAST DEPLOYED: Sun Nov 19 11:31:17 2023
NAMESPACE: cesium-ion
STATUS: deployed
REVISION: 1
NOTES:

The above indicates success. If you received an error instead, run microk8s helm uninstall cesium-ion --namespace cesium-ion to ensure any partially installed components are removed. Then review this section to ensure you didn’t miss a step and try again.

Once the output is successful, the NOTES: section will contain three commands to retrieve the URL of the application. These commands do not mention microk8s so copy and run the correct version below:

export NODE_PORT=$(microk8s kubectl get --namespace cesium-ion -o jsonpath="{.spec.ports[0].nodePort}" services cesium-ion-frontend)
export NODE_IP=$(microk8s kubectl get nodes --namespace cesium-ion -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT

And the output will be a fully qualified URL of the application:

http://10.152.183.244:8080

Visit this URL and the Cesium ion user interface should load. It will look similar to the below image:

If Cesium ion fails to load, uninstall the application and review the above steps again. If your license is expired or invalid, you will instead be redirected to a licensing page with additional information.
Cesium ion Self-Hosted Running with the default configuration
Figure 1. Cesium ion Self-Hosted Running with the default configuration

Verification

After loading the application, you can perform a few basic tasks to ensure everything is working correctly.

  1. Using a file manager, open the sampleData folder included in the installation zip

  2. Drag and drop House.laz into the browser window.

  3. Cesium ion should detect that you are uploading a Point Cloud, click Upload

  4. The asset should upload successfully and you will see an entry appear for it on the My Assets page. Progress information will appear in the preview window when the asset is selected.

  5. Once tiling completes, the asset will load. In this case it’s a small point cloud of a House.

Cesium ion Self-Hosted after tiling House.laz
Figure 2. Cesium ion Self-Hosted after tiling House.laz

After the initial install, additional changes to cesium-ion/values.yaml will not take effect unless you run the helm upgrade command after saving the modified file:

microk8s helm upgrade cesium-ion cesium-ion/ --wait --values cesium-ion/values.yaml --namespace cesium-ion

Next steps

Congratulations, you now have a working installation of Cesium ion running on Kubernetes. While we recommend you read through this documentation in its entirely at least once, where you go next is up to you:

  • The Additional Configuration section provides an overview of the most common and important options for Cesium ion, such as configuring Single sign-on, configuring an ingress and TLS, using an external PostgreSQL server, and installing Cesium Global 3D Content.

  • The Application architecture section provides an introduction to the overall system architecture, services, jobs, and other important information you should become familiar with.

  • The REST API reference documentation provides information on building clients that integrate directly with Cesium ion.

  • The Advanced Topics section describes how to use the tiling pipeline and asset server container images without Kubernetes, Cesium ion’s data management, user interface, or REST API. This includes instructions for running the tilers from the command line.

Additional configuration

Ingress and TLS

The default configuration for Cesium ion provides for IP-based access over HTTP/1.1. While this is acceptable for experimentation or local development, an ingress should be configured for production use to take advantage of DNS, TLS, caching, and the improved performance provided by HTTP/2 or HTTP/3. Follow the steps below to set one up:

Enable the ingress

  1. Open cesium-ion/values.yaml.

  2. Find the ingress section and set enabled to true.

  3. Directly above the ingress section, set localNoDNS to false.

  4. If you will be using an ingress other than the default, enter it in className and update annotations if required. Using a non-default ingress is outside of the scope of this documentation so consult your ingress documentation if needed.

  5. Under the assetServer, apiServer and frontendServer sections, under service:, change type: from NodePort to ClusterIP.

Configure DNS

The default Cesium ion configuration creates three user-facing services: One each for the API, the front end, and asset serving. Each of them will require a hostname to work with the ingress. While these names can be anything, it is recommended to provide them a shared domain. In this example we will use:

  • ion.example - User interface

  • api.ion.example - REST API

  • assets.ion.example - Asset server

Decide on the host names you would like to use or use the above. They can be changed later.

  1. Open cesium-ion/values.yaml.

  2. For each of the frontendServer, apiServer and assetServer sections:

    • Find the endpoint subsection.

    • Set host to the desired name.

Access Cesium ion via DNS

For multi-node clusters a DNS server will need to be configured, which is outside the scope of this document. For local testing and development on a single machine, the hosts file can be updated to point to your ingress. Let’s do that now to validate the above configuration.

Run the below command to apply the ingress and host changes made in the previous section

microk8s helm upgrade cesium-ion cesium-ion/ --wait --values cesium-ion/values.yaml --namespace cesium-ion

Then run the following command to get the IP address of the ingress

microk8s kubectl get --namespace cesium-ion ingress

The output will look similar to the following. In this case the Ingress IP is 127.0.0.1.

NAME         CLASS    HOSTS                                            ADDRESS     PORTS     AGE
cesium-ion   public   assets.ion.example,api.ion.example,ion.example   127.0.0.1   80, 443   20s

Open the hosts file in a text editor. You will need elevated permissions to edit this file:

  • On Linux this file is located in /etc/hosts

  • On Windows this file is located in <Root>\Windows\System32\drivers\etc\hosts

Add an entry for each host name with the IP address returned by the above command. For example:

127.0.0.1       ion.example
127.0.0.1       api.ion.example
127.0.0.1       assets.ion.example

Updates to hosts take effect immediately after the file is saved.

Importing a TLS certificate

While not strictly required, it is strongly recommended to configure TLS to enable support for secure communication. In addition to added security, newer protocols such as HTTP/2 and HTTP/3 provide greatly improved performance but require TLS to be enabled before an ingress can take advantage of them.

While each Cesium ion server can have its own configuration, for simplicity these instructions will use a single certificate for all three servers.

Creating the TLS certificate key pair is outside the scope of this document. Refer to your own internal processes or tools. Ensure the certificate includes the DNS host names you configured in the previous section. Wildcard certificates can also be used.

Once you have the certificate, follow the steps below:

  1. Create a file named certificate.yaml with the below content. Replace the public and private keys with the values from your certificate.

    apiVersion: v1
kind: Secret
metadata:
  name: cesiumion-tls-secret
stringData:
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    [INSERT YOUR PUBLIC CERTIFICATE]
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    [INSERT YOUR PRIVATE KEY HERE]
    -----END PRIVATE KEY-----
type: kubernetes.io/tls

  2. Install the new secret into your Kubernetes cluster by running the below command. The new secret will be named cesiumion-tls-secret with keys of tls.crt and with keys of tls.key. Remember to use the same namespace you used when installing Cesium ion.

    microk8s kubectl create --namespace cesium-ion -f certificate.yaml

  3. Open cesium-ion/values.yaml.

  4. For each of the frontendServer, apiServer and assetServer sections:

    • Find the endpoint subsection.

    • Set tls to true.

    • Set tlsSecretName to cesiumion-tls-secret

If you prefer to use multiple certificates, repeat the above process multiple times with a different certificate and secret name for each server.

Upgrade the application and verify the configuration

Once the ingress, DNS, and TLS are configured, upgrade the application by running:

microk8s helm upgrade cesium-ion cesium-ion/ --wait --values cesium-ion/values.yaml --namespace cesium-ion

  • Linux

  • Windows

Navigate to the configuration application URL, for example https://ion.example. The Cesium ion user interface should load and work the same as before. If you encounter an issue, review the above steps and try again.

In order to expose the kubernetes ingress running on your WSL2 VM, you will need to port forward the ingress pod to the host. To perform this action, you will need to have root privileges so that the network binding from kubectl works.

Run the following command to get the name of the ingress pod:

microk8s kubectl get pods -n ingress

Now expose the kubernetes pod with port-forwarding to the windows host, remember to replace ingress-pod-name with the name from above:

sudo microk8s kubectl -n ingress port-forward pod/ingress-pod-name --address 127.0.0.1 80:80 443:443

You should see output similar to the following:

Forwarding from 127.0.0.1:80 -> 80
Forwarding from 127.0.0.1:443 -> 443

Navigate to the configuration application URL, for example https://ion.example. The Cesium ion user interface should load and work the same as before. If you encounter an issue, review the above steps and try again.

The port-forward command does not exit until you press CTRL-C. Exiting the port-forward command also means ion will no longer be accessible from your Windows desktop. If you plan on doing Windows-based development, it is recommended you have this process run in the background on start-up.

Single sign-on (SSO)

The default configuration for Cesium ion does not include authentication and all users share a single account. To allow users to each have their own account, Cesium ion can integrate with your existing identity provider (IdP) to support SSO via SAML authentication. This can be accomplished by following the steps below:

  1. From within your identity provider, configure a new SAML application. This process will vary depending on your IdP. Cesium ion requires the login URL, entity ID, and SAML certificate.

  2. Cesium ion also has an Administrator user interface for configuring shared assets and application defaults. Access is granted by specifying a specific IdP attribute name and expected value that signals administrator access. For example if configuring a Google Workspace the attribute would be "Groups" and the value would be the name of the Google Workspace group you created for ion administrators. Administrator does not provide access to other user’s data and is only for configuring shared assets and defaults.

  3. Create a file named samlSecret.yaml and add a secret resource with the following contents, replacing the certificate body with your SAML certificate.

    apiVersion: v1
kind: Secret
metadata:
  name: saml-secret
stringData:
  saml.pem: |
    -----BEGIN CERTIFICATE-----
    [INSERT YOUR SAML CERTIFICATE]
    -----END CERTIFICATE-----

  4. Install the new secret into your Kubernetes cluster by running the below command. The new secret will be named saml-secret with a key of saml.pem. Remember to use the same namespace you used when installing Cesium ion.

    microk8s kubectl create --namespace cesium-ion -f samlSecret.yaml

  5. Open cesium-ion/values.yaml, find the authenticationMode property and change it to saml

  6. In the saml section update the certificateSecret, loginRequestUrl, entityId, nameIdFormat, adminAttributeName and adminAttributeValue fields to match those configured in step 1. The saml section should look similar to the following

      authenticationMode: saml

  # If authenticationMode=saml, these are required
  saml:
    # This secret must be created outside the cesium-ion chart
    # It should contain the SAML certificate for your identity provider
    certificateSecret:
      name: "saml-secret"
      key: "saml.pem"
    # The SAML URL for your identity provider
    loginRequestUrl: "https://login.for.your.provider.com/"
    # The entity ID that was configured in your identity provider when setting up the SAML application
    entityId: "your-entity-id"
    # The name ID format to use. Valid values are email, persistent or unspecified.
    nameIdFormat: "persistent"
    # Access to the Cesium ion Administrator user interface is granted to any
    # identity that matches the below name and value criteria
    # The attribute name to look up. Examples: "roles", "Groups"
    adminAttributeName: "Groups"
    # The value that is expected to be found for adminAttributeName.
    # This value is treated as a semicolon delimited list.
    # Examples: "admin", "my-admin-group", "users; admin; members"
    adminAttributeValue: "ion-administrators"

  7. Save your changes to cesium-ion/values.yaml

  8. Upgrade the application by running:

    microk8s helm upgrade cesium-ion cesium-ion/ --wait --values cesium-ion/values.yaml --namespace cesium-ion

  9. When you navigate to the Cesium ion user interface, you will be redirected to your identity provider for authentication before being granted access. If you encounter an issue, review the above steps and try again.

Data uploaded while SSO is disabled is unavailable once SSO is enabled. Similarly, any data uploaded while using SSO becomes unavailable when SSO is disabled.

Default and shared assets

Cesium ion does not contain any data by default. For example, creating a new story with Cesium Stories will show the Earth as a bright blue WGS84 ellipsoid without terrain or imagery: