Skip to main content

Procedurally Spawning Foliage

In the following tutorial, you'll learn how to use the Niagara particle system and custom C++ code to analyze Cesium World Terrain and generate foliage procedurally.

This tutorial may be complex for first-time users of Unreal Engine or Cesium for Unreal. If you're new, check out the Cesium for Unreal Quickstart to learn the foundations of Cesium for Unreal.

Want to learn more about foliage, or just want to place it by hand? Visit Placing Foliage on Cesium Tilesets.

Information

This tutorial was created by Cesium community member Aiden Soedjarwo - SquarerFive. Aiden is a Geospatial Data Analyst at GeoSynergy as well as a hobbyist game and software developer.

Want to write a tutorial of your own, or have suggestions for future tutorials? Let us know on the community forum.

A brief overview of the system

Abbreviations:

  • RT = Render target
  • HISM = Hierarchical instanced static mesh
  • RTDI = Render target data interface

The system we’ll be creating makes use of 3 components:

  • Cesium World Terrain - An orthographic scene capture component will be placed above the player camera. The orthographic width defines the grid where foliage instances will be spawned. 
  • Niagara Grid Simulation (making use of simulation stages) - Each pixel from the captured imagery will be processed here and then output onto a classifications RT.
  • FoliageCaptureActor - Extracts the pixels of the render target and reprojects them back to world position. The scene depth RT is used to define the elevation of each foliage instance and the scene normals RT defines how the foliage instances should be rotated.

Prerequisites

  • An installation of Unreal Engine 4.26 or later
  • Visual Studio 2019 (with the “Desktop Development for C++” workload installed) or Rider for Unreal Engine
  • An understanding of C++ development fundamentals
  • A project set up with the Cesium for Unreal plugin
  • A connection to Cesium Ion to add in the terrain tileset

1C++: Create the Actors

Inside the Unreal Editor, open the file context menu. Click New C++ Class and then select Actor. In this tutorial we’ll be naming it "FoliageCaptureActor".

We will also be creating an HISM component, which inherits Hierarchical Instanced Static Mesh Component.

Open the project solution using your IDE of choice.

Build.cs

Add the following modules to your PublicDependencyModuleNames in the project’s build.cs file:

  • RenderCore
  • RHI
  • CesiumRuntime
  • Foliage

The HTTP module is added as a dependency so the project is able to compile for shipping builds.

The C++ standard version will also need to be set with CppStandard = CppStandardVersion.Cpp17;

The file should then look like this:

using UnrealBuildTool;

public class aiden_geo_tutorial : ModuleRules
{
  public aiden_geo_tutorial(ReadOnlyTargetRules Target) : base(Target)
  {
     PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
 
     PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "Foliage", "CesiumRuntime", "RHI", "RenderCore", "HTTP" });

     PrivateDependencyModuleNames.AddRange(new string[] {  });

     CppStandard = CppStandardVersion.Cpp17;
  }
}

FoliageCaptureActor.h

1Set up the includes to contain the following in the header file, as we’ll be using classes defined in them throughout the project.

#include "CoreMinimal.h"
#include "GameFramework/Actor.h"

#include "Engine/TextureRenderTarget2D.h"
#include "FoliageType_InstancedStaticMesh.h"
#include "CesiumGeoreference.h"
#include "FoliageHISM.h"

#include "FoliageCaptureActor.generated.h"

2Subsequently, define the following structures and delegates.
As documented in the code comments, these data structures are the core of the system that defines how foliage should be spawned for each classification. As this project makes use of async tasks, delegates will come in useful especially if we want to execute a function after a task has completed on another thread.

/**
* @brief Used to store the reprojected points gathered from the RT.
*/
USTRUCT(BlueprintType)
struct FFoliageTransforms
{
  GENERATED_BODY()
  TMap<UFoliageHISM*, TArray<FTransform>> HISMTransformMap;
};

/**
* @brief Array of foliage type transforms gathered from the RT extraction task.
*/
USTRUCT()
struct FFoliageTransformsTypeMap
{
  GENERATED_BODY()
  TArray<FFoliageTransforms> FoliageTypes;
};

/**
* @brief Foliage geometry container
*/
USTRUCT(BlueprintType)
struct FFoliageGeometryType
{
  GENERATED_BODY()
 
  /* Placement */
  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Placement")
  float Density = 0.5f;
 
  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Placement")
  bool bRandomYaw = false;
 
  UPROPERTY(EditAnywhere, Category = "Placement")
  FFloatInterval ZOffset = FFloatInterval(0.0, 0.0);
 
  UPROPERTY(EditAnywhere, Category = "Placement")
  FFloatInterval Scale = FFloatInterval(1.0, 1.0);

  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Placement")
  bool bAlignToNormal = false;
 
  /* Mesh Settings */

  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Mesh")
  UStaticMesh* Mesh;
 
  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Mesh")
  bool bCollidesWithWorld = true;

  UPROPERTY(EditAnywhere, Category = "Placement")
  FFloatInterval CullingDistances = FFloatInterval(4096, 32768);

  /* Expensive */
  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Mesh")
  bool bAffectsDistanceFieldLighting = false;

  friend uint32 GetTypeHash(const FFoliageGeometryType& A)
  {
     return GetTypeHash(A.Density) + GetTypeHash(A.bRandomYaw) + GetTypeHash(A.ZOffset) + GetTypeHash(A.Scale) +
        GetTypeHash(A.Mesh) + GetTypeHash(A.bCollidesWithWorld) +
        GetTypeHash(A.bAffectsDistanceFieldLighting);
  }

  friend bool operator==(const FFoliageGeometryType& A, const FFoliageGeometryType& B)
  {
     return A.Density == B.Density && A.Mesh == B.Mesh && A.bCollidesWithWorld == B.bCollidesWithWorld && A.
        bAffectsDistanceFieldLighting == B.bAffectsDistanceFieldLighting
        && A.Scale.Max == B.Scale.Max && A.Scale.Min == B.Scale.Min && A.bRandomYaw == B.bRandomYaw &&
           A.ZOffset.Min == B.ZOffset.Min && A.ZOffset.Max == B.ZOffset.Max;
 
  }
};

/**
* @brief Container for a foliage type
*/
USTRUCT(BlueprintType)
struct FFoliageClassificationType
{
  GENERATED_BODY()

  UPROPERTY(BlueprintReadWrite, EditAnywhere)
  FString Type;
  UPROPERTY(BlueprintReadWrite, EditAnywhere)
  FLinearColor ColourClassification;
  UPROPERTY(BlueprintReadWrite, EditAnywhere)
  TArray<FFoliageGeometryType> FoliageTypes;

  /**
   * @brief If enabled, a line trace will be casted downwards from each point to determine surface normals.
   */
  UPROPERTY(BlueprintReadWrite, EditAnywhere)
  bool bAlignToSurfaceWithRaycast = false;

  UPROPERTY(BlueprintReadWrite, EditAnywhere)
  int32 PooledHISMsToCreatePerFoliageType = 4;
};

// Called after points have been gathered and reprojected from the classification RT.
DECLARE_DELEGATE_OneParam(FOnFoliageTransformsGenerated, FFoliageTransformsTypeMap);

// This is called after pixels have been extracted from input RTs
DECLARE_DELEGATE_OneParam(FOnRenderTargetRead, bool);

3 Inside the AFoliageCaptureActor class, declare the following methods and variables contained in the code snippet:

public:
  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Foliage Spawner")
  TArray<FFoliageClassificationType> FoliageTypes;

  UPROPERTY()
  ACesiumGeoreference* Georeference;

  /**
   * @brief Elevation (in meters) of the scene capture component that's placed above the player.
   */
  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Foliage Spawner")
  float CaptureElevation = 1024.f;

  /**
   * @brief Orthographic width of our scene capture components
   */
  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Foliage Spawner")
  float CaptureWidth = 131072.f;

  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Foliage Spawner")
  int32 UpdateFoliageAfterNumFrames = 2;

  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Foliage Spawner")
  int32 MaxComponentsToUpdatePerFrame = 1;

  /**
   * @brief Coverage grid.
   */
  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Foliage Spawner")
  FIntVector GridSize = FIntVector(0, 0, 0);

  /**
  * @brief Average geographic width in degrees.
  */
  double CaptureWidthInDegrees = 0.01;

public:
  /**
   * @brief Build foliage transforms according to classification types.
   * @param FoliageDistributionMap Render Target containing classifications.
   * @param NormalAndDepthMap Render Target with normals in the RGB channel and *normalized* depth in Alpha.
   * @param RTWorldBounds UE world extents of the render targets.
   */
  UFUNCTION(BlueprintCallable, Category = "Foliage Spawner")
  void BuildFoliageTransforms(UTextureRenderTarget2D* FoliageDistributionMap,
                              UTextureRenderTarget2D* NormalAndDepthMap, FBox RTWorldBounds);

  /**
   * @brief Create required HISM components, removing if outdated
   */
  void ResetAndCreateHISMComponents();

  /**
   * @brief Implementable BP event, called when the player moves outside of the capture boundaries.
   */
  UFUNCTION(BlueprintCallable, BlueprintNativeEvent, Category = "Foliage Spawner")
  void OnUpdate(const FVector& NewLocation);

  /**
   * @brief Is the foliage currently building?
   */
  UFUNCTION(BlueprintCallable, BlueprintPure, Category = "Foliage Spawner")
  bool IsBuilding() const;

FoliageTypes is an array containing the classification color and collection of geometries we want to spawn.

UpdateFoliageAfterNumFrames determines how often foliage should be added or removed from the scene.

MaxComponentsToUpdatePerFrame determines how many components should be updated per frame.

protected:
  /**
  * @brief Attempt to correct normals and elevation by raycasting
  */
  void CorrectFoliageTransform(const FVector& InEngineCoordinates, const FMatrix& InEastNorthUp,
                              FVector& OutCorrectedPosition, FVector& OutSurfaceNormals, bool& bSuccess) const;

  /**
   * @brief For each static mesh, we also want to have multiple HISM components to reduce
   * hitches when updating instances.
   */
  TMap<FFoliageGeometryType, TArray<UFoliageHISM*>> HISMFoliageMap;

  /**
   * @brief The scene depth value is multiplied by a small value so it remains within the range of 0.0 to 1.0.
   * In this function it is projected back to its (approximated) original value and then inverted.
   * @param Value Depth value
   * @return Height in meters.
   */
  double GetHeightFromDepth(const double& Value) const;


  /**
   * @brief Converts pixel coordinates back to geographic coordinates.
   */
  glm::dvec3 PixelToGeographicLocation(const double& X, const double& Y, const double& Altitude,
                                       UTextureRenderTarget2D* RT, const glm::dvec4& GeographicExtents) const;
  /**
   * @brief Converts geographic coordinates to pixel coordinates.
   */
  FIntPoint GeographicToPixelLocation(const double& Longitude, const double& Latitude, UTextureRenderTarget2D* RT,
                                      const glm::dvec4& GeographicExtents) const;

  /**
   * @brief Modified version of ReadRenderColorPixels
   */
  void ReadLinearColorPixelsAsync(
     FOnRenderTargetRead OnRenderTargetRead,
     TArray<FTextureRenderTargetResource*> RTs,
     TArray<TArray<FLinearColor>*> OutImageData,
     FReadSurfaceDataFlags InFlags = FReadSurfaceDataFlags(
        RCM_MinMax,
        CubeFace_MAX),
     FIntRect InRect = FIntRect(0, 0, 0, 0),
     ENamedThreads::Type ExitThread = ENamedThreads::AnyBackgroundThreadNormalTask);

  /**
   * @brief Don't run tick update if true.
   */
  bool bIsBuilding = false;
 
  /**
   * @brief Number of frames that have passed after updating foliage.
   */
  int32 Ticks = 0;

  static glm::dvec3 VectorToDVector(const FVector& InVector);

ReadLinearColorPixelsAsync is a modified version of the function FTextureRenderTargetResource::ReadLinearColorPixels that allows us to extract pixels from multiple render targets without blocking the game thread.

FoliageCaptureActor.cpp

Here is where we’ll implement the functions as declared in the header file.

Includes

Make sure to include KismetMathLibrary, as we’ll be using several functions within this library.

#include "Kismet/KismetMathLibrary.h"

Tick

This controls how often we want to update each of the foliage instances.

if (Ticks > UpdateFoliageAfterNumFrames && !bIsBuilding)
{
  Ticks = 0;
  int32 ComponentsUpdated = 0;

  for (TPair<FFoliageGeometryType, TArray<UFoliageHISM*>>& FoliageHISMPair : HISMFoliageMap)
  {
     for (UFoliageHISM* FoliageHISM : FoliageHISMPair.Value)
     {
        // if (!IsValid(FoliageHISM)) { continue; }
        if (FoliageHISM->bMarkedForClear)
        {
           FoliageHISM->ClearInstances();
           FoliageHISM->bMarkedForClear = false;
           ComponentsUpdated++;
        }
        else if (FoliageHISM->bMarkedForAdd)
        {
           FoliageHISM->PreAllocateInstancesMemory(FoliageHISM->Transforms.Num());
           FoliageHISM->AddInstances(FoliageHISM->Transforms, false);
           FoliageHISM->bMarkedForAdd = false;
           FoliageHISM->Transforms.Empty();
           ComponentsUpdated++;
        }
        if (ComponentsUpdated > MaxComponentsToUpdatePerFrame)
        {
           break;
        }
     }
  }
}

Ticks++;

BuildFoliageTransforms

void AFoliageCaptureActor::BuildFoliageTransforms(UTextureRenderTarget2D* FoliageDistributionMap,
                                                 UTextureRenderTarget2D* NormalAndDepthMap, FBox RTWorldBounds)
{
  // Need to check whether the CesiumGeoreference actor and input RTs are valid.
  if (!IsValid(Georeference))
  {
     UE_LOG(LogTemp, Warning, TEXT("Georeference is invalid! Not spawning in foliage"));
     return;
  }
  if (!IsValid(NormalAndDepthMap) || !IsValid(FoliageDistributionMap))
  {
     UE_LOG(LogTemp, Warning, TEXT("Invaid inputs for FoliageCaptureActor!"));
     return;
  }
  if (FoliageTypes.Num() == 0)
  {
     UE_LOG(LogTemp, Warning, TEXT("No foliage types added!"));
     return;
  }
 
  bIsBuilding = true;
  // Ensure the transforms array on the HISMs are cleared before building.
  for (TPair<FFoliageGeometryType, TArray<UFoliageHISM*>>& FoliageHISMPair : HISMFoliageMap)
  {
     for (UFoliageHISM* FoliageHISM : FoliageHISMPair.Value)
     {
        if (IsValid(FoliageHISM))
        {
           FoliageHISM->Transforms.Empty();
           FoliageHISM->bMarkedForClear = true;
        }
     }
  }

  // Find the geographic bounds of the RT
  const glm::dvec3 MinGeographic = Georeference->TransformUeToLongitudeLatitudeHeight(
     glm::dvec3(
        RTWorldBounds.Min.X,
        RTWorldBounds.Min.Y,
        RTWorldBounds.Min.Z
     ));

  const glm::dvec3 MaxGeographic = Georeference->TransformUeToLongitudeLatitudeHeight(
     glm::dvec3(
        RTWorldBounds.Max.X,
        RTWorldBounds.Max.Y,
        RTWorldBounds.Max.Z
     )
  );
  const glm::dvec4 GeographicExtents2D = glm::dvec4(
     MinGeographic.x, MinGeographic.y,
     MaxGeographic.x, MaxGeographic.y
  );

  // Setup pixel extraction
  const int32 TotalPixels = FoliageDistributionMap->SizeX * FoliageDistributionMap->SizeY;

  TArray<FLinearColor>* ClassificationPixels = new TArray<FLinearColor>();
  TArray<FLinearColor>* NormalPixels = new TArray<FLinearColor>();

  FOnRenderTargetRead OnRenderTargetRead;
  OnRenderTargetRead.BindLambda(
     [this, FoliageDistributionMap, ClassificationPixels, NormalPixels, GeographicExtents2D, TotalPixels](
     bool bSuccess) mutable
     {
        if (!bSuccess)
        {
           bIsBuilding = false;
           return;
        }

        const int32 Width = FoliageDistributionMap->SizeX;

        TMap<UFoliageHISM*, int32> NewInstanceCountMap;
        FFoliageTransforms FoliageTransforms;
       
        for (int Index = 0; Index < TotalPixels; ++Index)
        {
           // Get the 2D pixel coordinates.
           const double X = Index % Width;
           const double Y = Index / Width;

           // Extract classification, normals and depth from the pixel arrays.
           const FLinearColor Classification = (*ClassificationPixels)[Index];
           const FLinearColor NormalDepth = (*NormalPixels)[Index];
           // Convert the RGB channel in the NormalDepth array to a FVector
           FVector Normal = FVector(NormalDepth.R, NormalDepth.G, NormalDepth.B);
           // Project the Alpha channel in NormalDepth to elevation (in meters)
           const double Elevation = GetHeightFromDepth(NormalDepth.A);

           // Project pixel coords to geographic.
           const glm::dvec3 GeographicCoords = PixelToGeographicLocation(X, Y, Elevation, FoliageDistributionMap,
                                                                         GeographicExtents2D);
           // Then project to UE world coordinates
           const glm::dvec3 EngineCoords = Georeference->TransformLongitudeLatitudeHeightToUe(GeographicCoords);

           // Compute east north up
           const FMatrix EastNorthUpEngine = Georeference->InaccurateComputeEastNorthUpToUnreal(
              FVector(EngineCoords.x, EngineCoords.y, EngineCoords.z));

           for (FFoliageClassificationType& FoliageType : FoliageTypes)
           {
              // If classification pixel color matches the classification of FoliageType
              if (Classification == FoliageType.ColourClassification)
              {
                 bool bHasDoneRaycast = false;
                 FVector Location = FVector(EngineCoords.x, EngineCoords.y, EngineCoords.z);

                 if (FoliageType.bAlignToSurfaceWithRaycast)
                 {
                    CorrectFoliageTransform(Location, EastNorthUpEngine, Location, Normal, bHasDoneRaycast);
                 }
                 // Iterate through the mesh types inside FoliageType
                 for (FFoliageGeometryType& FoliageGeometryType : FoliageType.FoliageTypes)
                 {
                    if (FMath::FRand() >= FoliageGeometryType.Density)
                    {
                       continue;
                    }

                    // Find rotation and scale
                    const float Scale = FoliageGeometryType.Scale.Interpolate(FMath::FRand());
                    FRotator Rotation;
                   
                    if (FoliageGeometryType.bAlignToNormal)
                    {
                       Rotation = UKismetMathLibrary::MakeRotFromZ(Normal);
                    } else
                    {
                       Rotation = EastNorthUpEngine.Rotator();
                    }

                    // Apply a random angle to the rotation yaw if RandomYaw is true.
                    if (FoliageGeometryType.bRandomYaw)
                    {
                       Rotation = UKismetMathLibrary::RotatorFromAxisAndAngle(
                          Rotation.Quaternion().GetUpVector(), FMath::FRandRange(
                             0.0, 360.0
                          ));
                    }

                    // Find HISM with minimum amount of transforms.
                   
                    UFoliageHISM* MinimumHISM = HISMFoliageMap[FoliageGeometryType][0];
                    for (UFoliageHISM* HISM : HISMFoliageMap[FoliageGeometryType])
                    {
                       if (FoliageTransforms.HISMTransformMap.Contains(HISM) && FoliageTransforms.HISMTransformMap.Contains(MinimumHISM))
                       {
                          if (FoliageTransforms.HISMTransformMap[HISM].Num() < FoliageTransforms.HISMTransformMap[HISM].Num())
                          {
                             MinimumHISM = HISM;
                          }
                       }
                    }
                    if (!IsValid(MinimumHISM))
                    {
                       UE_LOG(LogTemp, Error, TEXT("MinimumHISM is invalid!"));
                    }

                    // Add our transform, and make it relative to the actor.
                    FTransform NewTransform = FTransform(
                       Rotation,
                       Location + (Rotation.Quaternion().
                          GetUpVector() * FoliageGeometryType.ZOffset.
                                                              Interpolate(FMath::FRand())), FVector(Scale)
                    ).GetRelativeTransform(GetTransform());
                    if (NewTransform.IsRotationNormalized())
                    {
                       if (!FoliageTransforms.HISMTransformMap.Contains(MinimumHISM))
                       {
                          FoliageTransforms.HISMTransformMap.Add(MinimumHISM, TArray{NewTransform});
                       } else
                       {
                          FoliageTransforms.HISMTransformMap[MinimumHISM].Add(NewTransform);
                       }
                    }
                 }
              }
           }
        }
        delete ClassificationPixels;
        delete NormalPixels;
        ClassificationPixels = nullptr;
        NormalPixels = nullptr;

        AsyncTask(ENamedThreads::GameThread, [FoliageTransforms, this]()
        {
           // Marked for add
           for (const TPair<UFoliageHISM*, TArray<FTransform>>& Pair: FoliageTransforms.HISMTransformMap)
           {
              Pair.Key->Transforms.Append(Pair.Value);
              Pair.Key->bMarkedForAdd = true;
           }
           bIsBuilding = false;
        });
     });
  // Extract the pixels from the render targets, calling OnRenderTargetRead when complete.
  ReadLinearColorPixelsAsync(OnRenderTargetRead, TArray<FTextureRenderTargetResource*>{
                                FoliageDistributionMap->GameThread_GetRenderTargetResource(),
                                NormalAndDepthMap->GameThread_GetRenderTargetResource()
                             }, TArray<TArray<FLinearColor>*>{
                                ClassificationPixels, NormalPixels
                             });
}

ResetAndCreateHISMComponents

This is called on BeginPlay and PostEditChangeProperty (when any properties in the details panel are modified).

void AFoliageCaptureActor::ResetAndCreateHISMComponents()
{
  for (FFoliageClassificationType& FoliageType : FoliageTypes)
  {
     for (FFoliageGeometryType& FoliageGeometryType : FoliageType.FoliageTypes)
     {
        if (FoliageGeometryType.Mesh == nullptr) { continue; }
        if (HISMFoliageMap.Contains(FoliageGeometryType))
        {
           for (UFoliageHISM* HISM : HISMFoliageMap[FoliageGeometryType])
           {
              if (IsValid(HISM))
              {
                 HISM->DestroyComponent();
              }
           }
           HISMFoliageMap[FoliageGeometryType].Empty();
        }
        HISMFoliageMap.Remove(FoliageGeometryType);

        for (int32 i = 0; i < FoliageType.PooledHISMsToCreatePerFoliageType; ++i)
        {
           UFoliageHISM* HISM = NewObject<UFoliageHISM>(this);
           HISM->SetupAttachment(GetRootComponent());
           HISM->RegisterComponent();

           HISM->SetStaticMesh(FoliageGeometryType.Mesh);
           HISM->SetCollisionEnabled(FoliageGeometryType.bCollidesWithWorld
                                        ? ECollisionEnabled::QueryAndPhysics
                                        : ECollisionEnabled::NoCollision);
           HISM->SetCullDistances(FoliageGeometryType.CullingDistances.Min, FoliageGeometryType.CullingDistances.Max);

           // This may cause a slight hitch when enabled.
           HISM->bAffectDistanceFieldLighting = FoliageGeometryType.bAffectsDistanceFieldLighting;
           if (!HISMFoliageMap.Contains(FoliageGeometryType))
           {
              HISMFoliageMap.Add(FoliageGeometryType, TArray<UFoliageHISM*>{HISM});
           }
           else
           {
              HISMFoliageMap[FoliageGeometryType].Add(HISM);
           }
        }
     }
  }
}

GetHeightFromDepth

double AFoliageCaptureActor::GetHeightFromDepth(const double& Value) const
{
  return CaptureElevation - (1 - Value) / 0.00001 / 100;
}

PixelToGeographicCoordinates and GeogrxaphicToPixelCoordinates

glm::dvec3 AFoliageCaptureActor::PixelToGeographicLocation(const double& X, const double& Y, const double& Altitude,
                                                          UTextureRenderTarget2D* RT,
                                                          const glm::dvec4& GeographicExtents) const
{
  // Normalize the ranges of the coords
  const double AX = X / static_cast<double>(RT->SizeX);
  const double AY = Y / static_cast<double>(RT->SizeY);

  const double Long = FMath::Lerp<double>(
     GeographicExtents.x,
     GeographicExtents.z,
     1 - AY);
  const double Lat = FMath::Lerp<double>(
     GeographicExtents.y,
     GeographicExtents.w,
     AX);

  return glm::dvec3(Long, Lat, Altitude);
}

FIntPoint AFoliageCaptureActor::GeographicToPixelLocation(const double& Longitude, const double& Latitude,
                                                         UTextureRenderTarget2D* RT,
                                                         const glm::dvec4& GeographicExtents) const
{
  // Normalize long and lat
  const double LongitudeRange = GeographicExtents.z - GeographicExtents.x;
  const double LatitudeRange = GeographicExtents.w - GeographicExtents.y;
  const double ALongitude = (Longitude - GeographicExtents.x) / LongitudeRange;
  const double ALatitude = (Latitude - GeographicExtents.y) / LatitudeRange;

  const double X = FMath::Lerp<double>(0, RT->SizeX, ALatitude);
  const double Y = FMath::Lerp<double>(RT->SizeY, 0, ALongitude);

  return FIntPoint(X, Y);
}

ReadLinearColorPixelsAsync

void AFoliageCaptureActor::ReadLinearColorPixelsAsync(
  FOnRenderTargetRead OnRenderTargetRead,
  TArray<FTextureRenderTargetResource*> RTs,
  TArray<TArray<FLinearColor>*> OutImageData,
  FReadSurfaceDataFlags InFlags,
  FIntRect InRect,
  ENamedThreads::Type ExitThread)
{
  if (InRect == FIntRect(0, 0, 0, 0))
  {
     InRect = FIntRect(0, 0, RTs[0]->GetSizeXY().X, RTs[0]->GetSizeXY().Y);
  }
 
 
  struct FReadSurfaceContext
  {
     TArray<FTextureRenderTargetResource*> SrcRenderTargets;
     TArray<TArray<FLinearColor>*> OutData;
     FIntRect Rect;
     FReadSurfaceDataFlags Flags;
  };

  for (auto DT : OutImageData) { DT->Reset(); }
  FReadSurfaceContext Context =
  {
     RTs,
     OutImageData,
     InRect,
     InFlags
  };

  if (!Context.OutData[0])
  {
     UE_LOG(LogTemp, Error, TEXT("Buffer invalid!"));
     return;
  }

  ENQUEUE_RENDER_COMMAND(ReadSurfaceCommand)(
     [Context, OnRenderTargetRead, ExitThread](FRHICommandListImmediate& RHICmdList)
     {
        const FIntRect Rect = Context.Rect;
        const FReadSurfaceDataFlags Flags = Context.Flags;
        int i = 0;
        for (FRenderTarget* RT : Context.SrcRenderTargets)
        {
           const FTexture2DRHIRef& RefRenderTarget = RT->
              GetRenderTargetTexture();
           TArray<FLinearColor>* Buffer = Context.OutData[i];

           RHICmdList.ReadSurfaceData(
              RefRenderTarget,
              Rect,
              *Buffer,
              Flags
           );
           i++;
        }
        // instead of blocking the game thread, execute the delegate when finished.
        AsyncTask(
           ExitThread,
           [OnRenderTargetRead, Context]()
           {
              OnRenderTargetRead.Execute((*Context.OutData[0]).Num() > 0);
           });
     });
}

VectorToDVector

glm::dvec3 AFoliageCaptureActor::VectorToDVector(const FVector& InVector)
{
  return glm::dvec3(InVector.X, InVector.Y, InVector.Z);
}

OnUpdate

void AFoliageCaptureActor::OnUpdate_Implementation(const FVector& NewLocation)
{
  // Align the actor to face the planet surface.
  SetActorLocation(NewLocation);

  const FRotator PlanetAlignedRotation = Georeference->InaccurateComputeEastNorthUpToUnreal(NewLocation).Rotator();

  SetActorRotation(
     PlanetAlignedRotation
  );

  // Get grid min and max coords.
  const int32 Size = GridSize.X > 0 ? GridSize.X : 1;
  const FVector Start = GetActorTransform().TransformPosition(FVector(-(CaptureWidth * Size) / 2, 0, 0));
  const FVector End = GetActorTransform().TransformPosition(FVector((CaptureWidth * Size) / 2, 0, 0));

  // Find the distance (in degrees) between the grid min and max.
  glm::dvec3 GeoStart = Georeference->TransformUeToLongitudeLatitudeHeight(VectorToDVector(Start));
  glm::dvec3 GeoEnd = Georeference->TransformUeToLongitudeLatitudeHeight(VectorToDVector(End));
 
  GeoStart.z = CaptureElevation;
  GeoEnd.z = CaptureElevation;
 
  CaptureWidthInDegrees = glm::distance(GeoStart, GeoEnd) / 2;
}

IsBuilding

bool AFoliageCaptureActor::IsBuilding() const
{
  return bIsBuilding;
}

CorrectFoliageTransform

InEngineCoordinates - the original location of the instance that’s projected from the RT.

InEastNorthUp - rotation matrix used to orientate the instance onto the planet surface.

OutCorrectedPosition - This is the hit position of the ray, which is tested against the collision mesh belonging to a tile.

OutSurfaceNormals - The surface normal of the ray, replaces the normal from the RT if raycasting is enabled.

bSuccess - Is set to true if there was a blocking hit, otherwise false. In BuildFoliageTransforms, we will fallback to the approximated normal and position if this is false.

void AFoliageCaptureActor::CorrectFoliageTransform(const FVector& InEngineCoordinates, const FMatrix& InEastNorthUp,
  FVector& OutCorrectedPosition, FVector& OutSurfaceNormals, bool& bSuccess) const
{
  UWorld* World = GetWorld();

  if (IsValid(World))
  {
     const FVector Up = InEastNorthUp.ToQuat().GetUpVector();
     FHitResult HitResult;
    
     World->LineTraceSingleByChannel(HitResult, InEngineCoordinates + (Up * 6000),
        InEngineCoordinates - (Up * 6000), ECollisionChannel::ECC_Visibility);
    
     if (HitResult.bBlockingHit)
     {
        bSuccess = true;
        OutCorrectedPosition = HitResult.ImpactPoint;
        OutSurfaceNormals = HitResult.ImpactNormal;
     }
  }
}

FoliageHISM.h

Transforms is an array of instance transforms to be added on the next frame.

bMarkedForAdd is a flag determining whether this component has instances to add.

bMarkedForClear is a flag determining whether instances should be cleared on the next frame.

public:
  UPROPERTY()
  TArray<FTransform> Transforms;

  UPROPERTY()
  bool bMarkedForAdd = false;

  UPROPERTY()
  bool bMarkedForClear = false;

2Niagara: Create a Niagara Function Script

The classifier will be built in Niagara, which provides write access to Render Targets and greater flexibility over regular materials.

1Within the Content Browser in the Unreal Editor, right click to open the context menu, hover over FX and select Niagara Function Script.  For this tutorial it’ll be named ‘NF_DetermineFoliageType’.

2Open the asset and set Library Visibility to Exposed. This makes the function accessible within the Niagara Editor context menu.

3Create inputs:

  • Input Data - Post-processed input imagery. - Vector4
  • If False - Pixel color if mask condition returns false. - Vector4
  • If True - Pixel color if mask condition returns true. - Vector4
  • Power - Contrast strength - float
  • Bias - Mask value, if Input Data.Y^Power > Bias then we return If True - float

4Create Outputs:

  • Output Color - Classification Color. - Vector4

Note: The names of the input nodes can also be set in the righthand side details panel:

5In the node graph, set up the function like the screenshot below.

6Click Compile, Apply, and Save so we can use the function in the Niagara System that will be created.

3Niagara: Create a Niagara System

1Create an empty Niagara System and name it “NS_FoliageAnalysis”.

2Inside the Niagara system, add the following user exposed parameters:

  • RT_Normals - Texture Render Target
  • RT_Output - Texture Render Target
  • TS_Depth - Texture Sample
  • TS_Imagery - Texture Sample
  • TS_Normals - Texture Sample

3Next, right click and select Add Emitter. We will be using the CompletelyEmpty template.

4With the emitter selected, set Sim Target to GPUCompute sim, enable Local Space, and set the Fixed Bounds to:

  • Min: "0, 0, 0"
  • Max: "1024, 1024, 1"

5We’ll also make use of Simulation Stages, which can be enabled by expanding the Simulation Stages menu and checking Enable Simulation Stages (Experimental).

6Set up the emitter attributes as following:

  • Data - Grid2DCollection 
  • RTDI_Normals - Render Target 2D
  • RTDI_Output - Render Target 2D

7Drag and drop the RTDI parameters into Emitter Spawn. As we’re not creating render targets on the fly, set Render Target User Parameter to their corresponding RT_user parameters.

8Drag and drop the Data parameter into Emitter Spawn. Num Cells X and Num Cells Y will both be set to 512. They will need to be the same as the width and height of the input render targets. Currently the grid collection will only store the classification of a pixel, so Num Attributes can be set to "1".

9Create a simulation stage and name it “Build Foliage Classifications”. Set the Iteration Source to Data Interface. Set the Data Interface to the Data grid emitter attribute.

10Create a Scratchpad by clicking the green add button on the Build Foliage Classifications module.

The Scratch Pad Module will contain the code where each pixel would be classified and assigned to the data grid.

In the module's inputs, add three texture samples parameters:

  • "Imagery Sample"
  • "Normals Sample"
  • "Depth Sample"

The module will also have the following emitter attributes:

  • Data
  • RTDI_Output

Click Apply to save and compile the above changes to the scratchpad.

The module attributes and inputs panel should then look like this:

Next, set up the scratchpad to have the same graph setup as below.

The NF_DetermineFoliageType function can be re-used to classify multiple types of vegetation by slightly adjusting the parameters.

In this graph, the first instance of NF_DetermineFoliageType is used to approximate where trees would be placed. It is less tolerant of the green channel, hence the lower Bias setting. 

The Power setting controls the contrast strength of the input data, where greener values would stand out more, and thus be masked. 

The second instance is a more tolerant version of the first instance, which results in larger clumps of yellow pixels.

The mask below is an example classification where:

  • GREEN (0, 255, 0) are tree canopies
  • YELLOW (255, 255, 0) are smaller bodies of vegetation (bush, grass).

11The Niagara emitter will need an emitter state module and particle spawn node. For now, use the Spawn Particles in Grid module with the following settings:

  • X Count: 512
  • Y Count: 512
  • Z Count: 1

Under Particle Spawn, add a Grid Location module with the default settings.

Elevation and Normals

A cheaper way to approximate elevation is to use the scene depth buffer. Values in the depth buffer are absolute, so we will need to multiply the values by a small number (0.00001) to keep them within the range of 0.0 and 1.0. 

It will then be inverted so pixels closest to the camera will have higher values and farther pixels have lower values; this gives us the alpha value, which can be used to interpolate between an elevation of 0 meters and the capture elevation.

The normalized elevation value will be stored in the alpha channel of RT_Normals.

12Create a second simulation stage called “Setup Elevation”. It will have the same settings as the first stage.

13Create a Scratch Pad Module called “SetElevation” in the simulation stage, with the following inputs:

  • RTDI_Normals - From Emitter Attributes
  • Data - From Emitter Attributes
  • TS_Depth - Texture Sample
  • TS_Normals - Texture Sample

14Create two Sample Texture 2D nodes and connect them to TS_Normals and TS_Depth. Create an Execution Index to Unit node and connect it to the UVs.

15We will only need the XYZ components of the normals and the X component of the scene depth. Create a new Make Linear Color node with corresponding XYZ components connecting. The alpha component will be "1 - Depth X * 0.00001".

16The output of Normals + Elevation will be connected to a Set Render Target Value node. Create a Linear to Index node with the grid input being the Data emitter attribute and linear input being the Execution Index.

The emitter should now look like this:

4Set up Render Targets

Render Targets (RTs) provide an intuitive way to transfer data to and from the GPU. Unlike regular textures, RTs can be modified at runtime, which makes them great for use cases such as erosion simulation and heightmap generation.

In the content browser, right click and go to Materials & Textures and then select Render Target.

We’ll be creating four render targets with the following settings:

RT_Classifications

  • Size: 512x512
  • Render Target Format: RTF_RGBA16f

RT_Depth

  • Size: 512x512
  • Render Target Format: RTF_R32f

RT_Imagery

  • Size: 512x512
  • Render Target Format: RTF_RGBA16f

RT_Normals

  • Size: 512x512
  • Render Target Format: RTF_RGBA16f

After creating the render targets, set the default values for the texture inputs in NS_FoliageAnalysis.

Select the scratch modules, and set sample input to linked inputs.

Test the Render Targets

To check that the Niagara Particle System and render targets have been set up properly, drag and drop a Scene Capture 2D actor into the scene.

Set the TextureTarget to RT_Imagery, Ortho Width to 65536 and Capture Source to Final Color (HDR).

Next, add a Niagara component to the actor, with the NS_FoliageAnalysis set as the Niagara system asset.

The output RT (RT_Classifications) should then update on the fly with a result looking like below:

If successful, the newly placed actor can then be deleted so the final results don’t get overwritten by the actor.

5ProceduralFoliageEllipsoid

1Lastly, create a C++ subclass of Cesium3DTileset called “ProceduralFoliageEllipsoid”. This actor will check whether the player has moved outside of the foliage capture radius, and if so will update the FoliageCaptureActor.

2Once created, open the header file and include FoliageCaptureActor.h with #include "FoliageCaptureActor.h"

3Inside the AProceduralFoliageEllipsoid class body, declare the following variables and function:

public:
  UPROPERTY(BlueprintReadWrite, EditAnywhere, Category = "Foliage")
     AFoliageCaptureActor* FoliageCaptureActor;

  virtual void Tick(float DeltaSeconds) override;

protected:
  // Initial spawn
  bool bHasFoliageSpawned = false;
};

4In the .cpp file, define the tick function with a call to the parent function as well

#include "Kismet/GameplayStatics.h"


void AProceduralFoliageEllipsoid::Tick(float DeltaSeconds)
{
  Super::Tick(DeltaSeconds);
}

5Inside the tick function, implement the code snippet below:

if (IsValid(Georeference) && IsValid(FoliageCaptureActor))
{
  // Don't start another build while the previous one is still running
  if (!FoliageCaptureActor->IsBuilding())
  {
     APlayerCameraManager* CameraManager = UGameplayStatics::GetPlayerCameraManager(this, 0);
     if (IsValid(CameraManager))
     {
        // Project the camera coordinates to geographic coordinates.
        const FVector CameraLocation = CameraManager->GetCameraLocation();
        glm::dvec3 GeographicCameraLocation = Georeference->TransformUeToLongitudeLatitudeHeight(
           glm::dvec3(CameraLocation.X, CameraLocation.Y, CameraLocation.Z));

        // Keep original camera elevation as a variable, and swap z with the capture elevation (this will be the new capture location).
        const double CurrentCameraElevation = GeographicCameraLocation.z;
        GeographicCameraLocation.z = FoliageCaptureActor->CaptureElevation;

        // Ensure CesiumGeoreference is valid
        if (!IsValid(FoliageCaptureActor->Georeference))
        {
           FoliageCaptureActor->Georeference = Georeference;
        }
    
        // Current geographic location of the foliage capture actor (used to measure the distance).
        const glm::dvec3 CurrentFoliageCaptureGeographicLocation = Georeference->TransformUeToLongitudeLatitudeHeight(glm::dvec3(
           FoliageCaptureActor->GetActorLocation().X, FoliageCaptureActor->GetActorLocation().Y, FoliageCaptureActor->GetActorLocation().Z
        ));

        const double Distance = glm::distance(GeographicCameraLocation, CurrentFoliageCaptureGeographicLocation);
        const double Speed = CameraManager->GetVelocity().Size();

        // New capture position
        const glm::dvec3 NewFoliageCaptureUELocation = Georeference->TransformLongitudeLatitudeHeightToUe(GeographicCameraLocation);

        // Only update the foliage capture actor if the player is outside of the capture grid, within elevation, and a speed less than 5000.
        if ((Distance > FoliageCaptureActor->CaptureWidthInDegrees && CurrentCameraElevation <= FoliageCaptureActor->CaptureElevation && Speed < 5000) || !bHasFoliageSpawned)
        {
           FoliageCaptureActor->OnUpdate(FVector(NewFoliageCaptureUELocation.x, NewFoliageCaptureUELocation.y, NewFoliageCaptureUELocation.z));
           bHasFoliageSpawned = true;
        }
     }
  }
}

The code snippet above runs every frame, which checks where the player is and whether it has moved outside of the capture radius. To improve performance, foliage will not update if the player is moving at a velocity of more than 5000 units per second or is above the capture altitude.

6Create Actor Blueprints

Create a Blueprint actor with ProceduralFoliageEllipsoid as the parent actor:

Drag the new BP subclass of ProceduralFoliageEllipsoid into the scene and untick Frustrum culling under the culling setting. This ensures that the tiles surrounding the player are visible to the capture component.

BP_FoliageCaptureActor

Open up BP_FoliageCaptureActor and add the following components with settings to the actor:

Scene (Scene Component)

Imagery (Scene Capture Component 2D)
  • Projection Type: Orthographic
  • Texture Target: RT_Imagery
  • Capture Source: BaseColor in RGB
  • Capture Every Frame: false
  • Capture On Movement: false
Normals (Scene Capture Component 2D)
  • Projection Type: Orthographic
  • Texture Target: RT_Normals
  • Capture Source: Normal in RGB
  • Capture Every Frame: false
  • Capture On Movement: false
Depth (Scene Capture Component 2D)
  • Projection Type: Orthographic
  • Texture Target: RT_Depth
  • Capture Source: Scene Depth in R
  • Capture Every Frame: false
  • Capture On Movement: false

Niagara

  • Niagara System Asset: NS_FoliageAnalysis

Ensure the components are parented as in the image below.

After adding the three Scene Capture Component 2Ds to the Scene Component, set the rotation of the scene component to:

X=540.0

Y=-90.0

Z=180.0

Create a macro called Wait5Frames (as render targets don’t always update immediately after a capture, but after a few frames).

In the Wait5Frames macro, add five Delay nodes with the Duration of each set to 0.0.

In the Event Graph, add an Event On Update node and make sure to also add a call to the parent function. Afterwards, connect a Capture Scene node to each of the scene capture components.

Add a call to the Build Foliage Transforms function. Set the Foliage Distribution Map input to RT_Classifications and Normal and Depth Map to RT_Normals. 

The RTWorldBounds is a box containing the captured area. The local minimum will be CaptureWidth * -0.5 and the local maximum will be CaptureWidth * 0.5. These offsets will then be transformed to world space (to later be georeferenced).

From the Event BeginPlay node, we would also want the foliage to update initially after 2 seconds with the world populated with foliage.

In the construction script, make sure to set the Ortho Width of the capture components to the Capture Width.

7Place Actors in the Scene

BP_FoliageCaptureActor can now be placed into the scene. The settings shown below can be used to spawn in trees.

Swap out the CesiumWorldTerrain actor with BP_ProceduralFoliageEllipsoid (make a note of your previous token/asset IDs so they can be used on the new actor).

Set the FoliageCaptureActor variable to the corresponding actor in the scene and then click save.

The setup above should result in many cube instances being spawned at runtime.

Spectacular results can easily be achieved by importing photogrammetry assets from Quixel or trees from the free Rural Australia environment pack.

Optimizations & Troubleshooting

Adding many instances in one frame may cause a hitch, with collision and distance fields being the most impactful when enabled.

Simply disabling distance fields and collisions for mesh types such as grass will reduce the impact. Another improvement would be to amortize the spawning of foliage over a number of frames, which can be done with the pooled FoliageHISM components.

In addition, the spawner has some limitations:

The presence of other tilesets in the scene, such as OSM buildings, may cause improper foliage placement.

  • A possible workaround for this is to set the Primitive Render Mode to Use ShowOnly List, with ShowOnlyActors containing the Cesium Terrain Tileset actor:

  • If bAlignToSurfaceWithRaycast is enabled for a foliage type, the FoliageCaptureActor and terrain tileset will need to use a unique trace channel which is set up in the project settings. See the Unreal Engine documentation for instructions.

The foliage classification system provided in this tutorial can result in foliage being placed on bodies of water.

  • If this is an issue for your project, try modifying the scratchpad you created in the Build Foliage Classifications Niagara module.

The foliage system may capture tiles that have a lower zoom level or level of detail, which results in some instances being elevated above the surface as higher detail tiles are streamed in.

  • Decreasing the CaptureWidth value would limit the components to capture a smaller area around the player, which contains higher levels of tiles.
  • Extend the system to update more frequently as the player moves through sections of the capture grid.

Use of custom normal maps in the tileset material causes instances to be rotated incorrectly.

  • This can be fixed by enabling bAlignToSurfaceWithRaycast, with a slight performance impact on the foliage builder (however, this depends on your system).

It’s also worth noting that materials used with the spawner will need to enable Use with Instanced Static Meshes in the material.

Full Source Code

The source code for the project is also available on GitHub (configured for Unreal Engine 4.26.2).

Currently the project is using the default token for the terrain tileset found in CesiumForUnrealSamples. To add tiles you’ll need to connect your Cesium ion account.

Content and code examples at cesium.com/learn are available under the Apache 2.0 license. You can use the code examples in your commercial or non-commercial applications.