Using Spatial Understanding to find custom room elements

Having just spent ~~two~~ nearly three days trying to get the Hololens to see a table, I decided to write this due to the lack of documentation surrounding these features available on the web.

This guide assumes you already have the Hololens Toolkit imported in your project, and know your way around Unity at least a little. I’m using Unity 2017.4 but a later version will also work (Apart from maybe 2018.2b for now).

Setting up your scene

Two prefabs are required in your scene before you can start any of this, and they are SpatialMapping and SpatialUnderstanding, both of which can be found inside the HoloToolkit (Use the search box to find them, it’s easier). Drag these into your scene, and then on the SpatialMapping object you will need to untick Draw Visual Meshes inside the Spatial Mapping Manger component. Optionally you can also change the Mesh material on the SpatialUnderstanding script to be Wireframe.mat which, while not necessary, will make it a lot easier to see what’s going on when it starts generating meshes over everything.

Now you should also create an Empty Object in the scene and call it MappingController or something equally meaningful. You will want to create a new script component on this in C# and call it ScanManager.

MappingController.cs

Inside your MappingController.cs file the first thing you want to do is to tell the compiler you will be using things from various packages, so replace the three default imports with the following:

using System.Collections.Generic;
using System;
using System.Collections.ObjectModel;
using HoloToolkit.Unity;
using HoloToolkit.Unity.InputModule;
using UnityEngine;

I will point out where each import is used as we reach that part in the code. After this, you want to implement IInputClickHandler from the HoloToolkit.Unity.InputModule package and then implement the function OnInputClicked(InputClickedEventData eventData). Inside this function just add the lines

Debug.Log("Requesting scan finish");
SpatialUnderstanding.Instance.RequestFinishScan();

This is used so when we airtap, the scan will begin to finish.

At the top of the class let’s just accept a prefab from the Inspector:

[SerializeField]
private GameObject prefab;

Inside your Start() function you want to add two lines:

InputManager.Instance.PushFallBackHandler(gameObject) What this does is makes this gameObject (notice the capitalisation) the handler for any click events that don’t have focus on any other object . This also comes from HoloToolkit’s Input module.
SpatialUnderstanding.Instance.ScanStateChanged += ScanStateChanged. This is going to throw an error but we will fix that in just a moment. This line adds our function ScanStateChanged() to be called whenever SpatialUnderstanding.Instance.ScanStateChanged gets called. This delegate pattern allows for functions to be called on classes that have no knowledge of each other.

Now we have to create our ScanStateChanged() function we referenced in Start(). This function has to be public (It’s being called from outside of our class) with a return type of void. Inside this function we will add the following lines:

public void ScanStateChanged() 
{
	switch (SpatialUnderstanding.Instance.ScanState)
	{
		case SpatialUnderstanding.ScanStates.Scanning:
			LogSurfaceState();
			break;
		case SpatialUnderstanding.ScanStates.Done:
			InstantiateObjectOnTable();
			break;
		default:
			break;
	}
}	

Now whilst the same could be done with an if/else if statement, this allows for further expansion if you wanted to listen for other states such as ReadyToScan or Finishing. Again you are going to get an error around the two function calls, so let’s create them with return type void and leave them empty for now. Creating your LogSurfaceState() function is completely optional, in fact you will almost definitely want to remove it from your application before publishing.

private void LogSurfaceState()
{
    IntPtr statPtr = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStatsPtr();
    if (SpatialUnderstandingDll.Imports.QueryPlayspaceStats(statPtr) != 0)
    {
        var stats = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStats();
        Debug.Log = String.Format("TotalSurfaceArea: {0:0.##}\n" +
                                  "WallSurfaceArea: {1:0.##}\n" +
                                  "HorizSurfaceArea: {2:0.##}", 
                                  stats.TotalSurfaceArea, 
                                  stats.WallSurfaceArea, 
                                  stats.HorizSurfaceArea);
    }
}

Whilst this all looks a little messy, all it is doing is gathering all of the data about how much data has been collected about the space and how much of it is horizontal vs vertical space. In my apps I have this as a little tagalong 3DTextMesh that I can check whilst wearing the Hololens.

Defining custom shapes

Let’s create a private function called AddShapeDefinition and give it the following signature.

void AddShapeDefinition(string shape name,
                        List<SpatialUnderstandingDllShapes.ShapeComponent> shapeComponents,
                        List<SpatialUnderstandingDllShapes.ShapeConstraints> shapeConstraints)

The shapeName is going top be the name the shape gets instantiated using later on.

Let’s populate the method with the following code:

IntPtr shapeComponentsPtr = (shapeComponents == null) ? IntPtr.Zero : SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(shapeComponents.ToArray());
IntPtr shapeConstraintsPtr = (shapeConstraints == null) ? IntPtr.Zero : SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(shapeConstraints.ToArray());
if (SpatialUnderstandingDllShapes.AddShape(shapeName,
    (shapeComponents == null) ? 0 : shapeComponents.Count,
    shapeComponentsPtr,
    (shapeConstraints == null) ? 0 :shapeConstraints.Count,
    shapeConstraintsPtr) == 0)
{
    Debug.Log("Couldn't create shape. Uh oh!");
}

As this is just a helper method, this code might look complicated, but it doesn’t actually do much. The first two lines check whether the parameters passed in were null and returns a pointer for them if they exist. The line inside the if query uses SpatialUnderstanding’s constructor and passes in our parameters. The function returns an int that will be zero if it failed to create the shape.

Let’s create a table. ┬──┬ ¯\_(ツ) Create a new function private void CreateTableShape() and we will populate it below. As we saw above there are two types which will define what a ‘table’ is to the Hololens: SpatialUnderstandingDllShapes.ShapeComponent and SpatialUnderstandingDllShapes.ShapeConstraint.

A component defines a plane of a shape and a shape can have several planes - for example a set of steps may have two or more components, one for each shape. Each component is made by defining a set of rules. Let’s have a look at the two simple rules used to define our table:

shapeComponents = new List<SpatialUnderstandingDllShapes.ShapeComponent>()
{
    new SpatialUnderstandingDllShapes.ShapeComponent(
        new List<SpatialUnderstandingDllShapes.ShapeComponentConstraint>()
        {
            SpatialUnderstandingDllShapes.ShapeComponentConstraint.Create_SurfaceHeight_Between(0.6f, 1.5f),
            SpatialUnderstandingDllShapes.ShapeComponentConstraint.Create_IsRectangle(),
        }),
};

As you can see here there are only two rules in play - one tells the Hololens to look for something between 0.6 and 1.5m off the ground, the second tells it to look for that something to be rectangular. The Create_IsRectangle() call does also accept a single parameter, a value between 0-1 that defines how similarly to a rectangle the shape is, it defaults to 0.5 and going too far over that will give you difficulty detecting the shape outside of perfect situations. It would probably also be wise to add size constraints such as Create_SurfaceAreaMin(float) or Create_RectangleLengthMin(float), but we’re going to ignore those here.

In SpatialUnderstanding, a constraint is a relationship between one or more components. In our example it only applies to the one component we have. Other constraints will relate the size or positions of two components, for example Create_RectanglesParallel(...) which takes the index of two components from the components List<> and ensures it only matches to two parallel shapes. This would be useful if someone was trying to identify a picnic bench, for example, but perhaps not a staircase that can turn corners. Add the below lines to our CreateTableShape function below the components.

shapeConstraints = new List<SpatialUnderstandingDllShapes.ShapeConstraint>()
{
    SpatialUnderstandingDllShapes.ShapeConstraint.Create_NoOtherSurface(),
};

As we can see, our table only needs one constraint, NoOtherSurface(), rule which tells the Hololens that there can’t be any other objects on top of the table. How forgiving that is isn’t clear.

Fortunately these few lines are all you need to define a very simple shape such as a table, and we can therefore pass these two lists into the helper function we created earlier:

AddShapeDefinition(“Table”, shapeComponents, shapeConstraints);

We can also add the following line to this function, but we will need to be aware that this then requires the function to be called only after the SpatialUnderstanding has finished running.

SpatialUnderstandingDllShapes.ActivateShapeAnalysis()

This function can only be called after shapes have been defined and your SpatialUnderstanding.Instance.ScanState is equal to Finished.

So in our ScanStateChanged() function, let’s add the line CreateTableShape() just before our call to InstantiateObjectOnTable.

Locating these shapes

Now we can finally create our private function InstantiateObjectOnTable() and get rid of that error. I’m going to go ahead a put the entire function here, then go through it line by line:

private void InstantiateObjectOnSurface()
{
    const int MaxResultCount = 512;
    var shapeResults = new SpatialUnderstandingDllShapes.ShapeResult[MaxResultCount];

    var resultsShapePtr = SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(shapeResults);
    var locationCount = SpatialUnderstandingDllShapes.QueryShape_FindPositionsOnShape("Table", 0.1f, shapeResults.Length, resultsShapePtr);
        
    if (locationCount > 0)
    {
        var shapeResult = shapeResults[0];
        Instantiate(prefab, shapeResult.position, Quaternion.LookRotation(shapeResult.position.normalized, Vector3.up));
        Debug.Log("Placed Hologram");
    }
    else
    {
        Debug.Log("Not enough space for the hologram");
    }
}

The first line simply is used to define an upper limit for the number of shapes we can store. Is 512 overkill? Of course, but modern computers and even the hololens have more than enough memory for for this.

The next line is to create an empty array of ShapeResult type, of the size we defined above. This line is then passed into the next line into PinObject which is used to reserve the memory to then be accessed in the future without risk of being written to by another program.

The next line is where the magic happens. Query_FindPositionsOnShape() accepts the name of our created shape, the minimum space on the shape, and then the length of our shape array, and a pointer to our array it can write to. This function returns the number of shapes it finds and this number will never exceed the shapeResult.length we passed in. Below this we check if this function returned anything. If so we grab that first shape (there are other ways of doing this, be creative) and Instantiate our prefab in that place.

This should then instantiate a prefab on your table if you try it out! Caveats: obviously you need a hololens to do this. That goes without saying. Also your prefab needs it’s origin at the bottom of it, unless you either A) transform it after instantiation, or B) don’t mind it being inside the table. 🤷‍♂️

Is this the most effective, extensible and reusable way to do things? No, of course not. Ideally you would have some kind of manager doing all of the shape stuff and make calls to that. But does this work? Yeah, and it works more than well enough for a small project or prototype.

The full code can be found below.

using System.Collections.Generic;
using System;
using System.Collections.ObjectModel;
using HoloToolkit.Unity;
using HoloToolkit.Unity.InputModule;
using UnityEngine;

public class ScanManager : MonoBehaviour, IInputClickHandler {
[SerializeField]
private GameObject prefab;

void start () {
    InputManager.Instance.PushFallbackInputHandler(gameObject);
    SpatialUnderstanding.Instance.ScanStateChanged += ScanStateChanged;
}

void OnDestroy () 
{
    if (SpatialUnderstanding.Instance != null) 
    {
        SpatialUnderstanding.Instance.ScanStateChanged -= ScanStateChanged
    }
}

public void OnInputClicked (InputClickedEventData eventData)
{
    Debug.Log("Requesting scan finish");
    SpatialUnderstanding.Instance.RequestFinishScan();
}

private void LogSurfaceState()
{
    IntPtr statPtr = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStatsPtr();
    if (SpatialUnderstandingDll.Imports.QueryPlayspaceStats(statPtr) != 0)
    {
        var stats = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStats();
        Debug.Log(String.Format("TotalSurfaceArea: {0:0.##}\n" +
                                  "WallSurfaceArea: {1:0.##}\n" +
                                  "HorizSurfaceArea: {2:0.##}", 
                                  stats.TotalSurfaceArea, 
                                  stats.WallSurfaceArea, 
                                  stats.HorizSurfaceArea));
    }
}

public void ScanStateChanged() 
{
	switch (SpatialUnderstanding.Instance.ScanState)
	{
		case SpatialUnderstanding.ScanStates.Scanning:
			LogSurfaceState();
			break;
		case SpatialUnderstanding.ScanStates.Done:
          CreateTableShape();
			InstantiateObjectOnTable();
			break;
		default:
			break;
	}
}	

void AddShapeDefinition(string shape name,
                        List<SpatialUnderstandingDllShapes.ShapeComponent> shapeComponents,
                        List<SpatialUnderstandingDllShapes.ShapeConstraint> shapeConstraints)
{
    IntPtr shapeComponentsPtr = (shapeComponents == null) ? IntPtr.Zero : SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(shapeComponents.ToArray());
    IntPtr shapeConstraintsPtr = (shapeConstraints == null) ? IntPtr.Zero : SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(shapeConstraints.ToArray());
    if (SpatialUnderstandingDllShapes.AddShape(shapeName,
       (shapeComponents == null) ? 0 : shapeComponents.Count,
        shapeComponentsPtr,
       (shapeConstraints == null) ? 0 :shapeConstraints.Count,
        shapeConstraintsPtr) == 0)
    {
        Debug.Log("Couldn't create shape. Uh oh!");
    }
}

private void CreateTableShape () 
{
    var shapeComponents = new List<SpatialUnderstandingDllShapes.ShapeComponent>()
    {
        new SpatialUnderstandingDllShapes.ShapeComponent(
            new List<SpatialUnderstandingDllShapes.ShapeComponentConstraint>()
            {
                SpatialUnderstandingDllShapes.ShapeComponentConstraint.Create_SurfaceHeight_Between(0.6f, 1.5f),
                SpatialUnderstandingDllShapes.ShapeComponentConstraint.Create_IsRectangle(),
        }),
    };
    var shapeConstraints = new List<SpatialUnderstandingDllShapes.ShapeConstraint>()
    {
        SpatialUnderstandingDllShapes.ShapeConstraint.Create_NoOtherSurface(),
    };
    AddShapeDefinition("Table", shapeComponents, shapeConstraints);
    SpatialUnderstandingDllShapes.ActivateShapeAnalysis();
}


private void InstantiateObjectOnTable()
{
    const int MaxResultCount = 512;
    var shapeResults = new SpatialUnderstandingDllShapes.ShapeResult[MaxResultCount];

    var resultsShapePtr = SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(shapeResults);
    var locationCount = SpatialUnderstandingDllShapes.QueryShape_FindPositionsOnShape("Table", 0.1f, shapeResults.Length, resultsShapePtr);
        
    if (locationCount > 0)
    {
        var shapeResult = shapeResults[0];
        Instantiate(prefab, shapeResult.position, Quaternion.LookRotation(shapeResult.position.normalized, Vector3.up));
        Debug.Log("Placed Hologram");
    }
    else
    {
        Debug.Log("Not enough space for the hologram");
    }
}
}