Using Nuitrack in Unreal Engine 4.25 (Windows)

In this tutorial, you'll learn how to create a new Unreal Engine project with Nuitrack and how to integrate Nuitrack with your own Unreal Engine project if you already have one. As a result, you'll get a simple project with an option of skeleton detection and tracking that can be further modified into a full-fledged desktop application.

Before you begin, make sure that you've downloaded and installed the Nuitrack runtime following our installation instructions.

Creating a New Project with Nuitrack

Download Unreal Engine 4.25 and run it.
Create a New Project: Blank → Next. Select the following settings for your project:
- Project Type: C++;
- Class of Hardware: Desktop / Console;
- Graphical Level: Maximum Quality;
- Additional Content: No Starter Content.

Set the project name: NuiSample

Delete all the Labels except Player Start from the World.

Set Player Location, Rotation and Scale values as in the screenshot below.

Save the current World.

Select Edit → Project Settings → Maps & Modes:

6.1. Set your World as Editor Startup Map and Game Default Map.

6.2. Set Default GameMode.
Set up your project as described in Setting up your project.
Follow the steps below to make your application quit when a user presses the Esc button:

8.1. Open Level Blueprint for the current Level (Blueprints → Open Level Blueprint)

8.2. Add the following Blueprint nodes (click the right mouse button to open the collection):
- Quit Game
- Escape
8.3. Add connections:

8.4. Click the Compile button.
Open Visual Studio for writing C++ code.

Files in the Unreal Projects/{ProjectName}/Source/{ProjectName} directory should have the following content:

10.1. NuiSample.Build.cs:

using UnrealBuildTool;

public class NuiSample : ModuleRules
{
	public NuiSample(ReadOnlyTargetRules Target) : base(Target)
	{
		PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
	
		PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore" });

		PrivateDependencyModuleNames.AddRange(new string[] { "NuitrackModule" });
	}
}

10.2. NuiSample.h:

#pragma once

#include <iostream>
#include <vector>

#include "nuitrack/Nuitrack.h"
#include "nuitrack/modules/SkeletonTracker.h"
#include "nuitrack/types/Skeleton.h"
#include "nuitrack/types/SkeletonData.h"

10.3. NuiSample.cpp:

#include "NuiSample.h"
#include "Modules/ModuleManager.h"

IMPLEMENT_PRIMARY_GAME_MODULE( FDefaultGameModuleImpl, NuiSample, "NuiSample" );

10.4. NuiSampleGameModeBase.h:

#pragma once

#include "NuiSample.h"
#include "GameFramework/GameModeBase.h"
#include "NuiSampleGameModeBase.generated.h"

using tdv::nuitrack::SkeletonTracker;
using tdv::nuitrack::SkeletonData;
using tdv::nuitrack::Joint;
using tdv::nuitrack::Vector3;

UCLASS()
class NUISAMPLE_API ANuiSampleGameModeBase : public AGameModeBase
{
	GENERATED_BODY()

	UWorld* World;

	ANuiSampleGameModeBase();

	void Tick(float dt) override;
	void BeginPlay() override;
	void BeginDestroy() override;

	SkeletonTracker::Ptr skeletonTracker;

	void OnSkeletonUpdate(SkeletonData::Ptr userSkeletons);
	void DrawSkeleton(int skeleton_index, std::vector<Joint> joints);
	void DrawBone(Joint j1, Joint j2);

	static FVector RealToPosition(Vector3 real);
};

10.5. NuiSampleGameModeBase.cpp:

#include "NuiSampleGameModeBase.h"
#include "DrawDebugHelpers.h"

using tdv::nuitrack::Nuitrack;
using tdv::nuitrack::JointType;

ANuiSampleGameModeBase::ANuiSampleGameModeBase()
{
	this->PrimaryActorTick.bCanEverTick = true;
}

void ANuiSampleGameModeBase::Tick(float dt)
{
	Nuitrack::update();
}

void ANuiSampleGameModeBase::BeginPlay()
{
	Super::BeginPlay();

	UE_LOG(LogTemp, Warning, TEXT("ANuiSampleGameModeBase::BeginPlay()"));

	World = GetWorld();

	if (World)
	{
		UE_LOG(LogTemp, Warning, TEXT("Nuitrack::init() CALLING..."));
		Nuitrack::init();

		UE_LOG(LogTemp, Warning, TEXT("SkeletonTracker::create() CALLING..."));
		skeletonTracker = SkeletonTracker::create();

		UE_LOG(LogTemp, Warning, TEXT(
			"skeletonTracker->connectOnUpdate() CALLING..."));
		skeletonTracker->connectOnUpdate(
			std::bind(&ANuiSampleGameModeBase::OnSkeletonUpdate,
				this, std::placeholders::_1));

		UE_LOG(LogTemp, Warning, TEXT("Nuitrack::run() CALLING..."));
		Nuitrack::run();
	}
	else
	{
		UE_LOG(LogTemp, Error, TEXT("NULL WORLD"));
	}
}

void ANuiSampleGameModeBase::BeginDestroy()
{
	Super::BeginDestroy();
	Nuitrack::release();
}

void ANuiSampleGameModeBase::OnSkeletonUpdate(SkeletonData::Ptr userSkeletons)
{
	auto skeletons = userSkeletons->getSkeletons();

	FlushPersistentDebugLines(World);

	if (!skeletons.empty())
	{
		for (auto skeleton : skeletons)
		{
			DrawSkeleton(skeleton.id, skeleton.joints);
		}
	}
}

void ANuiSampleGameModeBase::DrawSkeleton(int skeleton_index, std::vector<Joint> joints)
{
	if (joints.empty())
		return;

	DrawBone(joints[JointType::JOINT_HEAD], joints[JointType::JOINT_NECK]);
	DrawBone(joints[JointType::JOINT_NECK], joints[JointType::JOINT_TORSO]);
	DrawBone(joints[JointType::JOINT_RIGHT_SHOULDER], joints[JointType::JOINT_LEFT_SHOULDER]);
	DrawBone(joints[JointType::JOINT_WAIST], joints[JointType::JOINT_LEFT_HIP]);
	DrawBone(joints[JointType::JOINT_WAIST], joints[JointType::JOINT_RIGHT_HIP]);
	DrawBone(joints[JointType::JOINT_TORSO], joints[JointType::JOINT_WAIST]);
	DrawBone(joints[JointType::JOINT_LEFT_SHOULDER], joints[JointType::JOINT_LEFT_ELBOW]);
	DrawBone(joints[JointType::JOINT_LEFT_ELBOW], joints[JointType::JOINT_LEFT_WRIST]);
	DrawBone(joints[JointType::JOINT_RIGHT_SHOULDER], joints[JointType::JOINT_RIGHT_ELBOW]);
	DrawBone(joints[JointType::JOINT_RIGHT_ELBOW], joints[JointType::JOINT_RIGHT_WRIST]);
	DrawBone(joints[JointType::JOINT_RIGHT_HIP], joints[JointType::JOINT_RIGHT_KNEE]);
	DrawBone(joints[JointType::JOINT_LEFT_HIP], joints[JointType::JOINT_LEFT_KNEE]);
	DrawBone(joints[JointType::JOINT_RIGHT_KNEE], joints[JointType::JOINT_RIGHT_ANKLE]);
	DrawBone(joints[JointType::JOINT_LEFT_KNEE], joints[JointType::JOINT_LEFT_ANKLE]);
}

void ANuiSampleGameModeBase::DrawBone(Joint j1, Joint j2)
{
	DrawDebugLine(World, RealToPosition(j1.real), RealToPosition(j2.real),
		FColor::MakeRedToGreenColorFromScalar((j1.confidence + j2.confidence) * 0.5),
		true, -1, 0, 4);
}

FVector ANuiSampleGameModeBase::RealToPosition(Vector3 real)
{
	return FVector(-real.x, real.z, real.y) * 0.1f;
}

Build a Visual Studio project.
Run sample.

Integrating Nuitrack with an Existing Unreal Engine Project

Setting up your Project

Copy the NuitrackPlugin folder from {SDK Root}/UnrealEngine to the Plugins folder of your project (create the Plugins folder if does not exist). After that, restart Unreal Editor.
Generate Visual Studio project files by right-clicking an .uproject file.

Add NuitrackModule to the file with project dependencies Build.cs (Unreal Projects/{ProjectName}/Source/{ProjectName}/{ProjectName}.Build.cs):

PrivateDependencyModuleNames.AddRange(new string[] { "NuitrackModule" });

Linking Nuitrack

You can see an example of using the methods described in Unreal Projects/{ProjectName}/Source/{ProjectName}/NuiSampleGameModeBase.cpp (see the section Creating a New Project with Nuitrack, p. 10.5).

To initialize Nuitrack, create a skeleton tracker and subscribe to an event, you need to add the following code, for example, to the BeginPlay method of the GameMode class:

Nuitrack::init();
SkeletonTracker::Ptr skeletonTracker = SkeletonTracker::create();
skeletonTracker->connectOnUpdate(std::bind(&ANuiSampleGameModeBase::OnSkeletonUpdate, this, std::placeholders::_1));
Nuitrack::run();

To terminate Nuitrack, call the release method. You can add it to the BeginDestroy method of the GameMode class:

Nuitrack::release();

To raise events with new data from modules, call the update method. You can add it to the Tick method of the GameMode class:

Nuitrack::update();

Skeleton tracking data will be available inside the OnSkeletonUpdate method that was subscribed to the OnUpdate event from SkeletonTracker:

void ANuiSampleGameModeBase::OnSkeletonUpdate(SkeletonData::Ptr userSkeletons)
{
  auto skeletons = userSkeletons->getSkeletons();
  ...
}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unreal_Engine_Windows_Tutorial.md

Unreal_Engine_Windows_Tutorial.md

Using Nuitrack in Unreal Engine 4.25 (Windows)

Creating a New Project with Nuitrack

Integrating Nuitrack with an Existing Unreal Engine Project

Setting up your Project

Linking Nuitrack

Files

Unreal_Engine_Windows_Tutorial.md

Latest commit

History

Unreal_Engine_Windows_Tutorial.md

File metadata and controls

Using Nuitrack in Unreal Engine 4.25 (Windows)

Creating a New Project with Nuitrack

Integrating Nuitrack with an Existing Unreal Engine Project

Setting up your Project

Linking Nuitrack