Unity Gaming: Integrating Kinect (part 3)

Kinect Skeleton in Unity

So most of this code is used from the SDK, however I will be going into a detailed explanation about how it all works. As well as how to put it into Unity.

First things First. Create a new Folder inside your Scripts folder Called KinectScripts. Now in that folder create two new scripts; one called BodyManger and the other called BodyView.

UnityScriptsOrg

BodyManager Script

First we need a BodyManager script.  This object will Mange the Kinect sensor connection and read in all the body data coming from the Kinect. Import Kinect Library with:

using Windows.Kinect;

Then we need some fields for the manager to store and the data and get sensor data.

    private KinectSensor _Sensor;
    private BodyFrameReader _Reader;
    private Body[] _Data = null;

    public Body[] GetData()
    {
        return _Data;
    }

Okay so now in the Start method we want to establish the connection for the Kinect.

    void Start()
    {
        _Sensor = KinectSensor.GetDefault();

        if (_Sensor != null)
        {
            _Reader = _Sensor.BodyFrameSource.OpenReader();

            if (!_Sensor.IsOpen)
            {
                _Sensor.Open();
            }
        }
    }

Now that the connection is open and reading in the data we need to store it in the Body array. We will do this every frame of the game, therefore we need to edit the Update() method.
First we check to see if the _Reader has been established and the connection has been completed. If it has we will take the last frame, the reader read in and if it’s not null, we can then check to see if the data is there.

    void Update()
    {
        if (_Reader != null)
        {
            var frame = _Reader.AcquireLatestFrame();
            if (frame != null)
            {
                if (_Data == null)
                {
                }
            }
        }
    }

We still need to get the Body data from the Senor. To do this we will need to create a new Body array with data from the Sensor.BodyFrameSource.BodyCount.
At the end the method should look like this:

    void Update()
    {
        if (_Reader != null)
        {
            var frame = _Reader.AcquireLatestFrame();
            if (frame != null)
            {
                if (_Data == null)
                {
                    _Data = new Body[_Sensor.BodyFrameSource.BodyCount];
                }
            }
        }
    }

Then we need to refresh the stream of data from the Reader. By adding the following code to manipulate the frame.

    void Update()
    {
        if (_Reader != null)
        {
            var frame = _Reader.AcquireLatestFrame();
            if (frame != null)
            {
                if (_Data == null)
                {
                    _Data = new Body[_Sensor.BodyFrameSource.BodyCount];
                }

                frame.GetAndRefreshBodyData(_Data);

                frame.Dispose();
                frame = null;
            }
        }
    }

The last method in the Body Manager Class is OnApplicationQuit(), which Disposes the Reader, and closes the Sensor stream, sets it to null.

  void OnApplicationQuit()
    {
        if (_Reader != null)
        {
            _Reader.Dispose();
            _Reader = null;
        }

        if (_Sensor != null)
        {
            if (_Sensor.IsOpen)
            {
                _Sensor.Close();
            }

            _Sensor = null;
        }
    }

Now onto drawing the skeleton in the scene.

 

BodyView Script

The next Script to write is one to draw the skeletal structure. We won’t necessarily need to see the skeleton for the game, however, I’ll show you how to show skeletal body tracking. We also need the skeletal data to track the hands, whose state will dictate controller commands.

For this MonoBehavoir class we will need, a material to draw the bone in the Unity scene. A gameobject to store the BodyManger, to control the stream of the Kinect.

    public Material BoneMaterial;
    public GameObject BodyManager;

We also need a BodyManager object and a Dictionary to store the bodies being tracked.

    private Dictionary<ulong, GameObject> _Bodies = new Dictionary<ulong, GameObject>();
    private BodyManager _BodyManager;

Next we need to map out all the bones by the two joints that they will be connected to.

    private Dictionary<Kinect.JointType, Kinect.JointType> _BoneMap = new Dictionary<Kinect.JointType, Kinect.JointType>()
    {
        { Kinect.JointType.FootLeft, Kinect.JointType.AnkleLeft },
        { Kinect.JointType.AnkleLeft, Kinect.JointType.KneeLeft },
        { Kinect.JointType.KneeLeft, Kinect.JointType.HipLeft },
        { Kinect.JointType.HipLeft, Kinect.JointType.SpineBase },

        { Kinect.JointType.FootRight, Kinect.JointType.AnkleRight },
        { Kinect.JointType.AnkleRight, Kinect.JointType.KneeRight },
        { Kinect.JointType.KneeRight, Kinect.JointType.HipRight },
        { Kinect.JointType.HipRight, Kinect.JointType.SpineBase },

        { Kinect.JointType.HandTipLeft, Kinect.JointType.HandLeft }, //Need this for HandSates
        { Kinect.JointType.ThumbLeft, Kinect.JointType.HandLeft },
        { Kinect.JointType.HandLeft, Kinect.JointType.WristLeft },
        { Kinect.JointType.WristLeft, Kinect.JointType.ElbowLeft },
        { Kinect.JointType.ElbowLeft, Kinect.JointType.ShoulderLeft },
        { Kinect.JointType.ShoulderLeft, Kinect.JointType.SpineShoulder },

        { Kinect.JointType.HandTipRight, Kinect.JointType.HandRight }, //Needthis for Hand State
        { Kinect.JointType.ThumbRight, Kinect.JointType.HandRight },
        { Kinect.JointType.HandRight, Kinect.JointType.WristRight },
        { Kinect.JointType.WristRight, Kinect.JointType.ElbowRight },
        { Kinect.JointType.ElbowRight, Kinect.JointType.ShoulderRight },
        { Kinect.JointType.ShoulderRight, Kinect.JointType.SpineShoulder },

        { Kinect.JointType.SpineBase, Kinect.JointType.SpineMid },
        { Kinect.JointType.SpineMid, Kinect.JointType.SpineShoulder },
        { Kinect.JointType.SpineShoulder, Kinect.JointType.Neck },
        { Kinect.JointType.Neck, Kinect.JointType.Head },
    };

BodyView Update()

Now in the Unity Update() method we need to check to see if the Body Manager is not null and that it has data.

   void Update()
    {
        int state = 0;

        if (BodyManager == null)
        {
            return;
        }

        _BodyManager = BodyManager.GetComponent<BodyManager>();
        if (_BodyManager == null)
        {
            return;
        }

        Kinect.Body[] data = _BodyManager.GetData();
        if (data == null)
        {
            return;
        }
    }

Next, while still in the Update() method, we need to get the amount of bodies in the list of tracked bodies. Then delete unknown bodies.

        List<ulong> trackedIds = new List<ulong>();
        foreach (var body in data)
        {
            if (body == null)
            {
                continue;
            }

            if (body.IsTracked)
            {
                trackedIds.Add(body.TrackingId);
            }
        }

        List<ulong> knownIds = new List<ulong>(_Bodies.Keys);

        // First delete untracked bodies
        foreach (ulong trackingId in knownIds)
        {
            if (!trackedIds.Contains(trackingId))
            {
                Destroy(_Bodies[trackingId]);
                _Bodies.Remove(trackingId);
            }
        }

Now that we have the keys for tracking the bodies we need to create a body object with that tracking ID key. We need to write two more methods. A CreateBodyObject() method that will take a ulong id and a RefreashBodyObject() method that will take a Kinect.Body object and a GameObject for the body. We will use these methods after we go through the data, and find if bodies inside are being tracked or not. If it is tracked but doesn’t have a TrackingId, then we need to create a body with that TrackingID. If it is being tracked and has a TrackingId then we just need to refresh the drawn body.

  foreach (var body in data)
        {
            if (body == null)
            {
                continue;
            }

            if (body.IsTracked)
            {
                if (!_Bodies.ContainsKey(body.TrackingId))
                {
                    _Bodies[body.TrackingId] = CreateBodyObject(body.TrackingId);
                }

                RefreshBodyObject(body, _Bodies[body.TrackingId]);
            }
        }

    }

 

CreateBodyObject()

The CreateBodyObject takes an ID and returns a body gameobject. So we first need to create a gameobject that will store the appropriate data retrieved; then we need a for loop to go through every joint to draw the body.

   private GameObject CreateBodyObject(ulong id)
    {
        GameObject body = new GameObject("Body:" + id);

        for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
        {

        }

        return body;
    }

For every joint in the body we create a cube and add a lineRenderer to that cube. The cube will be drawn at each joint while the line renderer will be drawn to connect the joints.


   private GameObject CreateBodyObject(ulong id)
    {
        GameObject body = new GameObject("Body:" + id);

        for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
        {
            GameObject jointObj = GameObject.CreatePrimitive(PrimitiveType.Cube);

            LineRenderer lr = jointObj.AddComponent<LineRenderer>();
            lr.SetVertexCount(2);
            lr.material = BoneMaterial;
            lr.SetWidth(0.05f, 0.05f);

            jointObj.transform.localScale = new Vector3(0.3f, 0.3f, 0.3f);
            jointObj.name = jt.ToString();
            jointObj.transform.parent = body.transform;
        }

        return body;
    }

 

RefreashBodyObject()

Now to write the ResfreshBodyObject method. In this method we need to go through each joint type possible just like we did in the CreateBodyObject method. But this time we are passing in the current body, as well as the appropriate tracking number so we don’t draw the bones for the wrong person.

 

   private void RefreshBodyObject(Kinect.Body body, GameObject bodyObject)
    {
        for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
        {

        }
    }

Inside this for loop we need to get the key value pairs we made before in the bone loop for each joint.

   private void RefreshBodyObject(Kinect.Body body, GameObject bodyObject)
    {
        for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
        {
            Kinect.Joint sourceJoint = body.Joints[jt];
            Kinect.Joint? targetJoint = null;

            if(_BoneMap.ContainsKey(jt))
            {
                targetJoint = body.Joints[_BoneMap[jt]];
            }
        }
    }

We also need to update the skeletons position so it’s in the accurate place on the screen. To do this we need to write a method to get the Vetcor3 from the sourceJoint.

    private static Vector3 GetVector3FromJoint(Kinect.Joint joint)
    {
        return new Vector3(joint.Position.X * 10, joint.Position.Y * 10, joint.Position.Z * 10);
    }

The scale by 10 is to enlarge the skeleton, which will make it easier to work with. Now we have position to correct the gameObjects position.

   private void RefreshBodyObject(Kinect.Body body, GameObject bodyObject)
    {
        for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
        {
            Kinect.Joint sourceJoint = body.Joints[jt];
            Kinect.Joint? targetJoint = null;

            if(_BoneMap.ContainsKey(jt))
            {
                targetJoint = body.Joints[_BoneMap[jt]];
            }

            Transform jointObj = bodyObject.transform.FindChild(jt.ToString());
            jointObj.localPosition = GetVector3FromJoint(sourceJoint);
        }
    }

Next step in the for-loop is to get the linerenderer from the bodyObject, which was the cube we create for each joint. Then we need to see if target joint has a value. If it does we can then draw a line from the original joint to the target.

            LineRenderer lr = jointObj.GetComponent<LineRenderer>();
            if(targetJoint.HasValue)
            {
                lr.SetPosition(0, jointObj.localPosition);
                lr.SetPosition(1, GetVector3FromJoint(targetJoint.Value));
            }
            else
            {
                lr.enabled = false;
            }

Great! So we are almost done with drawing the skeleton. There is a bit more information that will be helpful that the SDK gives you, which is tracking status. There are three states to choose from, Tracked, Inferred, or NotTracked. We can have the line renderer show us the state of tracking by changing it’s color. To do this we need a method that will return a color based on the current state.

    private static Color GetColorForState(Kinect.TrackingState state)
    {
        switch (state)
        {
            case Kinect.TrackingState.Tracked:
                return Color.green;

            case Kinect.TrackingState.Inferred:
                return Color.red;

            default:
                return Color.black;
        }
    }

Now we add one more line to the for-loop of the RefreachBodyObject method and we are done.

    private void RefreshBodyObject(Kinect.Body body, GameObject bodyObject)
    {
        for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
        {
            Kinect.Joint sourceJoint = body.Joints[jt];
            Kinect.Joint? targetJoint = null;

            if (_BoneMap.ContainsKey(jt))
            {
                targetJoint = body.Joints[_BoneMap[jt]];
            }

            Transform jointObj = bodyObject.transform.FindChild(jt.ToString());
            jointObj.localPosition = GetVector3FromJoint(sourceJoint);

            LineRenderer lr = jointObj.GetComponent<LineRenderer>();
            if (targetJoint.HasValue)
            {
                lr.SetPosition(0, jointObj.localPosition);
                lr.SetPosition(1, GetVector3FromJoint(targetJoint.Value));
                lr.SetColors(GetColorForState(sourceJoint.TrackingState), GetColorForState(targetJoint.Value.TrackingState));
            }
            else
            {
                lr.enabled = false;
            }
        }
    }

And that’s it for drawing the skeleton!

Putting the Skeleton into Unity

Now in Unity I’ve created another scene by going to file new scene (it will prompt you to save your current scene if you haven’t already). This empty scene will make it easier for you to test and see what you’re doing.

In the empty scene, which I have saved as kinectTest, create two empty game objects. Call them Body Manager and Body View.

UnityObjectsBVBM

Attach the Body Manager script to the body manager object. Attach the Body View script to the BodyView object.

BodyManagerInspector

Select the Body View object. In the inspector you need to fill in the Body Manager slot and Bone Material slot. Click and drag the BodyManager from the hierarchy to the slot. Then in your materials folder click and drag the Bone Material into the bone material slot.

BodyViewObjectSettings

Now hit run. You should see nothing on your screen at first. This is because the Kinect hasn’t registered your skeleton yet. Stand back about 4 ft, and you should see the skeleton appear.

UnityKinectScene

If you look closely you can tell that the legs of the skeleton are red. This is because the Kinect is interpreting that’s where my legs should be. My desk was blocking the sensor from detecting where my legs actually were.

But wait! You say. We don’t need a skeleton in our infinite runner game, just the hand gestures and states! So was all this a waste? NO! Of course it wasn’t. We just don’t need to draw the whole skeleton in the scene anymore. But we now have the know-how if you need to in the future. YAY!

The next part of this mini series will be getting those hand gestures to work in the game!

 

HAPPY CODING!

-TheNappingKat


4 thoughts on “Unity Gaming: Integrating Kinect (part 3)”

  1. Hi there, thanks so much for making this tutorial, it’s been immensely helpful in my hackathons and projects! However, I just wanted to point out that when implementing the BodyView.cs, I had a problem where Visual Studio wouldn’t recognize the Kinect namespace and not build (even though the “using Windows.Kinect” was included). Just wanted to let you know that I solved this problem by replacing that using line with “using Kinect = Windows.Kinect” instead.
    Thanks!

  2. This is a great series of tutorials, thank you so much! Is part 4 coming anytime soon? I’m very interested hand gestures recognition with Kinect on Unity.

Leave a Reply

Your email address will not be published. Required fields are marked *