Kinect & HTML5 using WebSockets and Canvas

Kinect defined Natural User Interaction. HTML5 redefined the Web. Currenty, there are various tutorials describing how to interact with a Kinect sensor using Windows Forms or WPF for the user interface. But what about using a web interface for handling Kinect data? Trying to combine those two hot, cutting-edge technologies, I came up with a pretty and open-source solution, which I am going to describe in this blog post.

I am going to use the official Kinect SDK for my demo, but same principles apply to OpenNI SDK, too.

• Download source code and binaries

Prerequisites

• Kinect sensor
• Microsoft Kinect SDK (or OpenNI alternative)

Results

The project consists of two sub-projects: A server-side application which uses Kinect SDK and a client-side web page displaying the skeleton joints on an HTML5 canvas.

Client application:

Server application:

Tutorial

Here is, step by step, a way to achieve the above functionality:

Step 1: Server application

The server application's job is straightforward: Detect the users' joints, pack the data and send them to the clients using web sockets.

In order to detect the joints' coordinates, we need to add a reference to our prefered Kinect SDK and handle the skeleton events. I recommend Microsoft SDK over OpenNI because it's far less complicated:

static void Nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{

    List users = new List();

    foreach (var user in e.SkeletonFrame.Skeletons)
    {
        if (user.TrackingState == SkeletonTrackingState.Tracked)
        {
            users.Add(user);
        }
    }

    if (users.Count > 0)
    {
        string json = users.Serialize();

        foreach (var socket in _sockets)
        {
            socket.Send(json);
        }
    }
}

In order to send the data to the subscribed clients, we need to "pack" the joints' coordinates in a way that the clients will be able to understand and process. I decided to encode the coordinates in JSON format, as JSON is a lightweight and easy-to-understand way of transmitting data throught the web. You can find the JSON-encoder class in the source code files.

Considering the data transmission, I highly recommend the use of Fleck. Fleck, based on Nugget, is a C# web socket library that does what it says with minimum configuration effort: It broadcasts the desired data to the subscribed clients using lightweight web sockets. Here is how you can initialize a web socket server:

_sockets = new List();

var server = new WebSocketServer("ws://localhost:8181");

server.Start(socket =>
{
    socket.OnOpen = () =>
    {
        Console.WriteLine("Connected to " + socket.ConnectionInfo.ClientIpAddress);
        _sockets.Add(socket);
    };
    socket.OnClose = () =>
    {
        Console.WriteLine("Disconnected from " + socket.ConnectionInfo.ClientIpAddress);
        _sockets.Remove(socket);
    };
    socket.OnMessage = message =>
    {
        Console.WriteLine(message);
    };
});

After collecting, packing and transmitting the data, clients can now consume and process them accordingly.

Step 2: Client application

Time for HTML5 bits! We have the server set up, so let's add the web page clients. HTML5 spec recommends web socket support, currently implemented by Internet Explorer 10, Google Chrome and Mozilla Firefox. Web sockets are great for direct message communication between server and clients.

All we need is some JavaScript for receiving the server data and a canvas for drawing the points. Here is the JavaScript event handler for getting the data and drawing the joints to a <canvas> element:

socket.onmessage = function (evt) {
    status.innerHTML = "Kinect data received.";

    // Get the data in JSON format.
    var jsonObject = eval('(' + evt.data + ')');

    context.clearRect(0, 0, canvas.width, canvas.height);
    context.fillStyle = "#FF0000";
    context.beginPath();

    // Display the skeleton joints.
    for (var i = 0; i < jsonObject.skeletons.length; i++) {
        for (var j = 0; j < jsonObject.skeletons[ i ].joints.length; j++) {
            var joint = jsonObject.skeletons[ i ].joints[ j ];

            // Draw!!!
            context.arc(parseFloat(joint.x), parseFloat(joint.y), 10, 0, Math.PI * 2, true);
        }
    }

    context.closePath();
    context.fill();
};

Step 3: Mixin it up!

Now run the server application and then open the HTML file in Internet Explorer 10 or Google Chrome. Stand in front of a Kinect sensor and may the force be with you.

FAQ

• Do I really need a server application? Yes. You can't run the Kinect SDK directly from your browser. JavaScript cannot access your hardware. The server app is nothing but a Visual Studio Console project!
• Do I need to pack my data to JSON? No. You can use any format you'd like. JSON is a common format for transmitting data, especially when file size matters. You could use XML or any other custom format, but JSON is lightweight and needs no custom parser. After all, JSON is a subset of JavaScript, so JavaScript can handle it directly.
• Why not using WCF instead of web sockets? Because it's tougher. Using web sockets, messages can be sent after a server event occurs. And it's a lot easier!

Downloads

You will definitely need to download source code and binaries in order to develop your own HTML5 Kinect applications. Enjoy!

CodeProject