In this post we will begin to add TUIO (Tangible User Interface Object) support to our project by implementing a server module to parse OSC (Open Sound Control) packets for 2D cursor descriptions. This will allow us to import blob events from a client application. We will use the TUIOdroid application available for android devices to send UDP packets to our server module. The server module will parse these events, and our trackers will import the blob events. This post is intended to provide only a preliminary implementation of the TUIO protocol. There is information available in the packets that we will overlook for the time being. We'll concern ourselves with parsing the cursor locations from the UDP packets and allowing our tracker modules to process those events as though they were native touch events.
Below is a description of the data contained in a packet. This particular example packet only contains the cursor location under the 2D Interactive Surface profile (visit TUIO for more information).
/tuio/2Dcur source application@address
/tuio/2Dcur alive s_id0 ... s_idN
/tuio/2Dcur set s_id0 x_pos y_pos x_vel y_vel m_accel
.
.
/tuio/2Dcur set s_idN x_pos y_pos x_vel y_vel m_accel
/tuio/2Dcur fseq f_id
The source message specifies our client application and address, the alive message contains a list of identification numbers for the active components, and the fseq message contains the frame sequence identification number. We will detect the source, alive, and fseq messages, but we won't do anything with them. For now we will concern ourselves with extracting the cursor location (x_pos and y_pos) from the set messages.
Below is the declaration of our server module, cTUIO. The integers and floats arrive in big endian format (we will convert these to little endian), so we've defined a union for each. Within the module we have properties for the socket descriptor, structures for the server and client addresses, a buffer to store the packet, a thread, a mutex, and, among others, a vector of cursor locations.
#ifndef TUIO_H
#define TUIO_H
#include <pthread.h>
#include <sys/types.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h>
#include <netdb.h>
#include <sys/fcntl.h>
#include "blob.h"
const int PORT = 3333;
typedef unsigned int uint32;
typedef unsigned char uchar8;
typedef float float32;
union u32 {
uint32 i;
uchar8 c[4];
};
union f32 {
float32 f;
uchar8 c[4];
};
class cTUIO {
private:
int sock, n;
socklen_t clientlen;
struct sockaddr_in server, client;
char buf[1024];
pthread_mutex_t mutex;
pthread_t thread;
bool active;
u32 timestamp, timestamp_fraction, element_size;
u32 id;
f32 x, y;
std::vector<point> points, ret_points;
protected:
public:
cTUIO();
~cTUIO();
static void* loop(void* obj);
void process();
void processMessage(const char* parameters, const char* arguments);
std::vector<point>& getPoints();
void lock();
void unlock();
};
#endif
In the definition below our constructor creates a non-blocking socket, binds the socket, initializes the mutex, and creates the thread. Our destructor waits for the thread to terminate, destroys the mutex, and closes the socket. The start routine, loop(), passed to pthread_create() is a while loop calling the objects process() method. In the process() method we attempt to read from the socket. If the length of the message read is positive, we process the packet.
We will be receiving an OSC Bundle, so our packets will begin with "#bundle_[time tag]", where the underscore is the null character and the time tag is a 64 bit fixed point number, a 32 bit number indicating the number of seconds since midnight on January 1, 1900 followed by a 32 bit number specifying the fractional parts of a second. Next, we parse through the 2D Cursor profile messages. Each message begins with a 32 bit integer indicating the message length followed by "/tuio/2Dcur_" (underscore representing the null character). Next, we have a comma followed by a string containing a list of types. We'll be searching for types s, i, and f, indicating strings, integers, and floats, respectively. This string is null-terminated, and the message parts are aligned on 32 bit boundaries, so the type string is padded with null characters up to the next boundary. Finally, we have our values. We pass the type string and the value string off to our processMessage() method. Our processMessage() method loops through our type string and pulls out our cursor location. When we have obtained both the x and y position from the set message we push the cursor location onto the points vector to be made available to our tracker modules.
Lastly, we have a getPoints() method for retrieving the cursor locations and lock() and unlock() methods for locking and unlocking the mutex.
#include "tuio.h"
cTUIO::cTUIO() {
sock = socket(AF_INET, SOCK_DGRAM, 0);
int flags = fcntl(sock, F_GETFL, 0);
fcntl(sock, F_SETFL, flags | O_NONBLOCK);
bzero(&server, sizeof(server));
server.sin_family = AF_INET;
server.sin_addr.s_addr = INADDR_ANY;
server.sin_port = htons(PORT);
bind(sock, (struct sockaddr *)&server, sizeof(server));
clientlen = sizeof(struct sockaddr_in);
pthread_mutex_init(&mutex, 0);
active = true;
pthread_create(&thread, 0, &cTUIO::loop, this);
}
cTUIO::~cTUIO() {
active = false;
pthread_join(thread, 0);
pthread_mutex_destroy(&mutex);
close(sock);
}
void* cTUIO::loop(void *obj) {
while (reinterpret_cast<cTUIO *>(obj)->active) reinterpret_cast<cTUIO *>(obj)->process();
}
enum { UNDEFINED, SOURCE, ALIVE, SET, FSEQ };
void cTUIO::processMessage(const char* parameters, const char* arguments) {
int type = UNDEFINED, k, fcount = 0;
for (int i = 0, j = 0; i < strlen(parameters); i++) {
switch (parameters[i]) {
case 's':
if (type == UNDEFINED) {
if (strcmp(&arguments[j], "source") == 0) { type = SOURCE; }
else if (strcmp(&arguments[j], "alive") == 0) { type = ALIVE; points.clear(); }
else if (strcmp(&arguments[j], "set") == 0) { type = SET; }
else if (strcmp(&arguments[j], "fseq") == 0) { type = FSEQ; }
k = strlen(&arguments[j]);
k = k % 4 == 0 ? k : k + 4 - (k % 4);
j += k;
} else {//if (type == SOURCE) {
// source stored in &arguments[j]
}
break;
case 'i':
if (type == ALIVE) {
// id
} else if (type == SET) {
// id
} else if (type == FSEQ) {
// frame id
}
j += 4;
break;
case 'f':
//if (type == SET) {
// xyXYm (position, veloction, motion acceleration)
if (fcount == 0) {
x.c[3] = arguments[j+0];
x.c[2] = arguments[j+1];
x.c[1] = arguments[j+2];
x.c[0] = arguments[j+3];
} else if (fcount == 1) {
y.c[3] = arguments[j+0];
y.c[2] = arguments[j+1];
y.c[1] = arguments[j+2];
y.c[0] = arguments[j+3];
points.push_back(point(x.f, y.f));
}
fcount++;
j += 4;
break;
}
}
}
void cTUIO::process() {
memset(buf, 0, 1024);
n = recvfrom(sock, buf, 1024, 0, (struct sockaddr *)&client, &clientlen);
if (n > 0) {
int k;
if (strcmp(&buf[0], "#bundle") == 0) {
timestamp.c[3] = buf[8]; timestamp.c[2] = buf[9]; timestamp.c[1] = buf[10]; timestamp.c[0] = buf[11];
timestamp_fraction.c[3] = buf[12]; timestamp_fraction.c[2] = buf[13]; timestamp_fraction.c[1] = buf[14]; timestamp_fraction.c[0] = buf[15];
for (int i = 16; i < n; i += 4) {
element_size.c[3] = buf[i]; element_size.c[2] = buf[i+1]; element_size.c[1] = buf[i+2]; element_size.c[0] = buf[i+3];
if (strcmp(&buf[i+4], "/tuio/2Dcur") == 0) {
k = i + 17 + strlen(&buf[i+17]) + 1;
k = k % 4 == 0 ? k : k + 4 - (k % 4);
processMessage(&buf[i+17], &buf[k]);
}
i+= element_size.i;
}
pthread_mutex_lock(&mutex);
ret_points = points;
pthread_mutex_unlock(&mutex);
}
}
}
std::vector<point>& cTUIO::getPoints() {
pthread_mutex_lock(&mutex);
return ret_points;
}
void cTUIO::lock() {
pthread_mutex_lock(&mutex);
}
void cTUIO::unlock() {
pthread_mutex_unlock(&mutex);
}
We have modified the trackBlobs() method for each of our trackers to accept an additional argument. The parameter list now includes a reference to a vector of points. These points are appended to the blobs vector after we have applied our calibration. The coordinates for the cursor locations retrieved from our packets belong to the interval \([0,1]\) and need to be mapped to screen space. We simply multiply these values by our screen dimensions in the trackBlobs() method.
Below is an example of calling our trackBlobs() method with the cursor locations. We finish by calling the unlock() method to unlock the mutex.
tracker.trackBlobs(filter.thresholdFrame(), false, tuio.getPoints()); tuio.unlock();
This is only a preliminary implementation of the TUIO protocol. There is information in the 2D Cursor profile we have dropped in addition to other profiles. Further information on the TUIO protocol can be found at TUIO.org.
Download this project: tuio.tar.bz2
