The term Tracking is used in a lot of contexts. Since the tracking techniques that are introduced in the following chapters cover the tracking of motion in realtime one could define the term tracking in this context as:

the process of recording the movement of objects or people. ¹⁾

In this Seminar there are three different tracking techniques covered: Markerless Tracking, Marker Tracking and Gesture Tracking

Through popular media like movies such as The Lord of the Rings or games using the Kinect or Leap Motion these particular tracking techniques became known to a broader audience.

For almost all tracking techniques you need a camera. Some even require further hardware, such as a Kinect or a Leap Motion. In addition to it it is recommended that one uses powerful hardware because many tracking applications are using hardware heavy image processing algorithms to process the tracked environment.

This VVVV examples is made under the assumption that the reader already is familiar with the basic concepts of VVVV. If this is not the case please refer to VVVV - Advanced Techniques for an introduction to the concepts of VVVV.

Setting up your camera is simple. Add a Node to your Patch. This Node gets a video signal from a directshow compatible device.

Enabled: Enables or disables your camera
Property Page: Allows you to open the property page of the camera driver
Framerate: Sets the Framerate of the camera. For a smooth viewing experience the framerate should be higher than 25 fps.
Width, Height: Sets the Resolution of the camera. The higher the resolution, the sharper the picture.
Video Format: Sets the output format of the video. (eg. MJPG, YUY2)
Analog Video Standard, Analog Video Input: Settings for analog input.
Driver: Lets you choose the camera to be used.

Video: The video output of the camera.
Preview: Preview video output of the camera.
Stream Capabilities: Outputs every possible setting for the camera. This can be used to correctly define the input parameters of the Node.
Actual StreamFormat: If the inputsettings of the Node are not suitable for the camera it automatically uses another output format which is shown here.

Markerless Tracking is the application of tracking techniques that work without markers.
Therefore the article will track three different kinds of Markerless Tracking.

Colortracking is used to track one or multiple objects with specific colors in images. By doing so one can determine x and y position, as well as the size and angle of the objects.
Colortracking may either be used to track exact colorshades or track colors that are in a given scope around a colorshade.

One of the most popular Colortracking applications in the recent past is the kickstarter project Pixy²⁾. Click on the image to start the video:

For two VVVV patches see examples

This VVVV examples is made under the assumption that the reader already is familiar with the basic concepts of VVVV. If this is not the case please refer to VVVV - Advanced Techniques for an introduction to the concepts of VVVV.

The shown patch is a very basic example with no functionality except from tracking an object with red color and putting it's x and y position, width and height and angle into I/O Nodes. Yet, this patch contains all necessary functions to create a bigger project using colortracking (for an example patch, see examples )
Of great importance, as mentioned earlier, are good camera settings. For colortracking of specific colors one should use a distinquishable background (if you don't intend to track a white color than you can use a white background).
What one might notice while performing colortracking is a very heavy workload. This is due to the image processing algorithm that is used for colortracking.
Try scaling down either the framerate or resolution to get a smoother picture.
The camera output is then connected to the Video Input attribute of the Node.

This Node requires careful calibration on order to function properly. In the following we will address all input as well as output parameters. For a better understanding we will not follow the order of the parameters as they appear on the Node itself:

Colors to Track: The colors that are to be tracked. Can be one color (as in this case) or a Spread of colors.
TC Tolerance: This is the most important parameter. One defines the tolerance for the tracking colors as HSV value. For example, hue=1 would set a maximum color tolerance (all hues are selected then). One has to carefully calibrate this parameter for every new environment in order to get good results.
Show SearchPos: If this is set to 0,…,maxColor-1 shows a quad around the found object in the given color. If it's set to -1 shows quads for all objects that are found. This is very helpful for calibrating the TC Tolerance.
Set to -2 to deactivate it.
Show ThreshImage: Another helpful attribute for correctly calibrating the colors. This shows the image thresholded with the tracked colors and TC Tolerance. If TC Tolerance is correctly calibrated it should show only those object which are one of the tracking colors. The rest of the image is white. Set to -2 to turn off.
Noise Reduction: Reduces noise in the picture. As mentioned earlier this should be kept to a minimum because of the workload Noise Reduction produces.
ROI X,Y,Width, Height: Sets the room of interest. If one is only wants colortracking to process a specific area (room) of the image one can set the ROI parameters accordingly.
Scaled Values: Boolean attribute that can either be set to 1 or 0. Choose 1 if you want the ColorTracker to put out the object position values in an image coordinate system or 0 if you want the output to be in vvvv-specific coordinates.
Init Tracker: Bang to initalise tracking. The algorithm then starts anew, searching the whole image for an object and narrowing down the search frame later on.
Area Threshold: Needs either a value or Spread as input. If a tracked object gets too small, thus dropping under the given threshold, the tracker is re-initalised. Set to 0 to keep Tracker from re-initalising.
Enable Colortracking: Turn Colortracking On/Off.

Video Output: Has to be connected to a Renderer, otherwise the ColorTracker won't work! Also helpful to see the thresholded image for calibration purposes.
X,Y Position: Returns either a value(one color) or a Spread of values(multiple colors) of the x and y position of the objects that are tracked.
Width,Height: Returns either a value(one color) or a Spread of values(multiple colors) of the width and height of the objects that are tracked.
Angle: Returns either a value(one color) or a Spread of values(multiple colors) of the angle of the objects.
IsTracked: Returns either a value(one color) or a Spread of values(multiple colors). Is set to 1 when an object with a specific color is found. Otherwise 0.

Here you can see a well performing calibration for the colors red and green.
I should add that strangely the Show ThreshImage function seems to only work for red objects as only the red object is actually being displayed, even though the green object is also tracked correctly.

Facetracking is a technique to detect human faces in digital images (or videos). In simple facetracking applications the aim is to get the position and size of the face(s) in the digital image. In advanced facetracking applications it is also possible to get the age and gender of the tracked person or to distinguish one person from another person.

(Smartphone) Camera

Social Networks, Picture Libraries …

Advertisement

Amscreen is a system that use facetracking to get the gender and many more information about the person who stands in front of the display and shows fitting advertisements to the persons appearance.

Photo Booth

Facetracking is in VVVV realized by the node DetectObject. The node can track things depending on a trainings file. In this example we use the trainings file “haarcascade_frontalface_alt.xml” for facetracking.

First the DetectObject node needs a video input on the first pin. That could be a camera or a video. In this example we use a camera. The subpatch Camera should be configured by right clicking on it. We choose the camera we want to use and the video format MJPG. If we set the resolution and framerate to high the computational power needed will be very high. It is recommended to use these settings.

On the second pin of DetectObject we can enable a tracker that makes a red rectangle around the tracked face. The third pin is for the trainings file. On the sixth pin we set the min size of the object (or face). It is recommended to use 100 as min size. On the last input pin we can enable the tracking. If VVVV crashes at this point try to open VVVV as admin.

At this point the DetectObject node gets the camera input and track the face(s). To see results we must use the output pins. On the first output pin we get the camera input with red rectangles around the faces if the second input pin is enabled. On the other output pins we get the position and size of the face in the camera input. On the last pin the object id returns the number of the actually tracked object. If you go in front of the camera and then leave the camera and then go in front of the camera the object id will be +1 because the node doesn’t know that you are the same person. You’re just the second object who was tracked.

The rest of the patch should be clear. The video output will be modified (VideoTexture, Quad) to show it in a DirectX Renderer. The position and size will be used to make a white circle around the face. The first simple facetracking application is written.

simple_facetracking.zip

Blobtracking is a technique to detect regions (Blobs) in digital images that differ in properties (for example brightness or color) compared to the surroundings. This technique can be used for markertracking or multitouch systems. For further information about markertracking visit the Wiki page of Group 5b). In the following article we will describe how to use blobtracking for multitouch systems.

Mutlitouch is a technique who can recognize two or more touch inputs (for example fingers) at once. It is a very natural way of interacting with devices because the user just needs his hands or fingers. The combination of multitouch and output device called multitouchscreens is a very direct way of interacting because input and output device are the same. Today multitouch is often used. Examples for multitouch(screens) are smartphones, tablets, laptops (trackpads) and ticketing machines.

In the further article we want to build a multitouch trackpad with VVVV, the tracking software Community Core Vision (CCV) and the TUIO protocol.

CCV is a free tracking software who searches in a camera picture for fingers, objects or tracking markers and returns position, size and rotation of the objects (via TUIO). You can download it on http://ccv.nuigroup.com/

The TUIO protocol is special developed for multitouch. It transport multitouch values from touch input devices or tracking software to other software or devices. Because it is rampant a lot of programs have an interface that supports TUIO.

In this trackpad example we will use CCV to track the fingers and then transport the multitouch values via TUIO to VVVV. Instead of bulding a big trackpad you can also use TUIOpad or TUIOdroid to transport the multitouch inputs of the smartphone (via TUIO) to VVVV.

1. Cut the top of a proper box

2. Build a stand for a Webcam so that the webcam looks at top.

3. Tape the camera stand at the inside bottom of the box so that the camera can see the whole top hole of the box but not too much of the box (You can use a software that shows the camera output to find the best place for the camera). Make a hole in a corner for the cable.

4. Make a glass plate at the top of the box and lay a white paper on it (You can also use any other glass alike thing you can find. I have used a glass filing)

The hardware is ready. Now we have to calibrate our trackpad with CCV.

1. Open CCV (Download: http://ccv.nuigroup.com/)

2. Change the source image to „Use camera“

3. Activate Inverse at „Tracked Image“

4. Click on „Remove BG“ while no finger is on the trackpad and try to calibrate the picture so that the program just track you're fingers. You can also change you're Camera Settings on the right side for better result. It is also important that the room is not to bright or to dark. Our calibration settings:

5. If everything is working fine click on „Save Settings“ and close the program.

6. Open the „config.xml“ (inside the data folder) and change <LOCALHOST>192.168.178.100</LOCALHOST> to your local IP of your computer to configure the TUIO protocol.

7. Open CCV and enable „TUIO UDP“

Now CCV sends the multitouch values to your own computer. We can use these values in VVVV with the following Patch.

multitouch_fink.zip

The Patch decodes the TUIO Values by the TUIODecoder and render it as circles on a DirectX Renderer. We have added a Cursor so that we can use our trackpad as a „real“ trackpad. Now the trackpad is finished.

With CCV we can also build a whole Multitouch table. The following video shows how you can build it with a projector:

https://www.youtube.com/watch?v=sJU8sBt7eC8

photo_booth.zip

markerless_tracking.pdf

• Prototyping Interfaces: Interaktives Skizzieren Mit VVVV Jan Barth, Roman Stefan Grasy, Martin Lukas, Jochen Leinberger, Markus Lorenz Schilling (2013) Schmidt Hermann Verlag
• Blob Detection http://en.wikipedia.org/wiki/Blob_detection (13.06.2014, 8:45)
• http://www.koewe.de/uploads/tx_cbshops/fotoautomat-bild.jpg picture of the photo booth (12.06.2014, 16:15)
• http://www.amscreen.eu/optimeyes-real-time-insight-for-digital-media-advertising amscreen example picture (13.06.2014, 17:30)
• http://www.youtube.com/watch?v=sJU8sBt7eC8 multitouch table video (19.06.2014, 18:57)
• http://nxeassets.xbox.com/shaxam/0201/55/44/5544fbc6-b46e-4fe8-832e-c29f0a7808b4.JPG?v=1#hero-KinectGames.JPG (22.06.2014, 10:24)