Presentation Video: Link

Waiting inside the elevator until you get to your floor sometimes feels eternal. Usually there is no cellphone reception inside, that means no Whatsapp or Facebook (short Apocalypse for some people ). So what to do in the mean while?…..How about to to fall into a very familiar rhythm and to boost-up your mind with some drawing creativity? Well, here we present something to „rock and draw“.

Our main idea is to make the boring time inside an elevator somehow interesting, when people are going from one floor to another. People usually just stare at the roof or floor of the elevator and get bored (or claustrophobic ). But now you could draw and play music just by moving your hands/fingers and depending on the drawing pace, speed of the background music can change.

The beautiful thing about this project is that it could be implemented at any place where people need to burn out sometime while waiting for something to happen.

We took part on an inspiration card workshop, where we brought many domain cards and technology cards related to different areas of the university , to make them somehow more attractive. In this section we present some of the cards we considered during the ideation part of our project.

. .

. .

• Getting yourself entertained while waiting: The main goal of this project is to provide entertainment in a simple and intuitive way, while waiting at any given place where the system is installed.

• Draw and play music with hand/fingers movement: Drawing is probably one of the earliest things we learn when we are kids, and it is something we still enjoy even as grownups. The problem is that to do it, we need paper and color pencils, and unless you are a painter or a elementary school kid, you do not usually carry them around with you. Because of this, we wanted to design a system that allows you to draw in the air by using your hand and fingers. And the great thing is that it is suitable for all ages (we all are kids deep inside )

• Collaborative interaction: Drawing alone is kind of fun… but drawing with someone else is a blast!… except when they screw up your art … but even with that it is still fun, so because of that we wanted our design to be collaborative, so more than one person could used at the same time.

• Control velocity of music: We also wanted to add some background music but at the same time somehow we wanted the drawing to affect it, so we started first by using a very well known tune (Mario bros:-D) and speed up or down the rhythm according to drawing pace.

• Post-WIMP interaction (embodied interaction): Make use of our motor & bodily skills and spatial awareness. Embodiment has been purposed as the basis for a new foundational approach to HCI. It reflects both a physical presence in the world and a social embedding in a web of practices.

• Recognize hand-position and hand-gestures: Gesture recognition enables humans to communicate with the computer and interact naturally without any mechanical devices. Using the concept of gesture recognition, it is possible to point a finger at the computer screen so that the cursor will move accordingly.

• Audio feedback: Enhance the user experience by providing redundant information that helps him confirm the actions he is making.

• Instructions on-demand: Provide the user with information of how to use the system by simple hovering over the help button

As mentioned before, at the beginning we had many domains (all related to university areas) and technology cards. At last, we decided to choose the Elevator as our main domain.

• Elevator

It does not matter where you are, let it be waiting outside for the elevator to come, or being already inside while moving from one floor to the other. We focused on making this area an interesting place with the help of some creativity in form of colorful drawing and funny music.

Among all the technology cards, the one we considered to be the most useful on helping us achieve our goals was the one related to „Leap motion and tracking“. So in this section we provide a bit of information about this technology, and also about VVVV, the single software where we implemented the whole project.

• Leap motion hand tracking

On May 21, 2012 Leap Motion announced its first product, originally called The Leap. Airspace is the app store of the Leap motion, and there is where it sells the apps made by developers, having currently about 150 apps. ¹⁾

Leap motion is a small peripheral device which recognizes the hand and finger gestures. The leap motion is mainly used for navigating websites, pinch-to-zoom gestures on maps and 3D data visualization and for general controlling of desktop computers. It consists of two infrared cameras and three infrared LEDs. ²⁾

The leap motion controller is a small USB peripheral device which is designed to be placed on a physical desktop, facing upward. Using two monochromatic IR cameras and three infrared LEDs, the device observes a roughly hemispherical area, to a distance of about 1 meter. The LEDs generate a 3D pattern of dots of IR light and the cameras generate almost 300 frames per second of reflected data, which is then sent through a USB cable to the host computer, where it is analyzed by the leap motion controller software. Leap motion initially distributed thousands of units to developers who are interested in creating applications for the device. ³⁾

VVVV is a visual and data stream oriented, real-time programming language with physical interfces, that was developed around the year 2000 by the VVVV-group (Joreg, Max Wolf, Sebastian Gregor, Sebastian Oschatz). Their aim is to make the development of prototypes, artworks easily specially for those people who have so poor programming skills.

VVVV uses a data flow approach and a visual programming interface for rapid prototyping and developing.pplications written in vvvv are commonly called patches. Patches consist of a network of nodes. Patches can be created, edited and tested while they are running.⁴⁾

The control concept exists mainly out of three relevant hand/finger movements, which will be explained in detail in the following chapter.

Thanks to the leap motion technology, it is possible to recognize hand movements and gestures. We concentrate on the finger movement for the painting, the gesture of making a fist, as selection possibility without drawing and a swipe gesture to clean the screen from the actual drawing.

The explanation to the leap motion technology can be found in one of the previous chapters.

The finger represents the main control of the air drawing program. With the movement in the air, the drawing appears on the screnn. It needs a short time to get used to the control but through the fact, that it imitates drawing with a brush, the drawing goes smoothly. As additional functionality the thickness of the drawing line can be changed by moving the finger to or away from the screen.

When the user closes his palm and make a fist, the drawing funtction is turned off. A single dot will be displayed on screen which will be the output of the closed palm recognition and gives the user a feedback about the actual position. This creates the possibility for the user to make a selection of the different options, without coloring his path to the destination. Further it allows a more precise drawing by giving the possibility to create gaps.

If the user makes a fast swipe gesture, moving his flat hand from the bottom to the top, the picture on the screen will be deleted. This enables to start a fresh painting or makes the screen clear for the next user.

To make the drawing more attractive and interesting, color plays an important role. In the actual version the user can select between 6 different colors: black, blue, green, red, yellow and brown. The user is free to use them or stay with the default color, black.

As mentioned above the thickness of the drawing lines can be changed. The thickness is increasing proportionally according to distance. If the user moves away from the screen, the lines will be thin and will be thick gradually as he will move closure to it.

At the top left corner, a symbol can be found, that enables or disables the music funtionality. It is up the user if he likes to have a background music beeing played or not.

The instruction display can be activated by moving the hand or finger to the top right corner. It shows the main controls with a short explanation of the different possibilitys. All the instructions are represented in form of text and images.

The patch of the Software is quite complex, that’s why it is splitted into different parts. The following will give a brief explanation of the main parts of the patch and how they work. The complete patch is developed in a way that everything can be used collaboratively.

This part of the patch handles the recognition of the leap motion. It tracks several data like hand position, finger position, hand direction, finger direction, velocity etc. and it also recognizes different gestures of the hand. In this case we implemented swipe gestures, circle gestures and key tab gestures. Thanks to all this gestures we can get several data like the speed of the gesture, start-/end-point etc., which is depending on the gesture. The most important gesture in the patch is the recognition of the swipe gesture and it´s speed. It decides if the screen will be deleted or not. The other gestures are also implemented and could be used for future functionality.

This part represents the center of the whole node. The background board and the drawing is implemented in it, which contains all the necessary functionality to draw on the screen. Originally it was made for the mouse input, but we modified it to react on the finger position data of the leap motion. Several translations made it possible. It has a fixed 2D coordinate system, which allows to position the input always on the same coordinates. All that is grouped together and ends up in a renderer. Additionally a performance meter for the GPU is implemented which is turned off by default.

This part of the patch is enabling and positioning the color boxes. Every box is colored and has the according text that is represented by the box. The first layer shows the white border around every box. It is made by the belonging quad node. Second layer is the colored quad. The third layer is the text on the square and groups everything together, with the boxes as result. The question mark box follows the same pattern.

In this sub sequence of the patch we can track the cursors position which is relevant for our color boxes, the music on/off button and the instruction display at the top. Every box has a specified area in the 2D coordinate system. If the cursor (the hand of the user) enters the specified area it will trigger the pin input, that leads to the desired reaction. The possible reactions can be either a color change, showing the instruction screen or turning on/off the music. By entering in one of this areas, an audio feedback is played, which allows the user to recognize the change. The tracking part ends up as well in the group node, followed by the renderer.

This part shows how the background music is implemented. The WavePlayer-node plays the music according to several settings. Music is played only when the audio button is turned on and the leap motion is in use. The music speed is connected via several calculations to the velocity of the tracking. This enables the effect that a faster moving of the hand will fasten the music as well.

This simple construct of the ScreenShot- and the Writer node is also included in the patch. By triggering the ‘Save’ pin of the Write-node it takes a screenshot of the current screen and saves it in a desired place. This part is actually not in use but might be interesting for future developing. For example for the functionality to safe your pictures and print them.

Last but not least, the patch itself with the included audio files: Air drawing patch

In this section we provide some ideas for future improvement of the project.

• Erase specific sections: At the present we can't erase a specific area of drawing. Only complete deletion is possible. Adding the option to just erase small areas would be a must for future implementations.

• Saving the art: A nice add-on would be to have the option to save your work, so you can continue it on a different occasion. Also to implement the option for exporting it in to a USB stick.

• More colors: Drawing is more interesting if you have more colors to use. At the moment we have only six. Including a full palette of colors can boost up the visual experience.

• Enhance collaboration with more leap motion devices: Expanding the collaboration by adding more Leap motions would aloud more people to interact with the system at the same time.

• Improve visual interface: Enhance the user experience by making the interface more visual appealing.

• Here you can watch a video of our implementation. http://youtu.be/XdiMgqm7seY
• Here you can find the slides used in our presentation. air_drawing-slides.pdf