1. No category

Advanced Mechatronics: Propeller Project
Hands Free Piano using
Leap Motion to Detect
Hand Gestures
Anthony Brill, Jonghyun Bae, Matthew Moorhead
Hands Free Piano Introduction
● Inspired by the multi-core functionality of the Propeller BOE, we designed a
hands free piano controlled by gestures through Leap Motion
● For this project, the Propeller has two functions: receive data, and play
notes
○ Cog 0: receives serial data from the arduino
○ Cog 1-6: play specific tones for each gesture
● The arduino’s main functions: receive data, and send data
○ Receive data: Websocket via node.js
○ Send data: Software serial communication
● The Leap Motion’s functions: recognize gestures, and send data
○ Recognize gestures: Identify taps from specific fingers and hands
○ Send data: Websocket via node.js
Leap Motion
● Leap Motion: computer hardware sensor device that recognizes hand and
finger motion as inputs, yet requires no physical contact
● The device uses two cameras to track infrared light (850 nanometers)
projected onto the users hands from three infrared LED’s
● The interaction space is approximately 8 ft3,and is limited to roughly 2 ft
above the device
○ The height limitation is restricted by the LED light
propagation, since it is difficult to infer hand
position further than 2 ft away
● Once an object is tracked, a gray scale image is
sent to the computer for image processing
● Leap Motion Service (software) is then used to
process the raw sensor data and a tracking
algorithm is used to infer the position of objects
Leap Motion - Gestures
● There are 4 types of gestures the Leap Motion Service can recognize
○
○
○
○
Circle
Swipe
Tap
Screen Tap
Communication
● Leap Motion to Arduino: Websocket via node.js
○ Leap Motion has a built-in capability to export the data to a predesignated websocket
○ Web-socket is opened with node.js and establishes the connection
○ The data received from websocket is processed utilizing ‘johnny-five’
library and it will send commands to set Arduino Pin 2-7 high when
each finger performs a “tap”
Node.js Code
● Set up web socket to connect to Arduino
○ Set up Johnny-five library
○ Set up board object
● Open web socket
○ Enable gestures
○ Run frames in background
Node.js Code
● Tell arduino board what to do
Communication
● Arduino to Propeller: Software Serial
○ In the Arduino IDE we created a serial port on pins 10(RX) and 11(TX)
○ Data from the Leap Motion is masked from PORTD digital pins 0-7 of
the Arduino and sent to the Propeller
Arduino
...
Propeller
Data Processing
● Propeller
○ The propeller receives the data on pins 9(RX) and 8(TX)
■ The arduino sends the state of each gesture in the form of a byte,
in binary representation
■ Each cog is continuously monitoring the state of each bit
representing specific fingers
● Example: Cog 1 is monitoring bit 1; Cog 2 is monitoring bit 2
Problems
● Communication
○ Sending data from Leap Motion to Propeller
■ Node.js and Arduino
■ Processing
○ Pause times
○ Time pins being held high
● Hardware
○ Leap motion has some issues recognizing gestures