1. No category

Learning How to Dance Using a Web 3D Platform
Nadia Magnenat-Thalmann1, Dimitrios Protopsaltou1, and Evangelia Kavakli2
1
MIRALab, University of Geneva
CILab, University of the Aegean
{thalmann, protopsaltou}@miralab.unige.ch, [email protected]
2
Abstract. In this paper we present the European project Open Dance and in
particular our contribution to the 3D simulation of folk dances and their presentation on the web. Our aim is to provide a learning framework for folk dances.
First we describe the conceptual and learning model that we apply, focusing on
the requirements of dance education. Then we digitize folk dances, originating
from several regions of Europe, using as a recording device an optical motion
capture system. We allow dance teachers and students to use our web3D platform and interact with the animated dancers, aiming to the better understanding
of dances. Students interact with the platform and observe how the virtual dance
teachers perform. The evaluation of the system shows that the increased usability of our approach enhances the learning process. Our long term objective is to
create an online dance learning community and allow dance teachers to create
their own dance lessons online.
Keywords: Web3D, dance, motion capture, online learning.
1
Introduction
Dance has been characterised as an exciting and vibrant art that can be used in the
educational setting to assist the growth of the student and to unify the physical, mental, and emotional aspects of the human being [1]. It is an art form characterised by
the use of the human body as a vehicle of expression [2]. Others [3] suggest that
dance is abstract in the sense that it is not an expression of an act but an expression of
the feeling of an act. It is in this specific way that dance differs from creative dramatics and mime.
An educational dance programme is designed not only to give students experiences
in expressive movement, but also to develop their ability to express themselves in
movement. This is done through progressive experiences, which develop kinaesthetic
and cognitive awareness of movement [3].
Dance education has experienced numerous changes in content and identity
through its history [4]. Until recently dance was part of the physical education programme in many countries. It is now recognised as an art form comparable to music,
drama, and visual arts and equally worthy of study. However, it is true that of all the
art forms, dance receives the least attention.
It is interesting to note that the dance event constitutes part of our “intangible heritage” [5], because it, unlike other art forms, leaves few physical records behind. There
H. Leung et al. (Eds.): ICWL 2007, LNCS 4823, pp. 1 – 12, 2008.
© Springer-Verlag Berlin Heidelberg 2008
2
N. Magnenat-Thalmann, D. Protopsaltou, and E. Kavakli
are several limitations in the recording systems used by dance companies, organisations, folk clubs and bodies that perceive preservation of dances as part of their mission. For example, archival information in the form of documents and books offer rich
information about dance as an agent of cultural meaning but fail to present the movement and stylistic aspects of dance genres. On the other hand, video addresses movement/audio aspects, but at the same time it lacks information about the wider context,
meaning and cultural significance of the dance event recorded. Despite these limitations, video is currently the most efficient way of preserving dance of any kind. An
example of systematic work in this direction is the LADD project [6] in the USA
(LADD Dance Heritage Coalition). Furthermore, available information on the Web,
relevant to dance, as part of the cultural education, is more or less unstructured (in the
form of collections of HTML pages, PDF documents etc.). As such, it is hard to access and require additional processing before teachers and/or students can use it.
The objective of the Open-Dance project [7],[8] is the development of a web based
platform that bypasses these problems and offers a structured set of learning tools
based on a goal-driven learning framework. In the next section we review previous
work that analysed the application of interactive technologies to dance. Following
that, we describe the learning context that we adapt in our approach. We focus the
main discussion on the description of our dance digitization pipeline using an optical
motion capture system. The last part describes the web-based 3D visualization module and we give results from user evaluation sessions. For the sake of clarity, in the
context of this paper as well as in the Open-Dance project we use the term “traditional
dance” instead of “folk” as defined in [9] i.e. “Dances that have evolved spontaneously from everyday activities and are informally passed from one generation to another”.
2 Previous Work
Previous work [9] presented the progression of teaching or learning dance over the
Internet. In [10] the authors suggested that in order to learn or teach dance over the
Internet, there are several components that are required to makeup the entire system.
In general such system involves the use of Networked Virtual Environment (NVE)
systems. The required components consist of i) a platform or Virtual Environment
that is required in order to represent an area to teach in ii) realistic looking virtual
humans that are required in order for the real participants to be represented correctly
in the virtual world and iii) a network, to link together the teacher and the students,
and also a tracking system to accurately track the limbs of all participants.
The Cyber Dance [11] performance was the first real attempt for interactive dance
using such a NVE System. This performance was shown many times and involved the
interaction between many real and virtual humans. It was performed as a combination
of real-time and autonomous virtual humans. VLNET [12] was the Virtual Environment System that was used. The performance was based on a dance sequence where
Virtual Humans (Fig. 1) interacted with the real humans on stage. Obviously due to
the complexity of having multiple Virtual and Real Humans it was not possible to
track all the real dancers on the stage. The actual scenario involved a choreographed
dance sequence from real dancers (Fig. 2).
Learning How to Dance Using a Web 3D Platform
Fig. 1. Virtual humans interacting with real
3
Fig. 2. Real dancer on stage
Similarly [13] explored a variety of interfaces between the physical and virtual
worlds. While taking the theme of “dance” and technology as a starting point, they
supported a wider range of conceptions of the physical body or bodies. The focus was
on the virtual space as a networked space that can function as a performance space, a
shared, creative, social and playful space. Through exploring interference and mapping processes, the participants worked towards realising the transformative possibilities inherent in emerging technologies. Pfinder [14] is a real-time computer vision
program, (i.e. “person finder”), a system for body tracking and interpretation of
movement of a single performer. The model-building process is driven by the distribution of colour on the person's body, with blobs being added to account for each differently coloured region. DanceSpace was an interactive stage that took full advantage of Pfinder's ability to track the dancer's motion in real time. Finally on the
commercial side, it is worth to mention Danceforms [15], 3D software devoted entirely on realistic character animation. The stand-alone animation tool offers an easy
solution for motion capture editing or custom motion generation.
3 Traditional Dance Context
As we mentioned earlier the Open-Dance project aims to promote the use of interactive technologies in dance education and in particular (a) the use of interactive multimedia technologies (e.g., video, 2D and 3D graphics, interactive images and text) for
representing information about traditional dances and (b) the use of the Internet as the
learning medium.
From a methodological perspective, the design and development of the OpenDance Learning Environment for assisting traditional dance education requires: (a) to
identify the abstract concepts that define traditional dance together with the appropriate hypermedia forms for describing/visualizing each concept; (b) to organize the content and develop the teaching curriculum; (c) to design the structure and interactivity
of the learning environment; (d) to implement the user interface; (e) to proceed in
content production; (f) to develop e-learning modules; (g) to incorporate the separate
modules in the Open-Dance e-learning environment and (h) to produce the accompanying material with explicit instructions for the users. In the following section we
4
N. Magnenat-Thalmann, D. Protopsaltou, and E. Kavakli
mainly focus our discussion in the description of the conceptual model that we
adopted for traditional dances. This model will be our basis for the optimal documentation (recording) of the dance events. Following that, we describe the learning model
based on that we evaluate our approach and validate our results.
3.1 Conceptual Model
The conceptual model of the Open-Dance learning environment is based on the dance
conceptualization framework that has been developed within the Open-Dance Consortium [7]. This was motivated by the need to face the concept of dance in a holistic approach without following the usual discrimination between movement and context
[16], which only provides a fragmented view of the dance experience.
In particular, the dance conceptual model consists of three types of dance concepts:
concepts that focus either on the movement components of the dance, or the dance’s
context or both. We refer to these three categories as: Dance Activity, Dance Tradition and Dance Event, respectively. Each of these three categories is further divided
into a number of sub-concepts and characterizes the entity of a Dance.
This conceptual model has been integrated in the Open-Dance Curriculum and the
Learning Environment. Each sub-concept of the dance conceptual model forms a
dance lesson in the Open-Dance curriculum, e.g. the subchapter of Dance Event consists of the following lessons for the music, the costumes and the roles of participants
[16]. This relation of dance concepts and dance lessons is shown in Fig. 3.
Fig. 3. Open-Dance Conceptual model
3.2 Learning Model
The Open-Dance learning model describes the educational philosophy adopted within
the project and emphasises the need to identify the most appropriate uses of technology to support the overall learning experience. It is based on a three-stage process
with the following steps (Fig. 4): i) the learners are asked to appreciate the new concepts they have to learn ii) specific activities drive them to fit new knowledge to their
Learning How to Dance Using a Web 3D Platform
5
previous experience and knowledge (construction) and iii) extended activities allow
them to reflect on the new concepts and issues [17]. Vital to this model are two learning approaches. The first approach is the goal–based approach [18] whereby the learning process is driven by the goal the learner aims to achieve (e.g., to present a certain
dance in the school journal); this goal directs the learner’s activities serving as a motivator for learning. The second is the learning-by-doing approach that focuses on acting rather than memorising facts and concepts. This is achieved by incorporating
interactive components in the form of questions and answers, quizzes, and the use of
extended activities.
Fig. 4. Learning model
4 Dance 3D Digitisation
For the digitization of the dances we applied motion capture in order to recreate
dances in three dimensions and represent them online in a 3D environment. We organized a dance recording session at MIRALab in the University of Geneva where we
invited various dance groups from Greece, UK and Bulgaria. In the table below
(Table 1) we provide the list of dances that we recorded, listing their particular performance characteristics in terms of interaction.
4.1 Motion Capture
Capturing motion is a long and difficult process, which requires a lot of heavy equipment and sophisticated software. There exists a plethora of technologies that enable
one to record a motion. Among these, two classes of systems are more widespread
than the others. The first one is optic based. An array of cameras (at least two and up
to more than 20) can capture the trajectory of markers that are placed on the various
locations that one wants to record (see Fig. 5). The cameras are calibrated so that their
position in space and their internal configuration (focal length) are known in advance.
Applying geometrical computation the system calculates the location of the markers
with a fairly good accuracy (below one millimeter, depending on the quality of the
calibration) and a high frame rate (up to 1 KHz).
6
N. Magnenat-Thalmann, D. Protopsaltou, and E. Kavakli
Table 1. Motion capture session
Country
Dance Name
Description
UK
Landlord fill the flowing bowl
Set of six
UK
Shepherd’s Hey
Pair clapping
UK
Pat-A-Cake
Couple dance
UK
Gay Gordons
Couple dance
UK
Highland Fling
Solo dance
Bulgaria
Rachenitza
Solo dance
Bulgaria
Gankino Horo
Group dance
Greece
Syrtos
Group dance
Fig. 5. Optical motion tracking
Fig. 6. Magnetic motion tracking
The second class of system is magnetic. A box emits an intense magnetic field
which is analyzed by sensors attached to the body to track thus providing the position
and orientation of each sensor (Fig. 6). Such system is more used for real-time applications due to its quick setup capabilities (no calibration is required for instance) and
its transportability. The drawbacks, however, are a poor accuracy (around a centimetre), a small range of action (you mustn’t get too far nor too close from the emitter), a
huge sensitiveness to the surrounding environment (no metal allowed around the capture zone) and a lower frame rate (less than 200 Hz). Once the data was acquired, it
presents itself under the form of a collection of trajectories of either points or rigid
bodies.
Learning How to Dance Using a Web 3D Platform
7
In the context of Open-Dance we applied optical motion tracking using the VICON
[19] system. We use trackers, strategically placed on each body. To calibrate the system, each body is calibrated before any motion is recorded. Due to the system performance, recording two bodies simultaneously is a difficult process if the dancers are
close to each other. In this situation, the precision of our VICON system is not accurate enough and the workstation confuses some markers. To manage this kind of
problem we made each couple dance three times. First, each dancer was recorded performing alone. Then we recorded the couples (Fig 7).
Fig. 7. Motion capture of couples dance
Needless to say that such data is quite useless if not linked to an object to animate
(a body) and cleaned from all the artefacts that always occur during a capture session
(occlusion of optical markers, perturbation of the magnetic field). The process of
cleaning up the data is called post-processing.
4.1.1 Post-processing
There exists some powerful pieces of software for post-processing the data that use a
lot of data estimation and smoothing techniques from the simplest one (e.g. linear interpolation) to the more complex (e.g. Vicon IQ uses a skeleton calibrated on the subject and error minimization for estimating it’s pose at every frame and therefore the
location of the markers), plus many others (Kalman filtering, a-priori knowledge
about rigid bodies, spline interpolation and of course hand work).
4.1.2 Retargeting
Quite a lot of research has been conducted on the issue of motion retargeting (i.e. apply
one motion to various skeletons) but so far it didn’t achieve to be integrated into production software. Constraints based approaches aim at obtaining a visually plausible motion
when the target model doesn’t quite match the captured data. It formulates the problem
of retargeting more or less as follows: the captured motion is the starting point for satisfying a set of constraints to be enforced (e.g. always at least one foot on the floor, no
foot skating,) and the data is then optimized for satisfying the constraints while minimizing the change in the motion. However, this approach correct only what it is asked
8
N. Magnenat-Thalmann, D. Protopsaltou, and E. Kavakli
for: if, say, the captured character was walking on a flat area and the animated one is
walking uphill, then the final animation will dangerously bend downward, but the
character will not fall. Further adaptation has to be done when such a case occurs, and
Physics based approaches tend to address this issue. Physics based methods also aim at
satisfying constraints without changing too much the original animation, but this time
the constraints that are enforced are physical ones (ensure the right balance of the character,) either by non-linear optimization or more fancy methods like close-form [20].
Other problems may arise during the retargeting process. For instance parts of the
body sometimes self-penetrate with others parts when the motion is mapped on another character [21] (this range of cases arise when e.g. the person that was captured
was quite skinny, and the animated character is fat) or the recorded motion doesn’t fit
into the surrounding virtual environment. These issues are often edited by hand due to
the large number of solutions that are available for one single case (e.g. when the
hand of an animated virtual character penetrate the wall of a virtual house, how
should he avoid that e.g. by bending his elbow or wrist, by moving himself etc.), and
the research often focused on finding ways to make the editing by hand more easy
[22] rather than doing it automatically.
4.1.3 Post processing in Open-Dance
The post-processing of motion sequences primarily involves the following two stages:
trajectory reconstruction and labelling of markers/trajectories. Once those two steps
have been completed, it is possible to visualize the technical skeleton (Fig. 8) obtained by drawing segments between the marker positions. From this information the
subject skeleton and its animation are derived.
Fig. 8. Technical skeleton
It is relatively easy to construct the subject skeleton (or its approximation) from the
markers. However, the problem becomes much more complex when considering the
body meshes and its deformation parameters (Fig. 9). Skin deformation (or skinning)
is based on three inter dependant ingredients. First the skeleton topology and geometry, second the mesh or surface of the body, and third the deformation’s own parameters (very often consisting of vertex-level ratios describing the scope of deformation
with relation to joints). Good skinning results are achieved by a careful design of
these three components, where positions of joints inside the body surface envelope
and attachments ratios are very important issues.
Learning How to Dance Using a Web 3D Platform
9
Fig. 9. Skeleton attachment
4.1.4 Music Synchronization
One of the main issues in this approach is to identify key frames where the element of
time is evident in the performance of the dancer. (e.g. position of feet). The synchronization of frames generated from the dancer’s motion with respect to the music
frame, in such a way that it enables comparison, adds the element of rhythm in dance
learning. Synchronizing the recording of the music with the motion sequence is a
challenging and crucial task, especially for the evaluation the dance learning experience. A manual technique involving the dancers to indicate the starting frame of the
audio and of the motion sequence can be used, which may later be evolved into an
automatic system.
4.2 Web 3D Environment
The final step is the development of the web3D viewer [23] that allows the observation and manipulation of the 3D dancer online. The 3D viewer was developed with
Adobe Director 8.5 Shockwave Studio and integrates several functionalities (e.g.,
start, stop, Zoom in / Zoom out, Focus, Change camera position, etc.) through the
Lingo API [24]. The user is able to watch a dancing model, choosing the point of
view and the zoom level, and finally control the speed of the 3D animation (Fig. 9).
Fig. 10. Open-dance web3D viewer
5 Evaluation
In this section we discuss our evaluation results [25] based on the learning model that
we introduced previously. In our evaluation group we included internal experts from
10
N. Magnenat-Thalmann, D. Protopsaltou, and E. Kavakli
Fig. 11. Evaluation results of the 3D animation
the project’s participating institutions, experts from the English Folk Dance & Song
Society, teachers and students from the project’s pilot schools and a high number of
high school teachers and students as well as members of traditional dance related organizations. The evaluation criteria addressed the following aspects: (a) Quality of
content (b) Presentation of content (c) Pedagogical quality (d) Use of the 3D animated
dancer (interactivity and usability aspects). We provide an overview of the results
(Fig. 11) obtained so far with relation to the 3D animated dances on the web.
The 3D animated dancer received very positive comments, regarding its effectiveness and usability. Most users found that it assisted them in learning the steps of the
dance. They also made interesting suggestions regarding extra functionality they
would like to be added. These included the ability to move the dancer in space; the
ability to trace the dancer’s movement; and the ability to use different costumes for
different dances.
− Does the 3D animated figure look good? Do you find it enjoyable? Yes: 72.13%
− Are there any additional functions you would like to include in the 3D animation?
Yes:36.07%
− Is it easy to use the 3D animated character? Yes: 83.61%
− Is the quality of the 3D animated character good? Yes: 77.05%
Open-Dance has shown that (a) the same conceptual model can be used to record
different European traditional dances (b) web3D can be used to create dance resources
for the web (c) there is a great interest from teachers in formal and informal educational settings that would like to use the Open-Dance platform and (d) there is a great
Learning How to Dance Using a Web 3D Platform
11
interest from dance experts to use the platform in order to document traditional
dances. Future projects will build upon the technological results and the experience
gained through Open-Dance in order to (a) extend the usability of the web-learning
environment (b) increase the teaching resources offered the (c) expand the number
and type of users of the web platform. To this end, we are currently working on a new
traditional dance e-learning platform that will enable users to be part of an online
community and create their own lessons.
Acknowledgements
The work presented in this paper has been supported by the Swiss State Secretariat for
Education and Research (SER) Socrates-Minerva project (225471-CP-1-2005-1-GRMIVERVA-M). The authors wish to acknowledge the Open Dance project partners
for their collaboration. Special thanks are due to Etienne Lyard, Clementine Lo, Vincent Mugeot and Nedjma Cadi for their contribution in the motion capture session.
References
[1] Reston, V.A.: Dance Directions: 1990 and Beyond. National Dance Association (1988)
[2] Overby, L.Y.: Status of dance in education [Electronic version]. (Report No. EDO -SP91-5). Washington, D.C. : Eric Clearinghouse on Teacher Education. (ERIC Document
Reproduction Service No. ED348368) (1992)
[3] Siedentop, D., Herkowitz, J., Rink, J.: Elementary physical education methods. PrenticeHall, Englewood Cliffs (1984)
[4] Bannon, F., Sanderson, P.: Experience every moment: aesthetically significant dance education. Research in Dance Education 1(1), 9–26 (2000)
[5] UNESCO: Recommendations on the Safeguarding of Traditional Culture and Folklore,
adopted by the General Conference at its twenty fifth session, Paris, France (retrieved 1004-01) (1989), http://www.unesco.org/culture/laws/paris/html_eng/
page1.htm
[6] LADD Dance Heritage Coalition: LADD Project: Overview of LADD, http://www.
danceheritage.org/ladd1.html
[7] Open-Dance (retrieved 15-06-2007), http://www.aegean.gr/culturaltec/
opendance/
[8] Kavakli, E., et al.: Traditional Dance and E-Learning: The WebDance Learning Environment. In: Paper presented to the International Conference on Theory and Applications of
Mathematics and Informatics, Thessaloniki, Greece (2004)
[9] Raftis, A.: Dance teaching, D.O.L.T, Athens (in Greek) (1993)
[10] Magnenat-Thalmann, N., Joslin, C.: Learning how to Dance on the Internet. In: Interface
Conference, Hamburg (October 2000)
[11] Magnenat-Thalmann, N.: Cyberdance. In: Proc. of Virtuality and Interactivity, Firenze,
Italia, pp. 72–73 (1999)
[12] Pandzic, I., et al.: VLNET: A Body-Centered Networked Virtual Environment. Presence:
Teleoperators and Virtual Environments 6(6), 676–686 (1997)
[13] Transdance (retrieved 15-06-2007), http://www.sdela.dds.nl/transdance/
report/index.html
[14] Wren, C., et al.: Pfinder: Real-time tracking of the human body. Photonics East, Bellingham, WA. SPIE 2615 (1995)
12
N. Magnenat-Thalmann, D. Protopsaltou, and E. Kavakli
[15] DANCEFORMS (retrieved 15-06-2007), http://www.charactermotion.com/
danceforms/
[16] Karkou, V., Sanderson, P.: Dance movement therapy (DMT) in the UK: issues of theory
and assessment. The Arts in Psychotherapy 28, 197–204 (2001)
[17] Ferreira, M., et al.: A multimedia Telematics Network for On-the-Job Training, Tutoring
and Assessment. In: Conference proceedings ICEE 1998: International Conference on
Engineering Education, Rio de Janeiro, Brazil (1998)
[18] Schank, R.C.: Goal-Based Scenarios (retrieved 15-06-2007) (1992),
http://cogprints.org/624/00/V11ANSEK.html
[19] Vicon (retrieved 15-06-2007), http://www.vicon.com
[20] Shin, H.J., Kovar, L., Gleicher, M.: Physical Touch-Up of Human Motions. Pacific
Graphics (2003)
[21] Jeong, K., Lee, S.: Motion adaptation with self-intersection avoidance. In: International
Workshop on Human Modeling and Animation, Korea, pp. 77–85 (2000)
[22] Boulic, R., et al.: Experimenting Prioritized IK for Motion Editing. Eurographics (2003)
[23] Web3D viewer (retrieved 15-06-2007), http://www.vdu.lt/dancer/
[24] Adobe Director (retrieved 15-06-2007),
http://www.adobe.com/support/director/
[25] Web-Dance Evaluation, Technical report, University of the Aegean (2005) (retrieved
15-06-2007),
http://www.aegean.gr/culturaltec/webdance/reports/W5_UoA_01.
pdf