From Fortnite concerts to a coronation party in Frozen, there is a real need for high-quality dance animation in the gaming, animation and VFX industries. Creating these animations, however, remains a complex task. Moreover, the dance motions must match the style, rhythm and structure of the music.
This where a SIGGRAPH 2021 paper by Kang Chen, Zhipeng Tan, Jin Lei, Song-Hai Zhang, Yuan-Chen Guo, Weidong Zhang and Shi-Min Hu comes into play.
The first production-ready music-driven dance motion synthesis system
The authors present ChoreoMaster, a “music-driven dance motion synthesis system”. In other words : give ChoreoMaster a piece of music, and the software will automatically create dance moves to go along. This software has even already been used by Netease Games for several projects. According to the authors, this might even be “the first production-ready tool for this purpose in the industry”.
The following video will give you a good idea of what ChoreoMaster is capable of. Of course, you will want to watch it with the sound on, since the whole concept of ChoreoMaster is to match the style, rhythm and structure of the music.
How did they do it ?
As shown in the video, given a piece of music, ChoreoMaster is able to create dance moves to accompany the input song.
The authors explain that several breakthrough allowed them to achieve such results. In particular, the team presents a “cross-domain embedding framework, incorporating the introduced rules, to correctly and effectively characterize complex choreomusical relationships from limited available high-quality music/motion data, which successfully casts qualitative choreographic knowledge into computable metrics”. The framework is then used within ChoreoMaster. This tool relies on a database of music and dance motion sequences in order to generate dance motions. It should be highlighted that ChoreoMaster can be controlled by artists, as shown in the video. For example, you can replace unwanted motions without affecting other movements, control the dancing trajectory, or force a specific dance move to be used at a specific time/location.
How good is ChoreoMaster?
The authors compared their method to several existing dance generation methods. First, they did a quantitative evaluation, using several metrics to compare the methods. ChoreoMaster achieved the highest beat accuracy of all methods, according to the paper, as well as the highest diversity score. The authors also performed a user study with 35 participants, “10 of whom being choreographers or artists”. These people were asked to rate factors such as dance realism, music-to-dance style consistency, music-to-dance rhythm consistency. Once again, ChoreoMaster performed really well, well above the other dance generation methods and close to real dances.
As we explained earlier, ChoreoMaster already is production-ready. This tool has even been used for several projects by Netease Games. However, some limitations remain. For example, ChoreoMaster “cannot synthesize dance styles that are absent from the database, like ballet, waltzes, etc.”. As the authors explain, this could be corrected by expanding the dance database. It should also be highlighted that all dance motions in the current dataset are structured in four-beat bars, which means ChoreoMaster will produce incorrect results when dealing with three-beats-to-the-bar music. Lastly, the system “can’t handle the semantic relationships between dance motions and music lyrics” and human input is still required to ensure semantic consistency. The paper suggests that incorporating natural language processing modules could be an interesting improvement.
For more information, you can check out the paper using this link: ChoreoMaster : Choreography-Oriented Music-Driven Dance Synthesis. You will also find more results (dance videos) on this page.