“Feel-Good Game Sound” for the Game Music Composer

How can we define “feel-good game sound”? That’s the question that sound designer Joonas Turner attempted to answer with his recent GDC Europe talk entitled, “Oh My! That Sound Made the Game Feel Better!”  Joonas’ talk was a part of the Independent Games Summit portion of GDC Europe, which took place in Cologne Germany on Monday August 3rd 2015.

While much of Joonas’ talk focused on issues that would chiefly concern sound designers, there were several interesting points for game composers to consider.  I’ll be exploring those ideas in this blog.

Joonas is a video game sound designer and voice actor working within the E-Studio professional recording studio in Helsinki, Finland.  His game credits include Angry Birds Transformers, Broforce, and Nuclear Throne.  After briefly introducing himself, Joonas launched into his talk about creating an aural environment that “feels good” and also makes the game “feel good” to the player. He starts by identifying an important consideration that should guide our efforts right from the start.

Consider design first

JoonasTurner

Joonas Turner, sound designer at E-Studio.

In his talk, Joonas urges us to first consider the overall atmosphere of the game and the main focus of the player.  Ideally, the player should be able to concentrate on gameplay to the exclusion of any distractions.  The sound of a game should complement the gameplay and, if possible, deliver as much information to the player as possible.  If done perfectly, a player should be able to avoid consulting the graphical user interface in favor of the sonic cues that are delivering the same information.  In this way, the player gets to keep attention completely pinned on the playing field, staying on top of the action at hand.

Clearly, sound effects are designed to serve this purpose, and Joonas discusses a strategy for maximizing the utility of sound effects as conveyors of information… but can music also serve this purpose?  Can music deliver similar information to the player?  I think that music can do this in various ways, by using shifts in mood, or carefully-composed stingers, or other interactive techniques.  By way of these methods, music can let the player know when their health is deteriorating, or when they’re out of ammo.  Music can signal the appearance of new enemies or the successful completion of objectives.  In fact, I think that music can be as informative as sound design.

Music, sound design and voice-over: perfect together

As his GDC Europe talk proceeds, Joonas reminds us to think about how the music, sound design and voice-over will fit together within the overall frequency spectrum.  It’s important to make sure that these elements will complement each other, with frequency ranges that spread evenly across the spectrum, rather than piling up together at the low or high end.  With this in mind, Joonas suggests that the sound designer and composer should be brought together as early as possible to agree on a strategy for how these sonic elements will fit together in the game.

puzzle

(Here’s where Joonas brought up the first of two controversial ideas he presented during his talk.  While I’m not sure I agree with these ideas, I think the viewpoints he expresses are probably shared amongst other sound designers in the game industry, and therefore could use some more open discussion in the game audio community.)

While composers for video games always want to create the best and most awesome music for their projects, Joonas believes that this desire is not always conducive to a good final result.  He suggests that the soundtrack albums for video games are often more exciting and musically pleasing than the actual music from the game.  With this in mind, Joonas thinks that composers should save their best efforts for the soundtrack, while structuring the actual in-game music to be simpler and less aesthetically interesting.  In this way, the music can fit more comfortably into the overall aural design.

Your sonic brand

At this point in his presentation, Joonas urges the attendees to find aural styles that will be unique to their games.  He tells the audience to avoid using a tired sonic signature in every game, such as the famous brassy “bwah” tone that became pervasively popular after its use in the movie Inception.  If you are wondering what that sounds like, just hit the button below (courtesy of web developer Dave Pedu).

In 2012, Gregory Porter (an avid movie lover and creator of YouTube videos about the movies) created a fun video illustrating just how pervasive the infamous Inception “bwah” had actually become:

In my book, A Composer’s Guide to Game Music, I discuss the concept of creating a unique sonic identity for game in the chapter about the “Roles and Functions of Music in Games.”  In the book, I call this idea “sonic branding”(Chapter 6, page 112), wherein the composer writes such a distinctive musical motif or creates such a memorable musical atmosphere that the score becomes a part of the game’s brand.

Be Consistent

When recording music or sound design for a project, Joonas tells us that it’s important to remain consistent with our gear choices.  If a certain microphone has been used for a certain group of character voices, then that microphone should continue to be used for that purpose across the whole project.  Likewise, the same digital signal processing applications or hardware (compression, limiting, saturation, etc) should be used across the entire game, so that the aural texture remains consistent.  Carrying Joonas’ idea into the world of game music, we would find ourselves sticking with the same instrument and vocal microphones, and favoring the same reverb and signal processing settings throughout the musical score for a game.  This would ensure that the music maintained a unified texture and quality from the beginning of the game to the end.

Shorter is better

short-hand

In his talk, Joonas shares his personal experience with sound effects designed to indicate a successful action – a button press that causes something to happen.  Joonas tells us that for these sounds, shorter is definitely better.  The most successful sounds feature a quick, crisp entrance followed by a swift release. A short sound designed in this way will be satisfying to trigger, and won’t become tiresome after countless repetitions.

For the composer, the closest analogy to this sort of sound effect is the musical stinger designed to be triggered when the player performs a certain action.  In order to adhere to Joonas’ philosophy, we’d compose these stingers to have assertive entrances and quick resolves, so that they would be fun for the player even when repeated many times.

To clip or not to clip…

(This is the second of the two controversial ideas Joonas presented in his talk. Again, while I don’t necessarily agree with this, I think it’s an idea that hasn’t been expressed often and may need further discussion.)

VU-Meter

A volume unit (VU) meter registering some high audio levels.

The common wisdom amongst audio engineers is to avoid overloading the mix.  Such overloads can produce clipping and create distortion, which deteriorates the overall sound quality of the game.  However, Joonas suggests that for intense moments during gameplay, some clipping and distortion may actually enhance the sensation of anxiety and frenetic energy that such moments seek to elicit.  According to Joonas, this enhancement can actually be a desirable outcome, and the sound designer should therefore not be afraid of such overloads and clipping during intense moments in a game.

How would this idea relate to music?  Well, we’ve probably all heard examples of successful pop music that embraces sonic overload.  Lead vocalists sometimes scream into microphones to produce overloads, or a wailing guitar riff may be recorded with lots of overload artifacts.  As a deliberate effect placed carefully for the sake of drama, such brief moments of overload can add edginess to contemporary musical genres.  However, we’ve all likely heard other examples of overloads that seem more the product of high decibel levels rather than any deliberate processing. It’s important to differentiate a deliberate effect from an accidental one.  In music at least, we always want to control the final outcome of the mix, including the presence or absence of overload distortion.

Conclusion

Joonas wound up his talk by urging attendees to always give priority to the elements in the sound mix that are most important.  That would be a good guiding principle for music mixing as well.  Joonas is an interesting thinker in the area of game sound design.  He can be followed at his Twitter account, @KissaKolme.  Please feel free to comment below about anything you’ve read in this blog, and let me know how you feel about the ideas we’ve discussed.  I’d love to read your thoughts!

border-159926_640_white

Studio1_GreenWinifred Phillips is an award-winning game music composer.  Her most well-known projects include such famous and popular games as Assassin’s Creed Liberation, God of War, the LittleBigPlanet franchise, and many others.  She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press.  Follow her on Twitter @winphillips.

A Composer’s Guide to Game Music, now in Japanese!

 

A Composer's Guide to Game Music by Winifred Phillips, now on sale in Japanese! Published by O'Reilly Japan.

A Composer’s Guide to Game Music by Winifred Phillips, now on sale in Japanese!  Published by O’Reilly Japan.

I’m excited to share that my book, A Composer’s Guide to Game Music, was released today in Japan in its newly-published Japanese-language edition!  O’Reilly Japan has published the Japanese softcover of my book in Japan under the title, “Game Sound Production Guide: Composer Techniques for Interactive Music.”

This is the Japanese cover of the book. In Japanese, A Composer's Guide to Game Music is titled "Game sound production guide - composer techniques for interactive music," by Winifred Phillips.

Side-by-side, these are the covers of the two editions of the book. In Japanese, A Composer’s Guide to Game Music is titled “Game sound production guide – composer techniques for interactive music,” by Winifred Phillips.

I’m very excited that the Japanese language edition of my book has already hit #1 on the “Most Wished For” list on Amazon Japan!

The Amazon Japan "Most Wished For" list.

The “Most Wished For” list on Amazon.co.jp.

Coincidentally, the English-language version of A Composer’s Guide to Game Music is now #1 on the Kindle Top Rated list, too!

The Kindle "Top Rated" list on Amazon.com.

The Kindle “Top Rated” list on Amazon.com.

O’Reilly Japan is located in Tokyo, and is dedicated to translating books about technological innovation for Japanese readers.  They are a division of O’Reilly Media, a California publishing company that acts as “a chronicler and catalyst of leading-edge development, homing in on the technology trends that really matter and galvanizing their adoption by amplifying “faint signals” from the alpha geeks who are creating the future.  O’Reilly publishes definitive books on computer technologies for developers, administrators, and users. Bestselling series include the legendary “animal books,” Missing Manuals, Hacks, and Head First.”

oreilly

From what I’ve gathered, my book – A Composer’s Guide to Game Music – is the first English language book about game music to be translated into Japanese and sold in Japan.  There are a few other books available in Japan on the subject – but they were all originally written in Japanese.  These include a book exploring game sound by the audio hardware designer and sound developer Shiomi Toshiyukia text on creating sound for games with the CRI ADX2 middleware by Uchida Tomoya, and a book on producing game music and sound design by the artist “polymoog” of the dance music duo ELEKETL (pictured below, from left to right).

books-ama-jp

I’m tremendously excited about the Japanese edition of my book, and my excitement comes in large part from the venerable tradition of outstanding music in Japanese games.  From the most celebrated classic scores of such top game composers as Koji Kondo (Super Mario Bros.) and Nobuo Uematsu (Final Fantasy), to the excellent modern scores of such popular composers as Masato Kouda (Monster Hunter) and Yoko Shimomura (Kingdom Hearts), Japanese video game composers have set the creative bar very high.  I’m incredibly honored that my book will be read by both established and aspiring game composers in Japan!  I hope they’ll find some helpful information in my book, and I’m excited to contribute to the ongoing conversation about game music in the Japanese development community.

I’ve always loved Japanese game music.  In 2008, I participated in a compilation album in which successful game composers created cover versions of celebrated video game songs from classic games.  The album was called “Best of the Best: A Tribute to Game Music.”  I chose the music by Koji Kondo from Super Mario Bros., and recorded an a cappella vocal version.  It’s currently available for sale from the Sumthing Else Music Works record label, and can also be downloaded on iTunes.  You can hear the track on YouTube here:

If you’d like to learn more about the rich legacy of game music composition in Japan, you can watch an awesome free documentary series produced by the Red Bull Music Academy, entitled “Diggin’ in the Carts: A Documentary Series About Japanese Video Game Music.”  The series interviews famous game composers of Japan, which means that the interviews and narration are both in Japanese (with English subtitles).  Here’s an episode that focuses on modern accomplishments by Japanese game composers:

border-159926_640_white

Studio1_GreenWinifred Phillips is an award-winning game music composer.  Her most well-known projects include such famous and popular games as Assassin’s Creed Liberation, God of War, the LittleBigPlanet franchise, and many others.  She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press.  Follow her on Twitter @winphillips.

“Sound is Magic” – Insights for the Game Music Composer

BBC-SoundNow&Next

From May 19th to the 20th of this year, the British Broadcasting Corporation’s Research and Development department presented a two-day conference to explore the future of immersive sound.  Called “Sound: Now and Next,” the event featured a distinguished speaker list that included accomplished audio engineers, producers, educators, inventors, researchers, musicians and composers.  The event offered a wealth of fascinating presentations on the future of audio, and I recommend visiting the site and checking out the awesome video resources from the event, which include complete session videos made freely available for streaming from the site.

For game composers and sound designers, one of the best sessions was presented by Nick Ryan, an award-winning audio engineer/composer/audio consultant who is best known in the game industry for his sound design work on the Papa Sangre, Papa Sangre II and The Nightjar audio games for iOS.  These three games utilize binaural sound to immerse players in an audio-only interactive environment, which Nick Ryan calls “inhabitable audio.”

Nick’s presentation at the “Sound: Now and Next” conference was entitled “Sound is Magic.”  According to Nick, audio has a unique power to bring about an emotional and perceptual impact by virtue of the collaborative relationship between the sound source and the listener.  When a sound is separated from its original source (i.e. when it’s not possible to see the source that’s emitting the sound), listeners will instinctively use their imaginations to supply the nature of the sound’s origin.  This imaginative contribution on the part of listeners has the potential to draw them more fully into the experience.  “I profoundly believe that we are co-authors in everything that we listen to,” Nick tells us.

NickRyan

Nick Ryan, sound designer for Papa Sangre, Papa Sangre II and The Nightjar

Early in the presentation, Nick introduces us to his initial work in binaural / spatial audio by describing a project he produced in 2002 for BBC Radio 4. “The Dark House” was a popular interactive radio drama: a ghost story recorded on location in a large house.  The actors wore baseball caps with microphones embedded in the brims.  While the project was ostensibly a traditionally linear radio drama, it was structured so that the audience could decide from which character’s perspective the story would be told, and the audio mix would switch to the perspective of the character who had received the most votes.  In this way, the audio mix of the program changed drastically as the audience cast their votes during the broadcast.  The entire program is available for listening here:

Nick stresses that this project illustrates the power of adding interactivity to an audio experience.

Moving on to his work in video game development, Nick launches into a discussion of his work on Papa Sangre, a game set in a completely “non-sighted” realm of the afterlife, inhabited by vicious unseen monsters.  Sharing a few observations about gamers’ experiences in Papa Sangre, Nick points out that visually-impaired players would usually breeze through the game in an hour, whereas sighted players found it to be crushingly difficult.  Also, Nick describes a phenomenon whereby sounds associated with personal movement (such as footsteps) stimulated the motor cortex of the brain to be active, making listeners feel as though they were actually in motion. This motor cortex stimulation contributed to the immersive qualities of the gaming experience in Papa Sangre.  The effects of sound on the brain are tremendously fascinating, and I explored some of the effects of music on brain activity in my book, A Composer’s Guide to Game Music – so I was especially interested to hear more about that phenomenon in Nick’s talk.  To learn more about Nick’s work on Papa Sangre (and another audio-only game titled The Nightjar), check out this sound design mini masterclass that Nick gave for the British Academy of Film and Television Arts:

Continuing with his presentation for the BBC “Sound: Now and Next” conference, Nick described a collaboration with Volkswagen and the famous electronica duo known as Underworld to allow a car to essentially drive a piece of music, associating an interactive musical composition with the turning, braking, acceleration and de-acceleration of the vehicle.  While it isn’t a game-related project, it is fascinating when considered in terms of the interactive music possibilities that could be translated into gaming applications.  Here’s the final video result of “Volkswagen Golf GTI Play the Road.”

And here’s a behind-the-scenes video that explores the making of this interactive music system for driving:

Finally, Nick brings the entire concept of “Sound is Magic” to a culmination by describing his collaboration with John Matthias to create a four movement piece for string orchestra entitled “Cortical Songs.”  A computer simulates the way in which human neurons naturally behave, sending these signals to tiny flashing lights on the music stands of the string players.  The musicians respond to these flashes as they would respond to a conductor issuing cues – as though the simulated neural activity was leading the orchestra.  The magic of the human mind is now expressed through sound, expressing Nick’s concept of Sonification — the aural expression of silent phenomena.  Here is an excerpt from a performance of the composition:

The “Sound: Now and Next” conference offered an abundance of inspiring ideas from top practitioners in their fields, and I urge everyone to check out the site and see some of the other presentations that are available online.  Also, be sure to check out the complete presentation given by Nick Ryan — “Sound is Magic.”

 

border-159926_640_white

Studio1_GreenWinifred Phillips is an award-winning game music composer.  Her most well-known projects include such famous and popular games as Assassin’s Creed Liberation, God of War, the LittleBigPlanet franchise, and many others.  She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press.  Follow her on Twitter @winphillips.

Total War Battles: Kingdom

Total War Battles Kingdom - Winifred Phillips

Winifred Phillips won 2 Global Music Award Gold Medals for the music she composed for Total War Battles: Kingdom.

I’m happy to announce that one of my latest projects is Total War Battles: Kingdom, developed by Creative Assembly and published by Sega. I was happy to join the music composition team for this fantastic project! Along with my long-time music producer Winnie Waldron, I worked with Creative Assembly’s audio manager Richard Beddow to compose atmospheric medieval-inspired music for this awesome upcoming strategy game.

I’m also very happy to share that my music for Total War Battles: Kingdom has already been recognized with two Gold Medals from the Global Music Awards!  My music producer Winnie Waldron and I received a Gold Medal in the category of Game Music, and I received an additional Gold Medal in the category of Composition/Composer.

I won the two Global Music Awards Gold Medals for “Dark Ages” – a track I composed for Total War Battles: Kingdom.  I was hired by Creative Assembly to join a team of composers who each worked separately to compose their own tracks for the game. Each composer brought unique strengths to the project, and I was proud to work with my award-winning music producer Winnie Waldron to compose my own tracks for this terrific game!

GlobalMusicAwards
Two Global Music Award Gold Medals recognizing music composed by Winifred Phillips & produced by Winnie Waldron for the game Total War Battles: Kingdom.

 

Here is a YouTube video containing my award-winning music from Total War Battles: Kingdom:

Total War Battles: Kingdom is the latest game in the popular, multi-million-selling Total War franchise. Now in its 15th year, Total War is one of the most famous and critically-acclaimed series in gaming!  Here’s some info about the Total War franchise:

A drive for historical authenticity and superb gaming quality has helped establish the franchise as one of the most successful games of all time. The Total War franchise has won numerous awards, including two BAFTA Awards from the British Academy of Film and Television Arts, and an Ivor Novello Award. The video game franchise was also the basis of two television shows: Decisive Battles on the History Channel, and Time Commanders on the BBC. Alongside the core historical-based games, the Total War series has expanded to include the mobile title, Total War Battles: Kingdom.

total-war-battles-kingdom-open-beta-now-live_fkkg.1920

My music for Total War Battles: Kingdom was performed by a live ensemble comprised of some of the best and most accomplished musicians performing with historically authentic medieval instruments and techniques. The ensemble includes one of Europe’s top lute players, and members of this ensemble have graced the concert stages of such venues as the Royal Opera House, The Royal Festival Hall, and Kensington Palace, among many others. For Total War Battles: Kingdom, I combined evocative aural designs with a consort of period instruments and medieval modes. I used these techniques to transport listeners into the mysterious world of the Dark Ages. Here are the musicians who performed my music for Total War Battles: Kingdom:

Musicians:

  • Lute: Elizabeth Kenny
  • Recorders, Fife, Flute: Chloe Lochbaum
  • Vielle, Hurdy Gurdy: Sylvia Hallett
  • Hurdy Gurdy: Sue Eaton
  • Mandola: Andy Reynolds
  • Celtic Harp: Heather Wrighton
  • Cello: Richard Harwood
TWBK-Musicians

From left to right: Elizabeth Kenny (Lute), Richard Harwood (Cello), Sylvia Hallett (Hurdy Gurdy), and Heather Wrighton (Harp)

Here’s some more information about the game:

Creative Assembly’s Total War Battles: Kingdom is set during the chaotic turn of the 10th Century, as the world starts to emerge from the Dark Ages. Players will find themselves managing the needs of their own fiefdom and guarding against the machinations of neighboring kingdoms. Deception, spying and outright betrayal against enemies and friends alike will see the devious player rewarded. “We wanted to create a new way of playing Total War Battles. Whether that’s on the move or at work over lunch,” said Renaud Charpentier, Creative Assembly Digital Project Lead. “It shouldn’t matter where you want to play; we want to make it easy to come back again and again to your flourishing Kingdom. Then, we ramp up the complexity and challenge, adding more options to your Machiavellian schemes.”

Founded in 1987, Creative Assembly is one of the UK’s most successful and established game studios. Creator of the multi award-winning Total War strategy series, the studio has received numerous press, industry and consumer accolades, including BAFTAs and the Develop Industry Excellence awards.  Home to over 325 highly talented developers and counting, the studio continues to expand to cover a variety of triple-A console, PC and mobile projects.

border-159926_640_white

Studio1_GreenWinifred Phillips is an award-winning game music composer.  Her most well-known projects include such famous and popular games as Assassin’s Creed Liberation, God of War, the LittleBigPlanet franchise, and many others.  She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press.  Follow her on Twitter @winphillips.

Virtual Reality in the Uncanny Aural Valley

headphones-buried

Most visual artists in the game industry are familiar with a concept known as the “Uncanny Valley,” but it isn’t a problem that typically occupies the attention of sound designers and game music composers.  However, with the imminent arrival of virtual reality, that situation may drastically change.  Audio folks may have to begin wrestling with the problem right alongside their visual arts counterparts. I’ll explore that issue during the course of this blog, but first let’s start with a basic definition: what is the Uncanny Valley?

Here’s the graphic that is typically shown to illustrate the Uncanny Valley concept.  The idea is this: human physical attributes can be endearing.  We like human qualities when we see them attached to inhuman things like robots.  It makes them cute and relatable. However, as they start getting more and more human in appearance, the cuteness starts going away, and the skin-crawling creepiness begins.  The ick-factor reaches maximum in an amorphous no-man’s land right before absolute realism would theoretically be attained.  In this realm of horrors known as the “Uncanny Valley,” we see that the appearance of the human-like creature is not close enough to be real, but close enough to be really disturbing.  Don’t take my word for it, though.  Here’s a great video from the Extra Credits video series that explores the meaning of the Uncanny Valley in more detail:

So, now we’ve explored what the Uncanny Valley means to visual artists, but how does this phenomenon impact the realm of audio?

Spatial Audio – Reconstructing Reality or Creating Illusion?

The idea of an audio equivalent for the Uncanny Valley was suggested by Francis Rumsey during a presentation he gave in May 2014 at the Audio Engineering Society Chicago Section Meeting, which took place at Shure Incorporated in Niles, Illinois.  Francis Rumsey holds a PhD in Audio Engineering from the University of Surrey and is currently the chair of the Technical Council of the Audio Engineering Society.  His talk was entitled “Spatial Audio – Reconstructing Reality or Creating Illusion?”

Francis-Rumsey

Francis Rumsey, chair of the AES Technical Council

In his excellent 90 minute presentation (available for viewing in its entirety by AES members), Francis Rumsey explores the history of spatial audio in detail, examining the long-term effort to reach perfect simulations of natural acoustic spaces.  He examines the divergent philosophies of top audio engineers who approach the problem from a creative/artistic point of view, and acousticians who want to solve the dilemma mathematically by virtue of a perfect wave field synthesis technique. Along the way, he asks if spatial audio is really meant to recreate the best version of reality, or instead to conjure up an entertaining artistic illusion?  This leads him to the main thesis of his talk:

Sound Design in VR: Almost Perfect Isn’t Perfect Enough

Rumsey suggests that as spatial audio approaches the top-most levels of realism, it begins to stimulate a more critical part of the brain.  Why does it do this?  Because human listeners react very strongly to a quality we call “naturalness.”  We have a great depth of experience in the way environmental sound behaves in the world.  We know how it reflects and reverberates, how objects may obstruct the sound or change its perceived timbre. As a simulated aural environment approaches perfect spatial realism and timbral fidelity, our brains begin to compare the simulation to our own remembered experiences of real audio environments, and we start to react negatively to subtle defects in an otherwise perfect simulation.  “It sounds almost real,” we think, “but something about it is strange.  It’s just wrong, it doesn’t add up.”

Take as an example this Oculus VR video demonstrating GenAudio’s AstoundSound 3D RTI positional 3D audio plugin.  While the audio positioning is awesome and impressive, the demo does not incorporate any obstruction or occlusion effects (as the plugin makers readily admit).  This makes the demo useful for us in examining the effects of subtle imperfections in an otherwise convincing 3D aural environment.  The imperfections become especially pronounced when the gamer walks into the Tuscan house, but the sound of the outdoor fountain continues without any of the muffling obstruction effects one would expect to hear in those circumstances.

Voice in VR: The Uncanny Valley of Spatial Voice

During the presentation, Rumsey shared some of the research from Glenn Dickins, the Technical Architect of the Convergence Team at Dolby Laboratories.  Dickins had applied the theory of the Uncanny Valley to vocal recordings. The sound of the human voice in a spatial environment is exceedingly familiar to us as human beings, much in the same way that human appearance and movement are both ingrained in our consciousness.  Because of this familiarity, vocal recordings in a spatial environment such as 3D positional audio can be particularly vulnerable to the Uncanny Valley effect.  Very small and subtle degradation in the audio output of a spatially localized voice recording may trigger a sense of deep-rooted unease.

Glenn Dickins of Dolby Laboratories

Glenn Dickins of Dolby Laboratories

As we embark on three dimensional audio environments for virtual reality games, the sorts of sound compression typically used in video game design may become problematic, particularly in relation to voice recordings in games.  While a typical gamer might not recognize that a vocal recording had been compressed, the gamer might nevertheless feel that there was something “not quite right” in the sound of the character’s voices.  Compression of audio subtly changes the vocal sound in ways that are usually unnoticeable, but may become disruptive in a VR aural environment in which imperfections have the potential to nudge the audio into the Uncanny Valley.

Music in VR: Some Good News

While I’ve talked in this blog before about the importance of defining the role that music should play in the three-dimensional aural environment of a virtual reality game, Francis Rumsey offers an entirely different viewpoint in his talk.  He thinks that when it comes to music, listeners don’t really care about spatial audio.  That might be good news for game composers, because this may mean that music may play no role in the Uncanny Valley effect.

Describing a study that was conducted to determine how both naive and experienced listeners perceived spatial audio, Rumsey showed that when it came to listening to music, the spatial positioning wasn’t considered tremendously important.  Sound quality was held to be absolutely crucial, but this desire was neither heightened nor lessened by spatial considerations. So does this mean that when it comes to music, listeners have an enhanced suspension of disbelief?  Are they willing to accept music into their VR world, even if it isn’t realistically positioned within the 3D space?  If so, then this would mean that non-diegetic music (i.e. music that isn’t occurring within the fictional world of the game) may not need to be spatially positioned as carefully as either voice or sound design elements of the aural environment.  This may prove useful to audio teams, who may turn to music as a reassuring agent in the soundscape, binding the aural environment together and promoting emotional investment and immersion.  However, music’s role in virtual reality may not conform to the way in which listeners react to spatially positioned music in other situations.  At any rate, the issue certainly needs further study and experimentation to clarify the role that non-diegetic music should play in a VR game.

For other types of music in VR, the situation may be much simpler.  Music doesn’t always have to occupy the traditional “underscore” role that it typically serves during gameplay.  In a “music visualizer” VR experience, spatial positioning may become entirely unnecessary, because the music is serving the purpose of pure foreground entertainment (much the same way that music entertains listeners on its own).  Here’s a preview of a musically-reactive virtual world in the upcoming “music visualizer” game Harmonix Music VR, created by the developer of the famous and popular game series Rock Band and Dance Central:

In Conclusion

Rumsey concluded his talk with the observation that near accurate may be worse than not particularly accurate… in other words, if it’s supposed to sound real, then it had better sound perfectly real.  Otherwise, it might be better to opt for a stylized audio environment that exaggerates and heightens the world rather than faithfully reproducing it.  I hope you enjoyed this blog, and please let me know what you think in the comments below!
border-159926_640_white

Studio1_GreenWinifred Phillips is an award-winning game music composer.  Her most well-known projects include such famous and popular games as Assassin’s Creed Liberation, God of War, the LittleBigPlanet franchise, and many others.  She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press.  Follow her on Twitter @winphillips.

139th AES Convention for the Game Music Composer

AES139_NYC_Banner

I’m happy to share that I’ll be a speaker again this year at the Audio Engineering Society’s annual convention!  Last year, the convention took place at the Los Angeles Convention Center – a familiar stomping ground from my many visits to the famous Electronic Entertainment Expo over the years.  However, this year will take me somewhere entirely new: the Jacob Javits Center in New York City!

jacob-javits-convention-center

I imagine that most futuristic metropolitan buildings look best when the sky is purple.  Since it’s impossible to capture natural purple skies in the wild, I assume that someone helpfully photoshopped a purple firmament for this promo picture.  The convention center looks very impressive, and I’m looking forward to seeing it in person!

Attending last year’s AES in Los Angeles was a wonderful experience, and I was truly honored to have been chosen as a speaker for the event!  At last year’s AES, I gave an overview presentation about interactive music in video games – the talk was an expansion of the interactive music sections of my book, A Composer’s Guide to Game Music.  Here’s a video clip from my speech last year, entitled “Effective Interactive Music Systems: The Nuts and Bolts of Dynamic Musical Content.”  The entire speech is available for download from Mobiltape.com.

At this year’s AES, I’ll be speaking more specifically about my role as a member of the music composition team for the LittleBigPlanet franchise.  It will be fun to share my experiences as part of that wonderful music team at Sony Computer Entertainment Europe, and I’m looking forward to exploring some of the interactive music techniques of the LittleBigPlanet franchise!

4-LBP3-E3

This is a photo from the LittleBigPlanet 3 display in the Sony booth at E3 2014.  My presentation at the Jacob Javits Center will include lots of my music from the LittleBigPlanet franchise, and Sackboy will be making many appearances!

AES-MixBoardI’m also looking forward to seeing what’s new and hot in audio gear on the AES exhibit floor.  Last year’s show floor was crowded with humongous mixing desks like the one above, along with enough glittering gear to make a full-grown audio engineer cry tears of joy.  I’m looking forward to a similar spectacle this year.  In addition to the expo floor, the convention will include a comprehensive program of presentations, panels and workshops, and the popular Live Sound Expo will be returning this year to spread knowledge about audio solutions for live events.

On a more personal note – prior to attending my first AES, I read an article from the ONION (the world’s top news satire publication) which lead me to believe that, as an audio engineer attending such a convention, I would be able to gather with my fellow audio professionals and enjoy an in-depth discussion of our ponytails (warning: adult language).  I can report that this did not happen last year… which was a shame, because I made sure I wore a ponytail for the occasion.  ;-)

I submit the following photo as proof:

AES-AESDespite this minor disappointment, I had an awesome time at last year’s AES, and I’m very excited about this year’s event!  The convention will take place from Oct. 29th to Nov. 1st at the Jacob Javits Center in New York City.  Hope to see you there!

border-159926_640_white

Studio1_GreenWinifred Phillips is an award-winning game music composer.  Her most well-known projects include such famous and popular games as Assassin’s Creed Liberation, God of War, the LittleBigPlanet franchise, and many others.  She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press.  Follow her on Twitter @winphillips.

Aleatoricism for the Game Music Composer

dices-160005_1280

I recently read a research paper about game music that I came across via Google Scholar (always an excellent resource if you’re looking for food for thought).  The paper was entitled “Game Scoring: Towards a Broader Theory,” written by Mack Enns of the University of Western Ontario towards his Master of Arts degree.  While the paper’s intention is to show how composing music for gaming is “distinct from other types of scoring,” the author repeatedly asserts that game music “remains always inherently ‘aleatoric.'”

This was the first I’d heard anyone put forward a theory such as this, and it’s definitely a bold statement.  Is all game music intrinsically aleatoric?  In my book, A Composer’s Guide to Game Music, I discuss aleatoricism as “music in which some elements are left to chance.” (Chapter 2, pg. 30).  This means that some components of the music are left to the metaphorical roll of the dice.  Mack Enns further applies this concept to games by asserting that all game music “involves both chance operations and a degree of improvisation… the “performer” of a game score is not a musical performer but a “ludal” one, that is, a “gamer”…”

So, according to this, the gamer is essentially performing the music.  Well, let’s think about that a moment.  The gamer does perform actions which trigger musical reactions, so one might imagine the gamer operating as a sort of “conductor” for the musical score, indirectly cueing the music to begin, or determining which musical segments will be triggered at any given time by virtue of gameplay choices. But a performer and a conductor aren’t really the same thing… so the idea of the improvising “gamer” as the chance element in a game music composition may not hold up in all games.  But does it hold up in some?

Just to take a look at what the improvising performer brings to aleatoric music, I’ll first give you an example from one of the more famous aleatoric compositions by John Cage, after which we’ll take a look at some top video game examples.  In John Cage’s “Living Room Music,” a group of performers are instructed to use household objects as rhythmic instruments, but no instruction is given regarding the best objects to choose, or the desired tonal quality to be produced by the objects.  That’s left entirely to chance.  Here are two performances of “Living Room Music.”  The first, performed by the MET Orchestra Percussionists, uses a combination of fingertips and glasses.

As a contrast, the second version, performed by Square Peg Round Hole, uses a collection of cardboard boxes (music begins at the 1 minute mark).

You can see that the choices of the performers have a dramatic impact on the sound of the resulting composition.  Does a gamer have a similar awesome impact on the musical content of a game by virtue of the choices made during gameplay?  That’s a harder idea to mentally digest, but Mack Enns proposes that if we also include the game’s sound effects as a part of the musical score, then we can more easily perceive the gamer as an aleatoric musical performer.  “Game scoring,” Enns writes, “includes a recognition, accommodation and, even, an orchestration of sound effects which ordinarily
might also be considered “extramusical” to the untrained ear.”

So, if we’re willing to accept the game’s sound effects as a part of its musical score, then the entire experience becomes an aleatoric performance… but that presupposes the idea that sound effects should be accepted as musical events.  If we don’t perceive the sound effects as musical, then are there still circumstances in which the gamer can be considered an aleatoric musical performer?  It seems to me that the role of the gamer as a musician would depend on the nature of the game being played… and there are certainly games in which the player fills the role of the performer in a composition governed by chance.  Let’s take a look at a few popular video game examples.

border-159926_640_white

Flow

In the game Flow, players become a small organism glowing in a watery universe.  In this game, the sound effects are musical.  When your organism eats a glowing mote in the water, a tone sounds. Progressing through the game, the tones create a musical latticework according to the player’s choices, essentially weaving the fabric of the game’s musical score as the gamer plays.  This seems like a good example of an aleatoric score in which the sound effects could rightly be considered musical, and the player can be identified as an aleatoric performer.

border-159926_640_white

Electroplankton

Electroplankton is the most overt example of a game that casts the player in the role of a musician.  The game consists of an assortment of minigames that allow the player to participate in music creation through many clever graphical interfaces and procedures.  For instance, in the “Beatnes” game, the player is introduced to the art of “Live Looping” by rhythmically tapping the screen to trigger sounds that thereafter loop repeatedly with the beat, allowing the player to create more complex repeating layers of sounds as the music progresses.

border-159926_640_white

Rez

Rez is a rail shooter in which the player navigates a virtual environment within a computer network, destroying viruses and firewalls in order to rescue an artificial intelligence named Eden.  There are no sound effects in Rez – every action of the player is punctuated with a musical tone or rhythmic effect that melds with the electronic soundtrack to create the overall musical score.  When players “shoot,” they’re essentially playing the music.

border-159926_640_white

Lumines

The gameplay of this popular puzzle game is less directly musical, but still has a very strong effect on the progress of the musical score.  Players of Lumines manipulate falling blocks (in a manner vaguely reminiscent of Tetris).  The music consists of looping segments, during which the overall composition seems to halt any forward progress and dwell on a short internal segment repeatedly until the player racks up enough points to trigger the music to move ahead.  Because the player’s actions influence the progress of the music rather than its content, it seems that this game casts the player in a role more similar to a conductor than a musician.

border-159926_640_white

Flower

In the Flower video game, the player collects flower petals, which emit bell-like tones as they fly up into the air.  These tones join with the game’s musical score with a delicate, “wind-chime” effect.  The player is able to sound these bells each time a new flower petal is collected, and these sounds serve to provide foreground interest for the music and a sense of shape and forward movement to the composition.

border-159926_640_white

So, after reading Mack Enn’s paper and considering his thesis, I think it’s certainly reasonable to identify certain games as having soundtracks structured in such a way as to allow the player to become an aleatoric musical performer.  I’m not sure I’d agree that all games are structured this way, and the idea of considering all sound effects as musical is a highly experimental concept.  Nevertheless, it’s interesting to consider!  Let me know what you think in the comments, and feel free to post any other “aleatoric” game scores you’d like to share!

border-159926_640_white

Studio1_GreenWinifred Phillips is an award-winning game music composer.  Her most well-known projects include such famous and popular games as Assassin’s Creed Liberation, God of War, the LittleBigPlanet franchise, and many others.  She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press.  Follow her on Twitter @winphillips.