Video Game Composers: The Tech of Music in Virtual Reality (GDC 2018)

Video game composer Winifred Phillips, pictured in her music production studio.

By Winifred Phillips | Contact | Follow

The Game Developers Conference is almost here! I’m looking forward to giving my presentation soon on “Music in Virtual Reality” (Thursday, March 22nd at 3pm in room 3002 West Hall, Moscone Center, San Francisco).  Over the course of the last two years, I’ve composed a lot of music for virtual reality projects, some of which have already hit retail, and some of which will be getting released very soon!  As a result, I’ve spent a lot of time thinking about what role music should play in a virtual reality game. During my GDC talk in March, I’ll be taking my audience through my experiences composing music for four very different VR games –the Bebylon: Battle Royale game from Kite & Lightning, the Dragon Front strategy game from High Voltage Software, the Fail Factory comedy game from Armature Studio, and the Scraper: First Strike RPG-Shooter hybrid from Labrodex Inc.  In preparing my GDC presentation, I made sure my talk addressed some of the most important creative and technical hurdles facing video game composers working in VR.  However, time constraints ensured that some interesting info ended up ‘on the cutting room floor,’ so to speak.  So, I’ve written two articles that explore some of the best topics that didn’t make it into my GDC presentation.

My previous article focused on some abstract, creative concerns facing video game music composers and audio folks working in VR.  In this article, we’ll be turning our attention to more concrete technical issues.  Ready?  Let’s go.

New Binaural Developments

Illustration of popular binaural developments in VR audio, from the article by composer Winifred Phillips for video game composers.VR games currently focus on binaural audio to immerse players in the awesome soundscapes of their virtual worlds.  As we know, binaural recording techniques use two microphones, often embedded in the artificial ears of a dummy head (pictured right).  By virtual of the popular binaural recording technique and/or binaural encoding technologies, game audio teams can plunge VR players into convincing aural worlds where sounds are spatially localized in a way that conforms with real world expectations.  The technology of binaural sound continually improves, and recently the expert developers of the Oculus Rift VR headset have refined the quality of their VR sound with two significant upgrades.

First, they have introduced “Near-Field Head Related Transfer Function.”  We’re all probably familiar by now with the concept of Head Related Transfer Function (HRTF), in which sound interacts with our heads, bodies and ear canals on its way to our ear drums.  The subtle changes undergone by those sound waves endow them with the specific qualities that reflect the real-life aural world.  In the original Oculus Audio SDK, realistic HRTF was limited by distance.  Sounds that occurred a meter away or more would reach us with all the correct HRTF effects we’d expect, but when sounds were emitted within that meter-wide diameter around our heads, the HRTF effects would no longer function.  So now the Oculus team has introduced “Near-Field HRTF” to fill in that gap.   This allows sounds that might occur close to our heads to feel more realistic.

“With our recently introduced Near-Field HRTF, developers can model sounds much closer than one meter away with a greater degree of accuracy,” writes the Oculus team in a blog article announcing the new technology. “Now, if you’re holding an object that makes sound (like a ringing telephone) and bring it closer to your head, we’re able to replicate that experience in VR in a more believable way.”

Near-Field HRTF might have great applications for game composers interested in employing diegetic music that occurs close to the player, such as a musical instrument that the player is expected to ‘play,’ or a music-emitting device (a radio, a music box) that’s held in the player’s hand.  Here’s a video produced by the Oculus team that demonstrates Near-Field HRTF in action:

The second binaural audio upgrade from the Oculus team allows for “Volumetric Sound Sources.”  This is a simpler effect to understand, but with more interesting implications for music inside the VR space.  “Volumetric Sound Sources let sound designers model objects of virtually any size in a way that sounds realistic,” explains the Oculus team. “Rather than trying to pinpoint the source of a sound, designers can give a sound a radius—the larger the assigned radius, the larger the sound’s source.”

Theoretically, this effect should make a single sound source emit a more expansive breadth of sound, sidestepping the otherwise mono-like qualities that many object-based sound sources typically display.  When implementing music in ways that attach it to the environment (such as audio mixes that separate instrument sections and spread them out across the virtual landscape), this kind of “Volumetric” effect would prevent these sounds from feeling as though they each emanate from a single point.  For instance, a violin section might have the wider breadth expected from such a recording.  Here’s the video produced by the Oculus team that demonstrates this effect.  Unfortunately, in the video the developers focused on white noise-heavy sounds like rushing/falling water, hissing machinery noises, and ocean surf.  This is less than revealing for other types of sound (such as musical sources), but it might give us a general idea of the developers’ intentions:

Both of these recent innovations depend on the binaural delivery system in order to function.  Binaural is currently very important in VR audio, and is likely to remain important for some time to come.

Photo of video game music composer Winifred Phillips working in her music production studio.GDC 2018 Presentation Preview

In my upcoming GDC talk, I’ll be reviewing the history of the binaural recording method and its importance to virtual reality game development.  By virtue of both my own experiences with multiple VR projects, I’ll be describing several ways to spatially-localize music and audio content within a binaural audio delivery system.

However useful binary audio may be for VR audio design, binaural also has some logistical drawbacks for the consumer.  Let’s take a look at those, and examine some solutions that have been proposed.

The headphones problem

Illustration of the famous headphones problem in VR audio, from the article by Winifred Phillips (award winning video game music composer).In today’s modern high-tech environment, smart devices are everywhere.  One of the defining characteristics of a smart device is its ability to identify itself when used in combination with other technologies.  Smart devices are essentially plug and play.  We can plug a smart device such as the famous iPhone into a computer, and the computer will immediately know what it is.  Lots of peripheral devices are instantly recognizable, such as keyboards, game controllers, hard drives, etc.  Most standard headphones, however, aren’t smart devices.  When we plug them into a computer for use with a VR system, the system won’t be able to identify the headphone model, nor will it be able to detect any onboard signal processing such as noise cancellation or spatial rendering (surround sound or binaural).  That means that the VR system can’t automatically compensate for the headphone technology currently plugged into it.  This is where things can go wrong.

What if a pair of surround-sound headphones are plugged into a VR gaming device outputting in binaural sound?  The VR game’s binaural audio is going to get spatially reprocessed by the headphones, resulting in a serious degradation of the sound quality.  Consumers are accustomed to their devices communicating with each other and sorting these sorts of issues out autonomously.  If the VR system is smart, but the headphones are essentially dumb, what then?

Pictured: today's popular analog headphone models (an illustration from the article by composer Winifred Phillips for video game composers.“For any poorly perceived mix, the user may not know what the problem is or how to go about rectifying it. Even worse the user may even assume the poor mix is a fault of the game’s audio engine,” observed audio programmer Aristotel Digenis in his presentation “Challenges of the Headphone Mix in Games” at the 56th Audio Engineering Society Conference.  “Smart headphones” may be the natural evolution for headphones,” Digenis points out. “The analogue nature of conventional headphones means they have no way of identifying themselves in an equipment eco-system that is rapidly able to identify one another and each other’s capabilities.”

So, in an ideal world, the VR system would be able to identify the nature of the headphones and adjust its output accordingly.  We can all hope that smart headphones will eventually appear on the high-tech horizon, but what about the other end of the equation?  Can VR audio systems provide multiple audio mixes for different types of headphones?  Aristotel Digenis proposes a way in which this can be achieved… although it includes the use of a new audio format, and a dramatic re-evaluation of the way in which VR audio is currently delivered.


The MPEG-H Audio Alliance logo from the creators of the famous MPEG format, included in the article by Winifred Phillips for video game composers.First announced in 2013, the MPEG-H 3D Audio coding standard supports multiple audio channel output configurations including binaural rendering and higher order ambisonics.  One of the advantages of MGEG-H is its compatibility with multiple audio playback systems, from simple stereo, to complex surround configurations, to interactive virtual reality audio systems.  This flexible delivery mechanism, which the designers have coined Universal Delivery, allows the MPEG-H format to be cross-compatible with any type of listening equipment, from simple stereo headphones all the way to the most sophisticated home theater setups.  In other words, MPEG-H would allow VR audio to work on any playback system instead of its current limitation to headphones alone.

“Widespread adoption of the MPEG-H standard could mean game developers may not need to provide binaural mixes for their users,” Digenis asserts. “Instead they can continue delivering multi-channel mixes to the console/A V system, and it can provide a suitable binaural mix to the user if they are using headphones.”

Currently, the developers of MPEG-H are concentrating their focus on TV broadcast applications.  Here’s a short video demonstration of the television use of MPEG-H.  Notice how the audio content remains interactive on a highly detailed level, allowing the manipulation of individual sonic elements by the end-user.

While it seems like the MPEG-H format is a long way from practical implementation in VR applications, it shows some promise in providing future options for game audio folks working in VR.  We can imagine a future in which a virtual reality game may have its audio recorded and mixed in ambisonics, and then output in any number of configurations depending on the nature of the sound playback system.

In this photo, game music composer Winifred Phillips is pictured working in her music production studio.GDC 2018 Presentation Preview

In my upcoming GDC talk, I’ll be discussing the role of ambisonics in game audio development, starting with a historical overview and moving to the importance of the format in modern VR games.

Creating more opportunities for ambisonics to flourish in VR audio is an interesting topic to consider.  Let’s examine one intriguing possibility.

Ambisonics and the orchestra

In October of last year I wrote an article about some of the VR topics discussed at the Virtual Reality Developers Conference.  As a part of that article, I touched upon an idea that was briefly mentioned by Jay Steen of Criterion Games during his talk about the audio of Star Wars Battlefront Rogue One X-Wing VR Mission.  During the Q&A portion of his talk, Steen was asked about spatial positioning for a Star Wars musical score in VR.  “We did do a quick experiment on it, and we found that it’s like having an orchestra sitting around you,” Steen observed. “We didn’t want to evoke you sitting in the middle of an orchestral recording. We just wanted it to sound like the movie.” Even with that seeming dismissal of a spatially-positioned Star Wars score, Steen went on to add,  “Ambisonic recordings of orchestras for example, I think there’s something fun there. We haven’t experimented with it anymore than that, but yeah, definitely, we’d want to try.”

An illustration of popular ambisonic recording techniques for live orchestral performance, from the article by Winifred Phillips for game composers.Ambisonic orchestral recordings are not often encountered in VR games, but they’re becoming a bit more common in other forms of virtual entertainment.  As a way to imagine how ambisonic orchestral recordings might be deployed in future VR games, let’s take a look at an experiment undertaken by the Institute of Communication Systems at RWTH Aachen University in Germany.  In April of last year, the Institute began a cooperative venture with the Aachen Symphony Orchestra.  Over the course of several months, the orchestra was recorded with an em32 Eigenmike microphone array, which is capable of recording 32 channels of audio for fourth-order ambisonics.  They placed the microphone in the middle of the string section, and also positioned a 36o° camera at the microphone array location so that a VR video could be made.  The result replicates the experience of sitting right in the middle of the orchestra during a live concert.

The Institute of Communication Systems reports that “the recorded content will be used for future research activities in the area of audio signal processing for immersive audio systems, e.g., 3D audio formats, binaural signal processing and spatial audio playback.”  So these experiments may yield results that prove to be useful to game music composers working on VR projects.  Here’s a video of one of the recorded performances of the Aachen Symphony Orchestra.  Remember to wear headphones when listening, and feel free to swing the camera view in all directions!


In my past two articles, I’ve shared some of the additional research I’d encountered that didn’t make the cut for my GDC 2018 presentation, ‘Music in Virtual Reality  (Thursday, March 22nd at 3pm in room 3002 West Hall, Moscone Center, San Francisco).  The presentation I will give at GDC 2018 will include lots of practical and concrete techniques and strategies for video game music composers and audio folks looking to implement music strategically and effectively within VR. That being said, the more general research and techniques in these two articles can also provide helpful insight. I’ve included the official GDC description of my upcoming talk below.  Please feel free to share your thoughts and insights in the comments section at the end of this article!


Music in Virtual Reality

Illustration of the VR projects to be discussed in a GDC talk presented by Winifred Phillips for video game composers.This lecture will present ideas for creating a musical score that complements an immersive VR experience. Composer Winifred Phillips will share tips from several of her VR projects. Beginning with a historical overview of positional audio technologies, Phillips will address several important problems facing composers in VR.

Topics will include 3D versus 2D music implementation, and the role of spatialized audio in a musical score for VR. The use of diegetic and non-diegetic music will be explored, including methods that blur the distinction between the two categories.

The discussion will also include an examination of the VIMS phenomenon (Visually Induced Motion Sickness), and the role of music in alleviating its symptoms.  Phillips’ talk will offer techniques for composers and audio directors looking to utilize music in the most advantageous way within a VR project.


Through examples from several VR games, Phillips will provide an analysis of music composition strategies that help music integrate successfully in a VR environment. The talk will include concrete examples and practical advice that audience members can apply to their own games.

Intended Audience

This session will provide composers and audio directors with strategies for designing music for VR. It will include an overview of the history of positional sound and the VIMS problem (useful knowledge for designers.)

The talk will be approachable for all levels (advanced composers may better appreciate the specific composition techniques discussed).


Photo of Winifred Phillips in her video game composers music production studio.Winifred Phillips is an award-winning video game music composer whose most recent projects are the triple-A first person shooter Homefront: The Revolution and the Dragon Front VR game for Oculus Rift. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Follow her on Twitter @winphillips.






Understanding Audio in VR – A Game Music Composer’s Resource Guide

Video game music composer Winifred Phillips working in her game composers production studio.

By Winifred Phillips | Contact | Follow

When I’m not at work in my studio making music for games, I like to keep up with new developments in the field of interactive entertainment, and I’ll often share what I learn here in these articles.  Virtual reality is an awesome subject for study for a video game composer, and several of my recent projects have been in the world of VR.  Since I’m sure that most of us are curious about what’s coming next in virtual reality, I’ve decided to devote this article to a collection of educational resources.  I’ve made a point of keeping our focus general here, with the intent of understanding the role of audio in VR and the best resources available to audio folks.  As a component of the VR soundscape, our music must fit into the entire matrix of aural elements, so we’ll spend this article learning about what goes into making expert sound for a virtual reality experience. Let’s start with a few articles that discuss methods and techniques for VR audio practitioners.

Continue reading

Video game music systems at GDC 2017: tools and tips for composers

Photo of video game composer Winifred Phillips, working in her music production studio on the music of the SimAnimals video game.

By video game composer Winifred Phillips | Contact | Follow

Welcome back to this three article series that’s bringing together the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers explored discoveries they’d made while creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to broaden our viewpoint and gain a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. We’ve been looking at five interactive music systems discussed in these five GDC 2017 presentations:

In the first article, we examined the basic nature of these interactive systems. In the second article, we contemplated why those systems were used, with some of the inherent pros and cons of each system discussed in turn.  So now, let’s get into the nitty gritty of tools and tips for working with such interactive music systems.  If you haven’t read parts one and two of this series, please go do so now and then come back:

  1. Video game music systems at GDC 2017: what are composers using?
  2. Video game music systems at GDC 2017: pros and cons for composers

Ready?  Great!  Here we go!

Continue reading

Video game music systems at GDC 2017: pros and cons for composers

Video game composer Winifred Phillips, pictured in her music production studio working on the music of LittleBigPlanet 2 Cross Controller

By Winifred Phillips | Contact | Follow

Welcome back to our three article series dedicated to collecting and exploring the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers shared ideas they’d developed in the process of creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to cultivate a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. In the first article, we looked at the basic nature of five interactive music systems discussed in these five GDC 2017 presentations:

If you haven’t read part one of this article series, please go do that now and come back.

Okay, so let’s now contemplate some simple but important questions: why were those systems used?  What was attractive about each interactive music strategy, and what were the challenges inherent in using those systems?

Continue reading

Video Game Music Production Tips from GDC 2016

Game Composer Winifred Phillips during her game music presentation at the Game Developers Conference 2016I was pleased to give a talk about composing music for games at the 2016 Game Developers Conference (pictured left).  GDC took place this past March in San Francisco – it was an honor to be a part of the audio track again this year, which offered a wealth of awesome educational sessions for game audio practitioners.  So much fun to see the other talks and learn about what’s new and exciting in the field of game audio!  In this blog, I want to share some info that I thought was really interesting from two talks that pertained to the audio production side of game development: composer Laura Karpman’s talk about “Composing Virtually, Sounding Real” and audio director Garry Taylor’s talk on “Audio Mastering for Interactive Entertainment.”  Both sessions had some very good info for video game composers who may be looking to improve the quality of their recordings.  Along the way, I’ll also be sharing a few of my own personal viewpoints on these music production topics, and I’ll include some examples from one of my own projects, the Ultimate Trailers album for West One Music, to illustrate ideas that we’ll be discussing.  So let’s get started!

Continue reading

Interactive Music for the Video Game Composer

Game Composer Winifred Phillips works in her studio on the music of the popular Spore Hero video game As a speaker in the audio track of the Game Developers Conference this year, I enjoyed taking in a number of GDC audio sessions — including a couple of presentations that focused on the future of interactive music in games.  I’ve explored this topic before at length in my book (A Composer’s Guide to Game Music), and it was great to see that the game audio community continues to push the boundaries and innovate in this area! Interactive music is a worthwhile subject for discussion, and will undoubtedly be increasingly important in the future as dynamic music systems become more prevalent in game projects.  With that in mind, in this blog I’d like to share my personal takeaway from two sessions that described very different approaches to musical interactivity. After that, we’ll discuss one of my experiences with interactive music for the video game Spore Hero from Electronic Arts (pictured above).

Musical Intelligence

Baldur BaldurssonPhoto of Baldur Baldursson, the audio director for Icelandic game development studio CCP Games (part of the article by game composer Winifred Phillips) (pictured left) is the audio director for Icelandic game development studio CCP Games, responsible for the EVE Online MMORPG.  Together with Professor Kjartan Olafsson of the Iceland Academy of Arts, Baldursson presented a talk at GDC 2016 on a new system to provide “Intelligent Music For Games.”

Baldursson began the presentation by explaining why an intelligent music system for games can be a necessity.  “We basically want an intelligent music system because we can’t (or maybe shouldn’t really) precompose all of the elements,” Baldursson explains. He describes the conundrum of creating a musical score for a game whose story is still fluid and changeable, and then asserts,  “I think we should find ways of making this better.”

Continue reading

Music in the Manual: FMOD Studio Vs. Wwise


A few days ago, I downloaded and installed the latest version of a software package entitled FMOD Studio and was pleasantly surprised to discover that an oversight had been corrected. It’s not unusual for software updates to correct problems or provide additional functionality, but this update was especially satisfying for me. The makers of FMOD Studio had added the “Music” section to the software manual.

A brief explanation: FMOD Studio is a software application designed by Firelight Technologies to enable game audio professionals to incorporate sound into video games. The application focuses solely on audio, and is used in conjunction with game software. In essence, FMOD Studio is folded into the larger construct of a game’s operational code, giving the overall game the ability to do more sophisticated things with the audio side of its presentation.

When FMOD Studio was initially released in August of 2012, the manual did not include information about the music capabilities of the software. Admittedly, the majority of FMOD Studio users are sound designers whose interests tend to focus on the tools for triggering sound effects and creating environmental atmospheres. That being said, many composers also use the portions of the FMOD Studio application that are specifically designed to enable the assignment of interactive behaviors to music tracks. It was a bit puzzling that the manual didn’t describe those music tools.

One of the biggest competitors to FMOD Studio is the Wwise software from Audiokinetic. Wwise offers much of the same functionality as FMOD, and in working with the software one of the things I really like about it is its documentation. Audiokinetic put a lot of thought and energy into the Wwise Fundamentals Approach document and the expansive tutorial handbook, Project Adventure. Both of these documents discuss the music features of the Wwise software, offering step-by-step guidance for the creation of interactive music systems within the Wwise application. This is why the omission of any discussion of the music tools from the FMOD manual was so perplexing.

It’s true that many of the music features of the FMOD Studio software are also useful in sound design applications, and some are similar in their function to tools described in the sound design portions of the manual. Firelight Technologies may have assumed that those portions of the manual would be sufficient for all users, including composers. However, composers are specialists, and their priorities do not match those of their sound design colleagues. In using the FMOD Studio tools, the needs of composers would be sharply different from those driving the rest of the audio development community. Wwise understood this from the start, but FMOD seemed to be following a philosophy that hearkened back to the early days of the game industry.

In those days, the audio side of a game was often created and implemented by a single person. This jack-of-all-trades would create all the sound effects, voice-overs and music. Nowadays, the audio field is populated by scores of specialists. It makes sense for FMOD Studio to acknowledge specialists such as composers in their software documentation, and I’m very glad to see that they’ve just done so. If you’d like to learn more about FMOD Studio, you can see a general overview of the application in this YouTube video: