Composing video game music for Virtual Reality: 3D versus 2D

In this article written for video game composers, Winifred Phillips is here pictured working in her music production studio.

Welcome!  I’m videogame composer Winifred Phillips, and this is the continuation of our four-part discussion of the role that music can play in Virtual Reality video games.  These articles are based on the presentation I gave at this year’s Game Developer’s Conference in San Francisco, entitled Music in Virtual Reality (I’ve included the official description of my talk at this end of this article).  If you missed the first article exploring the history and significance of positional audio, please go check that article out first.

Are you back?  Great!  Let’s continue!

During my GDC talk, I addressed three questions which are important to video game music composers working in VR:

  • Do we compose our music in 3D or 2D?
  • Do we structure our music to be Diegetic or Non-Diegetic?
  • Do we focus our music on enhancing player Comfort or Performance?

While investigating these topics, we looked at some examples from VR games that provide great demonstrations, including four of my own VR projects –the Bebylon: Battle Royale arena combat game from Kite & Lightning, the Dragon Front strategy game from High Voltage Software, the Fail Factory comedy game from Armature Studio, and the Scraper: First Strike shooter/RPG from Labrodex Inc. In these articles, I’ll be sharing the discussions and conclusions that formed the basis of my GDC talk, including the best examples from these four VR game projects.  So now let’s turn our attention to the first of our three top questions:

Should our music be 3D or 2D?

We know that spatial delivery of sound design is critical, but does that extend to the music? Do most listeners care if the music is 3D?  It’s vital that we keep listener impact in mind – and some scholarly studies from expert researchers can help us throw light on that subject.

Illustration of audio research at the University of Hull, from the article by Winifred Phillips for video game composersMeasuring galvanic skin responses, a study conducted at the University of Hull in the UK tested for emotional reactions to both spatially treated music and standard stereo music recordings. They found that spatial treatment had no effect on the emotional impact and enjoyment of music. So, using a traditional stereo mix for a VR game’s music isn’t necessarily a bad thing… but spatial positioning for music can be beneficial, and fun too.

For instance:

  • We can use 3D elements to help integrate a 2D musical score into the VR world.
  • We can use 3D music to grab the player’s attention.
  • We can have music transition from 2D to 3D for dramatic effect.

3D music elements can help the musical score feel better connected to the environment in VR. Let’s take a look at an example from the Fail Factory VR game, which demonstrates how 3D music elements can share the stage with a conventional stereo music mix.

An illustration for the Fail Factory game on the popular VR platform, from the article for video game composers by Winifred Phillips (game music composer).In this comedic video game, players go to work in a zany robot factory. Players build massive robots, while keeping up with the ever-increasing complexity and speed of the assembly line. The result is often a series of hilarious failures, inspiring the game’s name – Fail Factory. When Armature Studio hired me to compose the music for their Fail Factory game for the Samsung Gear VR, they described a project in which music took center stage.

By necessity, Fail Factory is set on a gigantic factory floor – but what makes this factory uniquely awesome is the musical nature of the environment.  All the machinery in the factory moves rhythmically with the musical score. So, the dev team asked me to create a jazzy score for this music-driven gameplay. Apart from the score, all of the sound design of Fail Factory is also created specifically to be musical. The bleeps and bloops are pitched to integrate with the score, and the bangs and clangs are timed to emphasize the tempo. While much of the music is delivered to the player in traditional stereo, there are also lots of separate rhythmic and pitched elements that are spatially positioned on the game’s factory floor. The sound design team and I worked hard on getting the balance right between these 2D and 3D components.

For instance, in one minigame, heavy machinery slams down to a conveyor belt – this became the central downbeat for the music on this level. We tried just having that big metallic bang issue solely in 3D from its in-game position, but that didn’t work. As a spatialized sound that was rhythmically synced to the 2D music, the 3D metallic bang felt disconnected from the rest of the 2D score – plus, the bang just needed more oomph. The team and I went back and forth with iterations on this until we settled on both a spatialized impact sound and a simultaneous metallic clang integrated into the stereo music mix. Here’s how that sounded during gameplay:

So you can see that 3D music and audio in VR can be a complicated issue.  While the majority of the music in Fail Factory is mixed in stereo, there are percussive and tonal components (such as that big clang) that are spread out in 3D across the VR space. These elements in 3D allow us to have a nice stereo music mix that also integrates well into the three-dimensional soundscape.

Now let’s take a look at a different example that shows how music in VR can transition from 2D to 3D for dramatic effect.

In this article for video game composers, Winifred Phillips explains her music composition work for the Dragon Front game for the famous Oculus Rift VR platform.The popular Dragon Front VR strategy game for Oculus Rift is a mix of the famous tradition of high fantasy storytelling with a dieselpunk, World War II-inspired aesthetic. Each game session is a self-contained battle on a playing field loaded with monsters, missiles and the machinery of war. With all this in mind, the music of Dragon Front had to convey a suitably bold and dramatic style.  When High Voltage Software hired me to compose the music for Dragon Front , one of their biggest priorities was an epic main theme. So, I composed a big victorious anthem, with the stereo mix piped directly to the player’s headphones. The theme music was designed to continue into the hub, but bombastic music in the hub area could be distracting. So at that point the music moves from a direct channel to the player and takes up a position in the environment, as if it were issuing from in-game speakers. Here’s how that worked:

So we’ve now taken a closer look at the first of the three important questions for video game composers creating music for VR games:

  • Do we compose our music in 3D or 2D?
  • Do we structure our music to be Diegetic or Non-Diegetic?
  • Do we focus our music on enhancing player Comfort or Performance?

We’ve just explored what it means to compose music with both 2D and 3D considerations in mind.  The next article will focus on the second of the three questions: whether music in VR should be diegetic or non-diegetic.  Thanks for reading, and please feel free to leave your comments in the space below!

 

 


 

Music in Virtual Reality

Illustration of the popular VR projects featuring music by game composer Winifred Phillips, to be discussed in a GDC talk presented by Winifred Phillips for video game composers.This lecture presented ideas for creating a musical score that complements an immersive VR experience. Composer Winifred Phillips shared tips from several of her VR projects. Beginning with a historical overview of positional audio technologies, Phillips addressed several important problems facing composers in VR.

Topics included 3D versus 2D music implementation, and the role of spatialized audio in a musical score for VR. The use of diegetic and non-diegetic music were explored, including methods that blur the distinction between the two categories.

The discussion also included an examination of the VIMS phenomenon (Visually Induced Motion Sickness), and the role of music in alleviating its symptoms.  Phillips’ talk offered techniques for composers and audio directors looking to utilize music in the most advantageous way within a VR project.

Takeaway

Through examples from several VR games, Phillips provided an analysis of music composition strategies that help music integrate successfully in a VR environment. The talk included concrete examples and practical advice that audience members can apply to their own games.

Intended Audience

This session provided composers and audio directors with strategies for designing music for VR. It included an overview of the history of positional sound and the VIMS problem (useful knowledge for designers.)

The talk was intended to be approachable for all levels (advanced composers may better appreciate the specific composition techniques discussed).

 

 

Photo of video game composer Winifred Phillips in her music production studio.Winifred Phillips is an award-winning video game music composer whose most recent projects are the triple-A first person shooter Homefront: The Revolution and the Dragon Front VR game for Oculus Rift. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Follow her on Twitter @winphillips.

Composing video game music for Virtual Reality: The role of music in VR

In this article for video game composers, Winifred Phillips is pictured working in her music production studio.

By Winifred Phillips | Contact | Follow

Hey everybody!  I’m video game composer Winifred Phillips.  At this year’s Game Developers Conference in San Francisco, I was pleased to give a presentation entitled Music in Virtual Reality (I’ve included the official description of my talk at the end of this article). While I’ve enjoyed discussing the role of music in virtual reality in previous articles that I’ve posted here, the talk I gave at GDC gave me the opportunity to pull a lot of those ideas together and present a more concentrated exploration of the practice of music composition for VR games.  It occurred to me that such a focused discussion might be interesting to share in this forum as well. So, with that in mind, I’m excited to begin a four-part article series based on my GDC 2018 presentation!

Continue reading

Video Game Composers: The Tech of Music in Virtual Reality (GDC 2018)

Video game composer Winifred Phillips, pictured in her music production studio.

By Winifred Phillips | Contact | Follow

The Game Developers Conference is almost here! I’m looking forward to giving my presentation soon on “Music in Virtual Reality” (Thursday, March 22nd at 3pm in room 3002 West Hall, Moscone Center, San Francisco).  Over the course of the last two years, I’ve composed a lot of music for virtual reality projects, some of which have already hit retail, and some of which will be getting released very soon!  As a result, I’ve spent a lot of time thinking about what role music should play in a virtual reality game. During my GDC talk in March, I’ll be taking my audience through my experiences composing music for four very different VR games –the Bebylon: Battle Royale game from Kite & Lightning, the Dragon Front strategy game from High Voltage Software, the Fail Factory comedy game from Armature Studio, and the Scraper: First Strike RPG-Shooter hybrid from Labrodex Inc.  In preparing my GDC presentation, I made sure my talk addressed some of the most important creative and technical hurdles facing video game composers working in VR.  However, time constraints ensured that some interesting info ended up ‘on the cutting room floor,’ so to speak.  So, I’ve written two articles that explore some of the best topics that didn’t make it into my GDC presentation.

My previous article focused on some abstract, creative concerns facing video game music composers and audio folks working in VR.  In this article, we’ll be turning our attention to more concrete technical issues.  Ready?  Let’s go.

New Binaural Developments

Illustration of popular binaural developments in VR audio, from the article by composer Winifred Phillips for video game composers.VR games currently focus on binaural audio to immerse players in the awesome soundscapes of their virtual worlds.  As we know, binaural recording techniques use two microphones, often embedded in the artificial ears of a dummy head (pictured right).  By virtual of the popular binaural recording technique and/or binaural encoding technologies, game audio teams can plunge VR players into convincing aural worlds where sounds are spatially localized in a way that conforms with real world expectations.  The technology of binaural sound continually improves, and recently the expert developers of the Oculus Rift VR headset have refined the quality of their VR sound with two significant upgrades.

Continue reading

Video Game Composers: The Art of Music in Virtual Reality (GDC 2018)

Video game composer Winifred Phillips, pictured in her music production studio.

 

By Winifred Phillips | Contact | Follow

Once again, the Game Developers Conference is almost upon us!  GDC 2018 promises to be an awesome event, chock full of great opportunities for us to learn and grow as video game music composers.  I always look forward to the comprehensive sessions on offer in the popular GDC audio track, and for the past few years I’ve been honored to be selected as a GDC speaker.  Last year I presented a talk that explored how I built suspense and tension through music I composed for such games as God of War and Homefront: The Revolution.  This year, I’m tremendously excited that I’ll be presenting the talk, “Music in Virtual Reality.” The subject matter is very close to my heart!  Throughout 2016 and 2017, I’ve composed music for many virtual reality projects, some of which have hit retail over the past year, and some of which will be released very soon.  I’ve learned a lot about the process of composing music for a VR experience, and I’ve given a lot of thought to what makes music for VR unique.  During my GDC talk in March, I’ll be taking my audience through my experiences composing music for four very different VR games –the Bebylon: Battle Royale arena combat game from Kite & Lightning, the Dragon Front strategy game from High Voltage Software, the Fail Factory comedy game from Armature Studio, and the Scraper: First Strike Shooter/RPG from Labrodex Inc.  I’ll talk about some of the top problems that came up, the solutions that were tried, and the lessons that were learned.  Virtual Reality is a brave new world for game music composers, and there will be a lot of ground for me to cover in my presentation!

In preparing my talk for GDC, I kept my focus squarely on composition techniques for VR music creation, while making sure to supply an overview of the technologies that would help place these techniques in context.  With these considerations in mind, I had to prioritize the information I intended to offer, and some interesting topics simply wouldn’t fit within the time constraints of my GDC presentation.  With that in mind, I thought it would be worthwhile to include some of these extra materials in a couple of articles that would precede my talk in March.  In this article, I’ll explore some theoretical ideas from experts in the field of VR, and I’ll include some of my own musings about creative directions we might pursue with VR music composition.  In the next article, I’ll talk about some practical considerations relating to the technology of VR music.

Continue reading

VR Headphones Update: Video Game Music Composers

Video game composer Winifred Phillips, pictured in her music production studio working on the music of the Dragon Front virtual reality game for Oculus Rift and Samsung Gear VR.

By Winifred Phillips | Contact | Follow

Last year while working on the music of the Dragon Front virtual reality game for Oculus Rift (as pictured above), I gave a lot of consideration to the listening environment in which VR gamers would be hearing my video game music.  Since then I’ve served as the video game composer for several more virtual reality games (which will be released in the next few months).  I’ve also written a number of articles on this subject in order to share what I’ve learned with other game composers.  Last September I devoted two articles to a discussion of audio headphones designed specifically for the demands of virtual reality applications.  You can read those here:

In addition, two years ago I wrote an article that focused on some of the top difficulties associated with choosing the right headphones for VR.  You can read that article here:

Music Composers and Sound Designers in VR: The Headphones Problem

Now, I’d like to revisit the ideas discussed in those articles, so that we can see how the art of VR audio for headphones has progressed.

Continue reading

Video Game Music Composers: New VR Headphones

Video game composer Winifred Phillips, pictured in her music production studio.

By Winifred Phillips | Contact | Follow

As a video game composer, I’ve been working in my studio composing music for quite a few virtual reality projects lately (as pictured above), so I’ve been thinking a lot about issues related to audio in the VR environment.  Those issues include how gamers experience the audio content through various headphone models.  In this article, I thought we’d take a look at three newly-announced headphone models that are targeting the VR marketplace, and see what new technologies are being proposed to facilitate the best and most awesome VR audio experiences.  So, let’s get started!

Continue reading

Understanding Audio in VR – A Game Music Composer’s Resource Guide

Video game music composer Winifred Phillips working in her game composers production studio.

By Winifred Phillips | Contact | Follow

When I’m not at work in my studio making music for games, I like to keep up with new developments in the field of interactive entertainment, and I’ll often share what I learn here in these articles.  Virtual reality is an awesome subject for study for a video game composer, and several of my recent projects have been in the world of VR.  Since I’m sure that most of us are curious about what’s coming next in virtual reality, I’ve decided to devote this article to a collection of educational resources.  I’ve made a point of keeping our focus general here, with the intent of understanding the role of audio in VR and the best resources available to audio folks.  As a component of the VR soundscape, our music must fit into the entire matrix of aural elements, so we’ll spend this article learning about what goes into making expert sound for a virtual reality experience. Let’s start with a few articles that discuss methods and techniques for VR audio practitioners.

Continue reading