Understanding Audio in VR – A Game Music Composer’s Resource Guide

Video game music composer Winifred Phillips working in her game composers production studio.

By Winifred Phillips | Contact | Follow

When I’m not at work in my studio making music for games, I like to keep up with new developments in the field of interactive entertainment, and I’ll often share what I learn here in these articles.  Virtual reality is an awesome subject for study for a video game composer, and several of my recent projects have been in the world of VR.  Since I’m sure that most of us are curious about what’s coming next in virtual reality, I’ve decided to devote this article to a collection of educational resources.  I’ve made a point of keeping our focus general here, with the intent of understanding the role of audio in VR and the best resources available to audio folks.  As a component of the VR soundscape, our music must fit into the entire matrix of aural elements, so we’ll spend this article learning about what goes into making expert sound for a virtual reality experience. Let’s start with a few articles that discuss methods and techniques for VR audio practitioners.

Methods and techniques

3D Audio formats for VRAn illustration of popular methods and techniques for game composers - section from the article by Winifred Phillips, video game music composer.

3D Sound Labs takes us through the three most popular sound formats for spatial audio in VR: Multi-channel, Object-based, and Ambisonic.

How 3D Spatialized Audio Bottlenecks Virtual Reality Video

VR audio can consume enormous computational resources, resulting in a resource war between audio and video content.  This article discusses how audio demands in VR consume memory bus bandwidth, and proposes a short-cut that manipulates the frequency response of the audio content to enhance spatialization while consuming fewer resources.

Adapting Your DAW for VR Audio

A VR sound editor discusses his methods and workflow for virtual reality in this article.  Topics include capturing audio in the ambisonic format, and customizing the Pro Tools environment for spatialized audio projects.

Simple spatial audio for beginners

This article summarizes the available entry-level techniques and equipment that can help a newcomer jump into the world of spatial audio for virtual reality.

An audio post production house shares techniques for VR Audio

In this article, we learn about some of the top equipment and methods used by the VR-focused division of an experienced audio post production facility.

Technology and tools

Now that we’ve garnered some insight about what goes into creating VR audio, let’s take a look at some great technology and tools designed specifically for creating virtual reality sound.

The “Works” 3D Audio plugin for Pro ToolsAn illustration of famous technology and tools for game composers - section of the article by video game music composer Winifred Phillips.

This article explores the 3D rendering technology of the G’Audio Works plugin, which supports multi-channel, object-based and ambisonic spatialization within the Pro Tools application.

The Steam Audio Software Development Kit

This article focuses on the spatialized audio solution for VR developed by the famous Steam software distribution platform.  Available as a free download without any royalty requirements, Steam Audio is designed to assist both Unity and Unreal developers in creating and implementing spatialized audio in their projects.

Facebook 360 Spatial Workstation

The Two Big Ears audio company is known for its 3Dception software enabling audio folks to author spatialized audio for VR applications.  Now that Facebook has purchased the company, the software has been rebranded as the Facebook 360 Spatial Workstation with added compatibility for 360 videos hosted on the Facebook platform. The software is free for everyone to use in their projects. However, the previous plugin compatibility with Unity, Wwise and FMOD is no longer offered for new users.

Google’s Omnitone, the open source project for spatialized sound in VR

This article describes the Omnitone application, developed by Google to combine ambisonic decoding with binaural rendering.  Omnitone was designed to deliver spatialized audio for browser-based experiences and apps designed for Android and iOS.

NVIDIA VRWorks Audio

For Windows games and applications, NVIDIA now offers the VRWorks Audio Software Development Kit for implementing spatialized audio in VR for 64 bit Windows apps.  For developers working in Unreal Engine 4, the VRWorks Audio game engine plugin can be added directly to the UE4 engine, while future plugins are promised for other game and audio platforms.

The Google VR Audio Software Development Kit

This article runs through the capabilities of the Google VR Audio SDK, which was designed for use in developing applications for Google Daydream and Cardboard.  The article includes helpful suggestions for using the audio system to simulate indoor and outdoor environments, advice for creating spatialized sounds that users can easily pinpoint, and tips for ensuring that aural and visual content reinforce each other within the VR space.

Communities & Organizations

An illustration of popular communities and organizations for game composers - section of the article by Winifred Phillips (video game music composer).
Working in virtual reality can be a complicated and intimidating process.  Thankfully, there are plenty of online communities and organizations to help us.  I’ve assembled some of these below:

Conferences

An illustration for the famous conferences available for game composers - section of the article by video game music composer Winifred Phillips.

Finally, here’s a list of the virtual reality conferences, summits and conventions that have featured audio discussions and demonstrations in their past program schedules:

Conclusion

I hope you find the above resources interesting and helpful!  If you have any suggestions for additions to this list, please let me know in the comments section below!

 

Photo of video game composer Winifred Phillips in her game composers production studio.Winifred Phillips is an award-winning video game music composer whose most recent project is the triple-A first person shooter Homefront: The Revolution. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Follow her on Twitter @winphillips.

Resources For Video Game Music Composers

Video game music composer Winifred Phillips, at work in her music production studio.

By Winifred Phillips | Contact | Follow

I’m pleased to announce that my book, A Composer’s Guide to Game Music, is now available its new paperback edition! I’m excited that my book has done well enough to merit a paperback release, and I’m looking forward to getting to know a lot of new readers!  The paperback is much lighter and more portable than the hardcover.  Here’s a view of the front and back covers of the new paperback edition of my book (click the image for a bigger version if you’d like to read the back cover):

award-winning video game music composer Winifred Phillips' book, A Composer's Guide to Game Music, is now available in paperback.

From the article by Winifred Phillips (composer of video game music) - depiction of the book cover of A COMPOSER'S GUIDE TO GAME MUSIC.As you might expect, many aspiring game composers read my book, and I’m honored that my book is a part of their hunt for the best resources to help them succeed in this very competitive business.  When I’m not working in my music studio, I like to keep up with all the great new developments in the game audio field, and I share a lot of what I learn in these articles. Keeping in mind how many of my readers are aspiring composers, I’ve made a point of devoting an article once a year to gathering the top online guidance currently available for newcomers to the game music profession.  In previous years I’ve focused solely on recommendations gleaned from the writings of game audio pros, but this time I’d like to expand that focus to include other types of resources that could be helpful.  Along the way, we’ll be taking a look at some nuggets of wisdom that have appeared on these sites.  So, let’s get started!

Continue reading

Game Music Middleware, Part 3: Fabric

Middleware-Blackboard

Welcome back to my series of blogs that collect some tutorial resources about game music middleware for the game music composer.  I had initially intended to publish two blog entries on this subject, focusing on the most popular audio middleware solutions: Wwise and FMOD.  However, since the Fabric audio middleware has been making such a splash in the game audio community, I thought I’d extend this series to include it.  If you’d like to read the first two blog entries in this series, you can find them here:

Game Music Middleware, Part 1: Wwise

Game Music Middleware, Part 2: FMOD

Fabric is developed by Tazman Audio for the Unity game engine (which enables game development for consoles, PCs, mobile devices such as iOS and Android, and games designed to run within a web browser).  Here’s a Unity game engine overview produced by Unity Technologies:

The Fabric middleware is designed to expand the audio capabilities of the Unity game engine.  The complete product manual for the Fabric middleware is available online.  The video tutorials that I’m featuring below were created by two game audio professionals who have very generously walked us through the use of the software.  If you’d like a more nuts-and-bolts overview of the software features of Fabric, you can find it here.

The first video was shot in 2013 during the Konsoll game development conference in Norway, and gives an overview of the general use of Fabric in game audio. The speaker, Jory Prum, is an accomplished game audio professional whose game credits include The Walking Dead, The Wolf Among Us, Broken Age, SimCity 4, Star Wars: Knights of the Old Republic, and many more.

Making a great sounding Unity game using Fabric

border-159926_640_white

In the next two-part video tutorial, composer Anastasia Devana has expanded on her previous instructional videos about FMOD Studio, focusing now on recreating the same music implementation strategies and techniques using the Fabric middleware in Unity.  Anastasia Devana is an award-winning composer whose game credits include the recently released puzzle game Synergy and the upcoming roleplaying game Anima – Gate of Memories.

Fabric and Unity: Adaptive Music in Angry Bots – Part 1

Fabric and Unity: Adaptive Music in Angry Bots – Part 2

MIDI in Wwise for the Game Music Composer: Peggle Blast

PeggleBlastBanner

In a previous blog post, we took a look at a few tutorial resources for the latest version of the Wwise audio middleware.  One of the newest innovations in the Wwise software package is a fairly robust MIDI system.  This system affords music creators and implementers the opportunity to avail themselves of the extensive adaptive possibilities of the MIDI format from within the Wwise application.  Last month, during the Game Developers Conference in the Moscone Center in San Francisco, some members of the PopCap audio development team presented a thorough, step-by-step explanation of the benefits of this MIDI capability for one of their latest projects, Peggle Blast.  Since my talk during the Audio Bootcamp at GDC focused on interactive music and MIDI (with an eye on the role of MIDI in both the history and future of game audio development), I thought that we could all benefit from a summation of some of the ideas discussed during the Peggle Blast talk, particularly as they relate to dynamic MIDI music in Wwise.  In this blog, I’ve tried to convey some of the most important takeaways from this GDC presentation.

PeggleBlastSession

“Peggle Blast: Big Concepts, Small Project” was presented on Thursday, March 5th by three members of the PopCap audio team: technical sound designer RJ Mattingly, audio lead Jaclyn Shumate, and senior audio director Guy Whitmore.  The presentation began with a quote from Igor Stravinsky:

The more constraints one imposes, the more one frees oneself, and the arbitrariness of the constraint serves only to maintain the precision of the execution.

This idea became a running theme throughout the presentation, as the three audio pros detailed the constraints under which they worked, including:

  1. A 5mb memory limit for all audio assets
  2. Limited CPU
  3. 2.5mb memory allocation for the music elements

These constraints were a result of the mobile platforms (iOS and Android) for which Peggle Blast had been built.  For this reason, the music team focused their attention on sounds that could convey lots of emotion while also maintaining a very small file size.  Early experimentation with tracks structured around the use of a music box instrument led the team to realize that they still needed to replicate the musical experience from the full-fledged console versions of the game.  A simple music-box score was too unsatisfying, particularly for players who were familiar with the music from the previous installments in the franchise.  With that in mind, the team concentrated on very short orchestral samples taken from the previous orchestral session recordings for Peggle 2.  Let’s take a look at a video from those orchestral sessions:

Using those orchestral session recordings, the audio team created custom sample banks that were tailored specifically to the needs of Peggle Blast, focusing on lots of very short instrument articulations and performance techniques including:

  1. pizzicato
  2. marcato
  3. staccato
  4. mallets

A few instruments (including a synth pad and some orchestral strings) were edited to loop so that extended note performances became possible, but the large majority of instruments remained brief, punctuated sounds that did not loop.  These short sounds were arranged into sample banks in which one or two note samples would be used per octave of instrument range, and note tracking would transpose the sample to fill in the rest of the octave.  The sample banks consisted of a single layer of sound, which meant that the instruments did not adjust their character depending on dynamics/velocity.  In order to make the samples more musically pleasing, the built-in digital signal processing capability of Wwise was employed by way of a real-time reverb bus that allowed these short sounds to have more extended and natural-sounding decay times.

wwise-logo-empowers

The audio team worked with a beta version of Wwise 2014 during development of Peggle Blast, which allowed them to implement their MIDI score into the Unity game engine.  The composer, Guy Whitmore, composed the music in a style consisting of whimsically pleasant, non-melodic patterns that were structured into a series of chunks.  These chunks could be triggered according to the adaptive system in Peggle Blast, wherein the music went through key changes (invariably following the circle of fifths) in reaction to the player’s progress.  To better see how this works, let’s watch an example of some gameplay from Peggle Blast:

As you can see, very little in the way of a foreground melody existed in this game.  In the place of a melody, foreground musical tones would be emitted when the Peggle ball hit pegs during its descent from the top of the screen.  These tones would follow a predetermined scale, and would choose which type of scale to trigger (major, natural minor, harmonic minor, or mixolydian) depending on the key in which the music was currently playing.  Information about the key was dropped into the music using markers that indicated where key changes took place, so that the Peggle ball would always trigger the correct type of scale at any given time.  The MIDI system did not have to store unique MIDI data for scales in every key change, but would instead calculate the key transpositions for each of the scale types, based on the current key of the music that was playing.

The presentation ended with an emphasis on the memory savings and flexibility afforded by MIDI, and the advantages that MIDI presents to game composers and audio teams.  It was a very interesting presentation!  If you have access to the GDC Vault, you can watch a video of the entire presentation online.  Otherwise, there are plenty of other resources on the music of Peggle Blast, and I’ve included a few below:

Inside the Music of Peggle Blast – An Interview with Audio Director Guy Whitmore

Peggle Blast!  Peg Hits and the Music System, by RJ Mattingly

Real-Time Synthesis for Sound Creation in Peggle Blast, by Jaclyn Shumate

PopCap’s Guy Whitmore Talks Musical Trials And Triumphs On Peggle Blast

 

GameSoundCon Industry Survey Results

GameSoundCon

As the GameSoundCon conference draws closer, I thought I’d talk a little bit about the Game Audio Industry Survey that was designed by GameSoundCon Executive Producer Brian Schmidt.  The survey was prepared in response to the broader “Annual Game Developer Salary Survey” offered by industry site Gamasutra.  Since the Gamasutra survey suffered from skewed results for game audio compared to other game industry sectors (owing to lower participation from the game audio community), Schmidt set out to obtain more reliable results by adopting a different approach.

Instead of focusing on the yearly salaries/earnings of audio professionals, the survey concentrated on the money generated by the music/sound of individual projects. Each respondent could fill out the survey repeatedly, entering data for each game project that the respondent had completed during the previous year.  The final results of the survey are meant to reflect how game audio is treated within different types of projects, and the results are quite enlightening, and at times surprising.

GSC-SurveyThe financial results include both small-budget indie games from tiny teams and huge-budget games from behemoth publishers, so there is a broad range in those results.  Since this is the first year that the GameSoundCon Game Audio Industry Survey has been conducted, we don’t yet have data from a previous year with which to compare these results, and it might be very exciting to see how the data shifts if the survey is conducted again in 2015.

Some very intriguing data comes from the section of the survey that provides a picture of who game composers are and how they work.  According to the survey, the majority of game composers are freelancers, and 70% of game music is performed by the freelance composer alone.  56% of composers are also acting as one-stop-shops for music and sound effects, likely providing a good audio solution for indie teams with little or no audio personnel of their own.

A surprising and valuable aspect of the survey is to be found in the audio middleware results, which show that the majority of games use either no audio middleware at all, or opt for custom audio tools designed by the game developer.  This information is quite new, and could be tremendously useful to composers working in the field.  While we should all make efforts to gain experience with audio middleware such as FMOD and Wwise, we might keep in mind that there may not be as many opportunities to practice those skills as had been previously anticipated.  Again, this data might be rendered even more meaningful by the results of the survey next year (if it is repeated), to see if commercial middleware is making inroads and becoming more popular over time.

Expanding upon this subject, the survey reveals that only 22% of composers are ever asked to do any kind of music integration (in which the composer assists the team in implementing music files into their game). It seems that for the time being, this task is still falling firmly within the domain of the programmers on most game development teams.

The survey was quite expansive and fascinating, and I’m very pleased that it included questions about both middleware and integration.  If GameSoundCon runs the survey again next year, I’d love to see the addition of some questions about what type of interactivity composers may be asked to introduce into their musical scores, how much of their music is composed in a traditionally linear fashion, and what the ratio of interactive/adaptive to linear music might be per project.  I wrote rather extensively on this subject in my book, and since I’ll also be giving my talk at GameSoundCon this year about composing music for adaptive systems, I’d be very interested in such survey results!

The GameSoundCon Game Audio Industry Survey is an invaluable resource, and is well worth reading in its entirety.  You’ll find it here.  I’ll be giving my talk on “Advanced Composition Techniques for Adaptive Systems” at GameSoundCon at the Millennium Biltmore Hotel in Los Angeles on Wednesday, October 8th.

Many thanks to Brian Schmidt / GameSoundCon for preparing this excellent survey!