Understanding Audio in VR – A Game Music Composer’s Resource Guide

Video game music composer Winifred Phillips working in her game composers production studio.

By Winifred Phillips | Contact | Follow

When I’m not at work in my studio making music for games, I like to keep up with new developments in the field of interactive entertainment, and I’ll often share what I learn here in these articles.  Virtual reality is an awesome subject for study for a video game composer, and several of my recent projects have been in the world of VR.  Since I’m sure that most of us are curious about what’s coming next in virtual reality, I’ve decided to devote this article to a collection of educational resources.  I’ve made a point of keeping our focus general here, with the intent of understanding the role of audio in VR and the best resources available to audio folks.  As a component of the VR soundscape, our music must fit into the entire matrix of aural elements, so we’ll spend this article learning about what goes into making expert sound for a virtual reality experience. Let’s start with a few articles that discuss methods and techniques for VR audio practitioners.

Methods and techniques

3D Audio formats for VRAn illustration of popular methods and techniques for game composers - section from the article by Winifred Phillips, video game music composer.

3D Sound Labs takes us through the three most popular sound formats for spatial audio in VR: Multi-channel, Object-based, and Ambisonic.

How 3D Spatialized Audio Bottlenecks Virtual Reality Video

VR audio can consume enormous computational resources, resulting in a resource war between audio and video content.  This article discusses how audio demands in VR consume memory bus bandwidth, and proposes a short-cut that manipulates the frequency response of the audio content to enhance spatialization while consuming fewer resources.

Adapting Your DAW for VR Audio

A VR sound editor discusses his methods and workflow for virtual reality in this article.  Topics include capturing audio in the ambisonic format, and customizing the Pro Tools environment for spatialized audio projects.

Simple spatial audio for beginners

This article summarizes the available entry-level techniques and equipment that can help a newcomer jump into the world of spatial audio for virtual reality.

An audio post production house shares techniques for VR Audio

In this article, we learn about some of the top equipment and methods used by the VR-focused division of an experienced audio post production facility.

Technology and tools

Now that we’ve garnered some insight about what goes into creating VR audio, let’s take a look at some great technology and tools designed specifically for creating virtual reality sound.

The “Works” 3D Audio plugin for Pro ToolsAn illustration of famous technology and tools for game composers - section of the article by video game music composer Winifred Phillips.

This article explores the 3D rendering technology of the G’Audio Works plugin, which supports multi-channel, object-based and ambisonic spatialization within the Pro Tools application.

The Steam Audio Software Development Kit

This article focuses on the spatialized audio solution for VR developed by the famous Steam software distribution platform.  Available as a free download without any royalty requirements, Steam Audio is designed to assist both Unity and Unreal developers in creating and implementing spatialized audio in their projects.

Facebook 360 Spatial Workstation

The Two Big Ears audio company is known for its 3Dception software enabling audio folks to author spatialized audio for VR applications.  Now that Facebook has purchased the company, the software has been rebranded as the Facebook 360 Spatial Workstation with added compatibility for 360 videos hosted on the Facebook platform. The software is free for everyone to use in their projects. However, the previous plugin compatibility with Unity, Wwise and FMOD is no longer offered for new users.

Google’s Omnitone, the open source project for spatialized sound in VR

This article describes the Omnitone application, developed by Google to combine ambisonic decoding with binaural rendering.  Omnitone was designed to deliver spatialized audio for browser-based experiences and apps designed for Android and iOS.

NVIDIA VRWorks Audio

For Windows games and applications, NVIDIA now offers the VRWorks Audio Software Development Kit for implementing spatialized audio in VR for 64 bit Windows apps.  For developers working in Unreal Engine 4, the VRWorks Audio game engine plugin can be added directly to the UE4 engine, while future plugins are promised for other game and audio platforms.

The Google VR Audio Software Development Kit

This article runs through the capabilities of the Google VR Audio SDK, which was designed for use in developing applications for Google Daydream and Cardboard.  The article includes helpful suggestions for using the audio system to simulate indoor and outdoor environments, advice for creating spatialized sounds that users can easily pinpoint, and tips for ensuring that aural and visual content reinforce each other within the VR space.

Communities & Organizations

An illustration of popular communities and organizations for game composers - section of the article by Winifred Phillips (video game music composer).
Working in virtual reality can be a complicated and intimidating process.  Thankfully, there are plenty of online communities and organizations to help us.  I’ve assembled some of these below:

Conferences

An illustration for the famous conferences available for game composers - section of the article by video game music composer Winifred Phillips.

Finally, here’s a list of the virtual reality conferences, summits and conventions that have featured audio discussions and demonstrations in their past program schedules:

Conclusion

I hope you find the above resources interesting and helpful!  If you have any suggestions for additions to this list, please let me know in the comments section below!

 

Photo of video game composer Winifred Phillips in her game composers production studio.Winifred Phillips is an award-winning video game music composer whose most recent project is the triple-A first person shooter Homefront: The Revolution. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Follow her on Twitter @winphillips.

Resources For Video Game Music Composers

Video game music composer Winifred Phillips, at work in her music production studio.

By Winifred Phillips | Contact | Follow

I’m pleased to announce that my book, A Composer’s Guide to Game Music, is now available its new paperback edition! I’m excited that my book has done well enough to merit a paperback release, and I’m looking forward to getting to know a lot of new readers!  The paperback is much lighter and more portable than the hardcover.  Here’s a view of the front and back covers of the new paperback edition of my book (click the image for a bigger version if you’d like to read the back cover):

award-winning video game music composer Winifred Phillips' book, A Composer's Guide to Game Music, is now available in paperback.

From the article by Winifred Phillips (composer of video game music) - depiction of the book cover of A COMPOSER'S GUIDE TO GAME MUSIC.As you might expect, many aspiring game composers read my book, and I’m honored that my book is a part of their hunt for the best resources to help them succeed in this very competitive business.  When I’m not working in my music studio, I like to keep up with all the great new developments in the game audio field, and I share a lot of what I learn in these articles. Keeping in mind how many of my readers are aspiring composers, I’ve made a point of devoting an article once a year to gathering the top online guidance currently available for newcomers to the game music profession.  In previous years I’ve focused solely on recommendations gleaned from the writings of game audio pros, but this time I’d like to expand that focus to include other types of resources that could be helpful.  Along the way, we’ll be taking a look at some nuggets of wisdom that have appeared on these sites.  So, let’s get started!

Continue reading

Video game music systems at GDC 2017: tools and tips for composers

Photo of video game composer Winifred Phillips, working in her music production studio on the music of the SimAnimals video game.

By video game composer Winifred Phillips | Contact | Follow

Welcome back to this three article series that’s bringing together the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers explored discoveries they’d made while creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to broaden our viewpoint and gain a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. We’ve been looking at five interactive music systems discussed in these five GDC 2017 presentations:

In the first article, we examined the basic nature of these interactive systems. In the second article, we contemplated why those systems were used, with some of the inherent pros and cons of each system discussed in turn.  So now, let’s get into the nitty gritty of tools and tips for working with such interactive music systems.  If you haven’t read parts one and two of this series, please go do so now and then come back:

  1. Video game music systems at GDC 2017: what are composers using?
  2. Video game music systems at GDC 2017: pros and cons for composers

Ready?  Great!  Here we go!

Continue reading

Video game music systems at GDC 2017: pros and cons for composers

Video game composer Winifred Phillips, pictured in her music production studio working on the music of LittleBigPlanet 2 Cross Controller

By Winifred Phillips | Contact | Follow

Welcome back to our three article series dedicated to collecting and exploring the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers shared ideas they’d developed in the process of creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to cultivate a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. In the first article, we looked at the basic nature of five interactive music systems discussed in these five GDC 2017 presentations:

If you haven’t read part one of this article series, please go do that now and come back.

Okay, so let’s now contemplate some simple but important questions: why were those systems used?  What was attractive about each interactive music strategy, and what were the challenges inherent in using those systems?

Continue reading

Video game music systems at GDC 2017: what are composers using?

By video game music composer Winifred Phillips | Contact | Follow

Video game composer Winifred Phillips, presenting at the Game Developers Conference 2017.The 2017 Game Developers Conference could be described as a densely-packed deep-dive exploration of the state-of-the-art tools and methodologies used in modern game development.  This description held especially true for the game audio track, wherein top experts in the field offered a plethora of viewpoints and advice on the awesome technical and artistic challenges of creating great sound for games. I’ve given GDC talks for the past three years now (see photo), and every year I’m amazed at the breadth and diversity of the problem-solving approaches discussed by my fellow GDC presenters.  Often I’ll emerge from the conference with the impression that we game audio folks are all “doing it our own way,” using widely divergent strategies and tools.

This year, I thought I’d write three articles to collect and explore the ideas that were discussed in five different GDC audio talks.  During their presentations, these five speakers all shared their thoughts on best practices and methods for instilling interactivity in modern game music.  By absorbing these ideas side-by-side, I thought we might gain a sense of the “bigger picture” when it comes to the current leading-edge thinking for music interactivity in games. In the first article, we’ll look at the basic nature of these interactive systems.  We’ll devote the second article to the pros and cons of each system, and in the third article we’ll look at tools and tips shared by these music interactivity experts. Along the way, I’ll also be sharing my thoughts on the subject, and we’ll take a look at musical examples from some of my own projects that demonstrate a few ideas explored in these GDC talks:

So, let’s begin with the most obvious question.  What kind of interactive music systems are game audio folks using lately?

Continue reading

Video Game Music Production Tips from GDC 2016

Game Composer Winifred Phillips during her game music presentation at the Game Developers Conference 2016I was pleased to give a talk about composing music for games at the 2016 Game Developers Conference (pictured left).  GDC took place this past March in San Francisco – it was an honor to be a part of the audio track again this year, which offered a wealth of awesome educational sessions for game audio practitioners.  So much fun to see the other talks and learn about what’s new and exciting in the field of game audio!  In this blog, I want to share some info that I thought was really interesting from two talks that pertained to the audio production side of game development: composer Laura Karpman’s talk about “Composing Virtually, Sounding Real” and audio director Garry Taylor’s talk on “Audio Mastering for Interactive Entertainment.”  Both sessions had some very good info for video game composers who may be looking to improve the quality of their recordings.  Along the way, I’ll also be sharing a few of my own personal viewpoints on these music production topics, and I’ll include some examples from one of my own projects, the Ultimate Trailers album for West One Music, to illustrate ideas that we’ll be discussing.  So let’s get started!

Continue reading

Interactive Music for the Video Game Composer

Game Composer Winifred Phillips works in her studio on the music of the popular Spore Hero video game As a speaker in the audio track of the Game Developers Conference this year, I enjoyed taking in a number of GDC audio sessions — including a couple of presentations that focused on the future of interactive music in games.  I’ve explored this topic before at length in my book (A Composer’s Guide to Game Music), and it was great to see that the game audio community continues to push the boundaries and innovate in this area! Interactive music is a worthwhile subject for discussion, and will undoubtedly be increasingly important in the future as dynamic music systems become more prevalent in game projects.  With that in mind, in this blog I’d like to share my personal takeaway from two sessions that described very different approaches to musical interactivity. After that, we’ll discuss one of my experiences with interactive music for the video game Spore Hero from Electronic Arts (pictured above).

Musical Intelligence

Baldur BaldurssonPhoto of Baldur Baldursson, the audio director for Icelandic game development studio CCP Games (part of the article by game composer Winifred Phillips) (pictured left) is the audio director for Icelandic game development studio CCP Games, responsible for the EVE Online MMORPG.  Together with Professor Kjartan Olafsson of the Iceland Academy of Arts, Baldursson presented a talk at GDC 2016 on a new system to provide “Intelligent Music For Games.”

Baldursson began the presentation by explaining why an intelligent music system for games can be a necessity.  “We basically want an intelligent music system because we can’t (or maybe shouldn’t really) precompose all of the elements,” Baldursson explains. He describes the conundrum of creating a musical score for a game whose story is still fluid and changeable, and then asserts,  “I think we should find ways of making this better.”

Continue reading