Understanding Audio in VR – A Game Music Composer’s Resource Guide

Video game music composer Winifred Phillips working in her game composers production studio.

By Winifred Phillips | Contact | Follow

When I’m not at work in my studio making music for games, I like to keep up with new developments in the field of interactive entertainment, and I’ll often share what I learn here in these articles.  Virtual reality is an awesome subject for study for a video game composer, and several of my recent projects have been in the world of VR.  Since I’m sure that most of us are curious about what’s coming next in virtual reality, I’ve decided to devote this article to a collection of educational resources.  I’ve made a point of keeping our focus general here, with the intent of understanding the role of audio in VR and the best resources available to audio folks.  As a component of the VR soundscape, our music must fit into the entire matrix of aural elements, so we’ll spend this article learning about what goes into making expert sound for a virtual reality experience. Let’s start with a few articles that discuss methods and techniques for VR audio practitioners.

Methods and techniques

3D Audio formats for VRAn illustration of popular methods and techniques for game composers - section from the article by Winifred Phillips, video game music composer.

3D Sound Labs takes us through the three most popular sound formats for spatial audio in VR: Multi-channel, Object-based, and Ambisonic.

How 3D Spatialized Audio Bottlenecks Virtual Reality Video

VR audio can consume enormous computational resources, resulting in a resource war between audio and video content.  This article discusses how audio demands in VR consume memory bus bandwidth, and proposes a short-cut that manipulates the frequency response of the audio content to enhance spatialization while consuming fewer resources.

Adapting Your DAW for VR Audio

A VR sound editor discusses his methods and workflow for virtual reality in this article.  Topics include capturing audio in the ambisonic format, and customizing the Pro Tools environment for spatialized audio projects.

Simple spatial audio for beginners

This article summarizes the available entry-level techniques and equipment that can help a newcomer jump into the world of spatial audio for virtual reality.

An audio post production house shares techniques for VR Audio

In this article, we learn about some of the top equipment and methods used by the VR-focused division of an experienced audio post production facility.

Technology and tools

Now that we’ve garnered some insight about what goes into creating VR audio, let’s take a look at some great technology and tools designed specifically for creating virtual reality sound.

The “Works” 3D Audio plugin for Pro ToolsAn illustration of famous technology and tools for game composers - section of the article by video game music composer Winifred Phillips.

This article explores the 3D rendering technology of the G’Audio Works plugin, which supports multi-channel, object-based and ambisonic spatialization within the Pro Tools application.

The Steam Audio Software Development Kit

This article focuses on the spatialized audio solution for VR developed by the famous Steam software distribution platform.  Available as a free download without any royalty requirements, Steam Audio is designed to assist both Unity and Unreal developers in creating and implementing spatialized audio in their projects.

Facebook 360 Spatial Workstation

The Two Big Ears audio company is known for its 3Dception software enabling audio folks to author spatialized audio for VR applications.  Now that Facebook has purchased the company, the software has been rebranded as the Facebook 360 Spatial Workstation with added compatibility for 360 videos hosted on the Facebook platform. The software is free for everyone to use in their projects. However, the previous plugin compatibility with Unity, Wwise and FMOD is no longer offered for new users.

Google’s Omnitone, the open source project for spatialized sound in VR

This article describes the Omnitone application, developed by Google to combine ambisonic decoding with binaural rendering.  Omnitone was designed to deliver spatialized audio for browser-based experiences and apps designed for Android and iOS.

NVIDIA VRWorks Audio

For Windows games and applications, NVIDIA now offers the VRWorks Audio Software Development Kit for implementing spatialized audio in VR for 64 bit Windows apps.  For developers working in Unreal Engine 4, the VRWorks Audio game engine plugin can be added directly to the UE4 engine, while future plugins are promised for other game and audio platforms.

The Google VR Audio Software Development Kit

This article runs through the capabilities of the Google VR Audio SDK, which was designed for use in developing applications for Google Daydream and Cardboard.  The article includes helpful suggestions for using the audio system to simulate indoor and outdoor environments, advice for creating spatialized sounds that users can easily pinpoint, and tips for ensuring that aural and visual content reinforce each other within the VR space.

Communities & Organizations

An illustration of popular communities and organizations for game composers - section of the article by Winifred Phillips (video game music composer).
Working in virtual reality can be a complicated and intimidating process.  Thankfully, there are plenty of online communities and organizations to help us.  I’ve assembled some of these below:

Conferences

An illustration for the famous conferences available for game composers - section of the article by video game music composer Winifred Phillips.

Finally, here’s a list of the virtual reality conferences, summits and conventions that have featured audio discussions and demonstrations in their past program schedules:

Conclusion

I hope you find the above resources interesting and helpful!  If you have any suggestions for additions to this list, please let me know in the comments section below!

 

Photo of video game composer Winifred Phillips in her game composers production studio.Winifred Phillips is an award-winning video game music composer whose most recent project is the triple-A first person shooter Homefront: The Revolution. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Follow her on Twitter @winphillips.

VRDC 2017 takeaways: VR music for the game composer

Video game music composer Winifred Phillips, at work in her music production studio - from the article about music for virtual reality / VR.The Game Developers Conference is always an awesome opportunity for game audio experts to learn and share experiences.  I’ve given presentations at GDC for a few years now, and I’m always excited to hear about what’s new and notable in game audio.  This year, the hot topic was virtual reality.  In fact, the subject received its own dedicated sub-conference that took place concurrently with the main GDC show.  The VRDC (Virtual Reality Developers Conference) didn’t focus particularly on the audio and music side of VR, but there were a couple of notable talks on that subject.  In this article, let’s take a look at some of the more intriguing VR game music takeaways from those two talks.  Along the way, I’ll also share some of my related experience as the composer of the music of the Dragon Front VR game for the Oculus Rift (pictured above).

Inside and outside

The talks we’ll be discussing in this article are entitled “Audio Adventures in VR Worlds” and “The Sound Design of Star Wars: Battlefront VR.”  Here’s a common issue that popped up in both talks:

An illustration of music in the popular VR platform, from the article by Winifred Phillips (video game composer).Where should video game music be in a VR game?  Should it feel like it exists inside the VR world, weaving itself into the immersive 3D atmosphere surrounding the player?  Or should it feel like it’s somehow outside of the VR environment and is instead coasting on top of the experience, being conveyed directly to the player?  The former approach suggests a spacious and expansive musical soundscape, and the latter would feel much closer and more personal.  Is one of these approaches more effective in VR than the other?  Which choice is best?

Continue reading

Resources For Video Game Music Composers

Video game music composer Winifred Phillips, at work in her music production studio.

By Winifred Phillips | Contact | Follow

I’m pleased to announce that my book, A Composer’s Guide to Game Music, is now available its new paperback edition! I’m excited that my book has done well enough to merit a paperback release, and I’m looking forward to getting to know a lot of new readers!  The paperback is much lighter and more portable than the hardcover.  Here’s a view of the front and back covers of the new paperback edition of my book (click the image for a bigger version if you’d like to read the back cover):

award-winning video game music composer Winifred Phillips' book, A Composer's Guide to Game Music, is now available in paperback.

From the article by Winifred Phillips (composer of video game music) - depiction of the book cover of A COMPOSER'S GUIDE TO GAME MUSIC.As you might expect, many aspiring game composers read my book, and I’m honored that my book is a part of their hunt for the best resources to help them succeed in this very competitive business.  When I’m not working in my music studio, I like to keep up with all the great new developments in the game audio field, and I share a lot of what I learn in these articles. Keeping in mind how many of my readers are aspiring composers, I’ve made a point of devoting an article once a year to gathering the top online guidance currently available for newcomers to the game music profession.  In previous years I’ve focused solely on recommendations gleaned from the writings of game audio pros, but this time I’d like to expand that focus to include other types of resources that could be helpful.  Along the way, we’ll be taking a look at some nuggets of wisdom that have appeared on these sites.  So, let’s get started!

Continue reading

Video game music systems at GDC 2017: tools and tips for composers

Photo of video game composer Winifred Phillips, working in her music production studio on the music of the SimAnimals video game.

By video game composer Winifred Phillips | Contact | Follow

Welcome back to this three article series that’s bringing together the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers explored discoveries they’d made while creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to broaden our viewpoint and gain a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. We’ve been looking at five interactive music systems discussed in these five GDC 2017 presentations:

In the first article, we examined the basic nature of these interactive systems. In the second article, we contemplated why those systems were used, with some of the inherent pros and cons of each system discussed in turn.  So now, let’s get into the nitty gritty of tools and tips for working with such interactive music systems.  If you haven’t read parts one and two of this series, please go do so now and then come back:

  1. Video game music systems at GDC 2017: what are composers using?
  2. Video game music systems at GDC 2017: pros and cons for composers

Ready?  Great!  Here we go!

Continue reading

Video game music systems at GDC 2017: pros and cons for composers

Video game composer Winifred Phillips, pictured in her music production studio working on the music of LittleBigPlanet 2 Cross Controller

By Winifred Phillips | Contact | Follow

Welcome back to our three article series dedicated to collecting and exploring the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers shared ideas they’d developed in the process of creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to cultivate a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. In the first article, we looked at the basic nature of five interactive music systems discussed in these five GDC 2017 presentations:

If you haven’t read part one of this article series, please go do that now and come back.

Okay, so let’s now contemplate some simple but important questions: why were those systems used?  What was attractive about each interactive music strategy, and what were the challenges inherent in using those systems?

Continue reading

Video game music systems at GDC 2017: what are composers using?

By video game music composer Winifred Phillips | Contact | Follow

Video game composer Winifred Phillips, presenting at the Game Developers Conference 2017.The 2017 Game Developers Conference could be described as a densely-packed deep-dive exploration of the state-of-the-art tools and methodologies used in modern game development.  This description held especially true for the game audio track, wherein top experts in the field offered a plethora of viewpoints and advice on the awesome technical and artistic challenges of creating great sound for games. I’ve given GDC talks for the past three years now (see photo), and every year I’m amazed at the breadth and diversity of the problem-solving approaches discussed by my fellow GDC presenters.  Often I’ll emerge from the conference with the impression that we game audio folks are all “doing it our own way,” using widely divergent strategies and tools.

This year, I thought I’d write three articles to collect and explore the ideas that were discussed in five different GDC audio talks.  During their presentations, these five speakers all shared their thoughts on best practices and methods for instilling interactivity in modern game music.  By absorbing these ideas side-by-side, I thought we might gain a sense of the “bigger picture” when it comes to the current leading-edge thinking for music interactivity in games. In the first article, we’ll look at the basic nature of these interactive systems.  We’ll devote the second article to the pros and cons of each system, and in the third article we’ll look at tools and tips shared by these music interactivity experts. Along the way, I’ll also be sharing my thoughts on the subject, and we’ll take a look at musical examples from some of my own projects that demonstrate a few ideas explored in these GDC talks:

So, let’s begin with the most obvious question.  What kind of interactive music systems are game audio folks using lately?

Continue reading

Variation for the video game composer: the music of Little Lords of Twilight

Pictured: video game music composer Winifred Phillips at the BKOM booth during GDC 2017.

Since one of my most recent projects, Little Lords of Twilight, became available worldwide earlier this year and was recently greenlit on the famous Steam platform, I thought I’d write this article to share some of my creative and technical process in composing the music for this game. In particular, this project presents a great opportunity to look at how compositional variation (as we understand it from music theory) can be useful for the structure of interactive music.

Developed by BKOM Studios, Little Lords of Twilight won a Best in Play Award at GDC 2017, a Best Designed Mobile App Platinum Award from the BMA Awards, a Communicator Award for Best Mobile App, and has appeared on numerous “Best of” lists, including those published by PocketGamer, Explore Gadgets, and GameInOnline.  As a player-versus-player turn-based strategy game, Little Lords of Twilight offers a unique gameplay mechanic influenced by the in-game passage of time.  Day and night cycles dramatically alter your character’s appearance and abilities. Depending on whether it is currently day or night in the game, your character will have access to a completely different complement of awesome skills and spells to wield on the battlefield.

Continue reading