VRDC 2017 takeaways: VR music for the game composer

Video game music composer Winifred Phillips, at work in her music production studio - from the article about music for virtual reality / VR.The Game Developers Conference is always an awesome opportunity for game audio experts to learn and share experiences.  I’ve given presentations at GDC for a few years now, and I’m always excited to hear about what’s new and notable in game audio.  This year, the hot topic was virtual reality.  In fact, the subject received its own dedicated sub-conference that took place concurrently with the main GDC show.  The VRDC (Virtual Reality Developers Conference) didn’t focus particularly on the audio and music side of VR, but there were a couple of notable talks on that subject.  In this article, let’s take a look at some of the more intriguing VR game music takeaways from those two talks.  Along the way, I’ll also share some of my related experience as the composer of the music of the Dragon Front VR game for the Oculus Rift (pictured above).

Inside and outside

The talks we’ll be discussing in this article are entitled “Audio Adventures in VR Worlds” and “The Sound Design of Star Wars: Battlefront VR.”  Here’s a common issue that popped up in both talks:

An illustration of music in the popular VR platform, from the article by Winifred Phillips (video game composer).Where should video game music be in a VR game?  Should it feel like it exists inside the VR world, weaving itself into the immersive 3D atmosphere surrounding the player?  Or should it feel like it’s somehow outside of the VR environment and is instead coasting on top of the experience, being conveyed directly to the player?  The former approach suggests a spacious and expansive musical soundscape, and the latter would feel much closer and more personal.  Is one of these approaches more effective in VR than the other?  Which choice is best?

These two concepts share a lot in common with the traditional categories of diegetic and non-diegetic music in entertainment media.  Diegetic music exists inside the fictional world, perceived by the characters within it, whereas non-diegetic music is inaudible to the characters and only exists for the benefit of the audience.  VR presents an interesting twist to this usually straightforward dichotomy.  When the entertainment experience is doing everything in its power to make us forget that we’re an audience, to the point where we achieve a sense of complete presence within the fictional world… what role does non-diegetic music play then?  If we can now consider ourselves as characters in the story, how do we hear music that story characters aren’t supposed to hear?

An illustration of the game PlayStation VR Worlds, from the article by popular video game music composer Winifred Phillips“VR goes beyond picture sync. It’s about sync of the world,” says music producer Joe Thwaites of Sony Interactive Entertainment Europe.  In his talk about the music and sound of the game PlayStation VR Worlds, Thwaites explores the relationship between music and the VR environment. “The congruency between audio and visuals is key in maintaining that idea of believability,” Thwaites asserts, “which in turn makes immersiveness, and in turn makes presence.”  In virtual reality development, the term ‘presence’ denotes the sensation of actually existing inside the virtual environment.  According to Thwaites, a strong believable relationship between the aural and visual worlds can contribute to a more satisfying VR experience.

Music inside the world

An illustration of the Ocean Descent portion of famous PlayStation VR Worlds game, from the article by Winifred Phillips, video game composer.As an example, Thwaites describes an interactive music implementation that he integrated into the ‘Ocean Descent’ section of PlayStation VR Worlds.  In this portion of the game, Thwaites pulled the otherwise non-diegetic musical score more fully into the immersive world by creating an illusion that the in-game objects were reacting to the musical notes.  “There’s a part called The Jellyfish Cave, where you descend into this sea of jellyfish,” Thwaites describes.  “You get this 2D music,” he adds, “which bypasses the 3D audio plugin, so it goes straight to your ears.”  In other words, the music is recorded in a traditionally stereo mix and the output is fed directly to the player’s headphones without bothering with any spatial positioning in the virtual world.  “Then, as you look around, these jellyfish light up as you look directly at them,” Thwaites goes on, “and they emit a tone in 3D in space so the music tone stays where it is in the world.”  So, these tones have been attached to specific jellyfish in the virtual world, spatially positioned to emanate from those locations, as if special portions of the non-diegetic score have suddenly leapt into the VR world and taken up residence there. “And that has this really nice effect of creating this really immersive and magical moment which is really unique to VR,” Thwaite remarks.

So this method served to help non-diegetic music feel more natural within the VR environment.  But what happens when pure non-diegetic music is an absolutely necessity?

Music outside the world

An illustration of the famous Star Wars Battlefront VR game, from the article by video game music composer Winifred Phillips.In the game Star Wars Battlefront Rogue One X-Wing VR Mission, the audio team at Criterion Games were tasked with creating an authentic audio experience in a virtual reality environment dedicated to the eternally famous and popular Star Wars franchise.  In this case, according to audio lead Jay Steen, pure non-diegetic music was a must.  “Non-diegetic means not from a source in the scene. This is how most movies and flatscreen games handle the music. So the music plays through the direct out straight to the player’s ears and we were worried from what we’d heard about non-diegetic music that it would distract from immersion,” Steen confesses. “But we actually found the opposite. Maybe that’s because you can’t have a Star Wars story without the music. You don’t feel like you’re in Star Wars until the music kicks in.”  According to Steen, the non-diegetic music worked in this circumstance because the audio team was careful to avoid repetition in the musical score.  “We didn’t reuse or loop cues that much, and due to the linear structure of the mission we could kind of get away with this,” Steen points out. “We think that helps to not break immersion.”

My perspective on using non-diegetic music in VR:

Famous video game composer Winifred Phillips works in her music production studio.Sometimes non-diegetic music can be introduced into a VR game, and then quickly transformed into diegetic music within the immersive environment in order to enhance player presence.  In my musical score for the Dragon Front game for Oculus Rift, I composed a dramatic choral track for the opening main theme of the game.  During the game’s initial logo sequence, the music is channeled directly to the player’s ears without any spatial positioning.  However, this changes as soon as the player fully enters the initial environment (wherein the player navigates menus and prepares to enter matches).  Logo art from the popular Dragon Front game, featured in the article by video game music composer Winifred PhillipsOnce the logo sequence has completed, the music makes a quick transition, from a full-bodied direct stereo mix to the player’s headphones, to a spatially localized narrow mix located to the player’s lower right.  Upon turning, players see that the music is now coming from a battered radio, which the player is free to turn on and off.  The music is now fully diegetic, existing inside the game’s fictional world.  Here’s a video showing this sequence in action:

Music inside and outside

The logo for the VR Luge portion of the popular PlayStation VR Worlds game, from the article by Winifred Phillips (video game music composer).While non-diegetic music can be tricky in VR, sometimes its an important part of the overall aesthetic.  Plus, there can be ways to integrate non-diegetic music into the spatial environment.  Joe Thwaites of Sony Europe describes an interesting combination of diegetic and non-diegetic music that was integrated into the ‘VR Luge’ section of the PlayStation VR Worlds game.  In this gameplay sequence, players ride feet-first on a luge that’s racing downhill amidst heavy vehicle traffic.  The experience was designed to be a heart-stopping thrill ride.  “So one of the experiments we did around the synchronization of the world was using a combination of diegetic and non-diegetic music to build tension as you zoomed down the hill,” Thwaites describes. “We used 3D car radios to introduce elements of percussion into the 2D soundtrack that was playing.”  In the musical score for this sequence, the non-diegetic music presented a purely percussive rhythm, but as the player passed by other cars, the music would change.  “So as you passed a car with a radio playing, an element of that 3D music would transition from the car into the 2D soundtrack.”  In this way, the in-game radio music would briefly become a part of the game’s non-diegetic score, while still conveying spatial positioning inside the 3D world.

So in these examples from PlayStation VR Worlds and Star Wars Battlefront Rogue One X-Wing VR Mission, we see that audio teams grapple constantly with the contrasting natures of diegetic and non-diegetic music.  While it seems as though non-diegetic music has been relegated to a very traditional, non-spatially localized delivery, this may not always be the case.  Jay Steen of Criterion Games spent some time considering the possibility of delivering the non-diegetic music of his Star Wars game with a more enveloping spatial texture.  “We did do a quick experiment on it, and we found that it’s like having an orchestra sitting around you,” Steen says. “We didn’t want to evoke you sitting in the middle of an orchestral recording.  We just wanted it to sound like the movie.”  That being said, Steen doesn’t rule out the possibility of a more spatially-interesting mix for music in the future, including the use of ambisonic recordings for non-diegetic musical scores. “Ambisonic recordings of orchestras for example,” Steen speculates, “I think there’s something fun there. We haven’t experimented with it anymore than that, but yeah, definitely, we’d want to try.”

Conclusion

So this concludes our look at two presentations from GDC 2017 that focused on issues that complicate music creation and implementation in virtual reality.  I hope you’ve found this interesting, and please feel free to leave a comment in the space below!

 

Photo of video game composer Winifred Phillips in her music production studio.Winifred Phillips is an award-winning video game music composer whose most recent project is the triple-A first person shooter Homefront: The Revolution. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Follow her on Twitter @winphillips.

Video game composers can make you smarter! (The music of Dragon Front) Pt. 2

Pictured: Winifred Phillips (video game music composer) in her studio working on the music of the Dragon Front virtual reality game.

By Winifred Phillips | Contact | Follow

Welcome back to our three-part discussion of how video game composers (such as ourselves) can make strategy gamers smarter!  In these articles, we’re looking at ways in which our music can enhance concentration and tactical decision-making for players engrossed in strategic gameplay.  Along the way, I’ve been sharing my personal experiences as the composer for the Dragon Front strategy game for virtual reality.  Over the course of these articles we’ll be covering three of the top concepts that pertain to the relationship between music and concentration.  In part one, we discussed the concept of ‘music-message congruency,’ so if you haven’t read that article yet, please go check it out and then come back.

Are you back now?  Good!  Let’s move on to the second big technique for increasing the smarts of strategy gamers!

Cognition-enhancing tempo

As video game composers, we create music in a wide variety of tempos designed to support the energy of play and the pacing of the game’s overall design.  From leisurely tracks that accompany unstructured exploration to frenetic pieces that support the most high-stakes combat, our music is planned with expert precision to shape the excitement level of players and keep them motivated as they progress.

Continue reading

Video game composers can make you smarter! (The music of Dragon Front) Pt. 1

Video game composer Winifred Phillips, pictured in her music studio working on the original score for the Dragon Front virtual reality game.

Can video game composers make you smarter?  Well, video gaming can be a pretty cerebral activity, requiring astute problem-solving skills and disciplined concentration in order to excel.  That’s especially true for any game built around strategic and/or tactical gameplay, such as real-time or turn-based strategy, tactical shooters, multiplayer online battle arenas (MOBAs), and online collectible card strategy games.  To succeed in these types of games, players must assess the current situation and formulate a plan that accounts for future developments and variables.  Without this type of tactical forward-thinking gameplay, a gamer has little chance to win.  So, can music enable gamers to think tactically, stay focused and make smart decisions?  Over the next three articles, I’ll try to answer that question, while exploring the role of music in enhancing the concentration of strategic/tactical gamers.

Along the way, we’ll be taking a look at some scholarly research on the subject, consulting the opinions of experts, and I’ll be sharing my experiences creating the music for the recently released Dragon Front strategy game from High Voltage software.  We’ll check out some music tracks I composed for the popular Dragon Front game (pictured at the top of this article), and we’ll discuss methods for supporting and enhancing concentration for strategic/tactical game players.  But first, let’s take a closer look at the Dragon Front game.

Continue reading

VR Game Composer: Music Beyond the Virtual

Photo of video game music composer Winifred Phillips, from the article entitled "VR Game Composer: Music Beyond the Virtual."Welcome to the third installment in our series on the fascinating possibilities created by virtual reality motion tracking, and how the immersive nature of VR may serve to inspire us as video game composers and afford us new and innovative tools for music creation.  As modern composers, we work with a lot of technological tools, as I can attest from the studio equipment that I rely on daily (pictured left). Many of these tools communicate with each other by virtue of the Musical Instrument Digital Interface protocol, commonly known as MIDI – a technical standard that allows music devices and software to interact.

Image depicting VR apps from the article by Winifred Phillips, Game Music Composer.In order for a VR music application to control and manipulate external devices, the software must be able to communicate by way of the MIDI protocol – and that’s an exciting development in the field of music creation in VR!

This series of articles focuses on what VR means for music composers and performers. In previous installments, we’ve had some fun exploring new ways to play air guitar and air drums, and we’ve looked at top VR applications that provide standalone virtual instruments and music creation tools.  Now we’ll be talking about the most potentially useful application of VR for video game music composers – the ability to control our existing music production tools from within a VR environment.

We’ll explore three applications that employ MIDI to connect music creation in VR to our existing music production tools. But first, let’s take a look at another, much older gesture-controlled instrument that in ways is quite reminiscent of these motion-tracking music applications for VR:

Continue reading

VR Game Composer: Music Inside the Machine

Illustration for an article in the VR Game Composer series, written by video game composer Winifred PhillipsWelcome to part two of our ongoing exploration of some interesting possibilities created by the motion tracking capabilities of VR, and how this might alter our creative process as video game composers.

In part one we discussed how motion tracking lets us be awesome air guitarists and drummers inside the virtual space.  In this article, we’ll be taking a look at how the same technology will allow us to make interesting music using more serious tools that are incorporated directly inside the VR environment – musical instruments that exist entirely within the VR ‘machine.’

Our discussion to follow will concentrate on three software applications: Soundscape, Carillon, and Lyra.  Later, in the third article of this ongoing series, we’ll take a look at applications that allow our VR user interfaces to harness the power of MIDI to control some of the top music devices and software that we use in our external production studios. But first, let’s look at the ways that VR apps can function as fully-featured musical instruments, all on their own!

Soundscape

Let’s start with something simple – a step sequencer with a sound bank and signal processing tools, built for the mobile virtual reality experience of the Samsung Gear VR.

Video game composer Winifred Phillips demonstrating the Samsung Gear VR headset during the AES convention in NYC.I got a chance to demo the Samsung Gear VR during the Audio Engineering Society Convention in NYC last year, and while it doesn’t offer the best or most mind-blowing experience in VR (such as what we can experience from products like the famous Oculus Rift), it does achieve a satisfying level of immersion. Plus, it’s great fun!  The Soundscape VR app was built for Samsung Gear VR by developer Sander Sneek of the Netherlands.  It’s a simple app designed to enable users to create dance loops using three instruments from a built-in electro sound library, a pentatonic step sequencer that enables the user to create rhythm and tone patterns within the loops, and a collection of audio signal processing effects that let the user warp and mold the sounds as the loops progress, adding variety to the performance.

Continue reading

VR Game Composer: Music in the Air

Illustration for the article by game music composer Winifred Phillips, entitled "VR Game Composer: Music in the Air"Since I’ve been working recently on music for a Virtual Reality project (more info in the coming months), I’ve been thinking a lot about VR technology and its effect on the creative process.  Certainly, VR is going to be a great environment in which to be creative and perform tasks and skills with enhanced focus, according to this article from the VR site SingularityHub.  I’ve written in this blog before about the role that music and sound will play in the Virtual Reality gaming experience.  It’s clear that music will have an impact on the way in which we experience VR, not only during gaming experiences, but also when using the tools of VR to create and be productive.  With that in mind, let’s consider if the opposite statement may also be true – will VR impact the way in which we experience music, not only as listeners, but also as video game composers?

Game composer Winifred Phillips tries out the VR experience of Google Cardboard (pictured here in her music production studio).Simple VR technologies like the popular Google Cardboard headset can be a lot of fun – as I personally experienced recently (photo to the left).  However, they offer only the rudimentary visual aspects, which omits some of the most compelling aspects of the VR experience.  When motion tracking (beyond simple head movement) is added to the mix, the potential of VR explodes.  Over the next three articles, we’ll be exploring some interesting possibilities created by the motion tracking capabilities of VR, and how this might alter our creative process.  In the first article, we’ll have some fun exploring new ways to play air guitars and air drums in the VR environment. In the second article, we’ll take a look at ways to control virtual instruments and sound modules that are folded into the VR software.  And finally, in the third article we’ll explore the ways in which VR motion tracking is allowing us to immersively control our existing real-world instruments using MIDI. But first, let’s take a look at the early days of VR musical technology!

Continue reading

How Music Can Intensify Video Games: Part One

Tension has always served a crucial role in music composition and performance.  My next two blog articles will focus on how music works to shape tension and intensity in a dramatic presentation such as a video game.

Composer Winifred Phillips, working on music for Homefront: The Revolution in her music studio.During these blogs, we’ll be consulting with lots of top experts on the subject, and I’ll be sharing my experiences in regards to the tension-filled music that I composed as a member of the music team of Homefront: The Revolution – an open world, triple-A first person shooter game that was just released by Deep Silver/Dambuster Studios.  Along the way we’ll check out some excerpts from music tracks I composed (in my music production studio, pictured right) for Homefront: The Revolution, and we’ll talk about multiple techniques to build tension in a piece of music, with the goal of inciting the most emotional intensity possible in our audience. With that in mind, let’s start things off with a great quote from philosopher Henry David Thoreau:

“The fibers of all things have their tension and are strained like the strings of an instrument.”

Image illustrating anxiety (from the article by award-winning video game composer Winifred Phillips)Thoreau not only saw the connection between music and tension, but also made a good point about the stresses and strains in our lives – we all possess our own inner emotional pressure. The more fervently we pursue our goals and struggles, the higher the tension grows. Taken to the extreme, it can feel as though our insides are wound up as taut as clockworks. As game composers, our job has always been to induce players to care about what’s happening in the game, and that includes inciting and escalating the nervous anxiety associated with an awesome investment of emotion and empathy.  So let’s explore the best ways we can make players feel the tension!

Continue reading