Video game music systems at GDC 2017: pros and cons for composers

Video game composer Winifred Phillips, pictured in her music production studio working on the music of LittleBigPlanet 2 Cross Controller

By Winifred Phillips | Contact | Follow

Welcome back to our three article series dedicated to collecting and exploring the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers shared ideas they’d developed in the process of creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to cultivate a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. In the first article, we looked at the basic nature of five interactive music systems discussed in these five GDC 2017 presentations:

If you haven’t read part one of this article series, please go do that now and come back.

Okay, so let’s now contemplate some simple but important questions: why were those systems used?  What was attractive about each interactive music strategy, and what were the challenges inherent in using those systems?

The Pros and Cons

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).In this discussion of the advantages and disadvantages of musical interactivity, let’s start with the viewpoint of Sho Iwamoto, audio programmer of Final Fantasy XV for Square Enix.  He articulates a perspective on interactive music that’s rarely given voice in the game audio community.  “So first of all,” Iwamoto says, “I want to clarify that the reason we decided to implement interactive music is not to reduce repetition.”

From the article by game composer Winifred Phillips - an illustration of the game Final Fantasy XV.Those of us who have been in the game audio community for years have probably heard countless expert discussions of how crucial it is for video game composers to reduce musical repetition, and how powerful interactivity can be in eliminating musical recurrences in a game.  But for Iwamoto, this consideration is entirely beside the point.  “Repeating music is not evil,” he says. “Of course, it could be annoying sometimes, but everyone loves to repeat their favorite music, and also, repetition makes the music much more memorable.”  So, if eliminating repetition was not at the top of Iwamoto’s list of priorities, then what was?

“We used (musical interactivity) to enhance the user’s emotional experience by playing music that is more suitable to the situation,” Iwamoto explains, also adding that he wanted “to make transitions musical, as much as possible.”  So, if the best advantage of musical interactivity for Iwamoto was an enhanced emotional experience for gamers, then what was the biggest drawback?

For Iwamoto, the most awesome struggle arose from the desire to focus on musicality and melodic content, with the intent to present a traditionally epic musical score that maintained its integrity within an interactive framework. Often, these two imperatives seemed to smash destructively into each other.  “At first it was like a crash of the epic music and the interactive system,” he says.  “How can I make the music interactive while maintaining its epic melodies? Making music interactive could change or even screw up the music itself, or make the music not memorable enough.”

 

My perspective on epic interactive music

A photo of video game composer Winifred Phillips working in her music production studio on the music of LittleBigPlanet Cross Controller.Sho Iwamoto makes a very good point about the difficulty of combining epic musicality with an interactive structure.  For the popular LittleBigPlanet Cross Controller game for Sony Europe, I dealt with a very similar conundrum.  The development team asked me to create an epic orchestral action-adventure track that would be highly melodic but also highly interactive.  Balancing the needs of the interactivity with the needs of an expressive action-adventure orchestral score proved to be very tricky.  I structured the music around a six-layer system of vertical layering, wherein the music was essentially disassembled by the music engine and reassembled in different instrument combinations depending on the player’s progress.  Here’s a nine-minute gameplay video in which this single piece of music mutates and changes to accommodate the gameplay action:


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).Leonard J. Paul’s work on the platformer Vessel also hinged on a vertical layering music system. However, the biggest advantage of the vertical layering music system for Paul was in its ability to adapt existing music into an interactive framework.  From the article by game composer Winifred Phillips - an illustration of the game Vessel.Working with multiple licensing agencies, the development team for Vessel was able to obtain a selection of songs for their game project while it was still early in development.  The songs became rich sources of inspiration for the development team.  “They had made the game listening to those songs so the whole entire game was steeped in that music,” Paul observes.

Nevertheless, the situation also presented some distinct disadvantages.  “The licensing for those ten tracks took eight months,” Paul admits, then he goes on to describe some of the other problems inherent in adapting preexisting music for interactivity.  “It’s really hard to remix someone else’s work so that it has contour yet it stays consistent,” Paul says, “So it doesn’t sound like, oh, I figured out something new in the puzzle or I did something wrong, just because there’s something changing in the music.” In order to make the music convey a single, consistent atmosphere, Paul devoted significant time and energy to making subtle, unnoticeable adjustments to the songs.  “It’s very hard to make your work transparent,” Paul points out.


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).For sound designer Steve Green’s work on the music of the underwater exploration game ABZU, the main advantage of their use of an interactive music system was in the system’s ability to customize the musical content to the progress of the player by calling up location-specific tracks during exploration, without needing the make any significant changes to the content of those music files.  “So its mainly not the fact that we’re changing the music itself as you’re playing it, we’re just helping the music follow you along,” Green explains.  This enabled the music to “keep up with you as you’re playing the game, so it’s still interactive in a sense in that it’s changing along with the player.”

From the article by game composer Winifred Phillips - an illustration of the game ABZU.While this was highly-desirable, it also created some problems when one piece of music ended and another began, particularly if the contrast between the two tracks was steep.  “The dilemma we faced was going in from track one to track two,” Green observes.  For instance, if an action-oriented piece of music preceded a more relaxed musical composition, then “there was a high amount of energy that you just basically need to get in and out of.”

 

My perspective on interactive transitions

Photo of video game composer Winifred Phillips working in her music production studio on the music of the Speed Racer video game.Steve Green makes a great point about the need for transitions when moving between different energy levels in an interactive musical score.  I encountered a similar problem regarding disparate energy levels that required transitions when I composed the music for the Speed Racer video game (published by Warner Bros Interactive).  During races, the player would have the option to enter a special mode called “Zone Mode” in which their vehicle would travel much faster and would become instantly invincible.  During those sequences, the music switched from the main racing music to a much-more energetic track, and it became important for me to build a transition into that switch-over so that the change wouldn’t be jarring to the player.  I describe the process in this tutorial video:


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).While sometimes a game audio team will choose an interactive music system strictly based on its practical advantages, there are also times in which the decision may be influenced by more emotional factors.  “We love MIDI,” confesses Becky Allen, audio director for the Plants vs. Zombies: Heroes game for mobile devices.  In fact, the development team, PopCap Games, has a long and distinguished history of innovative musical interactivity using the famous MIDI protocol.  During the Plants vs. Zombies: Heroes project, MIDI was a powerful tool for the audio team.  “It really was flexible, it was something you really could work with,” Allen says.

From the article by game composer Winifred Phillips - an illustration of the game Planets vs. Zombies: Heroes.However, that didn’t mean that the MIDI system didn’t create some problems for the audio team.  Early on in development for Plants vs. Zombies: Heroes, the team decided to record their own library of 24 musical instrument sounds for the game.  But during initial composition, those instruments weren’t yet available. This led to an initial reliance on a pre-existing library (East West Symphonic Orchestra).  “We were undergoing this sample library exercise, knowing that we’d be moving over to those samples eventually,” Allen says. Although the East West sample libraries had been initially used, they were fundamentally different. “Our PopCap sample library is fantastic too, but it’s totally different,” Allen adds.  “So the sounds were not the same, and the music, even though they were the same cues, just felt wrong.”  Allen advises, “I think it’s very important, if you can, to write to the sample library that you’ll be using ultimately at the end.”


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).For Paul Weir’s work on the space exploration game No Man’s Sky, the motivation to use a procedural music system was also partly influenced by emotional factors.  “I really enjoy ceding control to the computer, giving it rules and letting it run,” Weir confides.  But there were other motivating influences as well. According to Weir, the advantages of procedural music rest with its unique responsiveness to in-game changes.  “Procedural audio, to make it different, to make it procedural, it has to be driven by the game,” Weir says.  From the article by game composer Winifred Phillips - an illustration of the game No Man's Sky.“What are you doing, game? I’m going to react to that in some way, and that’s going to be reflected in the sound I’m producing. In order to do that,” Weir adds, “it has to use some form of real-time generated sound.”  According to Weir, “procedural audio is the creation of sound in real-time, using synthesis techniques such as physical modeling, with deep links into game systems.”

While this gives a procedural music system the potential to be the most pliable and reactive system available for modern game design, there are steep challenges inherent in its structure.  “Some of the difficulties of procedural generated content,” Weir explains, “is to give a sense of its meaningfulness, like it feels like it’s hand crafted.” In a moment of personal reflection, Weir shares, “One of my big issues, is that if you have procedural audio, that the perception of it has to be as good as traditional audio. It’s no good if you compromise.”

 


 

So, for each of these interactive music systems there were distinct advantages and disadvantages.  In the third and final article of this series, we’ll get down to some nitty-gritty details of how these interactive systems were put to use.  Thanks for reading, and please feel free to leave your comments in the space below!

 

Photo of video game composer Winifred Phillips in her music production studio.Winifred Phillips is an award-winning video game music composer whose most recent projects are the triple-A first person shooter Homefront: The Revolution and the Dragon Front VR game for Oculus Rift. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Follow her on Twitter @winphillips.

Video game music systems at GDC 2017: what are composers using?

By video game music composer Winifred Phillips | Contact | Follow

Video game composer Winifred Phillips, presenting at the Game Developers Conference 2017.The 2017 Game Developers Conference could be described as a densely-packed deep-dive exploration of the state-of-the-art tools and methodologies used in modern game development.  This description held especially true for the game audio track, wherein top experts in the field offered a plethora of viewpoints and advice on the awesome technical and artistic challenges of creating great sound for games. I’ve given GDC talks for the past three years now (see photo), and every year I’m amazed at the breadth and diversity of the problem-solving approaches discussed by my fellow GDC presenters.  Often I’ll emerge from the conference with the impression that we game audio folks are all “doing it our own way,” using widely divergent strategies and tools.

This year, I thought I’d write three articles to collect and explore the ideas that were discussed in five different GDC audio talks.  During their presentations, these five speakers all shared their thoughts on best practices and methods for instilling interactivity in modern game music.  By absorbing these ideas side-by-side, I thought we might gain a sense of the “bigger picture” when it comes to the current leading-edge thinking for music interactivity in games. In the first article, we’ll look at the basic nature of these interactive systems.  We’ll devote the second article to the pros and cons of each system, and in the third article we’ll look at tools and tips shared by these music interactivity experts. Along the way, I’ll also be sharing my thoughts on the subject, and we’ll take a look at musical examples from some of my own projects that demonstrate a few ideas explored in these GDC talks:

So, let’s begin with the most obvious question.  What kind of interactive music systems are game audio folks using lately?

Continue reading

Composing video game music to build suspense, part 5: semi silence

Winifred Phillips - video game music composer - working on the music of The Da Vinci Code video game in her music production studio.

By Winifred Phillips | Contact | Follow

Welcome to the fifth and final installment of my five-part article series on music composition techniques for stimulating tension and suspense in video games.  These articles are based on the presentation I gave this year at the popular Game Developers Conference in San Francisco, entitled Homefront to God of War: Using Music to Build Suspense.  If you haven’t yet read the previous four articles, you’ll find them here:

Now that we’ve considered the power of Ominous Ambiences, Jarring Jolts, Creepy Clusters, and Drones of Dread, let’s take a look at the last item on our list of suspenseful music composition techniques – Semi Silence.

Continue reading

Composing video game music to build suspense, part 4: drones of dread

Winifred Phillips, video game music composer, at work in her studio on the music of the original God of War.

By Winifred Phillips | Contact | Follow

Welcome to the fourth installment of my five-part article series discussing music composition techniques that heighten tension and suspense for video game projects.  These articles are based on the presentation I gave at this year’s Game Developers Conference in San Francisco, entitled Homefront to God of War: Using Music to Build Suspense.  If you haven’t read the previous three articles, you’ll find them here:

Before we move on to the next music composition technique in our suspense-building arsenal, I’d like to briefly revisit a video game project we discussed in our last article; the popular Dragon Front VR game for the Oculus Rift, developed by High Voltage Software.

Continue reading

Composing video game music to build suspense, part 3: creepy clusters

Winifred Phillips (video game music composer) working in her studio on the music of the Dragon Front video game.

By Winifred Phillips | Contact | Follow

Welcome back to our five part discussion of the role that video game music can play in enhancing tension and promoting suspenseful gameplay!  These articles are based on the presentation I gave at this year’s Game Developers Conference in San Francisco, entitled Homefront to God of War: Using Music to Build Suspense.  If you haven’t read the previous two articles, you’ll find them here:

So, now that we’ve discussed ominous atmospheres and jarring jolts, let’s look at the next technique in our arsenal:

The Creepy Cluster technique

From game composer Winifred Phillips' article on suspenseful game music - an illustration of the 'clusters' technique.As we know, tone clusters are collections of notes packed together to produce unnerving dissonant effects. While it might seem like any cat can walk across a piano and produce unpleasant clusters, well-executed dissonance is actually one of the trickiest techniques we can employ.  It’s tremendously potent when used with expert precision.

Why do human beings respond so intensely to dissonance? Professor Michael Epstein of Northeastern University’s Auditory Modeling and Processing Lab has devoted over 20 years of expert research into why certain sounds have the power to instantly incite and deepen fear in listeners.  He tells Boston Magazine that “common musical intervals, changed slightly to create dissonance, are immediately disconcerting.” According to Epstein, “very precise noises trigger human fear and discomfort.”

Continue reading

Composing video game music to build suspense, part 2: jarring jolts

Winifred Phillips, composer of video game music, shown in her studio working on the music of the Assassin's Creed Liberation video game.

By Winifred Phillips | Contact | Follow

Welcome back to our five-part discussion of some of the best techniques that video game composers can use to enhance tension and promote suspenseful gameplay.  These articles are based on the presentation I gave at this year’s Game Developers Conference in San Francisco, entitled Homefront to God of War: Using Music to Build Suspense.  If you haven’t read our previous discussion of Ominous Ambiences in part one of this series, please go check that article out.

Are you back?  Good!  Let’s continue!

We’ve already talked about how to create an edgy, ominous atmosphere. By carefully nurturing the player’s suspense and anxiety, we can prime the player with an assortment of quietly unnerving sounds, until the player is perfectly ready for…

The Jarring Jolt technique

This is the second technique we’ll be discussing in our five-part article series on the role of music in building suspense. Like the Ominous Ambience (which we discussed in part one), the Jarring Jolt also owes a debt to the expert work of sound designers.  In fact, the Ominous Ambience and the Jarring Jolt are fairly interdependent. One doesn’t work that well without the other.

Continue reading

Composing video game music to build suspense, part 1: ominous ambience

Winifred Phillips (video game composer), working in her studio on the music of the Homefront: The Revolution video game.

By Winifred Phillips | Contact | Follow

At this year’s Game Developers Conference in San Francisco, I was honored to give a presentation entitled Homefront to God of War: Using Music to Build Suspense.  While I’ve certainly discussed techniques for building suspense in this blog before, the talk I gave at GDC expanded significantly on that discussion and included lots more research and practical examples that we haven’t previously examined here.  With that in mind, I’m excited to begin a five-part article series based on my GDC 2017 presentation!  During the course of these five articles, we’ll be taking a look at some of the best techniques that enable video game music composers to introduce suspense into their music, control tension levels during gameplay and keep players engaged.

So, let’s start by defining the core concept.  What exactly is suspense?

A physiological reaction

We all can agree that music is one of the most effective ways to produce emotional reactions. But suspense, particularly in the field of game development, isn’t just about an emotional state. It’s also a unique physiological reaction – a tension rising out of the uncertainty that we’re encountering during gameplay.

Continue reading