Video game music systems at GDC 2017: tools and tips for composers

Photo of video game composer Winifred Phillips, working in her music production studio on the music of the SimAnimals video game.

By video game composer Winifred Phillips | Contact | Follow

Welcome back to this three article series that’s bringing together the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers explored discoveries they’d made while creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to broaden our viewpoint and gain a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. We’ve been looking at five interactive music systems discussed in these five GDC 2017 presentations:

In the first article, we examined the basic nature of these interactive systems. In the second article, we contemplated why those systems were used, with some of the inherent pros and cons of each system discussed in turn.  So now, let’s get into the nitty gritty of tools and tips for working with such interactive music systems.  If you haven’t read parts one and two of this series, please go do so now and then come back:

  1. Video game music systems at GDC 2017: what are composers using?
  2. Video game music systems at GDC 2017: pros and cons for composers

Ready?  Great!  Here we go!

Tools and tips

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).To implement his interactive music system for the ocean exploration game ABZU, sound designer Steve Green relied on the popular Wwise middleware application in order to best facilitate transitions from one piece of music to another.  “This idea of changing music on the fly rather than bringing instruments in and out was a good example of how to do ABZU,” Green says.  In order to implement these transitions, Green relied primarily on simple horizontal resequencing, with some limited vertical layering to ease transitions.  At times, synchronization points were placed (usually at the beginning of each measure) in order to enable the music system to switch from one track to another.  From the article by game composer Winifred Phillips - an illustration of the game ABZU.Sometimes, this approach was too abrupt.  “Transitionary pieces are basically to help two tracks that are just not going to flow well together,” Green explains.

For instance, when the player is traversing a particular portion of a level (i.e. stage) the audio engine is able to keep track of the player’s progress.”We used a parameter that would gauge how far along you are in this stage of the level,” Green says.  “Once you reach the 90% range, (the audio engine) would call a transition.”  This musical transition would play simultaneously with the previous track, allowing the previous track to fade out and facilitating either a more gentle culmination into silence, or a more graceful segue to another piece of music.


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).During educator Leonard J. Paul’s GDC presentation, he discussed his work on the audio team of the Vessel platforming game, but he also shared insights into several other projects from his repertoire, along with lots of general advice that’s worth considering.  “When working in games, of course, you’ve got to think about code,” Paul says. “You’ve got to think about RAM – how much space this is all gonna to fit in – streams, if you’re streaming.  You’ve got to think about what your RAM window size is and how fast you can get information in and out – how many streams you can do at once, what kind of compression you’re going to use,” From the article by game composer Winifred Phillips - an illustration of the game Vessel.Paul continues, listing areas of technical concern for the game audio expert to think about when working on a project like Vessel.  “Because if you don’t,” he warns, “then you’re going to run into some issues.”

Paul also mentions some of the more famous software tools available to audio pros, including Wwise, FMOD Studio, Fabric, and Pure Data, along with a number of general words of advice.  “Technical tips – use spreadsheets, be organized, use tools to refine your process, embrace prototyping,” Paul urges. “Do audio sketches and get this stuff done early.”

 

My perspective on sketches and prototypes

Video game music composer Winifred Phillips, pictured in her music production studio working on the music of the SimAnimals video game.In his presentation, Leonard J. Paul brought up a good point about the importance of sketches and prototypes in the workflow of a video game composer.  Most game music composers have gone through this type of iterative process.  For instance, for the SimAnimals game from Electronic Arts, I composed lots of sketches and prototypes before settling on the ultimate style of the game.  The audio team had many varying ideas about what the music should sound like, so the initial brainstorming process involved quite a bit of trial and error.  It was a game about heartwarming animal relationships, so should the music feel old-fashioned and lyrical?  On the other hand, the game focused on strategy mechanics, so should the music feel more puzzle-like and intellectual?  Also, the game included darker underpinnings associated with suffering and dissolution, so should the music stick with a light and airy feel, or should it weave dark and sinister textures into the mix?  I tried tons of permutations on all these ideas until I hit upon the right balance of elements, and in the end my music for SimAnimals took me in unexpectedly challenging directions.  The style couldn’t have been achieved without the process of trial and error that characterized my early efforts at the beginning of music composition. As an example, I’m including here two videos showing the difference between the brightly cheerful main theme music of the game, and one of several grim and dissonant variations on the same melodic theme:


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).For Final Fantasy XV, audio programmer Sho Iwamoto created an audio engine he dubbed MAGI (Music API for Gaming Interaction).  While the audio engine accomplished tasks similar to the functionality of other applications such as Wwise and FMOD, the MAGI system also included some creative and original specializations.  “There are a lot of time signature variations or changes in JRPG music, so I designed MAGI to support them,” Iwamoto says.  Among its other capabilities, MAGI has the ability to adjust to changing tempo and time signature values across the length of a composition, allowing flexibility in the placing of synchronization points within the track.  This capability was born out of both a spirit of innovation and pure necessity.  Because Iwamoto was hired by Square Enix just two years earlier, much of the music of Final Fantasy XV had already been composed and recorded before he joined the team.  “Many of these (compositions) were not planned to be interactive,” Iwamoto admits.  Because of this, the musical compositions included variable tempos, dramatic pauses, and frequent shifts in time signature.

From the article by game composer Winifred Phillips - an illustration of the game Final Fantasy XV.In order to make these tracks interactive, Iwamoto used the MAGI system to insert custom synchronization points into each composition so that the transitions would be pleasing in a musical way.  “All you have to do is just move or delete (sync points) when you think the transitions are not musical, or add the sync points when you think the transition is musical,” Iwamoto comments.  “Sometimes these sync points can be hard to find, and you may have to wait sixteen bars or more to (find a workable) transition.”  This problem became a frequent issue during combat music, so to solve this problem, Iwamoto devised a strategy that he called the pre-end section. This was so named because it is always used as a preparation for the ending of the combat track.  “The pre-end section is designed to have more constant and dense sync points,” Iwamoto explains.  “That makes for really quick transitions.”  Existing as a separate music file, the pre-end section would always be in the same style as the main body of the composition, so the music could switch from the main body to the pre-end section smoothly.  Designed to be more consistent in terms of tempo, key and time signature, the pre-end section could accommodate lots of sync points, allowing the music to transition to a finishing flourish with almost instantaneous precision in accordance with the end of combat.  Despite its seeming complexity, Iwamoto meant for this system within the MAGI engine to be intuitive by design.  “We used very simple and basic approaches,” Iwamoto says.  “I wanted to make (MAGI) very easy and versatile.”


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).For audio director Becky Allen, simple and basic approaches are important for very different reasons.  “We are a mobile studio and we have to be very mindful of our size because of the devices we’re played on,” Allen says, describing her strategy for the MIDI music system of Plants vs. Zombies: Heroes.  “We’re always thinking about size, and ways to be smart.”  Part of that strategy involved a plan for limiting the amount of memory required for the MIDI files and their associated library of instrument sounds.  At first, those limitations were meant to be sensible but not extreme.  “80 megs,” Allen says. “40 for music, 40 for sound effects and VO.”  But then, the audio development process hit a snag when another mobile game came along and made a big splash in the marketplace.  “Along came Clash Royale, and that was a quick game, it loaded quickly,” Allen says.  “We had some pressure to come down to 30 megabytes.”

From the article by game composer Winifred Phillips - an illustration of the game Plants vs. Zombies: Heroes.In the end, the team held to 60 megabytes, but even this required sacrifices.  “We switched some things from stereo to mono but not everything. We streamed all the WAV music and we streamed larger WAV assets.” After making these and other modifications, the newly streamlined music system was ready to show how interactive it could be.  For instance, by associating the player’s relative health with a RTPC (real-time parameter control) in Wwise, the music system could adjust MIDI data on the fly in accordance with the player’s success or failure.  “If you’re up by five points all the MIDI music went up a whole step, and if you’re up by ten points all the MIDI music went up two whole steps. And it worked, it was awesome,” Allen enthuses.  “Utilizing your tools and an open mind and a flexible mind and a flexible team,” Allen says, “you can make these changes all the way along through the process.”


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).If flexibility is the ultimate ambition of an interactive music system, then a procedural music system could be considered the definitive realization of that goal.  While procedural music can be viewed as the height of interactivity, Paul Weir had built such systems for games before, and his viewpoint on the technology is much more down-to-earth. “Essentially they’re glorified random file players with a little bit of logic stuck on top,” Weir admits.  Together with the development team for the space exploration game No Man’s Sky, Weir set about to see what procedural music could bring to the project.  First, Weir hired a drum-driven electronica band called 65daysofstatic, then asked them to start creating music for their science fiction game.  “We were kind of really adamant that as a band, I did not want to interfere creatively with their process,” Weir says.  “So we always said, write an album, just write us an album, and we’re not going to start telling you how to do that, because you know how to do that – because you’re the band.  So we let them go off and write a very traditional album, but in the knowledge that we were going to come back to it later and just rip it all apart.”

From the article by game composer Winifred Phillips - an illustration of the game No Man's Sky.Using Pulse (his self-made procedural music generation tool) Weir set about disassembling the elements of the submitted music so that it could be incorporated into the procedural system.  He quickly learned that it would require more than just a retooling of the original content.  “So it wasn’t a question of saying just give us the stems and we’ll do it,” Weir says. “It was like, no no no, go right back and do us more performances, take out bits, give us more drum loops, perform new guitar riffs, create new stuff – almost like kind of remixing the original tracks.”  With these musical fragments, the procedural system could then perform calculations and assemble the library of elements into new combinations in accordance with the player’s locale and occupation. For instance, the music might be triggered depending on such in-game variables as proximity to buildings and time spent walking, utilizing what amounted to a large library of musical ‘soundscape sets’ that randomly alternated depending on how long any previous soundscape set had been playing.  “In the game you don’t get the album,” Weir insists.  “The game soundtrack is bits of what appear on the album. It’s lots of bits that aren’t on the album, but it feels relatively cohesive.”


 

Conclusion

So, there we have it!  We’ve compared the viewpoints of five game audio pros discussing the interactive music systems of their projects during their presentations at GDC 2017.  Interactive music is a fascinating subject, ranging from the simplest of solutions to the most complex and intricate of designs.  With ingenuity and patience, these audio developers have introduced their own creativity into the implementation of music in games.  Their strategies and solutions can help us to broaden our minds when we’re thinking about ways to make our own music interactive!  I hope you’ve enjoyed this three-article series, and please feel free to leave your comments below!

 

Photo of video game composer Winifred Phillips in her music production studio.Winifred Phillips is an award-winning video game music composer whose most recent projects are the triple-A first person shooter Homefront: The Revolution and the Dragon Front VR game for Oculus Rift. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Follow her on Twitter @winphillips.

Video game music systems at GDC 2017: what are composers using?

By video game music composer Winifred Phillips | Contact | Follow

Video game composer Winifred Phillips, presenting at the Game Developers Conference 2017.The 2017 Game Developers Conference could be described as a densely-packed deep-dive exploration of the state-of-the-art tools and methodologies used in modern game development.  This description held especially true for the game audio track, wherein top experts in the field offered a plethora of viewpoints and advice on the awesome technical and artistic challenges of creating great sound for games. I’ve given GDC talks for the past three years now (see photo), and every year I’m amazed at the breadth and diversity of the problem-solving approaches discussed by my fellow GDC presenters.  Often I’ll emerge from the conference with the impression that we game audio folks are all “doing it our own way,” using widely divergent strategies and tools.

This year, I thought I’d write three articles to collect and explore the ideas that were discussed in five different GDC audio talks.  During their presentations, these five speakers all shared their thoughts on best practices and methods for instilling interactivity in modern game music.  By absorbing these ideas side-by-side, I thought we might gain a sense of the “bigger picture” when it comes to the current leading-edge thinking for music interactivity in games. In the first article, we’ll look at the basic nature of these interactive systems.  We’ll devote the second article to the pros and cons of each system, and in the third article we’ll look at tools and tips shared by these music interactivity experts. Along the way, I’ll also be sharing my thoughts on the subject, and we’ll take a look at musical examples from some of my own projects that demonstrate a few ideas explored in these GDC talks:

So, let’s begin with the most obvious question.  What kind of interactive music systems are game audio folks using lately?

Continue reading

Video game composers can make you smarter! (The music of Dragon Front) Pt. 2

Pictured: Winifred Phillips (video game music composer) in her studio working on the music of the Dragon Front virtual reality game.

By Winifred Phillips | Contact | Follow

Welcome back to our three-part discussion of how video game composers (such as ourselves) can make strategy gamers smarter!  In these articles, we’re looking at ways in which our music can enhance concentration and tactical decision-making for players engrossed in strategic gameplay.  Along the way, I’ve been sharing my personal experiences as the composer for the Dragon Front strategy game for virtual reality.  Over the course of these articles we’ll be covering three of the top concepts that pertain to the relationship between music and concentration.  In part one, we discussed the concept of ‘music-message congruency,’ so if you haven’t read that article yet, please go check it out and then come back.

Are you back now?  Good!  Let’s move on to the second big technique for increasing the smarts of strategy gamers!

Cognition-enhancing tempo

As video game composers, we create music in a wide variety of tempos designed to support the energy of play and the pacing of the game’s overall design.  From leisurely tracks that accompany unstructured exploration to frenetic pieces that support the most high-stakes combat, our music is planned with expert precision to shape the excitement level of players and keep them motivated as they progress.

Continue reading

Video game composers can make you smarter! (The music of Dragon Front) Pt. 1

Video game composer Winifred Phillips, pictured in her music studio working on the original score for the Dragon Front virtual reality game.

Can video game composers make you smarter?  Well, video gaming can be a pretty cerebral activity, requiring astute problem-solving skills and disciplined concentration in order to excel.  That’s especially true for any game built around strategic and/or tactical gameplay, such as real-time or turn-based strategy, tactical shooters, multiplayer online battle arenas (MOBAs), and online collectible card strategy games.  To succeed in these types of games, players must assess the current situation and formulate a plan that accounts for future developments and variables.  Without this type of tactical forward-thinking gameplay, a gamer has little chance to win.  So, can music enable gamers to think tactically, stay focused and make smart decisions?  Over the next three articles, I’ll try to answer that question, while exploring the role of music in enhancing the concentration of strategic/tactical gamers.

Along the way, we’ll be taking a look at some scholarly research on the subject, consulting the opinions of experts, and I’ll be sharing my experiences creating the music for the recently released Dragon Front strategy game from High Voltage software.  We’ll check out some music tracks I composed for the popular Dragon Front game (pictured at the top of this article), and we’ll discuss methods for supporting and enhancing concentration for strategic/tactical game players.  But first, let’s take a closer look at the Dragon Front game.

Continue reading

Video Game Music Composer: Music and Sound in VR Headphones (Part Two)

Photo of game composer Winifred Phillips in her music production studio, from the article "Video Game Music Composer: Music and Sound in VR Headphones (Part Two)"My work as a video game composer has lately included some projects for virtual reality games (more info on that in the coming months), and as a result I’ve been thinking a lot about the awesome potential of VR, and have also been writing lots of articles on the subject.  Earlier this month I began a two-part article that focuses on the experience of the end user, and the gear with which they’ll be enjoying our video game music and audio content (you can read part one here). So, let’s now continue our discussion about the new generation of headphones designed specifically for VR!

In this article, we’ll be discussing two headphone models:

  • Entrim 4D
  • Plantronics RIG 4VR

So let’s get underway!

Entrim 4D headphones

Photo of the Entrim 4D, from the VR headphones article by Winifred Phillips (award-winning game music composer)This March at the famous SXSW convention in Austin, Samsung showed off a piece of experimental technology promising to bring a new dimension of immersion to virtual reality.  It’s designed specifically to complement their popular Samsung Gear VR device, and it works by virtue of electrodes that send electrical signals right into the wearer’s head!  As if virtual reality itself weren’t futuristic enough, now we’re talking about a device that zaps us to make the VR feel more real!  It’s called Entrim 4D (pictured right).  We’re talking about it here because (among other things) Entrim 4D is a pair of audio headphones built specifically for VR.

Continue reading

Communication Tips for the Video Game Composer

Video game composer Winifred Phillips, working in her music studio.A successful career as a video game composer involves much more than our day-to-day challenges in our music studios. In addition to our role as music experts, we need to be well-rounded business people and great members of a creative team.  As a speaker in the audio track of the Game Developers Conference this year, I had a chance to take in a wide variety of GDC sessions, and I noticed how often teamwork was discussed.  Along the way, a common idea emerged from many of these talks — good communication is key. This is a concept that I explored in my book (A Composer’s Guide to Game Music), so I was delighted to see a further discussion of the issue at GDC this year.  Far from just a valuable personality asset, the ability to communicate well must be considered a top priority: as intrinsically valuable as rock-solid competency, awesome artistry or compelling vision. Good communication amongst team members can make or break the development of a game. As game audio pros, we share this in common with our coworkers in other segments of the game development community. However, it becomes especially important for us to focus and emphasize good communication when we’re working remotely as independent contractors. With that in mind, I thought I’d use this article to briefly highlight some GDC 2016 sessions in the game audio track that discussed this popular topic, so we can think about more ways to enhance and improve our communication skills.  And later we’ll discuss a practical example from my work on the music of the SimAnimals game from Electronic Arts.

Continue reading