Video game music systems at GDC 2017: tools and tips for composers

Photo of video game composer Winifred Phillips, working in her music production studio on the music of the SimAnimals video game.

By video game composer Winifred Phillips | Contact | Follow

Welcome back to this three article series that’s bringing together the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers explored discoveries they’d made while creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to broaden our viewpoint and gain a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. We’ve been looking at five interactive music systems discussed in these five GDC 2017 presentations:

In the first article, we examined the basic nature of these interactive systems. In the second article, we contemplated why those systems were used, with some of the inherent pros and cons of each system discussed in turn.  So now, let’s get into the nitty gritty of tools and tips for working with such interactive music systems.  If you haven’t read parts one and two of this series, please go do so now and then come back:

  1. Video game music systems at GDC 2017: what are composers using?
  2. Video game music systems at GDC 2017: pros and cons for composers

Ready?  Great!  Here we go!

Tools and tips

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).To implement his interactive music system for the ocean exploration game ABZU, sound designer Steve Green relied on the popular Wwise middleware application in order to best facilitate transitions from one piece of music to another.  “This idea of changing music on the fly rather than bringing instruments in and out was a good example of how to do ABZU,” Green says.  In order to implement these transitions, Green relied primarily on simple horizontal resequencing, with some limited vertical layering to ease transitions.  At times, synchronization points were placed (usually at the beginning of each measure) in order to enable the music system to switch from one track to another.  From the article by game composer Winifred Phillips - an illustration of the game ABZU.Sometimes, this approach was too abrupt.  “Transitionary pieces are basically to help two tracks that are just not going to flow well together,” Green explains.

For instance, when the player is traversing a particular portion of a level (i.e. stage) the audio engine is able to keep track of the player’s progress.”We used a parameter that would gauge how far along you are in this stage of the level,” Green says.  “Once you reach the 90% range, (the audio engine) would call a transition.”  This musical transition would play simultaneously with the previous track, allowing the previous track to fade out and facilitating either a more gentle culmination into silence, or a more graceful segue to another piece of music.


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).During educator Leonard J. Paul’s GDC presentation, he discussed his work on the audio team of the Vessel platforming game, but he also shared insights into several other projects from his repertoire, along with lots of general advice that’s worth considering.  “When working in games, of course, you’ve got to think about code,” Paul says. “You’ve got to think about RAM – how much space this is all gonna to fit in – streams, if you’re streaming.  You’ve got to think about what your RAM window size is and how fast you can get information in and out – how many streams you can do at once, what kind of compression you’re going to use,” From the article by game composer Winifred Phillips - an illustration of the game Vessel.Paul continues, listing areas of technical concern for the game audio expert to think about when working on a project like Vessel.  “Because if you don’t,” he warns, “then you’re going to run into some issues.”

Paul also mentions some of the more famous software tools available to audio pros, including Wwise, FMOD Studio, Fabric, and Pure Data, along with a number of general words of advice.  “Technical tips – use spreadsheets, be organized, use tools to refine your process, embrace prototyping,” Paul urges. “Do audio sketches and get this stuff done early.”

 

My perspective on sketches and prototypes

Video game music composer Winifred Phillips, pictured in her music production studio working on the music of the SimAnimals video game.In his presentation, Leonard J. Paul brought up a good point about the importance of sketches and prototypes in the workflow of a video game composer.  Most game music composers have gone through this type of iterative process.  For instance, for the SimAnimals game from Electronic Arts, I composed lots of sketches and prototypes before settling on the ultimate style of the game.  The audio team had many varying ideas about what the music should sound like, so the initial brainstorming process involved quite a bit of trial and error.  It was a game about heartwarming animal relationships, so should the music feel old-fashioned and lyrical?  On the other hand, the game focused on strategy mechanics, so should the music feel more puzzle-like and intellectual?  Also, the game included darker underpinnings associated with suffering and dissolution, so should the music stick with a light and airy feel, or should it weave dark and sinister textures into the mix?  I tried tons of permutations on all these ideas until I hit upon the right balance of elements, and in the end my music for SimAnimals took me in unexpectedly challenging directions.  The style couldn’t have been achieved without the process of trial and error that characterized my early efforts at the beginning of music composition. As an example, I’m including here two videos showing the difference between the brightly cheerful main theme music of the game, and one of several grim and dissonant variations on the same melodic theme:


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).For Final Fantasy XV, audio programmer Sho Iwamoto created an audio engine he dubbed MAGI (Music API for Gaming Interaction).  While the audio engine accomplished tasks similar to the functionality of other applications such as Wwise and FMOD, the MAGI system also included some creative and original specializations.  “There are a lot of time signature variations or changes in JRPG music, so I designed MAGI to support them,” Iwamoto says.  Among its other capabilities, MAGI has the ability to adjust to changing tempo and time signature values across the length of a composition, allowing flexibility in the placing of synchronization points within the track.  This capability was born out of both a spirit of innovation and pure necessity.  Because Iwamoto was hired by Square Enix just two years earlier, much of the music of Final Fantasy XV had already been composed and recorded before he joined the team.  “Many of these (compositions) were not planned to be interactive,” Iwamoto admits.  Because of this, the musical compositions included variable tempos, dramatic pauses, and frequent shifts in time signature.

From the article by game composer Winifred Phillips - an illustration of the game Final Fantasy XV.In order to make these tracks interactive, Iwamoto used the MAGI system to insert custom synchronization points into each composition so that the transitions would be pleasing in a musical way.  “All you have to do is just move or delete (sync points) when you think the transitions are not musical, or add the sync points when you think the transition is musical,” Iwamoto comments.  “Sometimes these sync points can be hard to find, and you may have to wait sixteen bars or more to (find a workable) transition.”  This problem became a frequent issue during combat music, so to solve this problem, Iwamoto devised a strategy that he called the pre-end section. This was so named because it is always used as a preparation for the ending of the combat track.  “The pre-end section is designed to have more constant and dense sync points,” Iwamoto explains.  “That makes for really quick transitions.”  Existing as a separate music file, the pre-end section would always be in the same style as the main body of the composition, so the music could switch from the main body to the pre-end section smoothly.  Designed to be more consistent in terms of tempo, key and time signature, the pre-end section could accommodate lots of sync points, allowing the music to transition to a finishing flourish with almost instantaneous precision in accordance with the end of combat.  Despite its seeming complexity, Iwamoto meant for this system within the MAGI engine to be intuitive by design.  “We used very simple and basic approaches,” Iwamoto says.  “I wanted to make (MAGI) very easy and versatile.”


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).For audio director Becky Allen, simple and basic approaches are important for very different reasons.  “We are a mobile studio and we have to be very mindful of our size because of the devices we’re played on,” Allen says, describing her strategy for the MIDI music system of Plants vs. Zombies: Heroes.  “We’re always thinking about size, and ways to be smart.”  Part of that strategy involved a plan for limiting the amount of memory required for the MIDI files and their associated library of instrument sounds.  At first, those limitations were meant to be sensible but not extreme.  “80 megs,” Allen says. “40 for music, 40 for sound effects and VO.”  But then, the audio development process hit a snag when another mobile game came along and made a big splash in the marketplace.  “Along came Clash Royale, and that was a quick game, it loaded quickly,” Allen says.  “We had some pressure to come down to 30 megabytes.”

From the article by game composer Winifred Phillips - an illustration of the game Plants vs. Zombies: Heroes.In the end, the team held to 60 megabytes, but even this required sacrifices.  “We switched some things from stereo to mono but not everything. We streamed all the WAV music and we streamed larger WAV assets.” After making these and other modifications, the newly streamlined music system was ready to show how interactive it could be.  For instance, by associating the player’s relative health with a RTPC (real-time parameter control) in Wwise, the music system could adjust MIDI data on the fly in accordance with the player’s success or failure.  “If you’re up by five points all the MIDI music went up a whole step, and if you’re up by ten points all the MIDI music went up two whole steps. And it worked, it was awesome,” Allen enthuses.  “Utilizing your tools and an open mind and a flexible mind and a flexible team,” Allen says, “you can make these changes all the way along through the process.”


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).If flexibility is the ultimate ambition of an interactive music system, then a procedural music system could be considered the definitive realization of that goal.  While procedural music can be viewed as the height of interactivity, Paul Weir had built such systems for games before, and his viewpoint on the technology is much more down-to-earth. “Essentially they’re glorified random file players with a little bit of logic stuck on top,” Weir admits.  Together with the development team for the space exploration game No Man’s Sky, Weir set about to see what procedural music could bring to the project.  First, Weir hired a drum-driven electronica band called 65daysofstatic, then asked them to start creating music for their science fiction game.  “We were kind of really adamant that as a band, I did not want to interfere creatively with their process,” Weir says.  “So we always said, write an album, just write us an album, and we’re not going to start telling you how to do that, because you know how to do that – because you’re the band.  So we let them go off and write a very traditional album, but in the knowledge that we were going to come back to it later and just rip it all apart.”

From the article by game composer Winifred Phillips - an illustration of the game No Man's Sky.Using Pulse (his self-made procedural music generation tool) Weir set about disassembling the elements of the submitted music so that it could be incorporated into the procedural system.  He quickly learned that it would require more than just a retooling of the original content.  “So it wasn’t a question of saying just give us the stems and we’ll do it,” Weir says. “It was like, no no no, go right back and do us more performances, take out bits, give us more drum loops, perform new guitar riffs, create new stuff – almost like kind of remixing the original tracks.”  With these musical fragments, the procedural system could then perform calculations and assemble the library of elements into new combinations in accordance with the player’s locale and occupation. For instance, the music might be triggered depending on such in-game variables as proximity to buildings and time spent walking, utilizing what amounted to a large library of musical ‘soundscape sets’ that randomly alternated depending on how long any previous soundscape set had been playing.  “In the game you don’t get the album,” Weir insists.  “The game soundtrack is bits of what appear on the album. It’s lots of bits that aren’t on the album, but it feels relatively cohesive.”


 

Conclusion

So, there we have it!  We’ve compared the viewpoints of five game audio pros discussing the interactive music systems of their projects during their presentations at GDC 2017.  Interactive music is a fascinating subject, ranging from the simplest of solutions to the most complex and intricate of designs.  With ingenuity and patience, these audio developers have introduced their own creativity into the implementation of music in games.  Their strategies and solutions can help us to broaden our minds when we’re thinking about ways to make our own music interactive!  I hope you’ve enjoyed this three-article series, and please feel free to leave your comments below!

 

Photo of video game composer Winifred Phillips in her music production studio.Winifred Phillips is an award-winning video game music composer whose most recent projects are the triple-A first person shooter Homefront: The Revolution and the Dragon Front VR game for Oculus Rift. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Follow her on Twitter @winphillips.

Video game music systems at GDC 2017: pros and cons for composers

Video game composer Winifred Phillips, pictured in her music production studio working on the music of LittleBigPlanet 2 Cross Controller

By Winifred Phillips | Contact | Follow

Welcome back to our three article series dedicated to collecting and exploring the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers shared ideas they’d developed in the process of creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to cultivate a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. In the first article, we looked at the basic nature of five interactive music systems discussed in these five GDC 2017 presentations:

If you haven’t read part one of this article series, please go do that now and come back.

Okay, so let’s now contemplate some simple but important questions: why were those systems used?  What was attractive about each interactive music strategy, and what were the challenges inherent in using those systems?

Continue reading

Video Game Music Production Tips from GDC 2016

Game Composer Winifred Phillips during her game music presentation at the Game Developers Conference 2016I was pleased to give a talk about composing music for games at the 2016 Game Developers Conference (pictured left).  GDC took place this past March in San Francisco – it was an honor to be a part of the audio track again this year, which offered a wealth of awesome educational sessions for game audio practitioners.  So much fun to see the other talks and learn about what’s new and exciting in the field of game audio!  In this blog, I want to share some info that I thought was really interesting from two talks that pertained to the audio production side of game development: composer Laura Karpman’s talk about “Composing Virtually, Sounding Real” and audio director Garry Taylor’s talk on “Audio Mastering for Interactive Entertainment.”  Both sessions had some very good info for video game composers who may be looking to improve the quality of their recordings.  Along the way, I’ll also be sharing a few of my own personal viewpoints on these music production topics, and I’ll include some examples from one of my own projects, the Ultimate Trailers album for West One Music, to illustrate ideas that we’ll be discussing.  So let’s get started!

Continue reading

Interactive Music for the Video Game Composer

Game Composer Winifred Phillips works in her studio on the music of the popular Spore Hero video game As a speaker in the audio track of the Game Developers Conference this year, I enjoyed taking in a number of GDC audio sessions — including a couple of presentations that focused on the future of interactive music in games.  I’ve explored this topic before at length in my book (A Composer’s Guide to Game Music), and it was great to see that the game audio community continues to push the boundaries and innovate in this area! Interactive music is a worthwhile subject for discussion, and will undoubtedly be increasingly important in the future as dynamic music systems become more prevalent in game projects.  With that in mind, in this blog I’d like to share my personal takeaway from two sessions that described very different approaches to musical interactivity. After that, we’ll discuss one of my experiences with interactive music for the video game Spore Hero from Electronic Arts (pictured above).

Musical Intelligence

Baldur BaldurssonPhoto of Baldur Baldursson, the audio director for Icelandic game development studio CCP Games (part of the article by game composer Winifred Phillips) (pictured left) is the audio director for Icelandic game development studio CCP Games, responsible for the EVE Online MMORPG.  Together with Professor Kjartan Olafsson of the Iceland Academy of Arts, Baldursson presented a talk at GDC 2016 on a new system to provide “Intelligent Music For Games.”

Baldursson began the presentation by explaining why an intelligent music system for games can be a necessity.  “We basically want an intelligent music system because we can’t (or maybe shouldn’t really) precompose all of the elements,” Baldursson explains. He describes the conundrum of creating a musical score for a game whose story is still fluid and changeable, and then asserts,  “I think we should find ways of making this better.”

Continue reading

Music in the Manual: FMOD Studio Vs. Wwise

Wwise-FMOD

A few days ago, I downloaded and installed the latest version of a software package entitled FMOD Studio and was pleasantly surprised to discover that an oversight had been corrected. It’s not unusual for software updates to correct problems or provide additional functionality, but this update was especially satisfying for me. The makers of FMOD Studio had added the “Music” section to the software manual.

A brief explanation: FMOD Studio is a software application designed by Firelight Technologies to enable game audio professionals to incorporate sound into video games. The application focuses solely on audio, and is used in conjunction with game software. In essence, FMOD Studio is folded into the larger construct of a game’s operational code, giving the overall game the ability to do more sophisticated things with the audio side of its presentation.

When FMOD Studio was initially released in August of 2012, the manual did not include information about the music capabilities of the software. Admittedly, the majority of FMOD Studio users are sound designers whose interests tend to focus on the tools for triggering sound effects and creating environmental atmospheres. That being said, many composers also use the portions of the FMOD Studio application that are specifically designed to enable the assignment of interactive behaviors to music tracks. It was a bit puzzling that the manual didn’t describe those music tools.

One of the biggest competitors to FMOD Studio is the Wwise software from Audiokinetic. Wwise offers much of the same functionality as FMOD, and in working with the software one of the things I really like about it is its documentation. Audiokinetic put a lot of thought and energy into the Wwise Fundamentals Approach document and the expansive tutorial handbook, Project Adventure. Both of these documents discuss the music features of the Wwise software, offering step-by-step guidance for the creation of interactive music systems within the Wwise application. This is why the omission of any discussion of the music tools from the FMOD manual was so perplexing.

It’s true that many of the music features of the FMOD Studio software are also useful in sound design applications, and some are similar in their function to tools described in the sound design portions of the manual. Firelight Technologies may have assumed that those portions of the manual would be sufficient for all users, including composers. However, composers are specialists, and their priorities do not match those of their sound design colleagues. In using the FMOD Studio tools, the needs of composers would be sharply different from those driving the rest of the audio development community. Wwise understood this from the start, but FMOD seemed to be following a philosophy that hearkened back to the early days of the game industry.

In those days, the audio side of a game was often created and implemented by a single person. This jack-of-all-trades would create all the sound effects, voice-overs and music. Nowadays, the audio field is populated by scores of specialists. It makes sense for FMOD Studio to acknowledge specialists such as composers in their software documentation, and I’m very glad to see that they’ve just done so. If you’d like to learn more about FMOD Studio, you can see a general overview of the application in this YouTube video: