Video game music systems at GDC 2017: tools and tips for composers

Photo of video game composer Winifred Phillips, working in her music production studio on the music of the SimAnimals video game.

By video game composer Winifred Phillips | Contact | Follow

Welcome back to this three article series that’s bringing together the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers explored discoveries they’d made while creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to broaden our viewpoint and gain a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. We’ve been looking at five interactive music systems discussed in these five GDC 2017 presentations:

In the first article, we examined the basic nature of these interactive systems. In the second article, we contemplated why those systems were used, with some of the inherent pros and cons of each system discussed in turn.  So now, let’s get into the nitty gritty of tools and tips for working with such interactive music systems.  If you haven’t read parts one and two of this series, please go do so now and then come back:

  1. Video game music systems at GDC 2017: what are composers using?
  2. Video game music systems at GDC 2017: pros and cons for composers

Ready?  Great!  Here we go!

Tools and tips

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).To implement his interactive music system for the ocean exploration game ABZU, sound designer Steve Green relied on the popular Wwise middleware application in order to best facilitate transitions from one piece of music to another.  “This idea of changing music on the fly rather than bringing instruments in and out was a good example of how to do ABZU,” Green says.  In order to implement these transitions, Green relied primarily on simple horizontal resequencing, with some limited vertical layering to ease transitions.  At times, synchronization points were placed (usually at the beginning of each measure) in order to enable the music system to switch from one track to another.  From the article by game composer Winifred Phillips - an illustration of the game ABZU.Sometimes, this approach was too abrupt.  “Transitionary pieces are basically to help two tracks that are just not going to flow well together,” Green explains.

For instance, when the player is traversing a particular portion of a level (i.e. stage) the audio engine is able to keep track of the player’s progress.”We used a parameter that would gauge how far along you are in this stage of the level,” Green says.  “Once you reach the 90% range, (the audio engine) would call a transition.”  This musical transition would play simultaneously with the previous track, allowing the previous track to fade out and facilitating either a more gentle culmination into silence, or a more graceful segue to another piece of music.


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).During educator Leonard J. Paul’s GDC presentation, he discussed his work on the audio team of the Vessel platforming game, but he also shared insights into several other projects from his repertoire, along with lots of general advice that’s worth considering.  “When working in games, of course, you’ve got to think about code,” Paul says. “You’ve got to think about RAM – how much space this is all gonna to fit in – streams, if you’re streaming.  You’ve got to think about what your RAM window size is and how fast you can get information in and out – how many streams you can do at once, what kind of compression you’re going to use,” From the article by game composer Winifred Phillips - an illustration of the game Vessel.Paul continues, listing areas of technical concern for the game audio expert to think about when working on a project like Vessel.  “Because if you don’t,” he warns, “then you’re going to run into some issues.”

Paul also mentions some of the more famous software tools available to audio pros, including Wwise, FMOD Studio, Fabric, and Pure Data, along with a number of general words of advice.  “Technical tips – use spreadsheets, be organized, use tools to refine your process, embrace prototyping,” Paul urges. “Do audio sketches and get this stuff done early.”

 

My perspective on sketches and prototypes

Video game music composer Winifred Phillips, pictured in her music production studio working on the music of the SimAnimals video game.In his presentation, Leonard J. Paul brought up a good point about the importance of sketches and prototypes in the workflow of a video game composer.  Most game music composers have gone through this type of iterative process.  For instance, for the SimAnimals game from Electronic Arts, I composed lots of sketches and prototypes before settling on the ultimate style of the game.  The audio team had many varying ideas about what the music should sound like, so the initial brainstorming process involved quite a bit of trial and error.  It was a game about heartwarming animal relationships, so should the music feel old-fashioned and lyrical?  On the other hand, the game focused on strategy mechanics, so should the music feel more puzzle-like and intellectual?  Also, the game included darker underpinnings associated with suffering and dissolution, so should the music stick with a light and airy feel, or should it weave dark and sinister textures into the mix?  I tried tons of permutations on all these ideas until I hit upon the right balance of elements, and in the end my music for SimAnimals took me in unexpectedly challenging directions.  The style couldn’t have been achieved without the process of trial and error that characterized my early efforts at the beginning of music composition. As an example, I’m including here two videos showing the difference between the brightly cheerful main theme music of the game, and one of several grim and dissonant variations on the same melodic theme:


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).For Final Fantasy XV, audio programmer Sho Iwamoto created an audio engine he dubbed MAGI (Music API for Gaming Interaction).  While the audio engine accomplished tasks similar to the functionality of other applications such as Wwise and FMOD, the MAGI system also included some creative and original specializations.  “There are a lot of time signature variations or changes in JRPG music, so I designed MAGI to support them,” Iwamoto says.  Among its other capabilities, MAGI has the ability to adjust to changing tempo and time signature values across the length of a composition, allowing flexibility in the placing of synchronization points within the track.  This capability was born out of both a spirit of innovation and pure necessity.  Because Iwamoto was hired by Square Enix just two years earlier, much of the music of Final Fantasy XV had already been composed and recorded before he joined the team.  “Many of these (compositions) were not planned to be interactive,” Iwamoto admits.  Because of this, the musical compositions included variable tempos, dramatic pauses, and frequent shifts in time signature.

From the article by game composer Winifred Phillips - an illustration of the game Final Fantasy XV.In order to make these tracks interactive, Iwamoto used the MAGI system to insert custom synchronization points into each composition so that the transitions would be pleasing in a musical way.  “All you have to do is just move or delete (sync points) when you think the transitions are not musical, or add the sync points when you think the transition is musical,” Iwamoto comments.  “Sometimes these sync points can be hard to find, and you may have to wait sixteen bars or more to (find a workable) transition.”  This problem became a frequent issue during combat music, so to solve this problem, Iwamoto devised a strategy that he called the pre-end section. This was so named because it is always used as a preparation for the ending of the combat track.  “The pre-end section is designed to have more constant and dense sync points,” Iwamoto explains.  “That makes for really quick transitions.”  Existing as a separate music file, the pre-end section would always be in the same style as the main body of the composition, so the music could switch from the main body to the pre-end section smoothly.  Designed to be more consistent in terms of tempo, key and time signature, the pre-end section could accommodate lots of sync points, allowing the music to transition to a finishing flourish with almost instantaneous precision in accordance with the end of combat.  Despite its seeming complexity, Iwamoto meant for this system within the MAGI engine to be intuitive by design.  “We used very simple and basic approaches,” Iwamoto says.  “I wanted to make (MAGI) very easy and versatile.”


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).For audio director Becky Allen, simple and basic approaches are important for very different reasons.  “We are a mobile studio and we have to be very mindful of our size because of the devices we’re played on,” Allen says, describing her strategy for the MIDI music system of Plants vs. Zombies: Heroes.  “We’re always thinking about size, and ways to be smart.”  Part of that strategy involved a plan for limiting the amount of memory required for the MIDI files and their associated library of instrument sounds.  At first, those limitations were meant to be sensible but not extreme.  “80 megs,” Allen says. “40 for music, 40 for sound effects and VO.”  But then, the audio development process hit a snag when another mobile game came along and made a big splash in the marketplace.  “Along came Clash Royale, and that was a quick game, it loaded quickly,” Allen says.  “We had some pressure to come down to 30 megabytes.”

From the article by game composer Winifred Phillips - an illustration of the game Plants vs. Zombies: Heroes.In the end, the team held to 60 megabytes, but even this required sacrifices.  “We switched some things from stereo to mono but not everything. We streamed all the WAV music and we streamed larger WAV assets.” After making these and other modifications, the newly streamlined music system was ready to show how interactive it could be.  For instance, by associating the player’s relative health with a RTPC (real-time parameter control) in Wwise, the music system could adjust MIDI data on the fly in accordance with the player’s success or failure.  “If you’re up by five points all the MIDI music went up a whole step, and if you’re up by ten points all the MIDI music went up two whole steps. And it worked, it was awesome,” Allen enthuses.  “Utilizing your tools and an open mind and a flexible mind and a flexible team,” Allen says, “you can make these changes all the way along through the process.”


 

Illustration of a GDC 2017 presentation, from the article by Winifred Phillips (video game composer).If flexibility is the ultimate ambition of an interactive music system, then a procedural music system could be considered the definitive realization of that goal.  While procedural music can be viewed as the height of interactivity, Paul Weir had built such systems for games before, and his viewpoint on the technology is much more down-to-earth. “Essentially they’re glorified random file players with a little bit of logic stuck on top,” Weir admits.  Together with the development team for the space exploration game No Man’s Sky, Weir set about to see what procedural music could bring to the project.  First, Weir hired a drum-driven electronica band called 65daysofstatic, then asked them to start creating music for their science fiction game.  “We were kind of really adamant that as a band, I did not want to interfere creatively with their process,” Weir says.  “So we always said, write an album, just write us an album, and we’re not going to start telling you how to do that, because you know how to do that – because you’re the band.  So we let them go off and write a very traditional album, but in the knowledge that we were going to come back to it later and just rip it all apart.”

From the article by game composer Winifred Phillips - an illustration of the game No Man's Sky.Using Pulse (his self-made procedural music generation tool) Weir set about disassembling the elements of the submitted music so that it could be incorporated into the procedural system.  He quickly learned that it would require more than just a retooling of the original content.  “So it wasn’t a question of saying just give us the stems and we’ll do it,” Weir says. “It was like, no no no, go right back and do us more performances, take out bits, give us more drum loops, perform new guitar riffs, create new stuff – almost like kind of remixing the original tracks.”  With these musical fragments, the procedural system could then perform calculations and assemble the library of elements into new combinations in accordance with the player’s locale and occupation. For instance, the music might be triggered depending on such in-game variables as proximity to buildings and time spent walking, utilizing what amounted to a large library of musical ‘soundscape sets’ that randomly alternated depending on how long any previous soundscape set had been playing.  “In the game you don’t get the album,” Weir insists.  “The game soundtrack is bits of what appear on the album. It’s lots of bits that aren’t on the album, but it feels relatively cohesive.”


 

Conclusion

So, there we have it!  We’ve compared the viewpoints of five game audio pros discussing the interactive music systems of their projects during their presentations at GDC 2017.  Interactive music is a fascinating subject, ranging from the simplest of solutions to the most complex and intricate of designs.  With ingenuity and patience, these audio developers have introduced their own creativity into the implementation of music in games.  Their strategies and solutions can help us to broaden our minds when we’re thinking about ways to make our own music interactive!  I hope you’ve enjoyed this three-article series, and please feel free to leave your comments below!

 

Photo of video game composer Winifred Phillips in her music production studio.Winifred Phillips is an award-winning video game music composer whose most recent projects are the triple-A first person shooter Homefront: The Revolution and the Dragon Front VR game for Oculus Rift. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Follow her on Twitter @winphillips.

Video game music systems at GDC 2017: pros and cons for composers

Video game composer Winifred Phillips, pictured in her music production studio working on the music of LittleBigPlanet 2 Cross Controller

By Winifred Phillips | Contact | Follow

Welcome back to our three article series dedicated to collecting and exploring the ideas that were discussed in five different GDC 2017 audio talks about interactive music!  These five speakers shared ideas they’d developed in the process of creating interactivity in the music of their own game projects.  We’re looking at these ideas side-by-side to cultivate a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. In the first article, we looked at the basic nature of five interactive music systems discussed in these five GDC 2017 presentations:

If you haven’t read part one of this article series, please go do that now and come back.

Okay, so let’s now contemplate some simple but important questions: why were those systems used?  What was attractive about each interactive music strategy, and what were the challenges inherent in using those systems?

Continue reading

Video game music systems at GDC 2017: what are composers using?

By video game music composer Winifred Phillips | Contact | Follow

Video game composer Winifred Phillips, presenting at the Game Developers Conference 2017.The 2017 Game Developers Conference could be described as a densely-packed deep-dive exploration of the state-of-the-art tools and methodologies used in modern game development.  This description held especially true for the game audio track, wherein top experts in the field offered a plethora of viewpoints and advice on the awesome technical and artistic challenges of creating great sound for games. I’ve given GDC talks for the past three years now (see photo), and every year I’m amazed at the breadth and diversity of the problem-solving approaches discussed by my fellow GDC presenters.  Often I’ll emerge from the conference with the impression that we game audio folks are all “doing it our own way,” using widely divergent strategies and tools.

This year, I thought I’d write three articles to collect and explore the ideas that were discussed in five different GDC audio talks.  During their presentations, these five speakers all shared their thoughts on best practices and methods for instilling interactivity in modern game music.  By absorbing these ideas side-by-side, I thought we might gain a sense of the “bigger picture” when it comes to the current leading-edge thinking for music interactivity in games. In the first article, we’ll look at the basic nature of these interactive systems.  We’ll devote the second article to the pros and cons of each system, and in the third article we’ll look at tools and tips shared by these music interactivity experts. Along the way, I’ll also be sharing my thoughts on the subject, and we’ll take a look at musical examples from some of my own projects that demonstrate a few ideas explored in these GDC talks:

So, let’s begin with the most obvious question.  What kind of interactive music systems are game audio folks using lately?

Continue reading

My fellow speakers at the North American Conference on Video Game Music

NACVGM-ProgramCover-sm

The North American Conference on Video Game Music begins this Saturday, and I’m definitely looking forward to giving the keynote speech there!  It will be great to talk about some of the concepts from my book, A Composer’s Guide to Game Music, and  I’m also very pleased that I’ll have the opportunity to meet such a wonderful collection of scholars in the field of game music study.  Since not everyone will be able to travel to Fort Worth for the conference this weekend, I thought I’d provide you with some of the stimulating ideas that will be enlivening the forthcoming conference.  Below you’ll find a collection of links to research papers, articles, essays, PowerPoint presentations and YouTube videos that some of the speakers from the upcoming event have previously created on the subject of video game music.

border-159926_640_white

Guitar Hero: “Not like playing guitar at all”?

by Dominic Arsenault

(Profile on Academia.edu)

Dominic Arsenault is an assistant professor in the fields of video game design, history and musicology at the University of Montreal, Canada.  This weekend he’ll be presenting a paper at the conference entitled “From Attunement to Interference: A Typology of Musical Intertextuality in Video Games.”  Below you’ll find a link to his 2008 research paper from Loading… The Journal of the Canadian Game Studies Association.  This article explores the mechanics of guitar playing in the music simulation videogame Guitar Hero, comparing this gameplay mechanic to the musicianship of playing a real-world guitar.

Link to research paper at the Simon Fraser University Library

border-159926_640_white

Wrap Your Troubles in Dreams: Popular Music, Narrative, and Dystopia in Bioshock

by William Gibbons (Twitter @musicillogical)

William Gibbons is the organizing chair of the North American Conference on Video Game Music, and teaches musicology at Texas Christian University.  This weekend he’ll be presenting the paper “Navigating the Musical Uncanny Valley: Red Dead Redemption, Ni no Kuni, and the Dangers of Cinematic Game Scores” at the upcoming conference in Fort Worth.  Below you’ll find a link to his research paper on the music of the video game Bioshock, as published in 2011 in Game Studies: The International Journal of Computer Game Research.

Link to the research paper in Game Studies: The International Journal of Computer Game Research

border-159926_640_white

Links to Fantasy: The Music of The Legend of Zelda, Final Fantasy, and the Construction of the Video Game Experience

by Julianne M. Grasso (Twitter: @_juliannemarie)

Julianne M. Grasso is currently a doctoral candidate at the University of Chicago, pursuing her degree in music theory.  She’ll be presenting the talk “Intersections of Musical Performance and Play in Video Games” this weekend in Fort Worth.  What follows is a link to a fascinating and entertaining essay she wrote in 2009 about her experience writing her undergraduate thesis on the music of Zelda and Final Fantasy for her music degree from Princeton University.

Link to Essay on the Princeton University Website

border-159926_640_white

Maps and Legends: FPS-Based Interfaces for Composition and Immersive Performance

by Robert Hamilton (Twitter: @robertkhamilton)

Professor Robert Hamilton teaches in the Department of Music at Stanford University, and is also a lecturer at the California College of the Arts on Experimental Game Development.  His presentation this weekend will be “Designing Game-Centric Academic Curricula for Procedural Audio and Music.”  Below, you can read his 2007 paper exploring a new interactive music composition system triggered by a gamer’s position and actions within an in-game virtual space. This paper was presented at the International Computer Music Conference in Copenhagen, Denmark.

Link to Research Paper at the Stanford University Web Site

border-159926_640_white

SOUNDS Like Fun and Games: Exploring the Role and Development of the Video Game Sound and Music Designer

by Christopher J. Hopkins (YouTube: hopkinschris)

Professor Christopher J. Hopkins researches chiptune music while teaching in the music department of Long Island University in New York.  This weekend he’ll be presenting a paper entitled “Compositional Techniques of Chiptune Music.”  Below, you can read an interesting PowerPoint presentation from a speech that Professor Hopkins gave about the discipline of video game sound and music at the 2013 Summer Teaching with Technology Institute.

border-159926_640_white

There’s Always a Lighthouse: Commentary and Foreshadowing in the Diegetic Music of BioShock Infinite

by Enoch Jacobus (Twitter: @enochobus)

Professor Enoch Jacobus’ fields of research include ludomusicology and music theory pedagogy.  He teaches advanced musicianship and orchestration at Asbury University in Kentucky.  At the upcoming Fort Worth conference he’ll be presenting a paper on BioShock Infinite entitled “Lighter Than Air: A Return to Columbia.”  Happily, Professor Jacobus has previously given a speech on the music of BioShock Infinite at the inaugural North American Conference on Video Game Music that took place last year, and we can enjoy that speech via the YouTube video below:

border-159926_640_white

The Origins of Musical Style in Video Games: 1977 – 1983 (Chapter 12 of The Oxford Handbook of Film Music Studies)

by Neil Lerner (Email at Davidson College: nelerner at davidson dot edu)

Neil Lerner teaches a wide assortment of music courses as a professor in the music department of Davidson College in North Carolina. At the conference in Fort Worth this weekend he’ll be giving a presentation entitled “Teaching the Soundtrack in a Video Game Music Class.”  Neil Lerner has been active with several scholarly journals in the field of musicology.  He has served on the editorial board of Music, Sound, and the Moving Image, and is currently the secretary for the Society for American Music.  He also had the honor of holding the position of president of the American Musicological Society-Southeast Chapter.  Below is a link to a chapter he contributed to The Oxford Handbook of Film Music Studies, as excerpted on Google Books.

Link to Chapter on Google Books

border-159926_640_white

Chaos in the Cosmos: The Play of Contradictions in the Music of Katamari Damacy

by Steven B. Reale (Twitter: @StevenBReale)

Steven Reale is a music theorist, ludomusicology researcher, and associate professor at Youngstown State University in Ohio.  At the Fort Worth conference this weekend he’ll be serving as the program chair. Here’s a 2011 research paper he wrote on the music of the video game Katamari Damacy for the journal ACT, published by The Research Institute for Music Theater Studies in Thurnau, Germany.

Link to Research Paper at the University EPub Library Bayreuth