Virtual Reality has stepped out of the realm of science fiction and Hollywood pipe dreams. Sooner than we think, we’ll be gaming with VR systems like Sony’s Project Morpheus and Facebook’s Oculus Rift. As a user interface for gaming, the VR helmet promises to offer a spectacular aural environment by virtue of binaural audio (you’ll find a good explanation of the binaural system here).
I recently read a great article on the Designing Sound site about the audio possibilities inherent in the VR technology, and I highly recommend it. The article discusses some of the profound differences between VR audio and traditional surround-sound audio for modern games. The binaural technology involves the use of two microphones positioned carefully to mimic the natural placement of human ears. These two mono signals allow the resulting stereo recording to convey positional information with great accuracy, making it possible for the final result to deliver an immersive “surround-sound” experience without the use of multiple speakers.
Early in my career, when I was working as a composer/sound designer for a National Public Radio drama series, I captured some rudimentary quasi-binaural recordings by employing two microphones tied to my body while recording to a portable Digital Audio Tape recorder. Thus equipped, I walked through various outdoor environments, rode on public transportation, situated myself in noisy crowds and let the people stream around me, all while recording with my two microphones. I remember that the end result was powerfully immersive. Technically, I didn’t fully achieve the true binaural effect. My microphones were not at “head-level” but were instead at hip-height and spaced a bit wider than a human head would be… so perhaps it might be considered a true binaural recording designed for a hobbit with a huge noggin. Nevertheless, the principle was very similar, and I remember how excited I was about the immersive realism of those recordings. The use of binaural recordings in VR should be an amazing contribution to the “reality” portion of the virtual reality experience.
One important question isn’t answered by the Designing Sound article – how will music be incorporated into such a system? This question seems to pose similar difficulties to those faced when incorporating music into a surround-sound mix. Do we mix the music in surround sound as well, so that the movements of players directly impacts the physical positioning of the music in the 3D world – or will this be too confusing? Conversely, do we keep the music in traditional stereo, and if so, should the music always occupy a position directly in front of the players’ faces, no matter which way they may turn their heads? It’s a complicated decision, and it looks like VR technology will only make this issue more complex.
The Oculus Rift is expected to hit the marketplace at the end of 2015, so we’ll have at least a little time to consider these problems before they’ll have to be solved.
I got pingback to this from the DS article — which I wrote. The question on music is valid but I suspect it could be treated the way it ‘normally’ is in games. I’ve found that binaural audio works amazingly well for realistic sounds positioned in a 3D world, but no so much for 2D sounds (fixed ambience, voice overs, music). Most of us are used to listening to a soundtrack to our lives with iPod like devices so maybe music can still work in stereo? What would be more interesting is if the music can be deeply embedded into the narrative — getting the player to put on a pair of ear phones or some such in-game. Pure speculation! We need to experiment.
Great article! 🙂 Also, great point about the way music could be treated in VR. The only difference I can see between surround and VR is the ability of players to turn their heads to look around, and I wondered if music may seem to be “swinging around” with the player in that circumstance? With traditional surround, when the player’s character moves around, the gamer doesn’t also physically shift position in the same way, so keeping the music’s location static might not have the same impact as it could in VR.
And yes, I’d love to see music become part of the narrative — diegetic uses of music within a game world are always fun! 🙂 Non-diegetic uses (underscore) might have to be treated differently, I think.
I love the idea of music in VR being completely embedded in the 3D experience, and as such, positional and environmental. I can totally imagine a game world like Bio Shock, with its 1920’s source music being incredibly immersive with positional diegetic music, treated in the same way as a 3D sound source. In terms of “score” I really can’t imagine what will and won’t work. As Varun says, the only analog to real world experiences we have is hearing music in a car or through headphones – there are a lot of ways to present a scored experience that really takes advantage of the ‘3D-ness’ and the ‘presence’ aspect of the player/listener. I’m wondering if a straight ‘stereo’ score would have an odd ‘overlay’ or ‘flattening’ effect and not feel right. Is score as necessary in terms of helping the player feel a certain way? Lots of experimentation to do that’s for sure!
Great perspective on this, Rob — really appreciate that! It will be very interesting to see audio teams experiment with music implementation in VR. I’m sure there will be a lot of different solutions applied to the problem, and I’m wondering if the “headphones” analogy will become a standard, or if the music will actually be woven into the three dimensional space. I’m imagining a vertical layering system based on that ‘3D-ness’ effect, and wondering how that would work. Would players accept underscore that is so directly connected to the environment? Really cool possibilities to consider. 🙂