How to turn heads: Positional audio in immersive learning

The opportunities of 360 video and virtual reality (VR) are in designing highly immersive experiences for people to explore.

The ability to transport ourselves in this way extends the possibilities for engaging learners. But if the sound doesn’t align with the visuals, our brains will not be properly convinced. 3D positional audio greatly improves the delivery of a truly immersive experience. Combined with the visuals, it is the sound that creates that ultimate feeling of presence.

Our human ears pick-up audio in three dimensions and our brains process multiple cues to identify where a sound is and what it might mean. One of the most basic cues is how close something is and where it’s positioned, which we do far more effectively with our ears than our eyes. The auditory cortex is where all this information is processed. Other parts of the brain then identify a sound by comparing it to our memories of what have heard before, and then we decide our response.

Sounds can make us move – they can cause us concern, so we move away, or create a little intrigue that we look toward. Sound is able to help direct a narrative and guide the learner. Many people are still new to trying 360 video or VR material, so they may not realise they can look around. Early sound cues are a great way to get that movement happening.

In one of our recent 360 video projects, we applied a mono audio approach and compensated for the absence of full surround sound by adding bolder visual cues into the footage in post-production. But full 3D positional audio is capable of connecting sounds to objects as they move through a setting. So, even if something has moved out of sight and might be behind the learner, they can still be very aware of its presence and be compelled to turn around to see it. Conversely, if the sound is of poor quality and the learner is having to strain to hear, or the sound doesn’t fit correctly with the visuals, learners will quickly become fatigued and disengaged from the reality you are trying create. Great sound is the key to that ultimate suspension of disbelief that other methods can’t match.

Binaural recording is the creation of this realistic 3D sound. In production, it uses a range of small microphones placed around a model head. Because that head has density, these microphones capture sound as it would be received by human ears. Binaural recording captures the difference between the sound that a right ear hears and the sound that a left ear hears. Another aspect it can accommodate is how sound waves are affected by a head and body at the centre of the scene. This level of sound capture, when aligned correctly with the visuals, mentally guides us to where we need to be looking.