1: Workshop on six degrees of freedom (6DoF) Virtual Reality audio capture and playback using multiple Higher Order Ambisonics (HOA) microphones
By Tomasz Zernicki, CEO of Zylia
A statement of objectives
This workshop is addressed to all who wish to capture immersive audio scenes for virtual reality or music production. Especially, this workshop will focus on a strategy for recording sound and enabling six-degrees-of-freedom playback, making use of multiple simultaneous and synchronized Higher Order Ambisonics (HOA) recordings. Such a strategy enables users to navigate a simulated 3D space and listen to the six-degrees-of-freedom recordings from different perspectives. Additionally, during this workshop, we will describe challenges related to creating a Virtual Reality based navigable 3D audiovisual playback system.
The authors will also present practical considerations related to the usage of many spherical HOA microphone arrays, using the ZYLIA ZM-1S as an example of 3rd order HOA microphone. In that context, several examples of 6DoF audio will be shown. Some of them are from recording sessions that were made to capture full sound fields of musicians performing in a closed space. The recording sessions have different arrangements of several 19-capsule ZYLIA ZM-1S spherical microphone arrays. One of the demonstrated recordings will be an example of recordings done in August 2020 with the Poznan Philharmonic Orchestra using 30 units of ZM-1S.
Moreover, a navigable playback system implemented in WWISE and Unreal Engine 4 will be demonstrated, in which a user can navigate and listen to live recorded 6DoF sound. This was made possible by creating a grid of adjacent Ambisonics sound spheres organized in an analog way to the microphones’ positions during recording. The output of the combined Ambisonics spheres is then sent to a binaural decoder. Each Ambisonics spherical sound field contains spatial information regarding direction, but it cannot reproduce variable depth or distance between listener and sound sources. But in this setup, the sound from neighboring sound fields is projected beyond an established arbitrary radius. Therefore, the interpolation of multiple Ambisonics spheres provides additional volume and tonal balance variation, as the user navigates around the 3D space, enabling a sense of depth that a single sonic sphere could not provide.
Further examples of productions in the genre of opera and concert performance illustrate experiential gains of naturalistic renditions as well as creative applications of audiovisual volumetric simulations. Along these lines, the workshop also discusses how artifacts arising from 6DoF post-conversion may be mitigated for perceptual realism, as well as enhanced and highlighted for creating a hybrid format that embraces both the stage presence as well as the artificial nature of a concert setting. This will be weighed against the circumstances of the recording situation where limitations set by the content, location, staging, budget, and compliance of artistic team members contribute hierarchically to the selection, layout, and handling of recording equipment as well as post-production workflows. Likewise, we showcase the need to consider diegesis, agency, and attention centering of the visual and auditory field at the stage of the recording plan. In this respect, the workshop will also touch issues on planning for future audience experiences by tapping into the cognitive mechanisms of immersive experience, as well as its balanced application within the context of VR.
During the workshop, the authors also address challenges related to 6DoF audio capture, e.g. how to deal with a large volume of data, file synchronization, and optimal distribution of many microphone arrays.
A description of the intended audience
This workshop is addressed to all music enthusiasts who want to learn methods for navigable 3D audio recording and postproduction. Additionally, we welcome everybody interested in the recording of immersive audio for Virtual Reality experiences and games. Also, scientists interested in spatial sound can leverage the content of this workshop in their researches connected with Ambisonics audio.
A list of speakers with an outline of their planned presentations:
Tomasz Zernicki, co-founder, and CEO of Zylia (www.zylia.co) innovative company which provides tools for 3D audio recording and music production. Tomasz holds Ph.D. degrees in Electronics and Telecommunications. His professional interests concentrate on: 360/VR music production, spatial sound processing, and recording. As an audio expert, he takes an active role in the Moving Picture Expert Group (MPEG) standardization committee work. He has significant contributions to the audio compression standards such as MPEG-D USAC and MPEG-H 3D Audio. He was also awarded by The Polish National Centre for Research and Development for outstanding young scientists in Poland. Moreover, he was selected as one of the New Europe 100 challengers and innovators. More Info : http://pl.linkedin.com/in/tomaszzernicki
Tomasz Ciotucha, Creative Engineer at Zylia. Besides being a sound engineer, Tomasz is also a student of electroacoustic composition in Poznan. He is a graduate of bachelor studies in sound engineering and master studies in acoustics, also in Poznan. He focuses on audio and audiovisual art, encompassing different approaches in his works - from composition, to sound design and programming.