While virtual reality as a term has been around for quite some time, multiexperience, augmented reality and mixed reality are relatively new terms. But what do these technologies and terms encompass? And how will people-centric multiexperience impact the audio-visual business?
-
- What is multiexperience?
- How will the concept of multiexperience impact AV?
- What is virtual reality?
- What is augmented reality?
- What is mixed reality?
- How are virtual, mixed and augmented reality related to AV?
While drafting an article about virtual reality, I was reminded about Gartner group’s annual prediction of strategic technology trends for the coming years. At the Gartner IT symposium they announced the trends for 2020 and beyond. With a strategic technology trend, they mean “… one with substantial disruptive potential that is beginning to break out of an emerging state into broader impact and use, or which are rapidly growing trends with a high degree of volatility reaching tipping points over the next five years.”
Sometimes you find a technology shift or technology leap that is finally getting ready for mainstream and is relevant to AV. In this case, the technology trend that Gartner refers to is: multiexperience.
What is multiexperience?
Gartner describes multiexperience like this in their glossary: "Multiexperience refers to the various permutations of modalities (e.g., touch, voice and gesture), devices and apps that users interact with on their digital journey across the various touchpoints."
Technologies such as virtual reality (VR), augmented reality (AR) and mixed reality (MR) affect the way people perceive the digital world. A significant shift in how users both perceive and interact with the digital world will lead to a future multisensory and multimodal experience. Gartner says multiexperience currently focuses on: "… immersive experiences that use augmented reality (AR), virtual (VR), mixed reality, multichannel human-machine interfaces and sensing technologies."
Gartner has also written a definition about Multiexperience Development Platforms (MXDP), but the platforms themselves are not that AV-relevant yet: “MXDPs serve to centralize life cycle activities — designing, developing, testing, distributing, managing and analyzing — for a portfolio of multiexperience apps.”
How will the concept of multiexperience impact AV?
"Many a little makes a mickle" is a saying that could be applied to this emerging technology (which in fact is a merger of other technologies). I doubt that there will be a single moment, product or solution that we in the future will look back at and say “THERE, that was it”.
I believe the main driver for multiexperience is the development of wearables, devices that we carry around, and the need for entertainment/engagement in visual storytelling. This also ties in with the fact that we tote an incredible amount of computing power in our pockets. It is not that we have a PHONE that is also a computer, but we have a COMPUTER that also acts like a phone.
I also think that the term multiexperience is more relevant to the AV business as AV and IT
continue to merge. If you look at a theme park now scheduled to open in 2021, the LEGOLAND® New York Resort, you find some really cool attractions there. One in particular comes to mind: the LEGO Factory Adventure Ride. Here, LEGO has teamed up with Holovis (an AV company, specializing on immersive mixed reality experiences) and ETF Ride Systems/Engineering ) to create an all-new experience that will turn guests into Lego Minifigures by a combination of AI and motion-tracking sensors. And, of course, lots of displays…
Guests of the LEGO Factory Adventure Ride join Professor Brick on an adventure through the Lego Factory. At a point in the ride, the vehicles will start spinning until they end up face-to-face with a screen, where the guests will see their ride vehicle be transformed into Lego elements. After a while, the magic will happen as they will see themselves appear in the vehicle as Lego Minifigures! The recreation of their character is customized, based on the guest’s attributes, so if you have dark, short hair and glasses – your LEGO character will look the same.
The screenshot below is from the Inparkmagazine video from the IAAPA show where the ride was announced.
“The Lego Minifigure transformation is the textbook used for our HoloTrac technology, and since it’s completely invisible to the guest, the experience feels like real magic when the moment occurs,” Holovis creative director Peter Cliff told AV magazine.
Even if this is a kind of groundbreaking ride made for the entertainment industry, there is no doubt that today’s creative visual display technology will be part of the more mainstream AV solutions of tomorrow. It represents a combination of traditional screen displays and modern tracking systems with a high-end computing system to identify/help create similar looking avatars in the LEGO realm.
Let’s have a look at the developments that have led to this kind of praise for the multiexperience: virtual reality, augmented reality and mixed reality.
What is virtual reality?
According to Gartner Group: “Virtual reality (VR) provides a computer-generated 3D environment (including both computer graphics and 360-degree video) that surrounds a user and responds to an individual’s actions in a natural way, usually through immersive head-mounted displays. Gesture recognition or handheld controllers provide hand and body tracking, and haptic (or touch-sensitive) feedback may be incorporated. Room-based systems provide a 3D experience while moving around large areas, or they can be used with multiple participants.”
While most of us only have experienced VR through either games or rides in theme parks, the use of VR is rapidly growing in a number of applications. The origin of the term virtual reality is hard to trace. Elements of virtual reality appeared as early as in 1860s with Antonin Artaud who took the view that illusion was not distinct from reality, that spectators of a theatrical play should suspend disbelief and see the drama on stage as reality. However, the more modern virtual reality concept has roots in science fiction.
The first virtual reality installation was Morten Heilig’s “Experience Theatre” in 1950. Five short films were displayed in a device named Sensorama, where the human senses – vision, sound, smell and touch – were engaged to create an all-new experience. The Sensorama simulated a motorcycle ride through New York. The spectator sat in an imaginary motorcycle while experiencing the street through the films. Fans generated a breeze and olfactory effects were triggered when due, with a waft of pizza and petrol.
In the 70-90s, the industry of VR applications was mainly for medical, military, flight simulators and the automotive industry, and virtual reality headsets with decent field of view and sense of depth were developed. Eric Howlett developed the LEEP (Large Expanse, Extra Perspective) optical system and it was improved by NASA in their VIEW (Virtual Interactive Environment Workstation) mid-80s.
From the mid-80s onwards, there has been a tremendous refinement of the VR industry. For the last five years or so, headsets have been commercially available even if you don't have a huge budget, or are engaged in military or research sectors. For example, Sony announced their VR headset for PlayStation 4 in 2014. Numerous games/solutions have been released, but the big commercial success has yet to be seen.
If you want to read more in detail about virtual reality and the MANY steps from the 70-80s until today, I recommend the Wikipedia page on VR which is very comprehensive.
What is augmented reality?
Augmented reality, AR, is when virtual objects, such as text, graphics, audio, integrate with real-world objects in real-time. What makes AR different from VR is the “real-world” element. Augmented reality enhances and adds value to the user’s interaction with the real-world, while virtual reality does not integrate the real-world, but imitates or simulates it.
A very basic example of augmented reality is from the mobile game Pokémon Go, illustrated by a Combee flying in my office space. In this case, I use my mobile phone to create an enhanced real-world experience in my office using augmented reality.
With the rapid development of powerful smaller computers, optical systems and tracking sensors, the development of hardware that supports augmented reality has also seen a rise in later years.
One of the most hyped augmented reality pieces of hardware was Google Glass. It failed to achieve commercial success and caused quite a debate related to privacy (the glasses had a built-in camera that allowed people to record what they were seeing). Google later pulled the commercial version and announced Google Glass Enterprise Edition 2 in May 2019. The Google Glass is a small, lightweight wearable computer with a transparent display for hands-free work – in fact a traditional head-mounted display where it overlays information and you can give commands to the Glass. Take a look at this demonstration video that shows how it appears from the perspective of the eye.
While Google Glass is a traditional, augmented-reality set of glasses, Microsoft announced and shipped a pre-production version of HoloLens in 2016 and HoloLens 2 was announced February 2019. The main differences between HoloLens and Google Glass are:
- Size of glass for visual overlay (field of view)
- Gesture control
The area of the Google Glass that will playback computer-generated content is much smaller than in the HoloLens and the Google Glass is controlled by tapping/touching the physical device, while the HoloLens has a broader field-of-view for content and tracks the hand gestures in front of the glass – you do not need to touch anything physically to interact.
The latter part is what actually brings the HoloLens into the category "mixed reality" and not a pure augmented-reality device.
What is mixed reality?
Microsoft has a definition: “Mixed reality is the result of blending the physical world with the digital world. Mixed reality is the next evolution in human, computer, and environment interaction and unlocks possibilities that before now were restricted to our imaginations. It is made possible by advancements in computer vision, graphical processing power, display technology, and input systems.”
What is the difference between mixed reality and augmented reality? Mixed reality is AR with an additional system for handling input/control through gestures or other immersive technologies.
The first, immersive, mixed reality system that provided sight, sound and touch was developed in 1992 by US Air Force and named “Virtual Fixtures”. The project demonstrated that human performance can be enhanced by placing virtual objects on top of a person's direct view of a real physical environment. That's exactly what is done with Microsoft HoloLens. Another notable development in the realms of mixed reality is Magic Leap, founded in 2010 and one of the most hyped companies/products – having raised more than USD2.6 billions since its start. They released their first version of the “Magic Leap One” in 2018. You'll find a link to the first concept video here.
The benefits of a mixed reality device are many, from remote assistance, guides and tutorials, physical space walk-through with architect overlays, etc. A real-world application that utilizes HoloLens is Trimble, which provides precise alignment of holographic data on a job site, enabling workers to review their models overlaid in the context of the real, physical environment.
Other notable developments in the field of mixed reality are bionic contact lenses that can display augmented reality and virtual retina displays (the image is projected/scanned directly onto the retina of the viewer’s eye). While these two technologies seem to be further down the road, I think it is safe to say that they will inevitably be commercially available. Just don’t hold your breath.
How Are vR, mR and ar related to AV?
Audiovisual technology has been, and is still, an integral part of the development of virtual reality applications. No visual display, no virtual reality. In Morten Heilig’s Sensorama machine he needed to play back video (in 3D!) and audio in stereo.
With the rapid development of powerful computers and smaller and brighter (digital) projectors, the market for simulators, visualization centers using domes/planetariums, 360-degree projection or CAVE systems have grown tremendously. While most of these installations are handled by specialized companies, they are clearly within the AV market. And they are using everything from specialized to COTS (commercial-off-the-shelf) products.
There are companies in AV that have specialized in creative, immersive and multi-sensory experiences, such as the aforementioned Holovis, Alterface or Trio-Tech. And there are many AV companies that support them, delivering subcomponents, consultancy or installation and operation. The same applies quite naturally to augmented reality as well as mixed reality. Both technologies depend on a visual display system (often also audio) combined with or without interaction.
Conclusion
It seems quite clear that developments within the audiovisual industry impact the VR world – and not the other way around. The introduction of higher resolution projectors and displays has an impact on the quality of the many VR, AR and MR systems out there.
Higher resolution yields better, more lifelike, detail and as a consequence better experiences in the virtual world.
