For example, gaming-focused VR (Virtual Reality) headsets from companies including Meta, Sony, and Pico have sold millions of units, and AR (Augmented Reality) glasses from Vuzix, Microsoft, and more have found a valuable place in industry.
Earlier this year, Meta unveiled its Butterscotch prototype VR headset addressing several issues that have been holding back VR from feeling real and gave a glimpse of the experiences these devices might be able to offer by the end of the decade.
The Butterscotch display system approaches retinal resolution, which is the pixel density at which most people’s eyes cannot distinguish individual pixels and reduces the perception that the user is looking at screens. It is able to manage 56 pixels per degree (PPD) across the user’s field of view (FoV), close to the 60PPD heuristic commonly suggested, although the downside is that the FoV has been approximately halved compared to most other VR headsets.
A production-ready system would need a higher field of view, but this requires incredibly high-resolution screens and massive computing demands.
One solution, employed in Finnish VR specialist Varjo’s XR-3 headset, is to use a higher resolution foveal microdisplay in the centre of the vision, where the eye resolves the most detail, and use a lower resolution display for the rest of the visual field.
While this solution comes with added weight, complexity, and cost, it is possible that this approach will trickle down over the next decade as the technology matures.
The simpler approach will likely be a move towards large but higher pixel density microdisplay technologies in VR, with foveated rendering taking place at the software level, saving on computing power but not the number of pixels needed for the display.
According to research undertaken by IDTechEx OLED-on-silicon displays, which are more commonly used in camera viewfinders today, are likely to replace the LCDs used today to become the prevalent VR display technology in the 2030s, with VR-specific OLED-on-silicon releases from companies including BOE setting the stage for this already in 2023.
Butterscotch can also correct for the vergence-accommodation conflict which is a mismatch between the fixed focal (accommodation) distance of a binocular stereoscopic display system and the varied distances of objects from the viewer (vergence) it simulates by parallax.
This conflict can increase problems with motion sickness felt by users of XR headsets and decrease immersion since the eye is missing important visual cues for depth. Butterscotch can track where the eyes are looking using IR cameras, tweaking the scene rendering, and, most importantly, adjusting dynamically tunable lenses to focus the headset’s displays to match vergence and accommodation distances.
The lenses used in Butterscotch are moved physically by motors to vary focus, but Meta has also demonstrated geometric phase lens arrays that could be used for the same purpose. These lenses switch from a positive to a negative diopter depending on the handedness of circularly polarised input light: stack these lenses with switchable waveplates, and an extremely compact, dynamically tunable focal length lens with no moving parts and near-zero latency can be produced.
Although image quality issues remain, it seems more likely that Meta would take this approach in a commercial headset.
VR devices in 2034
The features shown off in Butterscotch will likely be required to boost immersion and comfort in VR. This is particularly true as interest develops in the use of VR headsets as a replacement for today’s workstations, since these factors are extremely important when working with text for long periods. This offers a true chance to broaden the utility of VR devices, not to mention their market size, but getting to this point will be an uphill struggle.
Meta spent $7.7 billion on its spatial computing-focused Reality Labs division in H1 2023 alone, and that’s indicative of the scale of R&D work required to make XR devices effective replacements for today’s computing systems.
Whilst Meta established itself as an XR hardware player early on, 2023 was the year when Apple finally made its first long-anticipated move here with its announcement of the Vision Pro. This headset is likely to be just the beginning of the spatial computing game for Apple.
The prize for Big Tech players in the XR market is likely establishing see-through AR devices as an eventual replacement for the smartphone, with passthrough MR, where cameras on a VR headset pass a view of the real world to the user, offering a useful stepping stone along this path.
Big Tech companies have been investing in AR software content for years, delivered through smartphones. Google’s ARCore and Apple’s ARKit have been highly influential in lowering the barriers to development here, with Amazon using the latter to develop its earliest AR e-commerce efforts back in 2017.
A stepping stone technology
In 2023, even lower-cost VR devices are routinely including passthrough MR features, which is beginning to democratise more immersive AR/MR experiences: for example, the Pico 4 includes colour passthrough to help users orientate themselves within the room.
This headset is a close competitor to Meta’s Quest 2 and, notably, Pico is owned by TikTok’s parent company ByteDance. It appears that Chinese tech giants have just as much interest in establishing themselves as XR players as their American counterparts.
Passthrough MR offers a commercially ready way to deliver MR experiences, but social acceptability is a significant problem since even the most compact and elegant VR device will always conceal the eyes behind displays.
The Vision Pro uses an outward-facing display to help address this, but even the most refined implementation of this system is likely to feel, at best, like a significant barrier from the wearer of the headset. The other significant issue is that these devices are fail-deadly and so could not be worn whilst driving, walking, in industrial settings, and so on since they are effectively blindfolds if something goes wrong.
See-through AR doesn’t suffer from these problems, offering the potential to replace smartphones as an all-day wearable access point to computing. However, so far, this approach has proven tough to get right, even for big players like Microsoft. These glasses should leave the eyes open to the real world: the crucial components used to project images to the eyes while leaving this view clear, which are known as optical combiners, have arguably offered the greatest technical challenge to AR success.
Optics roadblock
Waveguide combiners, which have generally looked like the combiner solution of choice for this since they can offer a relatively wide FoV while looking like normal spectacle lenses, have proved a challenging combiner technology to get right.
Apple, Microsoft, and Snap have all acquired firms with interests in waveguide design. One major problem is the lack of optical efficiency of many designs, increasing demands on display brightness, and therefore battery capacity and thermal management: solving this would make compact, all-day wearable AR devices more feasible.
The potential increased use of more efficient reflective waveguides over their diffractive counterparts that are dominant today is one solution here, but diffractive waveguide manufacturers counter that the manufacturability of reflective waveguides is inferior. A solution to this (extremely controversial) manufacturability problem could be a move towards moulded plastic reflective waveguides. This has the additional benefit of better mechanical properties for near-eye use compared to glass substrates but has the downside of limiting FoV, since this depends on the refractive index of a waveguide material, and polymers are inferior to glass here.
AR outlook in 2034
Waveguides are just one area of technological focus for Big Tech players with an interest in AR devices, not to mention myriad smaller players. The challenges remain significant, but the reward at the end could be huge: AR devices offer the potential for the most seamless access to digital content yet, offering even greater time in front of our eyes to the gatekeepers of this content.
While it seems unlikely that most of us will be wearing AR glasses instead of carrying a smartphone in 2034, early adopters will probably be beginning to make the switch by this point.
Author details: Sam Dale, Senior Technology Analyst, IDTechEx
