image
Kim Nevelsteen, PhD
Metaverse

XR in Metaverse

In Nevelsteen (2018), I argue that the Metaverse will necessarily be Mixed Reality (AR/AV), because the Internet is that already i.e., there are already virtual worlds which are augmented with the physical world. In Nevelsteen (2021), it was clarified how the current Metaverse definition (based on including the Internet and existing virtual worlds) is more expansive than the book definition. The current Metaverse definition (Dionisio, Burns III, and Gilbert 2013) contradicts the dystopian book definition (Bar-Zeev 2021b), because technology is the driver of the definition. This is emphasized by people calling for an ‘open’ Metaverse.

Here we can analyze how the Metaverse intersects with the set XR i.e., Augmented Reality (AR), Augmented Virtuality (AV) and Virtual Reality (VR). With the increased consumer popularity of Virtual Reality (VR) there is an increase in the number of peripherals to help users be immersed in the virtual world via VR e.g., head-mounted displays. Access to the Metaverse needs to be adjusted for each access type, similar to how web sites adjust to different display sizes. It is important to note that while the user is experiencing VR, their body remains in the physical world.

AR is also gaining popularity, with AR enabled phones in everyone’s pockets. Using the definition of a ‘virtual world’ (Nevelsteen 2018), it is possible to determine if the virtual of AR qualifies as a virtual world i.e., quantifying its ‘worldliness’. Information can be spatialized by coupling information with physical world locations (e.g., geospatial or indoor) and then presented to the user via AR. Such a virtual experience lacks the shared virtual spatiotemporal persistent environment of a virtual world. It is also possible to use a virtual world to mirror entities and objects in the physical world. It is not required to display all of the virtual for the user. This is similar to the technique of using a virtual world as a ‘behind the scenes resource’ in pervasive applications (Nevelsteen 2016).

If the "virtual spatiotemporal environment for interaction" (Nevelsteen 2018) of AR is not worldly enough to be classified as a virtual world, and the physical world is increasingly augmented by it, then AV is obtained, and eventually, a VR experience. However, the virtual spatiotemporal environment for interaction for that VR experience can still not be called a virtual world, unless all the characteristics (Nevelsteen 2018) of a virtual world are met. In other words, not all VR experiences are virtual worlds either.

AR can be referred to as "the virtual overlaid on top of the physical world" (Nevelsteen 2016). So as not to overload the user with information, dissimilar content can be overlaid on the physical world in ‘layers’ (Bar-Zeev 2021a). It is unclear whether AR should be considered one virtually augmented world or if multiple are possible. The majority of the world in AR is physical, otherwise it would be AV. Swapping out virtual content could potentially simulate a switching of worlds for the user, but if the majority of world is the physical world, then it is unclear whether the content switch is enough to constitute different worlds. The same question is applicable to AV. Virtual world techniques such as ‘instances’ and ‘phasing’ might be applicable to AR as well.

The Metaverse is the collection of virtual worlds. Perhaps the easiest example of connecting virtual worlds, is the teleportation between game worlds e.g., Minecraft and Doom. Such examples of such teleportation are plentiful; let’s try a different story from the perspective of AR.

Our user, Hiro is playing World of Warcraft (WoW); he started playing WoW on Desktop, but is now playing in VR, because it was updated with VR support back in 2023. Hiro receives a phone call in VR, mixed into his goggles; his friend Y.T. has invited him over to play in her holodeck. Hiro turns on the passthrough of his goggles; WoW disappears into the background, but the chat window from WoW is still displayed on his HUD (Heads-Up Display). Hiro leaves a virtual note, on the door that he will be back later. Hiro exits the building and contacts the taxi service for a ride to Y.T.’s location. His mobile carrier is using a virtual world for ‘spatial computing’ (Shekhar, Feiner, and Aref 2016) which tracks Hiro’s location. His location is passed to the taxi service, which is also using a different virtual world to track all autonomous taxis and all routing information shared between taxis. Hiro has some time to kill before the taxi arrives, so he has some transcribed chat in WoW in the meanwhile. Hiro enters the taxi, turns off passthrough, but enables proximity sensors and physical world audio tracking. All close proximity objects in the physical world have a ghostly shadow in Hiro’s virtual world. He’s back in WoW now. Hiro calls up a tiny taxi routing to be displayed in the corner of his HUD. While driving the Smart City is orchestrating city events such as traffic lights and emergency vehicles using a virtual world for spatial computing. Hiro bids farewell to WoW – the WoW chat window closes – and he teleports to a virtual shopping mall, a huge 3D subverse. Hiro finds a suitable gift for Y.T. and places a bid on one of the Blockchains. Before the taxi arrives the taxi service alerts Hiro that his trip is almost at the end. Hiro enables passthrough again and turns off proximity sensors. The autonomous taxi AI also signals his arrival and the taxi routing on the HUD closes. Hiro exits the cab and taps into the virtual world of Y.T.’s building and passes along his identity. It detects the standing invite from Y.T. As Hiro heads up to Y.T’s place the building monitors (through IoT devices) that Hiro is indeed going to the correct location. Hiro enters Y.T’s place and immediately shares the virtual world of Y.T’s holodeck, since he’s been there before. Hiro’s goggles can now overlay virtual data on top of the virtual data of the holodeck, which is an overlay on top of the physical world. The concept of ’pervasive virtuality’ (Valente et al. 2016) is used to enhance the physical world so that all of Hiro’s and Y.T.’s senses are accessible by the holodeck. With input from the physical world being simulated, Hiro and Y.T. are now free to be fully immersed in game worlds; they jump from one virtual world to the next as they desire. A notification appears on Hiro’s HUD that he won the bid; he can now present Y.T. with her gift.

Using passthrough on his goggles, Hiro was in AR several times during the story. Hiro was never overloaded with information. The Metaverse of the future, with respect to AR, is not about having all existing data ‘spatialized’ (Bar-Zeev 2021a) out in the world, but that entities and objects exist in one or more virtual worlds, which together make up the Metaverse. That is exactly why Blockchain is an interesting technology for the Metaverse; it is a registry of objects that can be used across virtual worlds. The Metaverse is not just an ‘Internet of People’ (Bar-Zeev 2021a), but interconnected virtual worlds having ‘agents’ (Nevelsteen 2018) i.e., human or software.

Using this story, we can count the number of virtual worlds, which together would form only part of the Metaverse: 1) WoW; 2) Hiro’s home world; 3) Hiro’s mobile carrier; 4) taxi service world; 5) the Smart city-wide world; 6) the shopping mall subverse of many virtual worlds; 7) Y.T.’s building including IoT devices; 8) Y.T.’s holodeck; 9) and finally, the many virtual worlds through Y.T.’s holodeck. While Hiro is waiting for the taxi, he is in 4 virtual worlds (WoW, the mobile carrier’s world, the taxi service’s world and the Smart city) and AR. While ‘in’ (Bar-Zeev 2021a) (i.e., immersed in) AR, Hiro was not immersed in the Metaverse, but a virtual representation of him was still present in multiple virtual worlds.

The Metaverse can never entirely including all of AR, because experiences not linked (i.e., offline) are not part of the Metaverse. But, AR can not ‘subsume’ (Bar-Zeev 2021a) (i.e., entirely include) the Metaverse, because there will exist virtual entities and objects beyond AR. Virtual worlds can, however, be used to drive AR, with those virtual worlds being linked to the Metaverse.

Errata

  • 2021-09-15 Added the usage of a virtual world as the ‘behind the scenes resource’ for AR. Thanks to comments from William G. Burns III.

References

* featured image by This is Engineering @thisiseng