Metaverse Interoperability Space

TL;DR

  • Users of the Metaverse will be spread across worlds; high concurrent user count is only required when users converge on a single (unsharded) area. Region-based scaling is already offered as a service by SpatialOS and Big World Technology.
  • Interoperability for a MetaGalaxy (i.e., a subverse of the Metaverse) is easier than for the Metaverse, because the authority of the subverse dictates the standard.
  • Updating standards and frameworks causes fragmentation of the interoperability space and risks the creation of legacy systems.
  • If web standards are to form the Metaverse, then native OS applications would have to be re-engineered for interoperability at cost.
  • If native OS applications are to be part of the Metaverse, the OS would be responsible for switching between worlds.
  • Client-side interoperability allows for Ubiquity i.e., users to access the Metaverse on all devices; Server-side interoperability allows for virtual worlds to use services provided by other virtual worlds.
  • It will be hard to support some low-power devices if the footprint for interoperability is large e.g., a peer-to-peer system.
  • There is already fragmentation in the blockchain interoperability space; virtual world owners can choose to rebuke blockchain leading to potentially more fragmentation in the Metaverse interoperability space.
  • It doesn’t matter if identity or assets are fraudulent on blockchain or through the authority of another virtual world, administration is required to correct the situation.
  • From an interoperability perspective, trusted virtual worlds can offer identities and assets, similar to how blockchain provides them; the Metaverse can form through the self-organization of various agreements, blockchain included.
  • IPSME can be used for both client-side and server-side interoperability, or between blockchains; systems can be integrated without being taken down and re-engineered; avoids the need for standardization and supports legacy systems; and, reduces the complexity of integrating systems.
  • IPSME can enable an ‘industry of integrations’; anyone knowing two systems could write a translation and market the translation.

Web servers, serving up web sites, are linked together through the Internet to form the World Wide Web. The ability to link through the Internet is referred to as technical ‘interoperability’ (Noura, Atiquzzaman, and Gaedke 2019). This article will concern itself with higher levels of interoperability e.g., syntactic, semantic. The Internet as an enabler of networking allows for other types of servers to also be connected e.g., email, file transfer and databases. This article will focus on the higher levels of interoperability to connect ‘virtual worlds’ (Nevelsteen 2018) and ‘subverses’ (Nevelsteen 2021a) together.

In Nevelsteen (2021a), I argue that interoperability "is the determining factor for individuals as to how close we are to obtaining the Metaverse".

Before diving into interoperability, I will first speculate that there are basically 3 ‘camps’ to which people align, with respect to how the Metaverse will evolve.

  • The first camp is the ‘Web camp’; web standards will continue to expand and evolve, and form the basis of the Metaverse (Louis 2021). This would mean that all virtual worlds in the Metaverse would be accessible via an Internet browser available, on each Operating System (OS).
  • Those in the second camp, the ‘Blockchain camp’, consider blockchain technologies so profound that blockchain will form the basis for the Metaverse. I think it is fair to say that the Metaverse OS (Burke 2021) and the Crucible Network fall in this camp.
  • The third and last camp is the ‘camp of Everything’, which harbors the idea that everything should have the opportunity to connect to the Metaverse, without costly re-engineering for interoperability e.g., converting to web applications. I speculate that the Metaverse Primer (Ball and Navok 2021) is written in this context e.g., "we want as much of the world to integrate into the Metaverse as possible".

Basics: serving up one world

A virtual world server can be a single machine that servers up a shared world for its users. To connect to the virtual world server users are often running special client software on their device. If the virtual world is web based, the Internet browser is the client software.

Instead of a single server, a clusters of servers can be used to scale up a service to serve a massive amount of users e.g., cloud computing. Such a cluster forms a multi-server architecture (Yahyavi and Kemme 2013). This technique works for web sites (e.g., Google search) as well as for virtual worlds. Virtual worlds can be scaled up by partitioning virtual space using ‘regions’ and/or ‘shards’.

“Using regions, the virtual space is divided into static or dynamic areas, with each area handled by a different group of servers; players can still move throughout the entire virtual space. With shards, players are divided up into groups and assigned to a unique copy (a shard) of the virtual space, with each shard handled by a different group of servers; players are prohibited from moving between shards. Shards are copies of the same virtual space that do not synchronize with each other. Note, that a shard can be divided up into regions.” (Nevelsteen 2018)

Region based scalability is now offered as a service by modern game engines e.g., SpatialOS and Big World Technology. We are still only serving up one virtual world, but to a massive amounts of users. World of Warcraft was divided up into shards; each virtual world was referred to as a ‘realm’. In Nevelsteen (2018), whether a world is contained in one shard determines if it is one or more virtual worlds. Because each realm is a different shard, each realm is a different virtual world of the same game.

Concurrent User Count, Continuous Space

Users of the Internet as a whole are spread out on different web sites, hosted by many servers. A web sites doesn’t usually go down unless an extreme amount of users try and access the same site simultaneously, overloading it i.e., essentially a Denial of Service Attack. Similarly, in the Metaverse, consisting of a collection of many virtual worlds/subverses, users will also be spread amongst all participating worlds and subverses. Server load becoming a problem when users converge on a single (unsharded) area e.g., in EVE Online, they used game design in an attempt to avoid a converging of users (Emilsson 2010). RP1 advertises a "persistent, unsharded, real-time platform with infinite scalability" for the Metaverse i.e., an infinity number of concurrent users. RP1 doesn’t provide any details for their platform, but the claim seems farfetched considering the complexity of the number of interactions is O(N2-N). They would have to have a form or interest management to keep the number of interactions in check.

Another successful technique is to simply limit the amount of users that are able to connect to a particular service or server. Ball (2020) mentions the Marshmello concert of 2019, where 11 million people experienced the same event. In order to avoid having all those people converge on a single concert space, more than 100,000 sharded ‘instances’ (Nevelsteen 2018) were orchestrated simultaneously, each with 100 people or less.

Interoperability Introduction

A ‘MetaGalaxy’ can be formed, by having "multiple virtual worlds clustered together as perceived collectives under a single authority" (Dionisio, Burns III, and Gilbert 2013). Users of the MetaGalaxy can then create a unique virtual world inside the galaxy, according to specifications provided by the MetaGalaxy. There are many of these MetaGalaxies in existence today, sometimes being referred to as ‘a metaverse’ (Batchelor 2021). Each virtual world inside the MetaGalaxy can be served by a single server or multiple servers. To achieve a MetaGalaxy, higher levels of interoperability are required between the various virtual worlds; participants must speak the same protocol. A common protocol can be achieved through ‘standardization’ or by requiring systems to adopt a ‘framework’ as a dependency (Nevelsteen and Wehlou 2021). Because a MetaGalaxy is controlled by a single authority, that authority can specify the standard protocol or framework, all worlds must conform to, in the MetaGalaxy.

To obtain ‘the Metaverse’ (Nevelsteen 2021a), there must be interoperability between all participating virtual worlds and MetaGalaxies (i.e., subverses of the Metaverse). There are, currently, many groups developing (possibly competing) standards or frameworks for a possible Metaverse e.g., Epic Games, the Open Metaverse Interoperability Group, those advocating the Open Metaverse OS, and more.

Standardization and Frameworks

When a protocol is specified to be a ‘standard’, those who want to participant must adhere to the standard in order to communicate. If a group of ‘stakeholders’ (Madni and Sievers 2014) can agree on a common protocol, then that protocol is a standard for that group. The process of standardization can be shorter in smaller groups. Within a MetaGalaxy, it is possible that the single controlling authority specifies the standard protocols and all virtual worlds in the MetaGalaxy conform. Specifying a public standard is a more arduous process as the number of stakeholders increases. Standardization is "a slow and limiting solution which can stifle fast paced innovation" (Nevelsteen and Wehlou 2021).

Probably the most successful public standard is that of HTML. According to W3Techs (2021), on the 6th September 2021, 88.9% of all web sites support HTML version 5; 4.3% support a different version and 6.8% support something different. That means 88.9% of web sites should render in a modern browser, with 11.1% not being supported. Each update to the web standard causes further fragmentation e.g., when an update is released, the interoperability space splits into those supporting the new standard and those that do not. All systems not updated risk becoming a legacy system (Selberg and Austin 2008). Legacy systems are difficult or impossible to re-engineer and bring up to standard (Madni and Sievers 2014; Selberg and Austin 2008).

People in the web camp support the idea that web protocols will form the Metaverse. This means existing native OS applications would have to be re-engineered (if possible) to web applications e.g., via WebAssembly (Louis 2021). Such re-engineering can be costly.

If disparate standards for the Metaverse arise, the interoperability space for the Metaverse also risks fragmentation, with each standard only gathering a smaller share. The stats by Levin (2021) can be used as an example of a fragmented interoperability space for Internet APIs in 2014; the share for REST, SOAP, JavaScript and XML-RPC being 69%, 18%, 5% and 2%, respectively; 6% other.

Frameworks are similar to standards except that, instead of a protocol, a framework (e.g., Software Development Kit) is provided which must be conformed to. The framework implements the common protocol. As long as systems use the latest framework, communication compatibility can be maintained. Updating the framework requires the re-engineering of systems to adhere to the update, or risk legacy systems.

Game Engines

Ball (2020) mentions that game engines such as Unity or Unreal engine could implement common protocols allowing for the formation of the Metaverse. Unity and Unreal are already frameworks for a large share of video games. A tactic, to form the Metaverse, could be to implement interoperability in these existing frameworks, to try and obtain unanimous adoption. Epic Games already offers "online services" via a framework for any game developer wanting support services such as voice chat, in-game achievements, matchmaking and more; it is only a small step to provide more interoperability services.

The problem is that competing standards and frameworks effectively fragment the Metaverse interoperability space e.g., if Unity and Unreal implement competing protocols.

If Unity and Unreal were to implement the same protocols, then the adoption share might be quite high. All worlds/subverses of the Metaverse would have to either use Unity/Unreal or implement the standard protocol themselves. The problem with this approach is that, after each new release, existing systems must be re-engineered to support the release; this re-engineering can be costly. Failure to update risks creating legacy systems. This tactic is in the camp of Everything. For the tactic to work, fragmentation would have to be low; a standard with a large share, that is not updated often e.g., like HTML. The slow updating of the standard could, however, stifle the fast paced innovation for the Metaverse.

If the protocol implemented by Unity/Unreal was the web standard, then the adoption share could be even higher. It is already possible to export a project for web and native platforms, using Unity. Unless all native OS applications are re-engineered to web applications, as described in the Web camp scenario above, this approach is in the camp of Everything. The problem being that many existing applications are not web applications. The OS would be responsible for switching between native application and the browser for web applications, for which there is currently little or no interoperability.

Client-side vs Server-side

‘Ubiquity’ (Nevelsteen 2021a) for the Metaverse means access via any device e.g., desktops, laptops, gaming consoles and more, using text descriptions, 2D, 2.5D or 3D. This would be the client-side of the Metaverse interoperability space. Fragmentation of this space is dependent on the degree of devices being supported, and is especially likely in early stages of interoperability. Users without a particular device could be excluded from accessing some virtual worlds e.g., "all of these virtual worlds [Cryptovoxels, Decentraland, etc] still require at least a smartphone, which still currently excludes 6/10 of the global population" (Burke 2021). Any interoperability solution client-side, will have to cater and maintain compatibility with various devices; failure to do so will also result in a greater degree of fragmentation. Interoperability client-side means protocols are in place so that clients can switch virtual worlds/subverses, similar to how a web browser switches web sites when the user clicks a link.

Ball (2021a) states that, "we need to think of the Metaverse as a sort of successor state to the mobile internet". With respect to the fact that interface devices (i.e., client-side) are getting smaller and more personal, this would be correct. However, with respect to interconnecting virtual worlds (Dionisio, Burns III, and Gilbert 2013), from the server- side, it would be more correct to consider the Metaverse as a successor of the Internet in general (Nevelsteen 2021a). Server-side interoperability allows virtual worlds to make use of services from other servers. The reusability and interoperability of such server-side services converge into strategies such as Service Oriented Architectures (Madni and Sievers 2014). Server-side interoperability includes the usage of Blockchain technologies and connects the Internet of Things (IoT).

Access via Device

If user devices are to be connected to the Metaverse, those devices must support client-side interoperability. If we specify a Virtual Reality (VR) head-mounted display as the access device, then such a device has an OS with native applications running on device; one of those applications can be a web browser. If we subscribe to the web camp, then native applications would be disconnected from the Metaverse and the web browser used by the user to navigate from one virtual world/subverse to the next (including possible local on devices worlds). If native applications are to access the Metaverse, then the camp of Everything is subscribe to, and interoperability must sought between native OS applications and the web browser e.g., to allow switching between browser and native applications when appropriate.

Augmented Reality (AR) can be used to access the Metaverse (Nevelsteen 2021b) e.g., through a mobile phone. The scenario on a mobile phone is similar to that of the VR HMD i.e., an OS running native applications, including a web browser. For other devices (e.g., smart glasses, contact lens, autonomous vehicles), a web browser might be absent and only an architecture with an SDK might be available. This is problematic for the Web camp, unless custom web browsers are created for those platforms. The conclusion for the camp of Everything is that the OS is the interface for the Metaverse, not the browser. If the OS is the interface, that means Steam, Apple, Facebook and SnapChat have control over their respective platforms: Valve Index, iPhone/macOS, Oculus and Spectacles. This means those companies have control to promote or stifle interoperability e.g., Apple being the gatekeeper for applications installed on their platform or Oculus blocking competing HMDs from accessing their content.

Some devices are such low-power that there are limits to the interoperability footprint possible on device e.g., wearables or IoT devices. A large interoperability footprint could be problematic for the Blockchain or Web camps. Low-power devices can make use of gateways as mediators, but this pushes the problem of interoperability to the gateways e.g., competing gateways (Noura, Atiquzzaman, and Gaedke 2019) already fragment the IoT interoperability space.

Decentralized or Distributed

It has been said that ‘decentralization’ is a characteristic of the Metaverse (Nevelsteen 2021a). It has already been discussed how a distributed system can be used to scale one virtual world to serve a massive amounts of users. The approach is still a centralized one, in the sense that all users connect to one centralized world, consisting of a distributed multi-server architecture. If the Metaverse is "a collection of many virtual worlds/subverses" (Nevelsteen 2021a), then that collection is a decentralized one; many user, each behind a client device, connected to one or more virtual worlds, with each world perhaps a distributed system of servers.

The Metaverse would still not be a fully distributed system, unless clients connected to each other i.e., a ‘peer-2-peer’ (Yahyavi and Kemme 2013) system where every compute node is both client and server. Those in the Blockchain camp might argue that the Metaverse will be an ultra-large peer-2-peer system. Such an architecture could prove problematic for some devices e.g., mobile devices without constant availability are not ideal for server functions, and neither are low-powered devices lacking performance. The interoperability footprint is too high for such devices. The Metaverse would lack ubiquity, meaning the interoperability space would not be covered.

One might suggest a hybrid architecture; peer-to-peer combined with a server-based architecture (Yahyavi and Kemme 2013). But, the purpose of such an architecture is not to lighten the load of each node, but to ensure proper operation of the system e.g., avoid cheating.

Blockchain

The next generation of the web is said to be "increasingly decentralised and based on user centricity and the sovereignty of their data and wealth ... a paradigm ultimately based on blockchains" (Burke 2021). Blockchain technologies have the potential to allow users to utilize a single identity with assets, in one virtual world/subverse to another, across the Metaverse. Ball (2021b) states the shortcomings of blockchain, "specifically because of their decentralization", to currently be: costly transaction fees, slowness and energy use.

Blockchain is a decentralized peer-to-peer technology, and Burke (2021) has peer-to-peer as the basis for the Metaverse OS. But, peer-to-peer is not suitable for all devices (e.g., low-powered), making it hard for blockchain to obtaining ubiquity, risking fragmentation of the Metaverse interoperability space. This makes it problematic for blockchain to be the basis for the Metaverse, as prescribed by those in the Blockchain camp. Ball (2021b) already notes that "all of the major NFT platforms and blockchain-based worlds are browser-based and lack console releases and mobile applications".

Supporting blockchain requires implementing standard protocols or frameworks. There are already many different blockchains in existence today (i.e., competing standards), with blockchain interoperability as an open issue. If there is already fragmentation of the blockchain interoperability space, then why would there not be an increase in fragmentation when considering the entire Metaverse i.e., fragmentation of the Metaverse interoperability space. With an open Metaverse, anyone can add a server to the Metaverse and be the authority of the virtual world on that server. That authority can choose whether to support or rebuke blockchain leading to potentially more fragmentation.

In Nevelsteen (2018), virtual worlds are distinguished based on their single shared data space, a shard. Blockchain technology can be a horizontal layer overlaid across the vertical silos of virtual worlds. In the extreme case, a virtual world could primarily be comprised of only blockchain elements. This means virtual worlds become "increasingly interoperable and interconnected to the point it will be hard to distinguish them as separate but rather different instances of a whole" (Burke 2021). It is not difficult to understand why those in the Blockchain camp already call this collection of blockchain connected worlds ‘the Metaverse’.

Although blockchain has incredible potential, there are still some fundamental problems with respect to interoperability. An identity can be registered on a blockchain allowing worlds to authenticate identity. From an interoperability standpoint, this is identical to using another trusted virtual world as identity authenticator, provided server-side interoperability exists allowing one virtual world to query the other. If an identity is spoofed, then it doesn’t matter if it is on blockchain or maintained by another virtual world, administration will be required to correct the fraudulent behavior. This is currently the case for many artists who are having their art "stolen" e.g., minted on the blockchain and sold by a fraudulent artist. If a virtual world in the Metaverse is malicious, it could use blockchain as a source of data, but have a blatant disregard for rights and permissions e.g., allowing users to import any blockchain data into the world, refuse to import a users data or neglect to revoke access to an item when ownership was transferred. The scenario is the same if another virtual world was the authority over the virtual item e.g., World of Warcraft dictating who knows an ancient artifact.

Having blockchain as an authority for identity and assets across worlds could be powerful, especially since the authority would not be under the control of a single company or organization. However, I think it would be naive to discount the possibility that other virtual worlds/subverses can also offer their identities and assets out to the Metaverse. This is exactly what server-side interoperability offers; virtual worlds can provide services to other worlds. This is especially true for virtual worlds that do not want to support blockchain or those worlds that would like a hybrid approach. Instead of trying to blanket the Metaverse interoperability space with one overarching technology, the Metaverse could form through the self-organization of various agreements between worlds, blockchain included. This is exactly why server-side interoperability is important.

IPSME

What does IPSME (Nevelsteen and Wehlou 2021) have to offer?

IPSME is not a protocol or framework, but a set of conventions, with a super small footprint, making it possible to implement on most any system (i.e., even low-powered devices); the pubsub dependency forming the messaging environment usually already exists on most platforms. This means IPSME can be utilized for both client-side and server-side interoperability, interoperability between competing blockchains, and the interoperability for an OS to switch between web and native OS applications. IPSME describes a dynamic evolutionary architecture for a system of systems e.g., the Metaverse.

Integrations between systems are made external to the systems being integrated. When interfaces utilized in an integration are updated, the systems being integrated do not have to be re-engineered, meaning those system don’t even have to be take down. The translations can be updated or translations can added to the system dynamically and externally incorporating the updates. In the analogy by Ball (2021a), industrial plants "continued to use a lumbering network of cogs and gears that were messy and loud and dangerous, difficult to upgrade or change". Rather than requiring existing system to be re-engineered to support a particular protocol or framework, IPSME allows systems to keep using their "cogs and gears", but adds translations for interoperability.

The role of the reflectors of IPSME is allow any organization topology for communicating participants i.e., promote scalability. Reflectors provide interoperability between competing platforms (e.g., different operating systems), by leaving the protocol by which reflectors communicate as undefined. Reflectors also separate complexity so that authors of participants have a very limited scope of other communicating participants they must take into account during development.

IPSME avoids the need for standardization, by allowing any number of protocols to be used simultaneously. Deviations from a standard can be supported by adding translations. Legacy systems can be supported without requiring them to be re-engineered, which is sometimes costly or impossible.

IPSME reduces the complexity of the system of systems, by reducing the number of integrations required for a fully connected system. Translations are reusable and transitively applicable. If a someone creates a translation between interface A & B, and installs that on system X; a system Y can also install that translation to gain the same translation. If three systems, X, Y and Z, and two translations, A : X↔Y and B : Y↔Z, are communicating via IPSME, then a translation X↔Z is implicit. System X broadcasts out a message which is translated from X to Y by A, and again from Y to Z by B. As the number of translations in the system of systems grows, complexity can be detected and translations added dynamically, to increase performance and better message translation quality.

Rather than require organizations behind each system to develop an integration for each system they wish you interoperate with, IPSME offers the possibly of an ‘industry of integrations’. Anyone who knows the interfaces of two systems could write a translation, integrating the systems, and market that translation. If a translation is poorly written, then others can write a competing translation. Ball (2021a) states the success of the iPhone to be based on that "iPhone had ‘an app for that’ because millions of developers built them"; the success of the Metaverse could be because IPSME allows developers to build a translation for that.

References

XR in Metaverse

In Nevelsteen (2018), I argue that the Metaverse will necessarily be Mixed Reality (AR/AV), because the Internet is that already i.e., there are already virtual worlds which are augmented with the physical world. In Nevelsteen (2021), it was clarified how the current Metaverse definition (based on including the Internet and existing virtual worlds) is more expansive than the book definition. The current Metaverse definition (Dionisio, Burns III, and Gilbert 2013) contradicts the dystopian book definition (Bar-Zeev 2021b), because technology is the driver of the definition. This is emphasized by people calling for an ‘open’ Metaverse.

Here we can analyze how the Metaverse intersects with the set XR i.e., Augmented Reality (AR), Augmented Virtuality (AV) and Virtual Reality (VR). With the increased consumer popularity of Virtual Reality (VR) there is an increase in the number of peripherals to help users be immersed in the virtual world via VR e.g., head-mounted displays. Access to the Metaverse needs to be adjusted for each access type, similar to how web sites adjust to different display sizes. It is important to note that while the user is experiencing VR, their body remains in the physical world.

AR is also gaining popularity, with AR enabled phones in everyone’s pockets. Using the definition of a ‘virtual world’ (Nevelsteen 2018), it is possible to determine if the virtual of AR qualifies as a virtual world i.e., quantifying its ‘worldliness’. Information can be spatialized by coupling information with physical world locations (e.g., geospatial or indoor) and then presented to the user via AR. Such a virtual experience lacks the shared virtual spatiotemporal persistent environment of a virtual world. It is also possible to use a virtual world to mirror entities and objects in the physical world. It is not required to display all of the virtual for the user. This is similar to the technique of using a virtual world as a ‘behind the scenes resource’ in pervasive applications (Nevelsteen 2016).

If the "virtual spatiotemporal environment for interaction" (Nevelsteen 2018) of AR is not worldly enough to be classified as a virtual world, and the physical world is increasingly augmented by it, then AV is obtained, and eventually, a VR experience. However, the virtual spatiotemporal environment for interaction for that VR experience can still not be called a virtual world, unless all the characteristics (Nevelsteen 2018) of a virtual world are met. In other words, not all VR experiences are virtual worlds either.

AR can be referred to as "the virtual overlaid on top of the physical world" (Nevelsteen 2016). So as not to overload the user with information, dissimilar content can be overlaid on the physical world in ‘layers’ (Bar-Zeev 2021a). It is unclear whether AR should be considered one virtually augmented world or if multiple are possible. The majority of the world in AR is physical, otherwise it would be AV. Swapping out virtual content could potentially simulate a switching of worlds for the user, but if the majority of world is the physical world, then it is unclear whether the content switch is enough to constitute different worlds. The same question is applicable to AV. Virtual world techniques such as ‘instances’ and ‘phasing’ might be applicable to AR as well.

The Metaverse is the collection of virtual worlds. Perhaps the easiest example of connecting virtual worlds, is the teleportation between game worlds e.g., Minecraft and Doom. Such examples of such teleportation are plentiful; let’s try a different story from the perspective of AR.

Our user, Hiro is playing World of Warcraft (WoW); he started playing WoW on Desktop, but is now playing in VR, because it was updated with VR support back in 2023. Hiro receives a phone call in VR, mixed into his goggles; his friend Y.T. has invited him over to play in her holodeck. Hiro turns on the passthrough of his goggles; WoW disappears into the background, but the chat window from WoW is still displayed on his HUD (Heads-Up Display). Hiro leaves a virtual note, on the door that he will be back later. Hiro exits the building and contacts the taxi service for a ride to Y.T.’s location. His mobile carrier is using a virtual world for ‘spatial computing’ (Shekhar, Feiner, and Aref 2016) which tracks Hiro’s location. His location is passed to the taxi service, which is also using a different virtual world to track all autonomous taxis and all routing information shared between taxis. Hiro has some time to kill before the taxi arrives, so he has some transcribed chat in WoW in the meanwhile. Hiro enters the taxi, turns off passthrough, but enables proximity sensors and physical world audio tracking. All close proximity objects in the physical world have a ghostly shadow in Hiro’s virtual world. He’s back in WoW now. Hiro calls up a tiny taxi routing to be displayed in the corner of his HUD. While driving the Smart City is orchestrating city events such as traffic lights and emergency vehicles using a virtual world for spatial computing. Hiro bids farewell to WoW – the WoW chat window closes – and he teleports to a virtual shopping mall, a huge 3D subverse. Hiro finds a suitable gift for Y.T. and places a bid on one of the Blockchains. Before the taxi arrives the taxi service alerts Hiro that his trip is almost at the end. Hiro enables passthrough again and turns off proximity sensors. The autonomous taxi AI also signals his arrival and the taxi routing on the HUD closes. Hiro exits the cab and taps into the virtual world of Y.T.’s building and passes along his identity. It detects the standing invite from Y.T. As Hiro heads up to Y.T’s place the building monitors (through IoT devices) that Hiro is indeed going to the correct location. Hiro enters Y.T’s place and immediately shares the virtual world of Y.T’s holodeck, since he’s been there before. Hiro’s goggles can now overlay virtual data on top of the virtual data of the holodeck, which is an overlay on top of the physical world. The concept of ’pervasive virtuality’ (Valente et al. 2016) is used to enhance the physical world so that all of Hiro’s and Y.T.’s senses are accessible by the holodeck. With input from the physical world being simulated, Hiro and Y.T. are now free to be fully immersed in game worlds; they jump from one virtual world to the next as they desire. A notification appears on Hiro’s HUD that he won the bid; he can now present Y.T. with her gift.

Using passthrough on his goggles, Hiro was in AR several times during the story. Hiro was never overloaded with information. The Metaverse of the future, with respect to AR, is not about having all existing data ‘spatialized’ (Bar-Zeev 2021a) out in the world, but that entities and objects exist in one or more virtual worlds, which together make up the Metaverse. That is exactly why Blockchain is an interesting technology for the Metaverse; it is a registry of objects that can be used across virtual worlds. The Metaverse is not just an ‘Internet of People’ (Bar-Zeev 2021a), but interconnected virtual worlds having ‘agents’ (Nevelsteen 2018) i.e., human or software.

Using this story, we can count the number of virtual worlds, which together would form only part of the Metaverse: 1) WoW; 2) Hiro’s home world; 3) Hiro’s mobile carrier; 4) taxi service world; 5) the Smart city-wide world; 6) the shopping mall subverse of many virtual worlds; 7) Y.T.’s building including IoT devices; 8) Y.T.’s holodeck; 9) and finally, the many virtual worlds through Y.T.’s holodeck. While Hiro is waiting for the taxi, he is in 4 virtual worlds (WoW, the mobile carrier’s world, the taxi service’s world and the Smart city) and AR. While ‘in’ (Bar-Zeev 2021a) (i.e., immersed in) AR, Hiro was not immersed in the Metaverse, but a virtual representation of him was still present in multiple virtual worlds.

The Metaverse can never entirely including all of AR, because experiences not linked (i.e., offline) are not part of the Metaverse. But, AR can not ‘subsume’ (Bar-Zeev 2021a) (i.e., entirely include) the Metaverse, because there will exist virtual entities and objects beyond AR. Virtual worlds can, however, be used to drive AR, with those virtual worlds being linked to the Metaverse.

Errata

  • 2021-09-15 Added the usage of a virtual world as the ‘behind the scenes resource’ for AR. Thanks to comments from William G. Burns III.

References

* featured image by This is Engineering @thisiseng

A Metaverse definition using Grounded Theory

TL;DR

  • In 2003, Second Life was one of the first virtual worlds to be called a "metaverse", fulfilling the books description;
  • I argue that interoperability "is the determining factor for individuals as to how close we are to obtaining the Metaverse";
  • The majority (at least 13 of 20 responders) provide a definition of the Metaverse that is encompassed by the existing scientific definition;
  • No need (yet) to advance the currently accepted scientific definition, in order to combine the Metaverse with other technologies
  • And yes, the Metaverse might be the behind the scenes resource, for spatial computing, for all in the future.

The definition for the Metaverse has been a topic of discussion lately. Many people have provided a Metaverse definition, but few have been able to explain the relationship of all the tech terms with the greater Metaverse. In order to shed some light on how these definitions relate and perhaps why they are relative, this article will distill the Metaverse definitions using Grounded Theory. As an exploratory piece, Cathy Hackl (2021) surveyed professionals and provides 20 different definitions for the Metaverse. In her article, she asks if the currently accepted Wikipedia definitions, which are based on scientific publications, "really reflect what many in the technology and business world refer to when they use the word metaverse?" (Hackl 2021)

Before we can discuss what the Metaverse is, we must first know what a virtual world is.

Virtual World defined

To be able define "Virtual World" from a technical perspective, I used grounded theory (Johannesson and Perjons 2014) allowing words/concepts to be added to a candidate definition; if an added word/concept did not advance the candidate definition, the word/concept was superfluous. The technical definition by myself (Nevelsteen 2018) states a virtual world to be:

"A simulated environment where MANY agents can virtually interact with each other, act and react to things, phenomena and the environment; agents can be ZERO or MANY human(s), each represented by MANY entities called a virtual self (an avatar), or MANY software agents; all action/reaction/interaction must happen in a real-time shared spatiotemporal nonpausable virtual environment; the environment may consist of many data spaces, but the collection of data spaces should constitute a shared data space, ONE persistent shard."

Bartle provides another iteration of his definition of "Virtual World" in his book MMOs from the Inside Out. Bartle’s (2015) definition states "a virtual world is something with the following characteristics:

  • It operates using an underlying automated rule set-its physics.
  • Each player represents an individual "in" the virtual world-that player’s character.
  • Interaction with the world takes place in real time-if you do something, it happens pretty much when you do it.
  • The world is shared-other people can play in the same world at the same time as you.
  • The world is persistent-it’s still there when you’re not.
  • It’s not the real world."

The two definitions are essentially equal.

The Metaverse described by Snow Crash

As an exercise, let us apply Grounded Theory towards a Metaverse definition.

In the book Snow Crash (Stephenson 1992), the protagonist Hiro enters the Metaverse, which has physics and allows people to react in real-time in the shared virtual space, "people are pieces of software called avatars", and the Metaverse continues to exist when Hiro exits to "Reality" (i.e., the physical world). At this point we can start our grounded theory equating the Metaverse to a virtual world. If the words/concepts added to the definition don’t advance the definition, we can safely drop the added word/concept.

The professionals surveyed by Hackl (2021) characterized the Metaverse to be: persistent, live digital, have a sense of agency, social presence, and shared spatial awareness (Piers Kicks); a sensory leap as ‘place’, motion and presence, support our belief in that shared illusion as a space, be more closely aligned with stereoscopic perception (Kenneth Mayfield); a digital space, enabling humanity to experience the things that make us human, provide hyper social co-experiences, allow people to connect in authentic ways (Samuel Jordan); support non-fungible and infinite items and personas, be not bound by conventional physics and limitations (Luke Shabro); have voice interaction (Neil Redding); be interactive and real-time rendered (Rafael Brown). Hackl herself adds (Bhaduri 2021) that the Metaverse should allow people to "share virtual experiences" and should include Cloud Computing technologies. All these characteristics are superfluous, because these characteristics are captured by the concept "virtual world". Piers Kicks adds the Metaverse should enable the "the ability to participate in an extensive virtual economy with profound societal impact"; the economy of some virtual worlds already surpasses that of some nations (Diaz 2018). Luke Shabro’s "non-fungible and infinite items and personas" have existed in virtual worlds, just not across many virtual worlds. Neil Redding adds that there should be "effectively infinite space"; one can imagine that they mean the Metaverse to be truly expansive, but the criteria is satisfied by the virtual world in Minecraft. JB Grasset provides the criteria of "a social network based on gaming", but both "social worlds" and "game-like worlds" (Bartle 2015) exist as virtual worlds and the social networks inside those worlds are no less technically expansive then, for example, Facebook. Samuel Jordan adds people should be able to "create memories that rival physical experiences", but this is already the case for virtual worlds as observed by Pearce and Artemesia (2009). Lucas Rizotto specifies the Metaverse should be "a mass delusion", to which MMOs cater, a subset of virtual worlds.

It could be argued that virtual worlds on flat desktop or mobile screen do not provide the "hyper social" experiences and the "stereoscopic perception" specified above. To remedy this, we add Virtual Reality (VR) to the candidate Metaverse definition. Hiro enters the Metaverse through his goggles and earphones, a head-mounted display providing a VR interface to the Metaverse. Hackl and Kenneth Mayfield mention VR specifically, with others mentioning VR, in combination with Augmented Reality (AR) or in the superset XR. Our candidate definition is then: a virtual world with VR.

We have not completely satisfied the vision of the Metaverse described in Snow Crash. Initial virtual worlds were run on a single centralized server by one organization, but the book describes the Metaverse as "the graphic representation ... of a myriad different pieces of software that have been engineered by major corporations ... the money these corporations pay ... pays for developing and expanding the machinery enabling the [Metaverse] to exist" (Stephenson 1992). Second Life by Linden Labs set out to build the Metaverse in 2003; they allowed anyone to buy land/servers that would link into their virtual world. The owners of land could build whatever they desired and the Second Life platform provided the tools for content generation, including the scripting of behaviors. Second Life earned the title of "metaverse" by some, but is still only a virtual world with "no interworld transit capabilities" (Dionisio, Burns III, and Gilbert 2013). VR technology was not in the hands of many consumers, so Second Life could not entirely provide the interaction with the Metaverse described in the book. To my recollection, the book doesn’t specify any more technological aspects that would disqualify Second Life. I don’t recall if Snow Crash mentions "spatial audio" or "tactile feedback" as Neil Redding mentions (Hackl 2021). At this point it is no longer the book, but modern technological advances that are pushing the candidate definition.

A myriad pieces of software

The "myriad different pieces of software" above can be interpreted differently. Someone wanting to architect the Metaverse could try and provide a common platform which everyone would adopt, allowing various "major corporations" to setup their own virtual world and add it to the would-be Metaverse. This is what Dionisio, Burns III, and Gilbert (2013) describes as a MetaGalaxy, "multiple virtual worlds clustered together as perceived collectives under a single authority"; the someone creating the Metaverse platform being that authority. This is the dystopian future Stephenson described in Snow Crash, i.e., major corporations had "to get approval from the Global Multimedia Protocol Group". At this point, we can expand the candidate Metaverse definition to be: a MetaGalaxy with VR.

Without providing a reference, I think it is safe to say that the number of virtual worlds exploded in the last decade and we have many MetaGalaxies (subverses of the Metaverse today). The reason, the current status of the Metaverse isn’t the dystopian future described in the book is because there are already many of these virtual worlds and MetaGalaxies/subverses to choose from today. People are also calling for an "open" Metaverse in order to explicitly avoid a single authority. In Snow Crash, the Metaverse was "made available to the public over the world-wide fiber-optics network"; this would be the Internet. To avoid a single regulating organization, the candidate Metaverse definition can be specified as: a collection of many virtual worlds and MetaGalaxies/subverses with VR, linked together through the Internet. Returning to the various Metaverse definitions provided by Hackl (2021), the Metaverse is further characterized as: decentralized (Ryan Gill); a universe (Piers Kicks); where people can create their own world (Tom Allen); millions of digital galaxies (Claire Kimber); and, the next iteration of the Internet (Elena Piech). Hackl (Bhaduri 2021) corroborates the latter characteristic as "the future of Internet". Given the current candidate Metaverse, these characteristics are incorporated. Tom Allen further adds that the Metaverse should be an "exponentially growing virtual universe"; I don’t think the Internet has exponential growth and the Metaverse might achieve exponential growth initially, but doubtfully be sustained. Bosco Bellinghausen states the Metaverse should be "a true technological democracy"; I do not think the modern Internet can be considered a democracy and although there might be democracy technologies in the works, it is still each nation that determines the democracy for its people or not.

In my publication (Nevelsteen 2018), I argue that "Because the Internet is already mixed reality ... the Metaverse will be necessarily mixed as well". By including Mixed Reality (MR), the candidate definition becomes: a collection of many virtual worlds and subverses in XR, linked together through the Internet; with XR being the super set of VR/AR/MR. XR is mentioned several times, as a characteristic for the Metaverse, by the professionals (Hackl 2021) as: accessing the real in the digital and vice versa (Kenneth Mayfield); for the good of the real and virtual world (Esther O’Callaghan OBE); crossing the physical/digital divide between actual and virtual realities (Eric Redmond); a digitally mixed reality (Luke Shabro); the gradual convergence of the digital world with the physical world (Elena Piech); gateway between the real and the virtual reality (Bosco Bellinghausen); and, closer to reality (Ryan Gill). Richard Ward states that XR "is just in the development stages" for a would-be Metaverse. Tom Allen specifies "adapting experiences and knowledge from the physical world"; if a Mirror World is meant, this is handled below, otherwise the criteria is satisfied. Neil Redding requires "location-independent presence"; it is not entirely clear what is meant by this, but current candidate definition certainly allows a user to be in multiple places at once, but being independent of physical location will be difficult. Kenneth Mayfield says the Metaverse will be "a reconfiguring of our assumptions regarding sensory input, definitions of space, and points of access to information"; sight, sound and touch are definitely being reconfigured, but taste and smell are still research topics. Neil Redding similarly mentions, "everything we do in physical space but in a multisensory stimulation". Claire Kimber states the Metaverse is where "all digital experience sits". Hackl specifies the Metaverse will connect people, places and things, which virtual worlds already do, but with the addition of XR it is possible to connect the physical with the virtual i.e., the "convergence of physical and digital" (Bhaduri 2021).

We are HERE.

Michael Robbins considers that the word Metaverse "literally says that in the future we will live in separate universes" (Hackl 2021); I would argue that Michael Robbins’ Metaverse is here today. The candidate definition describes well the current status of the Metaverse today; little to no interoperability and limited ubiquity. I would argue that the desired level of interoperability (technical interoperability also caters to ubiquity) is the determining factor for individuals as to how close we are to obtaining the Metaverse. Richard Ward and Kenneth Mayfield consider the Metaverse to already exist (Hackl 2021). Surprisingly, no one mentioned ubiquity as an attribute of the Metaverse i.e., access via desktop, phone, head-mounted display, or other low powered devices. To incorporate ubiquity, requires tweaking the definition to: a collection of many virtual worlds and subverses, intersecting with XR, linked together through the Internet. Not all access to the Metaverse will be the same. Before mobile phones gained sufficient power to run MMO clients, World of Warcraft created a Mobile Armory app which tied low-powered mobile flat screen experiences to the immersive virtual world e.g., with chat, character profiles, monster and item lists.

Lucas Rizotto states that the Metaverse should "look like Ready Player One" (Hackl 2021); this means that just linking virtual worlds and subverses together is not sufficient. There needs to be some level of interoperability to link virtual spaces into a single perceived virtual universe e.g., in the movie Parzival could move seamlessly between the worlds of Minecraft and Doom, as demonstrated via the portal between the main universe and The Race. When a web server is setup on the World Wide Web, the server is immediate readable and links can be created to and from the web server, "teleporting" the user to another web server. Teleporting between virtual worlds today mostly involves installing specialized client software that is walled-off from other virtual worlds. In this regard, the Metaverse has not be attained. Having interoperability with respect to social networking would answer Karinna Nobbs’s (Hackl 2021) criteria for the Metaverse to be "the anchor of community life and where you meet with old and new friends", with a capability reaching beyond what Facebook currently supports. Blockchain promises various aspects of interoperability, but the existence of multiple Blockchains runs the risk of having fragmentation; there is also no guarantee that world/subverse owners will want to support Blockchain-based technology at all.

The Metaverse in Science

There are actually widely cited definitions of the Metaverse in science. In 2012, the IEEE Virtual World Standard Working Group had the following entry in their glossary.

"Multiple MetaGalaxy & Metaworld systems linked into a perceived virtual universe, although not existing purely by a central server or authority. Originally coined by Neil Stephenson in the book Snow Crash as a single Metaworld known as "The Street", the Metaverse today describes a collection of Metaworlds that are seamless and interconnected but largely decentralized which comprise the perceived virtual universe (metadata universe, metaverse) but follows closer to the overall persistence of a Metaverse from the roots of Cyberpunk genre in which systems much like the Metaverse existed but within the scope of many worlds and uses, foreshadowing a decentralized system of operation and scope. Theoretical Example: Interoperability between such systems as ActiveWorlds, SecondLife, BlueMars, and others by which a stan- dardized protocol and set of abilities exist within a common interface which can traverse among virtual world spaces seamlessly regardless of controlling entity." (P1828 2012)

A more distilled version was published in the comprehensive Metaverse publication by Dionisio, Burns III, and Gilbert (2013). And, Wikipedia editors have consolidated this into what Hackl noted as "a collection virtual shared space, created by the convergence of virtually enhanced physical reality and physically persistent virtual space, including the sum of all virtual worlds, augmented reality, and the Internet ... typically used to describe the concept of a future iteration of the internet, made up of persistent, shared, 3D virtual spaces linked into a perceived virtual universe". The candidate definition coincides to the scientific definitions of the Metaverse.

The definitions (Hackl 2021) by Claire Kimber, Eric Redmond, Esther O’Callaghan OBE, Luke Shabro, Piers Kicks, Karinna Nobbs, Tom Allen, Ryan Gill, Richard Ward, Kenneth Mayfield, Samuel Jordan, JB Grasset and Lucas Rizotto have now been reconciled. To answer Hackl’s question "But do those definitions really reflect what many in the technology and business world refer to when they use the word metaverse?" (Hackl 2021). Yes, if thirteen of Hackl’s responders would refer to the current scientific definition of the Metaverse, it would encompass their own definition.

The definition by Neil Redding highlights the shortcomings of current technology to provide a sufficiently immersive experience. We’ve seen hints of this in other definitions e.g., the one provided by Kenneth Mayfield. Dionisio, Burns III, and Gilbert (2013) name four central components to a viable Metaverse. One of those components is Realism. I argue above that interoperability "is the determining factor for individuals as to how close we are to obtaining the Metaverse"; for some realism seems to be the determining factor e.g., Neil Redding’s criteria includes: 3D photo-real immersive visuals, spatial audio and tactile feedback. Extreme examples of realism include the concepts of Digital Twins and Mirror Worlds; digital twins replicating physical world humans and mirror worlds replicating the physical world, respectively. If multiple mirror worlds exist, will there be interoperability between them? I categorized Brain-Computer Interfaces and Smart Lenses (Elena Piech) as part of the "Wearables" suggested by Hackl (Bhaduri 2021); input/output devices closest to the body to aid in achieving interaction realism. To deem realism the determining factor is entirely fine, but begs the first question: is a low-fi Metaverse allowed, before the level of realism is increased? I would argue yes, if interoperability between worlds/subverses can be achieved. Even Snow Crash describes the usage of cheap low-fi public terminals to access the Metaverse. When Dionisio, Burns III, and Gilbert (2013) refer to realism, they are referring to ‘immersive realism’, "believability rather than devotion to detail" i.e., text virtual world MUDs could satisfy that criteria. Second, we can question if the opposite is true: If there was one world/subverse that provided extreme realism, but that world/subverse was a closed system, siloed off from other worlds, would it be the Metaverse? I would argue no, since it would not constitute a universe.

Spatial Computing and beyond

Greenhalgh et al. (2001) exploited "the coextensive virtual world as a ‘behind the scenes’ resource for coordinating and managing devices and interaction in the physical space" i.e., using the ability of a virtual world for "spatial computing" (Shekhar, Feiner, and Aref 2016) in physical world. In 2012, I extended their work by successfully combining a virtual world with Internet of Things (IoT) (Nevelsteen 2016). To extend even further, it is by no means a stretch of imagination to, instead of using a virtual world, combine the interoperable Metaverse with IoT instead (Nevelsteen 2016); many virtual worlds controlling many IoT ecosystems. This coincides with the work by Rehm, Goel, and Crespi (2015) and is currently part of Cyber Physical Systems. The Metaverse would become the interconnection of, the more general, virtual environments or collections thereof, instead of virtual worlds and subverses. The ability for the Metaverse to be a behind the scenes resource for spatial computing is applicable to Robotics, Autonomous Vehicles and ultimately Smart Cities. I imagine this is what Hackl meant by "the next wave of computing" (Bhaduri 2021). Combining increased realism with the uses of spatial computing, Elena Piech’s criteria for the Metaverse to be "the next iteration of life" (Hackl 2021) can be achieved and possibly Emma-Jane MacKinnon-Lee’s "everything we have always dreamed of" (Hackl 2021).

The only criteria which remains unaddressed is that of Artificial Intelligence (AI) offered by Hackl (Bhaduri 2021). Robotics, Autonomous Vehicles and Smart Cities all require at least some level of AI so its relevance to the Metaverse is obvious. What is interesting is that Rafael Brown (Hackl 2021) has seemingly applied Larry Tesler’s Theorem (ca. 1970) that, AI is whatever machines have not done yet, to the Metaverse. Perhaps that the Metaverse will be combined with AI and AI is such a futuristic term, means the Metaverse becomes a futuristic term as well.

It should be noted that there was no need to advance the candidate definition, beyond the currently accepted scientific one, in order to combine the Metaverse with other technologies. Yes, the Metaverse might be the behind the scenes resource, for spatial computing, for all the in the future. If, for every new technology connected to the Metaverse, we change the definition of the Metaverse, then the term will indeed be as futuristic as Larry Tesler’s Theorem about AI. Nevertheless, the question that remains is, what must be resolved today so that the Metaverse can exist, as soon as possible, as an interconnected whole, in contrast to fragments.

References

  • Bartle, Richard A (Dec. 2015). MMOs from the Inside Out. Apress.
  • Bhaduri, Abhijit (June 2021). Live with Cathy Hackl, Futurist. URL: https://www.youtube.com/watch?v=2f1njigiN2E
  • Diaz, Andrea (May 2018). ’World of Warcraft’s’ virtual gold is seven times more valuable than Venezuela’s real money. URL: https://edition.cnn.com/2018/05/08/world/world-of-warcraft-token-worth-more-than-venezuelas-currency-trnd/index.html
  • Dionisio, John David n., William G. Burns III, and Richard Gilbert (2013). “3D
    Virtual Worlds and the Metaverse: Current Status and Future Possibilities.” In: ACM
    Computing Surveys 45.3, 34:1–34:38. DOI: 10.1145/2480741.2480751.
  • Greenhalgh, Chris et al. (2001). The EQUIP platform: Bringing together physical and
    virtual worlds. Tech. rep. Mixed Reality Laboratory - University of Nottingham-UK.
    URL: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.22.3793.
  • Hackl, Cathy (May 2021). Defining The Metaverse Today. URL: https://www.forbes.com/sites/cathyhackl/2021/05/02/defining-the-metaverse-today/.
  • Johannesson, Paul and Erik Perjons (2014). An Introduction to Design Science.
    Switzerland: Springer International Publishing. DOI: 10.1007/978-3-319-10632-8.
  • Nevelsteen, Kim J. L. (May 2016). “Distributed Technology-Sustained Pervasive Applications.”
    Doctoral dissertation. Stockholm University, Department of Computer and System Sciences. URL: http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-129151.
  • Nevelsteen, Kim J. L. (May 2018). “Virtual World, Defined from a Technological Perspective, and Applied to Video Games, Mixed Reality and the Metaverse.” In: Computer Animation & Virtual Worlds 29.1, e1752. ISSN: 1546-427X. DOI:
    10.1002/cav.1752.
  • P1828, IEEE Virtual World Standard Working Group (2012). April 6 2012, IEEE P1828/D1 Draft Trial-Use Standard for Virtual Components. Tech. rep. IEEE Standards Asso- ciation Department.
  • Pearce, Celia and Artemesia (2009). Communities of play: emergent cultures in
    multiplayer games and virtual worlds. MIT Press.
  • Rehm, Sven-Volker, Lakshmi Goel, and Mattia Crespi (2015). “The Metaverse as Mediator between Technology, Trends, and the Digital Transformation of Society and Business.” In: Journal For Virtual Worlds Research 8.2. DOI: 10.4101/jvwr.v8i2.7149.
  • Shekhar, Shashi, Steven K. Feiner, and Walid G. Aref (Dec. 2016). “Spatial Computing.” In: Communications of the ACM 59.1, pp. 72–81. DOI: 10.1145/2756547.
  • Stephenson, Neal (1992). Snow Crash. Bantam Dell.