Microsoft Mesh will teleport you in AR and VR to play with others across lots of devices

I entertained a guest at home last week, but he wasn’t really there. 

I had a Microsoft HoloLens 2 on my head, visor down, in my dusty upstairs office. Hovering in the room was a holographic table. And then, a cartoon avatar of Alex Kipman, Microsoft’s technical fellow in charge of mixed reality. I met him in the real world two years ago, when I tried the HoloLens 2 for the first time at Microsoft’s headquarters in Seattle. Now, he was teleporting into my home. We talked about collaboration and telepresence. He conjured up sharks and jellyfish that float around us. We began to doodle and annotate together.

Now playing:
Watch this:

Meeting with Microsoft’s Alex Kipman in the HoloLens…

2:18

I used my fingers to grab holographic objects, passing them across a table that wasn’t there. In his office, somewhere else, I was a cartoon avatar at his virtual table. Later, Microsoft sent me a video of what I looked like on the other end, and I wondered: Who’s dreaming whom?

We were using Microsoft Mesh, a technology that aims to allow virtual telepresence in VR and AR across the HoloLens, VR headsets, iOS, Android and even the upcoming world of smart glasses. Microsoft’s tech is exploring how we’ll do 3D work remotely, but it also raises the question of how we’ll appear in each other’s spaces when we do.

Virtual work, in 3D

Yes, you’ve heard the promise of collaborating in VR or AR before. I’ve tried demos promising collaborative experiences for years, whether it’s in AR glasses, VR via apps like Spatial and immersive VR art experiences. Microsoft’s been trying to crack the idea of collaborative work in mixed reality for years, that was the pitch when I got my 2019 HoloLens demo. But in a 2020 that was all virtual, little of that work was done in VR or AR at all.

Right now, my virtual work happens on 2D screens with laptops, phones, computers, Zoom and Google and Slack, and a lot of other things. Most of the time I’m myself, staring into a camera. My colleagues and I see each other eye-to-eye in a grid. In VR, I’m usually a cartoon, free to move, but unable to see real people’s faces.

We still don’t have widespread consumer AR smart glasses yet, but that could start changing based on recent advancements by Qualcomm. Facebook has smart glasses in the works, and Samsung and Apple may have them, too. Microsoft’s tech aims to solve the multidevice collaborative problem ahead of those devices arriving.

Microsoft Mesh will bridge both real and virtual worlds, but not necessarily be able to show our real faces yet. For most people, cartoon-ish avatars will appear. But some people could also scan themselves with cameras and appear as “holoportations” of themselves.

Alex Kipman does this during Microsoft’s Ignite keynote, using an array of depth-sensing Azure Kinect cameras to create a volumetric 3D video capture of himself. It’s tech I’ve seen once before, during last summer’s Burning Man virtual event in Microsoft’s social VR app, AltspaceVR. (There was a concert featuring Diplo that involved similar projection tech.) Kipman acknowledges that Burning Man used some early versions of the tools involved in Mesh’s rollout. 

Microsoft’s Ignite keynote was designed to be watched in VR through its AltspaceVR app. 

So, we stopped our HoloLens demo and switched over to a VR headset connected to a PC: the HP Reverb G2. In AltspaceVR, Alex’s avatar and mine carried over (although switching systems took some time). That’s the idea of Microsoft Mesh: finding worlds that people could work in across devices. 

Building for a world that doesn’t exist yet

Some of Microsoft’s first Mesh partners are familiar faces in the AR world, using the tech to find a way to bridge to smart glasses that aren’t quite here. Pokemon Go developer Niantic has an experimental example of what Pokemon hunting on smart glasses could feel like, done with people wearing the HoloLens 2. Pikachu and other creatures run around, but a friend could beam in too as a cartoon-like companion, walking alongside you wherever you are.

Niantic CEO John Hanke admits that the explorations are just starting. Niantic is exploring smart glasses with Qualcomm that may use AR when paired with phones, but in the meantime, the Microsoft HoloLens 2 will be used like a prototype to test how future glasses could work. Niantic has already made multiplayer AR work on phones, but headsets are the next frontier.

“Making this whole persistent shared AR world work at a world scale, city scale, park scale, that’s something we’ve been able to experiment with on phones. But this gives us a chance to start playing with that with the headset, which is a completely different experience,” Hanke says in a conversation over Zoom. Niantic is using the HoloLens 2 outdoors, in areas where phones would normally be used to play.

Now playing:
Watch this:

Niantic’s concept of Pokemon Go in a HoloLens

1:33

For Niantic, the idea of playing with a friend who seems to be right next to you is the really new element. “In our neighborhood, everyone’s talking on the phone, or listening to a podcast. This idea of having companionship while you’re out in the world is very, very natural. Taking that to the next level, having that be a sort of holographic companionship while you’re out … in a sense, it’s novel and sci-fi, but you know, in other ways, we’re kind of already doing it.”

Meanwhile, Cirque du Soleil founder Guy Laliberté is developing an immersive theater and performance social platform called Hanai World that will explore blending the virtual and real, in ways that aren’t quite clear yet. Laliberté is using the HoloLens 2 as a way to play with bridging real locations and virtual experiences. Microsoft’s Ignite keynote ended with an odd, Burning Man-like dance space with 3D scans of Laliberté, Kipman and others around a fire, representing maybe an example of shows that could happen in the future. The idea sounds like virtual theater I’ve seen, and also maybe an AR version of VR arcade spaces like The Void. Also, maybe, a way of exploring virtualizing theater beyond Zoom. Laliberté promises a virtual demo of its concept by the end of this year. Cirque du Soleil was, incidentally, one of the earliest creative partners for the HoloLens.

OceanX, an oceanographic nonprofit organization founded by billionaire Ray Dalio that filmmaker James Cameron has done a variety of documentary projects with, is one of Microsoft’s other early partners. OceanX’s deep-sea research vessel OceanXplorer is adding a holographic room where people on deck will be able to see live 3D maps of what the deep dives are discovering, and holographically beam others in to collaborate. It literally sounds like a James Cameron movie. 

OceanX’s Vice Chairman Vincent Pieribone said over a phone call that James Cameron has been a leading visionary in helping imagine how to integrate the vision of AR on the boat. “We’ve always used the tree scene from Avatar, where they’re all standing around looking at the [holographic] tree, that’s been our model of how we’re going to create our holograms. Even more than all the Marvel and Iron Man [holograms].”

How will it play with the rest of the metaverse?

Kipman sees Mesh as a way for VR and AR collaboration to become multiperson, fast. I bring up the multiplayer-enabled open-source version of VR art app Tilt Brush, called MultiBrush. “Imagine very quickly being able to take Tilt Brush, mesh enabling it, and now you can be painting together,” he says. “Tilt Brush HoloLens edition, Tilt Brush virtual reality edition, doing it at the same time in the same physical location, or like we’re doing right now in separate locations.”

Essentially, Mesh aims to be the VR/AR cloud computing ecosystem that doesn’t exist yet. There are an increasing number of multiplayer and collaborative apps, from Spatial to multiperson Tilt Brush, but there isn’t a common glue between them. Companies like Facebook are looking to build a common social world around its VR and AR that knits together with the rest of Facebook and Instagram. Apple and Google, most likely, will have similar plans for how AR and VR will work with their operating systems. 

Kipman says that the Microsoft Mesh private preview being introduced right now needs a Microsoft identity at its core to enable the collaborative cloud features — that means logging in with a Microsoft account, much as a Facebook account is now needed for Oculus. It will be interesting to see whether the tools open up to allow more ways to log in and connect across a variety of ecosystems, and how that would play out — Microsoft promises that Mesh will work for a variety of ecosystems, Oculus included. There are already a lot of companies trying to be the pathway to a global interconnected metaverse of VR and AR. Microsoft could be one of the first to make it all work, but how well Microsoft Mesh ends up playing with all of Facebook’s collaborative VR/AR tools, and eventually Google and Apple’s, is the big unknown.