Emotional Metaverse
Are facial expressions enough to convey emotion? What does that mean for the metaverse and the companies trying to build it?
In February 2021, Mark Zuckerberg said, in an interview with The Information, that Facebook would invest significant resources into a new generation of realistic avatars for their VR platform, with the goal being to allow users to show emotions instantly by tracking their facial muscles movements and translating such facial data into the avatars, mimicking (via real time 3D rendering) the attempts of a face at communicating an emotion.
Sounds cool but, to me, there’s something really archaic about taking 3D facial movement data and turning it into, well, 3D facial movement data. It feels very similar to when they took newspaper ads and put them onto the web —unchanged, making the online reading experience terrible and, by extension, ruining the efficacy of those ads.
I hope we get rid of these soon because, even though it’s obvious in hindsight, taking real-world ads and simply translating them into digital is a terrible idea. The same is true for most ‘real world’ experiences translated into digital.
It’s different
Fortnite is thought of as a video game but it has, in fact, replaced the entire social outdoor playground experience for millions of teenagers around the planet, not by being an outdoor playground experience translated into digital, but by being an abstraction of it. And yes, Fortnite has a lot of outdoor playground elements but it's not the similarities that drove its success, it was the differences.
Who would want a 'digital outdoor experience' to be as limited as one would be in the real world? In Fortnite you can jump way higher than you could in real life, you can build structures to hide in in minutes, and so on. This game is limited to a mouse and keyboard or controller, and that is a good thing, because it allows players to do much more than what they could do with their own legs, arms, and torso. This simplified way of interaction is as old as video games themselves, and even though it stems from technical limitations of the 70s, I still think it's a better input method.
The 'digital outdoor experience' is improved by abstracting physical input into key strokes and mouse clicks. What that means is that adding a treadmill as a controller to move your character in the game is not an upgrade, it's a downgrade. This sort of project looks great and a lot of us have been dreaming of getting to play in one of those for years, but they're never going to replace mouse and keyboard because they remove the abstractions that make most digital experiences so great.
It's not that gamers are lazy, it's that 'digital' is, well, digital. It's great because most of the real world is removed from the experience, and that opens the door to a different kind of reality1.
Showing Feelings
This is why I think that Zuckerberg's idea of capturing 3D facial data to drive realistic 3D faces in digital spaces is not going to work. We are used to conveying emotion through our faces because that's what we evolved with. But not only is that not necessarily the endgame for emotive expression2, we've been already changing the rules of the game for several centuries.
The painting above is emotionally powerful, not because it contains a screaming face, but because the face in it almost doesn't look like a human face at all.
And what about this one? Looking at it is certainly an emotional experience for a lot of us. It doesn't look like anything from the real world, yet we recognize a feeling in it, through its abstract shapes and arrangement of colors.
These are some very popular and somewhat recent examples, but artists have been doing this emotionally expressive work for ages. And not just through pictures, but through sounds, sculptures, garments, even architecture.
The amount of praise and admiration that these artworks receive from the masses is, to me, proof that our faces and the words we spit out of them just aren't enough to convey what we're feeling. We need more.
A Strange New World
In my original reaction to Zuckerberg's comment, I said that I'm glad the face-reading technology was in development, even though Facebook's intended application of it was stupid. Nine months later, I am still glad it's coming, but I also hope that other data points will be able to drive emotional expression in the digital world3. My heart rate can tell you more about my feelings than my face, and so can the clenching of my toes, the tension in my back, or the heaviness of my gut.
The metaverse is coming, there's no doubt about it. So when millions4 of us start spending most of the day in it, we will have to find ways to connect emotionally. And when I think of 'digital feelings', they don’t look like this:
They look like this:
Not necessarily a better one, but certainly a different one.
Most mammals, including us, have also relied on dancing and making noises to achieve this for thousands of years.
Yes yes yes, I am also worried about the privacy implications of making this data available.
And, eventually, billions.