When Facebook revealed the new hand tracking feature on the Quest during Oculus Connect 6 in 2019, it looked like magic. Within just a couple of months, the feature was added to existing Quest headsets, and in 2020, hand tracking came as a standard feature on the Quest 2. Over a year later, the feature is really only used for simple menu navigation and one or two games. There is so much untapped potential in virtual hand tracking, it would be such a shame if developers never find a way to take advantage of it.
Don’t get me wrong, the current implementation of hand tracking on the Quest 2 is really impressive. You can boot up the headset and start navigating the menus without ever picking up a controller. This includes browsing the store, downloading apps, and changing settings. But as soon as you want to open a game or app — even a streaming app like YouTube — you’re going to have to use a controller. Menu navigation is cool, but if you have to pick up a controller anyway as soon as you actually want to do something with the Quest, it feels kind of pointless.
There are some games that feature hand tracking support, but not many. The premiere hand tracking game is called Waltz of the Wizard, and it’s a great example of what hand tracking can offer that controllers can not. In WotW, you use different gestures to cast spells and interact with objects in the world in imaginative ways. For example, you can make the ILY gesture with one hand and your virtual fingers will stretch out and form a slingshot, then you can use your other hand to pull back a string and fire magic darts. Another spell lets you snap your fingers to make objects explode. There are more than a dozen different spells you can learn, and all of them feel enhanced by hand tracking; in other words, using controllers would detract from the experience.
Waltz of the Wizard isn’t much more than a toy box experience, and it at times emphasizes some of the limitations of the Quest’s hand tracking. The Quest uses inside-out tracking, meaning the cameras that track your movement are inside the headset, rather than external like the sensors for the Valve Index. This means that gestures need to be done palm up and within a certain range of the headset so that the cameras can see your fingers. If the headset can only see the back of your hands or if your hands move too away, the hand tracking simply won’t work. Further, physically interacting with objects can be someone finicky. Obviously, when you pick something up you can’t actually feel the weight or the surface of the object, so even something as simple as closing your hand around a virtual ball and throwing it can feel somewhat unnatural. Still, there are plenty of potential uses within the Quest’s admittedly limited hand tracking tech, still to be explored.
Hand tracking has the power to create more immersive VR experiences. One game that desperately needs hand tracking support is Tabletop Simulator. While you can already play your favorite virtual board games in VR, you’re limited to the pointer control of the controllers. Imagine if you could hold your hand of cards, roll dice, and move miniatures around a board without needing to use controllers. Interacting with small objects can be difficult, certainly, but overall it would be a much more immersive experience.
One of the best ways to overcome the dissonance that occurs when your real hands interact with virtual objects is to make the objects familiar so that it’s easier to imagine you’re actually holding them, even though you can’t feel them. Imagine this: a virtual comic book store, with all the newest releases up on the wall. You can browse the store, pick up issues, and flip through them. The app could have integration with a digital comic service like Comixology or Marvel Unlimited, giving VR users a way to read virtual comic books and actually hold the books in their hands.
What’s the best feature in any video game? Yes, petting the dog. How about a virtual petting zoo that uses hand tracking to let you pet and feed the animals. Imagine pouring a little bit of kibble into the palm of your hand and letting a virtual billy goat nibble at it. The only thing missing would be the unmistakable odors of barn animals, and would anyone really miss it?
VR also has the ability to tap into our memories and have experiences that are of a different time. On Twitter, Upload VR’s managing editor Ian Hamilton came up with the brilliant idea of turning Netflix’s streaming library into a virtual Blockbuster. VR users could waltz up and down the aisles of Blockbusters, casually browsing the shelves full of VHS tapes for rent. These kinds of virtual experiences would be significantly enhanced with hand tracking because you could pick up the tapes and read the back of the box just like we used to do so many years ago.
VR is still a relatively young medium and hand tracking itself is barely a year old, but it’s exciting to think about how much potential the feature has. It really feels like the next step forward for VR, and I can’t wait to see where it goes.